The project will elaborate a conceptual framework for how the research process is conducted and evaluated (e.g., study design, conduct, analysis, interpretation) and different researcher personas (e.g., peer review, replication, and discovery) to develop specific tasks that can be benchmarked effectively and are amenable to testing with LLM agents.
This is a really interesting position - working on eval frameworks to determine to what extent LLMs can do research.
I see a couple of potential directions for a role like this.
- The LLMs are terrible, and remain terrible, at the core steps in research and this role quickly is redundant.
- The LLMs are better than we imagined, and improve faster than we can keep track of, and this role is playing catchup all the time.
- Most likely - these tools have value if applied appropriately, and creating an LLM agnatic open eval framework will be high leverage work.
---
EMS press is looking for a software developer - https://ems.press/updates/2025–01–21-hiring-software-developer). EMS is a great niche publisher with a great team.
---
---
Crossref is hiring for some data scientists - https://www.crossref.org/jobs/2025–01–16-data-scientist/ - they recently restructured some roles in tech, so it’s interesting that these roles are opening up now.
---
Open Scholarship Catalytic Awards Program - https://www.orcaopen.org/work/cap - a $5K to $15K funding round to.
The program aims to stimulate a culture of open scholarship (encompassing open science, open research, open data, and a range of other open-sharing practices that promote transparency, reproducibility, community engagement, and collaboration) at US-based traditionally under-resourced institutions.
Application deadline 15th of March.
---
All of these though, feel very contingent on current events, for now at least, let’s see how the next few months pan out.