This morning inspired by listening to https://www.youtube.com/watch?v=wc8FBhQtdsA and reading this thread on linked in - https://www.linkedin.com/posts/dartar_opensource-aiforscience-share-7448170944151355392-2bNP?utm_source=share&utm_medium=member_desktop&rcm=ACoAAAEImlkB8SaKRnNedw-vBQuAiicIBoz5PWQ, I was struck by the section in the podcast with Simon about Dark Factories for software development. This is where code is neither written nor read by humans, and is derived from the idea of dark factories in manufacturing- where the automation is so complete you don't need humans in the facility so you can turn off the lights.
It made me wonder whether we might coin the term "Dark Knowledge" - which might be an oxymoron. It would be were LLMs make accurate and usable claims about the world, but we don't have any insight into how they are generated, or what the underlying motivating mechanisms are beneath them. The process of generation is obscured from our sight. This is the antithesis of science, but it's a path that might be possible.
If there are classes of research questions that can be framed, executed and resolved entirely by LLMs to generate claims, with no sight from humans, then this will happen. This generates effective claims about the world, but not through the paradigm of science. Is that even knowledge? I think that's the endpoint we have to avoid. But what are the risks here?
I think there are two risks we might contemplate.
1) The unintended consequence of applying a lever in the world. If we have a fact that operates on the world, but the operation on the world has hidden side effects, then we might end up in a situation where the consequences of acting lead to bad outcomes. This might be most risky in areas like synthetic biology.
2) The squeezing of space for creative reasoning. If we can't create our own mental models of the world - albeit imperfect, do we risk reducing the our ability for creative leaps that can lead to new knowledge?
Those are the two the immediately strike me. There must be others.
It made me wonder whether we might coin the term "Dark Knowledge" - which might be an oxymoron. It would be were LLMs make accurate and usable claims about the world, but we don't have any insight into how they are generated, or what the underlying motivating mechanisms are beneath them. The process of generation is obscured from our sight. This is the antithesis of science, but it's a path that might be possible.
If there are classes of research questions that can be framed, executed and resolved entirely by LLMs to generate claims, with no sight from humans, then this will happen. This generates effective claims about the world, but not through the paradigm of science. Is that even knowledge? I think that's the endpoint we have to avoid. But what are the risks here?
I think there are two risks we might contemplate.
1) The unintended consequence of applying a lever in the world. If we have a fact that operates on the world, but the operation on the world has hidden side effects, then we might end up in a situation where the consequences of acting lead to bad outcomes. This might be most risky in areas like synthetic biology.
2) The squeezing of space for creative reasoning. If we can't create our own mental models of the world - albeit imperfect, do we risk reducing the our ability for creative leaps that can lead to new knowledge?
Those are the two the immediately strike me. There must be others.