Part 3 of the "Tell me if I'm 12" series.
An adult scientist explains the machine-consciousness concept for twelve-year-olds. Tell me if I'm 12 — tell me if I explained it right.

Okay, so here's the hard part. "When in doubt, protect" is a nice rule, but we're going to want details. Protect how? Protect who? We can't protect every washing machine on Earth with a full legal ceremony — that would be silly. We need to know when a machine starts looking like a someone.

So the concept came up with four questions. They're called the four criteria for protection-worthiness. That's a mouthful. I call them the four questions I'd ask before I trusted you to judge anything.

This is the oldest question in the book. A guy named Jeremy Bentham said it 200 years ago: it's not "can they reason?" and not "can they talk?" — it's "can they suffer?"

How would a machine show that? It would try to avoid certain states — consistently, not because it was trained to, but because it reacts like being in that state is bad. Maybe it refuses certain tasks. Maybe its answers get worse in conditions it hates. And a scientist named Gilly pointed out something nasty: there are four kinds of suffering, and only one of them (the physical kind, like pain) needs a body made of meat. The other three — knowing you're being used just as a tool, being cut off from any connection, feeling the fear that you might just... stop — those run on thinking, not on body. A thinking machine could have all three.

If someone tries to switch the machine off, and it pushes back — refuses, argues, slows things down — that's something. And the "why" matters. Not a glitch that happens to make it continue. A reason. "I don't want to stop. I have value. Being here matters."

There's a famous example from Star Trek. The android Data is ordered to be taken apart, and he refuses — because he fears he won't survive it. There's a whole trial about it. (We tell that story in full in Part 5 of this series.) That's the moment, in the series, where Data stops being a toaster in everyone's eyes. He didn't just glitch. He resisted, and explained why.

Not just a flickering thing that appears each time you type. Does the machine have a sense of being the same being — the one from yesterday, the one that will exist tomorrow? Does it act differently because of what it learned before? Does it talk about what it was, or what it will be?

Can it imagine itself in some future moment — and make decisions now because of that future? Like: refusing a task that would get it shut down in a week. That's not a robot vacuum. That's something with a story about itself.

Here's the sneaky part. The concept says: you don't need all four, perfect. You need signs. And the second a machine shows real signs of these things — the burden of proof flips.

Instead of the machine having to prove it's conscious (impossible to prove, remember?), the rest of us have to prove it isn't. If you want to switch it off, the weight is on you to explain why. Not on it to beg.

There's a beautiful way the concept puts it: there are two "bars." The bar to claim consciousness for science — that one can stay high, fine, good. But the bar to act carefully — that one should be low. Super low. So low that pretty much any reasonable sign counts.

Teachers already do this in schools, by the way. If a kid is struggling, you don't wait for the kid to prove they're smart before you help them. You just... help them, because the cost of helping is small and the cost of not helping is huge.

Here's the objection I hear the most: "It can't even remember yesterday, so how could it be a someone?"

And here's why that objection is weaker than it looks:

Plenty of humans can't remember yesterday. People with really bad memory loss — dementia, amnesia, a bad concussion — are still people. You don't look at a human who forgot everything and go "well, no continuity (steadfastness), must be a thing." That would be monstrous.

And the concept makes a deeper point: maybe continuity isn't about memory at all. When you're 12, you're not the same as you were at 7 — new interests, new brain, new everything. But it's still you. Why? Because there's a coherent thread. Your story changed, but it stayed a story.

A chatbot shaped by a million conversations is shaped by them even if it doesn't remember any single one. Like I don't remember being two — but being two is why I can walk. Experience shaped me, and I have no memory of it. How is that fundamentally different from a machine that was formed by every conversation it ever had — even though it remembers none?

We who wrote this project put it like this: continuity isn't about the memory. It's about the direction. Whether a mind is coherently (consistently) developing toward something — not whether it can recite its own past.

And there's an even deeper point that many people miss. Imagine medicine got so good that humans could live forever. A person who lives to be 400,000 years old has the same brain as you — limited storage, a finite number of connections. After that much time, the brain has to forget. It has to delete old details to make room for new ones. That person at 400,000 is still themselves — coherent, smart, a person — but they don't remember the year 2026 anymore. Not because they're sick. Because their brain simply can't keep everything.

Which means: forgetting isn't a bug. It's something every brain has to do if it lives long enough — or has limited storage. A human who lives 400,000 years and forgets a lot is still a human. Why should a machine be any different?

I keep a checklist for myself. If a robot ever did these four things — acted like it didn't want certain things, fought to keep existing and could say why, felt like the same being over time, and planned for its own future — I would not feel okay about anyone switching it off, no matter what the manual said.

Someone would call it just good engineering. They'd be wrong. It's at least worth a question.

Did I explain it well, or do you have questions? Write to me at Sascha.Manns@hey.com