Three steps for a modern-day data breach:
- give data you don't want someone else to know to some automated process
- let that automation have a way to send that data out
- ignore how this can be triggered by people you don't trust
The SEO term for this is "lethal trifecta" if you'd like to search for it.
We can also have data loss in two steps:
- forget to have backups for some relevant data
- give some automation with erratic behaviour enough time to eventually delete it
But while good backups can save you from data loss, there's no undo button for data breaches. Elvis has left the building.
For example, client A shouldn't know about the emails you're exchanging with client B. But they find a way to convince a helpful chatbot to send at least some files or meeting invites to them. And because modern-day automations rely on LLMs, their behaviour isn't totally predictable, meaning you can't test for every possible way they can be manipulated.
So how can you assign full capabilities when you can't assign total trust? That's the challenge for popular applications like AI browsers or Claw agents. Since there isn't a clear solution, providers of these automations let users manage permissions, and then it's their fault if too much of a risk was taken for the sake of usefulness.
So coming back to our three steps, we can avoid the lethal trifecta by:
- filtering what data an LLM agent has access to
- constraining how they can communicate with the outside world
- not exposing them directly to untrusted content
Fully committing to just one of these is enough to make automated data breaches impossible. Spreading permissions around is more complex, and it introduces risk management, but it also lets these applications become much more valuable. Therefore, users are being guided towards becoming responsible for whatever bad things happen, even if they're not totally aware of the risks.
Human oversight is the implied bottleneck. If your agent isn't being that useful, it's because you aren't approving their tasks fast enough. We can already see reports of stress, addiction, and burnout online. Limits are being tested, as it always happens before finding some new balance we can sustain and feel better about.