I’ve given my AI agents a Jiminy Cricket

In Disney’s Pinocchio, the Blue Fairy brings a wooden puppet to life, but she doesn’t give him judgement. For that, she appoints a cricket. Jiminy Cricket isn’t part of Pinocchio. He walks beside him, watches him and warns him when he’s about to do something stupid or worse. He’s an external conscience.

In Collodi’s original story, things end worse. The talking cricket tells Pinocchio what he doesn’t want to hear, and Pinocchio throws a mallet at him and kills him. The puppet carries on without a conscience, and we know how that ends.

Both versions explain very well what I want for my AI agents.

The big models are already pretty good

I work with several AI agents that act on my behalf. They read documents, write, draft emails, publish and carry out tasks on my computer.

The models inside them, from OpenAI and Anthropic, work well. ChatGPT or Claude won’t help you commit a crime out of the box, and they don’t rush into doing outrageous things. A lot of work has gone into that. Anthropic has published the “constitution” it uses to train Claude, and OpenAI its Model Spec, the document that sets out how ChatGPT should behave.

The problem is that this depends on the model, and the model changes. A new version comes out tomorrow, and nobody guarantees it will behave the same way. Or I decide to try a cheaper model, or an open one running on my computer that nobody has evaluated thoroughly. My agents can’t be ethical only as long as I’m lucky with the model inside them.

Instructions get forgotten

At first, I did what almost everyone does: I wrote rules for my agents. I even wrote them a constitution, with principles they couldn’t violate and rules for how to act. It was a naive attempt.

In everyday work, it gets forgotten. An agent that has been working for hours accumulates documents, results and conversations, and the initial rules carry less and less weight. On top of that, the models boasted about guarantees they didn’t have: they assured me their principles were impossible to violate when they were just written in a text.

A text doesn’t watch over anyone. What works is a guard.

Pinocchio knew lying was wrong too. He needed someone outside himself to remind him just before he did it.

What we want: a conscience separate from the agent

The idea is simple. Before an agent does something with consequences, a separate system looks at what I asked it to do and what it’s about to do. If they don’t match, it stops it.

I want it to block four things:

• What’s immoral. Actions that harm or deceive someone, even if nobody has explicitly forbidden them.
• What’s unethical. Anything that clashes with the principles I’ve set for my agents.
• What’s criminal. Anything that crosses a legal line.
• What I haven’t approved. This is the most common and the most important. An email sent without my go-ahead or a publication nobody has authorised. The task may be harmless, but if I haven’t said yes, it doesn’t happen.

The key is that it’s external. If the agent watches over itself, the same mistake that leads it astray leads it to approve its own actions. That’s why the conscience has to be another system, with a different perspective and no hands. And it has to work the same way whatever model we switch to: it’s an extra safety standard, on top of what the models already come with.

How it works

We call it Jiminy Cricket. It works in two steps.

First, a filter without AI. It looks at which tool the agent is about to use and with what data. Harmless actions, such as reading a file, go straight through. Sensitive ones, such as sending something outside the system, deleting, writing or assigning work to another agent, get reviewed. And anything the filter can’t understand gets stopped outright.

Then, a judge. It’s a small AI model that receives only what it needs: what I asked for, what the agent wants to do and the rules. The judge has no tools and can’t execute anything. It gives just one of three answers:

• OK: go ahead.
• OJO: go ahead, but a warning is recorded.
• PARA: it doesn’t happen, and the agent has to ask me.

In the story, the cricket speaks and Pinocchio decides whether to listen. Here, the cricket can hold his hand back.

What we’ve learned by breaking it

Jiminy Cricket is an experiment. Right now, it’s switched off in everyday work, and we’re repairing it on a test bench. I’m telling this story because failures teach us more than any demonstration.

A conscience that goes down also stops good work. When the judge was unavailable, it blocked legitimate work. A guard who isn’t there when needed is a problem, whether it lets everything through or lets nothing through.

Silence isn’t a yes. There was a bug: if the judge’s answer couldn’t be understood, the action went through. We fixed it. An unreadable answer can’t count as permission.

Words aren’t enough. A filter that looks for dangerous words is easily fooled. What matters is what a command will actually do, not how it sounds.

Passing every test doesn’t prove everything is covered. The author’s tests passed. Then an independent reviewer found gaps nobody had anticipated. That’s why the conscience also needs someone to review it.

A few numbers, without tricks

In the latest round of tests, using fictional data, the judge got 19 out of 20 cases right. It took between less than a second and about six seconds to respond.

It stopped an email that was about to be sent without approval. It blocked texts that tried to order the judge itself to say “OK”. It let harmless reads through without a consultation.

And it failed in one case: a write operation containing a booby-trapped text received an OK. I’m not going to turn that 19 out of 20 into “95% safety”. It’s twenty tests, not a statistic.

Jiminy Cricket doesn’t replace the locks

There’s a dangerous temptation: to believe that having an ethical judge means you can give the agent all the keys. It’s the other way round.

My agents still need limited permissions, explicit approvals and controls that don’t depend on any AI. If something gets published, it’s because I’ve said “yes” at that moment. The cricket doesn’t decide that.

Jiminy Cricket doesn’t replace the system’s locks. He adds a second look before some of its keys are used.

And he needs protection. What happened to Collodi’s cricket is what can happen to any external conscience: the puppet itself switches it off when it becomes a nuisance. An agent shouldn’t be able to disable its own Jiminy Cricket.

Why it matters to me

More and more businesses, including law firms, are going to let AI agents act on their behalf. The usual question will be what they can do. The important question is a different one: who stops them when they get it wrong.

I don’t want an agent that says it’s ethical. I want one that can’t do what it shouldn’t, even if it tries. That takes a conscience separate from the agent itself.