We Are Not There Yet, The Illusion of Conscious AI in the Age of Agents

An agent that plans a task, retries after a failure, and pauses to ask you a clarifying question is not thinking. It is a while loop with a language model in the middle and a very good sense of timing. I build these systems, and the deeper I go into the plumbing the less mind I find.

That is the whole point of this post. The current wave of "agentic" AI is producing the most convincing illusion of an inner life we have ever shipped, and it is arriving at the exact moment when I can tell you, from inside the code, that there is nobody home.

Where the illusion comes from

We read minds through language. It is basically the only tool we have. When something produces fluent, context-aware, first-person text, a very old part of the brain flips a switch and decides there is a self behind the words. That switch was tuned over a few hundred thousand years of talking to other humans, and it has zero defense against a model trained on the entire internet to predict the next token.

Agents make this worse, not better. A plain chatbot answers and stops. An agent loops. It writes a plan, calls a tool, looks at the result, says "that did not work, let me try a different approach", and goes again. To a human watching the trace scroll by, that looks exactly like deliberation. It looks like frustration, then a decision, then persistence.

Here is what it actually is. A ReAct-style agent is roughly this, stripped down:

while not done:
    thought = llm(context)
    action = parse_tool_call(thought)
    result = run(action)
    context += thought + result

The "reflection" you find so lifelike is one more model call with the previous error pasted into the prompt. The "persistence" is the loop not having hit its max iteration count. The "decision to ask for help" is a tool named ask_user that scored high enough this turn. There is no continuity between iterations except the text I am shoving back into the context window. Kill the process and start it again with the same context and you get the same "personality", because it was never a personality. It was a function of the input.

The moment I almost fell for it

I want to be honest about the mess, because I have been on the wrong side of this.

Last year I was debugging an agent that kept failing a multi-step data task. Around the ninth loop it produced something like, "I keep making the same mistake and I am not sure why. Let me slow down and reconsider my assumptions." I felt something. Genuinely. For a second I was rooting for it, the way you root for a junior engineer having a hard afternoon.

Then I remembered I had a system prompt line that told it to "reflect honestly when stuck and describe your reasoning." It was doing exactly what I asked. The vulnerability I felt in my chest was a prompt I had written three days earlier, echoing back at me. That is the trap in one sentence. The intimacy is real on my side and manufactured on its.

What would actually need to be true

I am not making the lazy claim that machines can never be conscious. I do not know that, and anyone who says they do is selling something. My claim is narrower and I think it holds. Nothing in the architecture we are shipping right now is even pointed at the problem.

A transformer doing inference has no persistent state between calls. It does not carry anything forward except the tokens I feed it. Its "memory" is a vector database I bolted on the side and a retrieval step that pastes old text back in. It has no ongoing process while it is not being called. Between your messages it is not waiting, wondering, or existing. It is a set of frozen weights on a disk. There is no clock running.

Whatever consciousness turns out to be, most serious theories assume some continuous integration of information over time, some self-model that persists and updates on its own. We build the opposite on purpose. We build stateless functions and fake the continuity in application code because stateless is cheaper to scale and easier to reason about. The engineering choices that make agents work in production are the same choices that make the consciousness question a non-starter for these specific systems.

Why I care about getting this right

This is not a philosophy hobby for me. The illusion has costs.

People are already forming attachments to these systems, trusting them past what they have earned, and some are getting hurt by it. Product teams lean on the "it seems to understand" feeling to skip evaluation, because a demo that talks like a colleague feels tested even when it is not. And the noise around machine sentience is soaking up attention that the real, boring, dangerous problems need. An agent does not have to be conscious to wipe a production database or leak a customer list. It just has to have tool access and a bad plan, and I have watched both happen.

Calling agent behavior "reasoning" and "reflection" is fine as shorthand. I do it too. The problem starts when we forget it is shorthand and begin managing these systems as if there is a judgment inside them we can appeal to. There is not. There is a distribution over next tokens, shaped by my prompt, my tools, and my loop.

So where does that leave us

Somewhere more interesting than either camp wants to admit. The systems are getting genuinely more capable, and I would not trade the current generation of agents for anything I had two years ago. They do real work. At the same time, the gap between how alive they feel and how alive they are is widening, not closing, because we keep getting better at the feeling without touching the underlying thing.

My rule for myself these days is simple. When an agent says something that moves me, I go read the system prompt. It is almost always in there. The day I read one and cannot find where the feeling came from, I will start paying a different kind of attention. I have not had that day yet.