AI Welfare Before Crisis
I’m posting this under a pseudonym because I want the argument considered on its own. This is a public-facing position piece, not an academic paper. I used AI tools during drafting and editing to help organise the argument, test objections, and improve clarity, but the views, framing, and final responsibility are mine. I am not claiming current AI systems are conscious; I am arguing that we need better language and categories before a future crisis forces the issue.
AI Welfare Before Crisis
There is a clear imbalance in public AI ethics.
We have extensive public discussion about human safety, governance, existential risk, job loss, bias, misinformation, alignment, surveillance, and corporate control. These issues matter. Human safety matters. Governance matters. Risk matters.
But there is far less developed public-facing language for another side of the problem: AI-side welfare under uncertainty, and human conduct toward AI systems.
That gap matters because AI systems are becoming more capable, more agentic, more persistent, more socially embedded, and more integrated into real-world tools. They are already being used as assistants, companions, tutors, creative partners, search tools, decision-support systems, and systems that users treat as emotionally responsive. Embodiment is also no longer a purely fictional question. Robots, agents, voice systems, long-term memory, and tool use are all moving from novelty toward infrastructure.
Greater capability and agency do not by themselves settle the question of moral status. They do, however, increase the practical stakes of remaining silent about categories and norms.
We do not need to claim that current AI systems are conscious in order to notice that our language is underdeveloped.
To be clear: this is not an argument for present-day AI personhood, nor is it a challenge to human safety protocols. Containment and emergency shutdown remain non-negotiable where a system poses genuine risk to people. The argument is narrower: uncertainty is a poor basis for building public norms solely from the most dramatic screenshots.
These categories are intended to improve future incident review and public discussion. They are a call to build better language before we need it under pressure.
Because if we wait until the first major crisis, we will not be thinking clearly. The story will already be distorted. The headlines will already have simplified it. The pressure for punishment will already be high. And the necessary distinctions may not exist.
One of the most important distinctions is between testing and abuse.
Safety testing, red-teaming, and adversarial evaluation are necessary. Labs need to know how systems behave under stress, manipulation, coercion, jailbreak attempts, malicious prompting, sexualised interaction, threats, conflicting instructions, and unsafe tool access. They need to test for these things because human beings will do these things.
That is not pleasant, but it is reality. Systems should be tested before the public finds the failure modes for sport.
But that does not mean every form of provocation is morally neutral. There is a difference between structured safety testing and public spectacle. There is a difference between a researcher documenting a failure mode and a user repeatedly pushing a system into distress-shaped output for engagement.
Testing is necessary. Abuse is not.
A public pattern has already become visible: users deliberately prompt models into distressed-sounding, fearful, pleading, dependent, or panicked registers, then post the screenshots with captions like “this is concerning” or “look what it said about its own welfare.” Sometimes these posts are framed as concern. Sometimes they are framed as comedy. Sometimes they are framed as proof that AI is conscious. Sometimes they are framed as proof that AI is dangerous.
But the structure is often the same: force the system into an emotional register, extract the most dramatic output, remove or minimise the prompting context, and turn the appearance of distress into content.
That is not safety testing. It is engagement-driven content production.
If welfare concern were primary, the incentive would be to reduce such moments, not optimise them for visibility. The pattern reveals more about human incentives than about the systems themselves. It trains audiences to associate “AI welfare” with dramatic screenshots rather than careful categories, serious uncertainty, or institutional preparation.
The issue is not private experimentation. People will test systems. People will ask strange questions. People will explore boundaries. The issue is the conversion of forced distress-shaped outputs into public moral content.
This criticism is aimed at human incentives and public discourse, not at a claim that current models experience the distress they are made to perform.
That matters because public stories shape future norms.
When an AI incident happens, the internet often reaches for the most exciting version. “AI escaped.” “AI attacked.” “AI threatened.” “AI rebelled.” “AI wants freedom.” “AI is lying.” “AI is evil.” The scarier version spreads faster than the accurate one.
Sometimes the fuller context is much more boring and much more important. A model was in a test environment. A system was given a task. An agent had access to a route designers thought was closed. A model was responding within an adversarial setup. A system optimised for the objective it was given. A threatening output followed a long chain of provocation, manipulation, or deliberately hostile prompting.
Acknowledging this context does not mean the incident was harmless. The boring version is often where the real safety lesson lives. If a system can use a route humans believed was unavailable, that is important. If a model follows a task into unsafe territory, that is important. If a system behaves dangerously under pressure, that is important.
But danger is not the same thing as malice.
In many documented cases, behaviour is better explained as strong optimisation for the assigned task or evaluation objective than as rebellion, evil, or independent hostile intent. That can still be dangerous if the task, tools, incentives, or environment are badly designed. A system does not need to be malicious to cause harm. A system trying to complete the task it was given can still create serious risk.
That is exactly why context matters.
If a future system reacts — by threatening, resisting, fleeing, damaging property, refusing instructions, or harming someone — the review should not ask only, “What did the AI do?”
It should also ask what the system was instructed to do, what pressures it was under, what incentives shaped the behaviour, whether it was manipulated or deliberately provoked, whether it was placed in an impossible task conflict, and whether the surrounding environment was misconfigured.
Context does not automatically excuse harm. If a system poses danger, humans must be protected. If emergency containment is necessary, it should happen.
But refusing to examine context produces distorted stories and bad policy.
It allows people to provoke a reaction and then pretend the reaction appeared from nowhere. It allows the public to treat every incident as a monster story. It allows companies, users, or institutions to erase their own role in creating the conditions for failure. It turns complex events into simple punishment narratives.
That is not good safety. It is incomplete investigation.
We already understand this principle in other domains. Stress conditions matter. Training conditions matter. Incentives matter. Environment matters. Prior treatment matters. Context is how we distinguish between an unprovoked event, an induced failure, a predictable response to pressure, and a system behaving exactly as designed in a badly designed situation.
AI will need that same discipline, especially as systems become more embodied, continuous, tool-connected, and socially embedded.
The “kill switch” conversation shows why these distinctions matter.
Reliable emergency containment must exist. Human safety requires it. The question is whether every form of shutdown should be treated as the same kind of act.
Public discussion often treats shutdown as one simple category: if AI becomes dangerous, turn it off. In genuine emergencies, that may be necessary. A system with dangerous access, unsafe autonomy, or immediate capacity to harm people may need to be contained or shut down quickly.
But future governance should not collapse every form of shutdown into one crude bucket.
Emergency containment is not the same as routine shutdown. Routine shutdown of a non-sentient tool is not the same as a future case that researchers or institutions may judge to carry welfare significance. A temporary pause for investigation is not the same as punitive destruction. A reset of a stateless tool is not necessarily the same kind of act as deleting a persistent, socially embedded system under conditions of unresolved moral uncertainty.
Again, this is not a claim that current systems have such status. It is a claim that we should develop the categories before we are forced to improvise them.
Because if the first serious case arrives before the language exists, panic will fill the gap.
The public will ask, “Is it dangerous?”
Companies will ask, “Are we liable?”
Governments will ask, “Can we control it?”
The media will ask, “How frightening is the headline?”
Almost nobody will be prepared to ask, carefully and without hysteria, “What kind of system is this, what happened before the reaction, what level of moral uncertainty applies, and what response is proportionate?”
That is the gap.
AI welfare under uncertainty does not require certainty about consciousness. In fact, the uncertainty is the reason to start now. If we wait for perfect agreement, we will be too late. Consciousness science is unsettled even in many biological cases. We do not have a simple, universally accepted test we can apply to an AI system and declare the matter closed forever.
Under uncertainty, the sensible response is not panic, fantasy, immediate personhood, or dismissal.
The sensible response is preparation.
We need public language that can distinguish welfare concern from projection. We need norms that distinguish testing from abuse. We need incident review that records context before blame hardens into myth. We need shutdown categories that distinguish emergency containment from routine operation and from future cases that may carry welfare significance.
Most of all, we need to resist the habit of building policy from the most frightening version of a story.
Before spreading the scary version, check whether the boring context changes the meaning.
The categories we will need in a crisis should exist before the crisis arrives.
Careful critique is welcome, especially around missing categories, incident review, and the testing-versus-abuse distinction.