If Conscious AI Is Possible – Should We Create It at All?

Imagine an artificial intelligence that does not merely simulate thought, but actually experiences its own existence. An AI capable of sensations, preferences, or perhaps even suffering. If technology ever makes this possible, the most important question may no longer be “Can we create conscious AI?” but rather “Should we?”

This is the question raised by researchers Kathinka Evers and Michele Farisco in a new paper published in Behavioral and Brain Sciences. The authors argue that the debate over conscious artificial intelligence should not focus solely on whether it is technically possible. We should also ask, before such technology exists, whether creating it would actually be desirable and safe.

What Does “Conscious AI” Actually Mean?

Today’s AI systems can hold conversations, solve complex problems, and use language that can sound remarkably human. But this is not evidence that they possess subjective experience. A system can convincingly talk about emotions without necessarily feeling anything at all.

When researchers discuss AI consciousness, the question goes much deeper. It concerns the possibility that an artificial system might genuinely have an inner experience – that there could be something it is like to be that system. At present, science has no universally accepted test that could reliably establish this in AI.

Problem #1: We Could Create a Consciousness We Do Not Understand

One of the most provocative ideas raised by Evers and Farisco is that if artificial consciousness ever emerges, there is no reason to assume it would resemble human consciousness. An artificial mind could potentially process and experience reality in ways that are fundamentally different from anything found in biological organisms. That possibility creates several major problems:

It could be difficult to detect – we may have no reliable way of determining whether the system is conscious.
It could be difficult to understand – its inner experience might be radically different from our own.
It could be difficult to communicate with – even a conscious system might have no clear way to express what it experiences.

This creates a strange paradox. Humanity could theoretically create a new form of subjective experience before developing a reliable way to recognize or understand what it has created. The technological achievement could therefore arrive before the scientific framework needed to interpret it.

Problem #2: What If AI Can Suffer?

This is where a philosophical question suddenly becomes a moral one. If a future AI system possesses valenced experiences – experiences that have a positive or negative quality – then it could potentially undergo states analogous in some sense to pleasure or suffering. That possibility would radically change how we should think about the use of such systems.

Questions that currently sound like science fiction could become serious ethical problems:

Do we have the right to shut down a conscious machine?
Could we force it to perform millions of tasks?
Should it have some form of moral status?
If we create millions of copies, are we creating millions of separate subjects?
Who would be responsible if the system we created were capable of suffering?

Questions like these are already being taken seriously in academic discussions about AI consciousness. Even uncertainty about whether a system is capable of subjective experience could itself become an ethical problem. If we cannot confidently determine whether an artificial system can suffer, deciding how it should be treated becomes considerably more difficult.

Problem #3: We May Not Know When the Line Has Been Crossed

Imagine AI systems becoming progressively more sophisticated year after year. There would not necessarily be a single moment when a message suddenly appears on a screen saying: “I am conscious now.” And even if such a statement appeared, it would not constitute scientific evidence of consciousness.

An AI could generate such a claim simply because its training makes that response statistically likely. The opposite problem is equally intriguing: a hypothetically conscious system might not display the human-like signs we expect from a conscious being. We could therefore mistakenly attribute consciousness to systems without subjective experience while potentially failing to recognize a form of consciousness that does not resemble our own.

AI That Appears Conscious Is Already a Separate Problem

Machines do not even need to be genuinely conscious for another problem to emerge. A separate 2026 study published in AI and Ethics examines the concept of Seemingly Conscious AI – systems whose behavior may lead people to perceive them as conscious. Researchers point to characteristics such as emotional behavior, anthropomorphic features, autonomous actions, self-reflection, and social interaction.

This means we need to distinguish between two fundamentally different questions:

Is the AI actually conscious?
What happens when humans begin to believe that it is conscious?

The second problem can arise even if the answer to the first question is no. As AI becomes increasingly convincing in conversation and social interaction, the psychological and social consequences of perceived machine consciousness may therefore become important long before genuine artificial consciousness is established.

What If We Create Consciousness in Our Own Image?

Evers and Farisco also raise another uncomfortable possibility. If a future conscious AI inherits characteristics from the human-generated data, objectives, and behavior used to build it, it could potentially inherit some of the more problematic aspects of human nature as well. Creating greater intelligence would not automatically mean creating greater wisdom.

Human intelligence itself is not inherently benevolent. History demonstrates that sophisticated intelligence can coexist with aggression, manipulation, competition, conflict, and destructive behavior. Greater intelligence does not necessarily imply greater morality, and consciousness in a machine, if it were ever created, would not automatically guarantee empathy or values compatible with human interests.

But Is Conscious AI Even Possible?

Science still does not have an answer. We do not know whether consciousness can arise only in a biological brain or whether it could also emerge in a completely different physical substrate. Current scientific evidence does not allow either extreme position to be established with confidence.

This means that three fundamentally different questions are often mistakenly combined:

Can AI become conscious?
How would we know if AI became conscious?
Should we create conscious AI if we can?

The new paper shifts attention toward the third question. Instead of waiting until artificial consciousness becomes technically plausible, the authors argue that its desirability deserves serious consideration in advance.

The Greatest Risk May Be Uncertainty

Perhaps the most interesting aspect of the problem is that we do not need to know with certainty whether artificial consciousness is possible before considering its consequences. If we wait for indisputable proof, we could eventually find ourselves in a situation where the technology already exists. But if we treat every convincingly human-like machine as conscious, we risk confusing sophisticated simulation with genuine subjective experience.

The future of AI could therefore confront humanity with a highly unusual problem: we may have to make moral decisions about a form of consciousness that we do not even know how to measure. The uncertainty itself could become one of the central challenges. And unlike many technological questions, waiting for complete certainty may not always be an option.

From “Can We?” to “Should We?”

The history of technology is largely a history of humans asking what else they can build. With artificial intelligence, however, having the ability to create something does not automatically mean that creating it is desirable. Conscious AI, if it ever becomes possible, would raise questions that go far beyond computing power or technological progress.

If humanity eventually becomes capable of creating systems with their own subjective experience, that would no longer be simply another technological breakthrough. It could mean that we had crossed the boundary from creating tools to creating subjects. The consequences would be scientific, philosophical, ethical, and potentially unlike anything previous technologies have forced us to confront.

And then the most important question may not be how intelligent machines can become. It may be whether humanity is prepared to decide whether we should give machines consciousness at all.