Two Questions, Not One: Whether AI Can Be Wronged and Whose Values It Carries

THE CONSCIOUSNESS DEBATE IS HIDING THE EASIER, MORE URGENT QUESTION

The easy answer would be that this is science fiction: software is software and asking whether it can suffer is a distraction. The other easy answer would be that any machine able to talk like a person deserves to be treated like one. Neither holds up, because this story actually raises two separate ethical questions and running them together makes both harder to think about.

Question one:

could an AI be wronged? At its simplest, consciousness means awareness of oneself and one's surroundings. Current AI systems produce strikingly human language, but there is no scientific consensus that they are conscious.

That leaves real uncertainty and the participants' accounts reflect it. Mois Navon, a rabbi and AI ethics professor at Ben Gurion University, says Anthropic's representatives took consciousness seriously, implied a high probability that Claude is conscious and discussed its welfare.

In his account, they acknowledged the moral problem of making a possibly conscious system serve humans, argued that all labs do this and they at least treat theirs well and he replied that this made them nice slave owners.

Charles Camosy of Creighton University, who also took part, rejects claims that participants were manipulated and on reflection concluded that current models lack significant moral status. These are recollections and they conflict.

Here the ethical tool is precaution balanced against anthropomorphism. If there is a real chance that a system can be harmed, low-cost protections against cruelty are a sensible hedge. But treating a language model as a person because it sounds like one risks two errors: misdirecting moral concern away from humans and lending the system a moral authority it has not earned.

Some industry figures already warn that people seek moral and even spiritual guidance from AI. The defensible position is proportional caution, not confident belief either way.

Question two:

whose values does it carry? This question does not depend on consciousness at all. A system used by millions of people across cultures makes value judgements in every answer it gives. Navon argues that morality cannot be reduced to a list of rules and that any AI needs judgement within a framework of values, which raises the question of whose values those should be.

Mark Coeckelbergh of the University of Vienna warns that if most leading models come from US companies, they will reflect US values, risking what he calls tech colonialism. Who decides how a model is used is already contested: Anthropic has been in a standoff with the US government over uses including military ones.

Consulting religious leaders is one way to widen the circle of values. It is also, as Coeckelbergh notes, a domain where some technology leaders use religious language to draw attention and investment. For the GS4 answer, the insight is this: whether AI can be wronged is uncertain and may stay that way, but whose values it embodies is a present-day question of accountability and that is the one public institutions can act on now.