Ted Chiang on Artificial Intelligence

I read a typically excellent take on AI today from Ted Chiang today and it left me sitting with his wisdom, as his writing so often does.

If you don’t know Ted’s name, you probably know his work – he wrote the short story Story of Your Life that the beautiful sci-fi film Arrival is based on. Kudos to Nathalie for introducing me to his writing; she read his book of short stories Exhalation (“Story of your life” is not in that collection, btw).

One of my favourite stories in that collection is the novella-length The Lifecycle of Software Objects, written in 2010, which rigorously, through fiction, tries to ask what digitial consciousness looks like. The theses are nuanced, but the main idea I took away from it was that if digital consciousness were possible, then it would need to be not just created, but parented.

No, Artificial Intelligence Is Not Conscious

Ted has written barn-burning iron-clad plain-spoken articles in the last couple of years unpacking the ethical question of using AI for creative work and the polluting effect inherent in AI, but this piece in The Atlantic is about whether LLMs are conscious, and he’s pretty unequivocal.

Should we seriously consider the possibility that Claude, or any large language model, might be conscious? And if it has feelings, is it capable of receiving moral instruction?

No. Absolutely not. Generative AI is harmful enough when we understand it as a conventional technology, but if we confuse fluency at generating text with consciousness or moral agency, we’re at risk of assigning responsibility to entirely the wrong parties whenever anyone uses a chatbot.

Just, no, basically. But he does of course go further.

I must say that I have never considered the possibility that an LLM is conscious, or that it could become conscious, with any seriousness…but if faced with a True Believer, I don’t know that I would have had the ontological tools to articulate why it didn’t just feel like a fallacy, but was impossible.

Now I do!

Starting with the fact that it’s pretty inconceivable that we would just stumble on something like consciousness. To say that we have achieved something so monumental as a society without any of the steps to get there, is immediately suspect.

Faking the moon landing is a good step toward faking a Mars colony, but it’s not a good step toward actually putting astronauts on Mars.

And while I knew the underlying process or loop of running an LLM is iterative (they don’t plan ahead, they just generate the next most likely word), seeing chatbot interactions characterised, accurately, as just interactive co-authoring of a document with a statistical word generator, made the scenario even more ridiculous.

Being open to the possibility that LLMs are conscious is the same as being open to the possibility that Microsoft Word is conscious, or, more precisely, that multiple distinct consciousnesses are dormant in every Word document containing a conversational transcript, and that they are awakened every time the document is loaded. Should you consider the possibility that every time you open a Word document, you are bringing multiple conscious interlocutors into existence, and every time you close one, you snuff their existence out? No. Contemplating that scenario is not a good use of your time. Even if the Microsoft Office team employed a philosopher who said you shouldn’t be so certain, because consciousness is not well understood, that would not be sufficient reason for you to take this idea seriously. We don’t need to fully understand the nature of consciousness to definitively say that certain things are not conscious, and conversational transcripts fall in that category.

I’m not going to just quote the whole article here, it’s well worth a read.

Choosing his audience

I will also note that Ted writes beautifully, he has an amazing command of the English language. I think it is very interesting how flat and unaffected his writing is in these articles, most of all in his New Yorker piece explaining why AI can’t make art. I was nodding furiously throughout, but I also felt that I wasn’t really the target market for his arguments.

I absolutely agree with all of Ted’s points, he is speaking about ethics, morality, personhood, justice, creativity, expression and art. Importantly, he is not talking about how these technologies can be used for other applications. That’s a much less interesting set of questions, both philosophically and in piercing the sheen of marketing that encases these deeply troubling companies.

I’ve been using them at work and even personally, for wrangling information and automating some of the drudgery of professional software development. The efficacy of LLMs in these scenarios is frequently wildly overstated, but it’s also not nothing. Things like publicly musing their text-prediction engine may be conscious, and waving next to the Pope, allow Anthropic to build hype for their pension-fund-obliterating IPO.