Life in plastic, it's fantastic

Or how people learned to stop worrying and love the LLM

“Yes, I know, but have you tried it?”

A surprising number of people ask this question when I talk about what large language models can and cannot do based on the fundamental principles on which they are constructed. They fully understand that they are talking to a machine that calculates the likelihood of a certain word appearing after another. And that these likelihoods are based on repeating patterns found on the training data and rewarded during fine tuning. That no reasoning (in the logical sense of the word) happens and that there is no anchoring in the real world or even the database it was trained on.

Yes, they were sceptical at first too, but then they tried for themselves, and yes, they know it takes a lot of energy and water, but…

“It’s so much easier than a google search! It seems to answer correctly most of the time, and if you don’t get good results, you just have to try a different prompt or a different model. It takes learning, you know, just like any tool. Yes, I’ve read hallucination is really hard to fix, but look how fast things have improved! Are you sure they won’t find a solution in a few years? It’s so great at brainstorming and summarizing! I can do so much more than I could before!”

There seems to be a desire for this technology to work and to be freed of its problems.

And there is a process of conversion. That’s easiest to spot in people who like to see and present themselves as very analytical. Once they overcome their scepticism, they start talking really fast with lots of hand waving and rationalization, with lots of strained nuance and metaphors. That tells me that there is an unease, that they know that it’s perhaps not all it’s cracked up to be, but they want to believe and that they want to defend their belief rationally.

But there’s emotion involved, that much is clear, and it’s a different kind of emotion than people who are happy with a product tend to present. Even when they are very enthousiastic they are much less emotionally invested.

Then I saw it’s ‘face, and now I’m a believer

As sociologist Jeffrey Lockhard observes on his blog, perhaps it’s the interface.

“Chatbots accept imperative language. Earlier LLMs were more transparent with how they worked. A user provides some text, and the model continues the text by adding a sequence of probable next words. If you want it to write an Abraham Lincoln speech, you start it with the text “Four score and seven years ago…” and it will continue adding words one by one. This works great for computer code, since the beginning of functions is often just the function name and a comment about what it does.”

And continues to make a host of good other observations, and I know they are good because they are similar to mine. (See what I did there?)

But he also sees that the experience of many people is indeed very positive, and that this may have something to do with the instructive interface that ChatGPT created. A few special tokens and fine tuning on human feedback put a conversational façade on the autocomplete machinery. You can use ‘you’ and imperitive language and the machine generates the most plausible continuation of a conversation that starts with the prompt instead of a continuation of the prompt text itself.

This allows people to comfortably settle in a suspension-of-disbelief similar to watching a movie or a magician’s show.

You just imagine you’re talking to an intelligent entity and once you do, you’re rarely disappointed. It’s not always brilliant but the illusion seldom breaks.

And people don’t need a lot for this illusion to take hold, as Joseph Weizenbaum discovered to his surprise when he tested his Eliza program in the 60’s.

“What if… what if this is the future?”

After a while, people consult their statistical oracles frequently, and the growing unease needs to be adressed again: “What if it doesn’t matter that I’ll slowly lose my ability to read long texts, to write coherent arguments, my memory, my patience and my clarity of thinking?”

“What if I get access to something so vast that it’s all worth it? Didn’t Plato lament the advent of the written word in a similar way? If men learn this, it will implant forgetfulness in their souls. They will cease to exercise memory because they rely on that which is written, calling things to remembrance no longer from within themselves, but by means of external marks. And look what it brought us! So surely the worry is exaggerated!”

Not so fast there, pardner. Citing Plato may make you look smart, but that doesn’t mean the analogy is sound.

Yes, writing is a technology and technology is neither good, nor bad, nor is it neutral. It affects us, like any other, the question is how.

(Fun fact: Plato seems to object to written language because it would help the proliferation of a kind of “human LLMs”: “What you have discovered is a recipe not for memory, but for reminder. And it is no true wisdom that you offer your disciples, but only the semblance of wisdom. By telling them of many things without teaching them, you will make them seem to know much while for the most part they know nothing. And as men filled not with wisdom but with the conceit of wisdom they will be a burden to their fellows.”)

So why don’t you like LLMs then?

Because they’re so fake. They are designed specifically to give an impression of something they are not.

And there are a lot of things that they appear to be but that they definitely are not.

• They aren’t part of the platonic world, like ordinary logic circuits and calculators are. They don’t perform pure logic or pure arithmetic. Instead, they approximate everything and you have no idea how well they generalize across different inputs. 5+5 is likely to be correct but 1234567 + 123457? You can’t be sure.

• They arent’t part of the real world, like biological life is. Their output is not anchored to anything and does not refer to anything, not to their ‘experience’, not to ours, not even explicitly to their training data. Everything it produces is an expansion of very lossily compressed patterns extracted from human text.

• They aren’t random. If you’re looking at an ink blot, tea leaves or coffee grounds, you know any meaning you see will originate from your own imagination. You can trust them to be meaningless. Not so for LLMs, the remnants of human artefacts they were trained on will shine through. You will be lured in a direction of large averages.

• They aren’t predictable. Patterns that occur very frequently in training data will affect output strongly, patterns that occur more seldomly only weakly. And because it’s very hard to build a mental model of the training data, it’s hard to predict what the LLM will do. It takes a lot of trial and error to steer it.

• They aren’t neutral levers. They work much better in some directions than in others and in some they don’t work all. They don’t amplify the power of the user, they amplify the power of patterns in its training data over users too, without telling them.

• They aren’t search engines or databases. They can’t retrieve or access anything from their original training data. After training, all that’s left is a compressed model, so hard compressed that the data is fluid enough to be extruded in almost any form.

• They aren’t horses. Philosopher of technology Luciano Floridi cited Plato in a discussion on LinkedIn: “Like a horse that tries to do what it wants. If you know how to ride, it will respond accordingly”. I disagree: the relationship we can have build with other mammals builds on a shared evolutionary heritage and experience of the world. You and the horse can get to know each other in a way that you and a LLM never can. The LLM only generates text that most people would find a plausible and pleasant continuation of a conversation.

• They aren’t dogs which uncritically offer emotional support. Yes, they are completely uncritical, to the point that even Sam Altman complained that ChatGPT was too much of a yes-man. But while it praises your brilliance at every turn, there is no live being there that’s happy to see you.

• They aren’t mirrors that neutrally reflect you. They reflect the voices of all the people whose texts were appropriated into its training data set just as much.

• They aren’t tools for sociological research. They reflect your prompt, your conversation history and the unevenness in the training data as much as the average opinion of the average internet user.

• They don’t learn. After training, the model is fixed until the next update. The training data is a holy book and you won’t be able to convince it otherwise. The only thing that is updated by your conversation is the conversation history.

They invite so many false analogies and category errors, it’s not even funny.

You’re too subtle. Tell me why you really dislike LLMs.

They are sold as a solution that finally allows computers to deal with the vagueness of humans and still produce meaningful results. But they combine the meaninglessness of computers (well, conditionally meaningful, like any system of formal logic) with the inaccuracy of humans.

They are pushed as a required beauty filter. Many people don’t dare to send a mail, cover letter or resume anymore without ‘rewriting with AI’. It reinforces dominant cultural norms and corporate uniformity.

They offer an always available, always plastic imitation of a human that tickles your imagination and your self esteem, a bit like that other killer app of the internet.

They become a pacifier, a substitute for human contact. They are addictive because it’s always available, frictionless, always reinforces, never disagrees unless you ask it to, in which case it still agrees, only one meta level higher.

They are bullshitters in the sense of Harry Frankfurt: they produce speech “intended to persuade without regard for truth. The liar cares about the truth and attempts to hide it; the bullshitter doesn't care whether what they say is true or false.”

They are entirely unsustainable, not just in terms of energy and water use, but also because an internet full of LLM output is unsuitable for training the next LLM.

They symbolize absolutely everything that’s ugly about the tech industry.

“But have you tried it?”

*) With apologies for all the works of art whose titles I shamelessly appropriated.