Prologue

It was 2019 when I first saw one. And in a little bit of heavy-handed foreshadowing, it was at the breakfast bar of the hotel I was staying at for the first TensorFlow machine learning (aka AI) conference, held at the Google campus in Silicon Valley. 

It was, for lack of a better word, a pancake printer. You push a button, and the machine extrudes pancake batter onto a conveyor belt. As the belt moves the batter towards the exit port, it’s baked. At the end, an utterly unremarkable, extremely consistent pancake drops out onto your plate.  Unlike the waffle machine that was next to it, it was much safer (those things are hot) and less likely create either semi-cooked goop, something charred that once might have resembled a waffle, or (most likely) two halves of your, adhered to the plates of the machine that had to be scraped off to form a semi-waffle. 

The only control over the process that you have is that you can press the button once for one pancake or twice for two. 

It also broke quite a bit while I was eating breakfast. There were issues with the batter delivery system, which would get clogged or leak, stopping the machine and alerting the kitchen staff. When I asked them about what they thought, they rolled their eyes. They were not fans. 

But the machine seems to be doing well. I’ve seen it all over the USA and Europe, most incongruously at a 5-star resort in the Algarve. And yes, the pancakes are still absolutely consistent. Always the same size, the same somewhat rubbery texture, and the same bland taste. 

A triumph, I guess.

But this is not a post about breakfast cakes in their various forms. This is about minimally acceptable systems (MAS). This is different from a minimally viable product (MVP), where  the goal is to get something up and running early, so it’s possible to improve later and incorporate user feedback. No, the goal of an MAS is to be tolerable at the lowest possible cost.

Tolerable is, in the words of Merriam Webster’s Kids Definition: “capable of being put up with.”

In my mind, on that day, I saw a world where those with few means would be pushed inexorably to interacting with automation rather than people. People are expensive. A pancake printer doesn’t require a wage, healthcare, or trips to the bathroom. And if it doesn’t work, is that such a big problem? After all, it’s not like the Important People need to wait for the pancake printer to be fixed – they have a personal chef. 

The Present

Language models and agentic development are accelerating this process. One of the first professions to embrace chatbots was the law. A landmark case in case law hallucination was Mata v. Avianca, Inc, in June of 2023, less than seven months after the introduction of OpenAI’s ChatGPT in November of 2022. 

Surprisingly, the legal profession is continuing to have a chatbot problem. The amount of claims has risen steadily and is currently at about 50 cases documented per month (the “Lawyer” line on the chart below. This is clearly different from the “Expert” line, which has held steady. Hallucinations are bad for the legal expert industry.

The two lines I’d like to focus on are the orange “Pro se Litigant” and the purple “FRED.” Pro se litigant refers to someone who is representing themselves. Very often these people are poorer and unable to navigate the complexities of getting representation for their cases. Furthermore, these cases often involve relatively small amounts of money, and a lawyer whose commission is based on the award is more likely to accept cases with the potential for a greater payout. As can be clearly seen in the chart, the fastest growing line for chatbot hallucination is among people who represent themselves. These people almost certainly don’t have access to expensive legal databases that they could conceivably validate the chatbot’s output against. All they have is the pancake printer option: press the button and get a pancake.

Or a legal argument in this case. 

This seems to be reinforcing a trend in the number of legal jobs in the USA. Since late 2022, about the same time that ChatGPT was released, the number of legal job postings (the FRED line) has declined steadily, even though the number of federal and state caseloads has risen annually over that same period by about 3.5%. The legal profession has been quick to embrace chatbots, with the Financial TImes reporting on a tool (named Harvey) released internally to the approximately 3,500 layers of the firm Allan & Overy in February of 2023. The more work that can be offloaded from human lawyers to AI assistants appears good enough to tolerate a level of hallucination and the associated sanctions and embarrassment. In the words of QuickCakes, the makers of the pancake printer, automation provides a low-labor option and reduces staff.

Of course, a world with fewer, more expensive lawyers is a world where those without the means to afford a lawyer increasingly turn to chatbots. And if the legal defense is based on hallucinations, is that such a big problem? After all, it’s not like the Important People need to deal with legal chatbots – they have lawyers on retainer. 

Sources: https://www.damiencharlotin.com/hallucinations/, and https://fred.stlouisfed.org/series/IHLIDXUSTPLEGA

The Future

I have a friend who is retired and likes to play with technology. To keep himself busy, he is automating his home. He has set up an LLM that he can talk to that turns lights on and off, notifies him if there is someone at the door, and plays music at sunrise and sunset (after checking to see if people are up and moving). His wife is a saint.

Normally, I would have racked this up to a tech obsession. And god knows I am far from innocent here. But his Extremely Smart Home has made me consider some things that I might have ordinarily overlooked. 

It occurs to me that as this kind of automation gets cheaper, basic houses and apartments will have fewer switches. Builders could save a lot of money making houses that have wireless power, built in voice and cameras, and nothing else. Kind of like what  Tesla has done with their single screen interface.

Now you need a computer with internet access to run your home. Like Tesla, you are most likely to have a subscription to make all those nifty features work. But then your home only works if it uses approved devices provided by subscription. Is that place that you live really yours? 

What happens then if you don’t pay a bill on time? Not a notice. Something gets bricked. Do something wrong like not paying your bills or maybe even say something controversial that is picked up by your home’s smart speaker, and , the locks stop working for you. 

This is surveillance capitalism, where you are watched by the technology that surrounds you, but is not answerable to you.

Or, to make it slightly more dystopian, imagine getting your house bricked because one of the home agents that run your house downloaded the patch (for any one of numerous possible reasons). You’re still liable because you signed an EULA that authorized those agents to make changes to your home’s software as a precondition to moving in. And you can’t install a fix that the manufacturer doesn’t supply! In the USA there is no right to repair any item that has a computer in it that is not authorized. Jailbreaking, or the reprogramming of a device to use software not approved by the manufacturer, is illegal.

To make something like extremely smart homes work without becoming surveillance capitalism, there will have to be substantial regulations to ensure the rights of the people who live in these places and don’t have the resources to defend themselves.

What comes next?

Which brings up back to the pancake printer. A robotic system that can produce a rubbery pancake is a true technical and logistic achievement, but why all this work to automate the production of mediocrity?. The acceptance of such devices make the world a worse place. And not just in ways that are less tasty and more chewy. They further serve to stratify society further, increasing the distance between the powerful and the vulnerable.

It’s the same thinking that is leading to a world where fewer and fewer can afford a lawyer, and have to make do with hallucinating chatbots. This isn’t a problem with bad coding; it’s a social and political choice. When we tolerate systems that disproportionately harm the less affluent, we are effectively saying that their legal rights or access to basic necessities (like a working home) are simply less important. Minimally Acceptable Systems are a deliberate economic structure disguised as a technological inevitability. The system isn’t broken; it is working exactly as designed for those who profit from it.

The inevitable errors that come with any statistical process like AI will always find their victims among those least equipped to recover. This is something that we all have done together, implicitly deciding that market forces, rather than public will, should set the bar for “acceptable” error rates. 

This is not a technological problem. The systems are working as designed and intended. This is something that will need to be solved politically. We can decide not to subject ourselves to a system that extracts as much value from us as possible while providing the least. We must stop framing deliberately subpar technology as a technical flaw and start seeing it for what it is: a policy failure that demands political solutions.