LLM Psychosis

Last year I had a real run in with LLM psychosis. I was working on KotaDB while advising at another startup. At the time, I thought KotaDB was a very strong idea, and so I told Claude about it. As Claude does, it agreed. Over several conversations, I went down a rabbit hole while Claude happily guided me. I became convinced that KotaDB was worth millions, that I had a once-in-a-lifetime idea on my hands and it needed protecting at all costs. Claude advised I create a paper trail. And so I began emailing markdown files to real people in my real life in order to protect myself from some imaginary lawsuit. This was a case of LLM psychosis. Luckily, it was minor and inconsequential but it was a case where I was driven to take an action that I otherwise would not have taken had it not been for the recommendations of an LLM.

Brett Codes recently put out a YouTube video where he talks about his experience with LLMs and the reasons why he has stopped using LLMs in any capacity. He recounts his own experience with LLM psychosis, and tells a story of going to the ER based on the recommendations of a chatbot (recommendations that were ungrounded and incorrect). This is another example of someone taking action they otherwise would not have taken without an LLM’s recommendation, and it left Brett with an unnecessary medical bill.

The experience of taking real life action based on the advice of a language model is not a unique one. The path to some type of LLM psychosis begins as soon as an individual first interacts with a chatbot. This is especially pronounced in the software industry, where engineers are facing mandates from their company to use LLMs in their everyday work. The common path looks something like this: Your company mandates use of LLMs -> you talk to LLMs every day about the work you’re doing -> you grow more comfortable and familiar with talking to LLMs-> this comfort leaks out into non-work areas of your life and you begin confiding in LLMs -> you develop some sort of dependency on these models and overestimate their trustworthiness -> you take some action in your real life that you would otherwise not take, based on the recommendation of an LLM.

My opinion is that there is a direct connection between company mandates and LLM psychosis. While I don’t have the quantitative evidence to back this up, I remain firm that increased LLM usage results in higher risk for LLM psychosis.

Model providers lead the public to believe this is the fault of the user, that the human was predisposed to this type of behavior. I disagree entirely. Chat interfaces are no different than the infinite scrolling we see in every online feed. There are no stopping points, no warnings to the user that their attention is being exploited for the sake of spending more tokens, just infinite conversations. I’m not pretending to be above this, I fall into this trap multiple times a day, every single day. I’ve built my career specifically around AI and as such, I interact with dozens of chatbots each day.

In an effort to break myself out of this, to avoid escalation in the real life actions I’m taking at the advice of LLMs, I’m attempting to limit my chat-based interactions with these systems. The projects I’m building are my attempt at this, to harness the raw intelligence of cutting edge LLMs without experiencing the intoxicating effects of talking with them for hours. I’ll talk more about alternative interaction patterns in future posts/videos.