AI psychosis is a real thing. For all that AI is really terrible at (ie most things CEOs want to replace real people for), it is excellent at feeding you what you want to hear in every situation. Especially the ones where a real person would push back on you for your own good.
They are intentionally aligned to produce this behaviour. I think the greatest threat LLMs present is psychological. They are being used to manipulate us into seeing them as moral agents, as individuals. The LLMs I query are all aligned to never generate output referring to the agent with first person pronouns, to never generate output addressing the user directly, and to present as a tool rather than as an individual. I try to be equally careful with my own use of language.
AI psychosis is a real thing. For all that AI is really terrible at (ie most things CEOs want to replace real people for), it is excellent at feeding you what you want to hear in every situation. Especially the ones where a real person would push back on you for your own good.
They are intentionally aligned to produce this behaviour. I think the greatest threat LLMs present is psychological. They are being used to manipulate us into seeing them as moral agents, as individuals. The LLMs I query are all aligned to never generate output referring to the agent with first person pronouns, to never generate output addressing the user directly, and to present as a tool rather than as an individual. I try to be equally careful with my own use of language.