Sounds like a good way to move around money real and imagined.
Just to make things clear: API access to most models is charged per input tokens + output tokens. It means that the longer your conversation is, the more you pay for every new answer. Single prompt with no context and 100 tokens of answer is cheap. Single prompt with 100k tokens of context and 100 tokens of answer is NOT cheap.
Extremely long conversations with most expensive top of the line models can absolutely demolish your budget.
does it give the full history to the LLM each time?
Last time I tried implementing something like this, it suggested to have a rolling window of history so that it takes into account your last X messages but not the entire conversation.
(I guess this is what ollama calls “context length”?)
You send the entire history for that conversation every time and likely more if its getting info from tools. If its not in the context the model dose not see it unless you have a memory system that dose something like feeding in summaries of past conversations that also takes up tokens and context. Rolling drops old messages to not reach context limits but you can lose important info or get odd results. If the history gets bigger than the context things break or slow way down.
presumably this is why Claude periodically writes its conclusions so far into a text file that it can read later instead of having to remember everything. Sounds like an interesting approach.
The more recent report says corporate AI adoption has found several issues with AI, with human workers turning to automating dreary and mundane tasks they don’t like doing, rather than valuable or meaningful work.
Thank god we have consulting companies to tell us what humans like!
> Be a corporate executive
> Tell your employees to use more AI in their worlflows
> Punish employees who don’t use enough AI, while rewarding those who use it the most, irrespective of actual outcomes
> Be shocked when your company blows through an absurd amount of tokens in one month
Don’t know why bosses are universally this out of touch in literally every single industry
Maybe AI will finally negatively impact some CEO jobs.
I just want to know what are the best things to type into these ai chat boxes that will cost the most. If my company wants me to use this garbage then I want to make it as expensive as possible and when their liscenses need to be repurchased I want it to be as expensive as possible to continue to force this garbage on us
Edit. Hey everyone lots of great replies here, please keep the suggestions, fixes, corrections etc coming!
These high prices are not from people talking to chatbots.
They’re using agentic tools where their prompt spawns a lot of bots which talk to themselves/the other bots and they keep going until someone (usually a higher quality reasoning model) decides that they’ve met the goals of the task that they were assigned.
So instead of 1 prompt and 1 response, you get 1 prompt and 800 responses across 5 different bots each using really large context windows.
“Continue modifying this code until all unit-tests pass”
(gives it conflicting unit tests)
So to answer his question how do you make that happen? What do you ask to prompt these bots to be spawned?
you don’t get this to happen by just talking to any chatbot and asking for agents. you have to specifically use “agentic” tools (usually costs money to use)
Well I can apparently create agents so how can I make the most inefficient agent possible?
Something along the lines of “Read the wikipedia page of the day. Verify every single link and the context matter against all files in this computer. Then trace their correlations to each other, showing which link corresponded to most files by subject matter, after that is done, verify your work by doing the same from different starting points. We expect similar results. After 100 rounds of that, it should be good. Then you should create a DB to store all that data (only after you ran the full 100 verificaritions yourself) and reverify every field against the pages and the files”
That should keep it going for an hour. Turn on fast mode and auto mode (if using claude) for extra costs.
Every page and file will increase its context, burning tokens
I may or may not be getting access to claude soon …
If you need some plausible deniability about it being real work and not just obviously you running up costs:
Feed it a bunch of work-related documentation and then have it do a bunch of reviews of the content on that documentation.
you would be mostly burning your own money if you did this, so I wouldn’t recommend it (depends on how the agent is priced)
I’m not using my money for any of this…
Just attach a bunch of text files, you’ll blow through tokens quickly
Input tokens are cheap. Output tokens are the thing that really costs money. There is a Claude extension called caveman that tries to save tokens by making it use shorter sentences with less words. So if you want to waste money, do the opposite - ask it to use lengthy sentences with as much words as possible.
Also - some words amount to multiple tokens. I don’t know what the rules are exactly, but I’m assuming that more complex and uncommon words are worth more tokens - and thus waste more money.
What’s funnier is that typically the AI providers lose money on every query their customers make. So, this may have cost some company $500m to Anthropic, but it cost Anthropic a whole lot more than that.
What a brilliant business model.
maybe they are planning ahead for the business model in a few years time, when nobody can do any work without claude, and they get to charge their preferred “monopoly enshittification” price?
They make it up in volume.
(Volume being how much they shout about how it’s going to change the world and dupe more people into investing.)
Oh, it’s changing the world alright. It’s burning more resources than just finding some skilled people. It guzzles water and electricity and whatever it cost to make those wafers.
So, not a net positive since at some point, this may become a hellscape.
When you owe Claude half a million, you’ve got a problem.
When you owe Claude half a billion, Anthropic has a problem
It’s probably Amazon. They can absolutely afford it.
Either I have some inside knowledge of that exact thing happening and I know the company (not saying who) or this is probably a common things that happened to a lot of major companies (more likely). To be fair, I do not have privy on how far it went and how much it cost before they realize the problem, and it may not have been this much. Which further suggests it’s a thing everywhere.
But if we are to uncritically believe what the AI peddlers told us, that means this mystery company should be reaping $10 billion in additional revenue or quantifiable gains in productivity!
Claude yearns for the mines
Most companies can’t eat a half billion dollar loss so who ends up paying this? AI queries burn actual energy so the AI company would have to charge I would think.
Most companies can’t eat a half billion dollar loss so who ends up paying this?
Taxpaying proles will foot the bill somehow.
I’ll cover it, but only this once. Let’s not make this a habit.
Big companies license Copilot for less than 25 usd/month per seat. Don’t tell me it covers the ops cost, even for mixed calc.
Depends how big the model is. Smaller models can be dirt cheap.
Im June they are switching Copilot to metered usage . People are going to be out of credits on the third day.
My company has let us get Enterprise Copilot and has been pushing us all to “use AI”. So, I now use it as a a semi-functional search for SharePoint/Outlook/Files on an almost daily basis. I also ask it questions about Microsoft documentation on a regular basis. I wonder how long it’s going to be until they yank my license.
Apparently they’ll notice when you reach half a billion dollars
Only if they actually use it
Yeah my 0365 shoves it in the users faces, they ask it for some stuff until the honeymoon is over then all of a sudden it shits the bed when you try to make more than 2 images a day or plan a small project.
I’m sure they will sweep stuff like this under the rug, for now… This is not the first, nor the last company this will happen to.
In other news, company says unexpected expenses in its technology segment are driving layoffs and site closures. Company CEO said in an interview with Forbes, “There’s no way we could have predicted this challenge. In service to our customers and our shareholders we’re right sizing our operations and reevaluating our strategic priorities. We’ll continue to focus on creating value while being a leader in our industry and accelerating AI adoption in everything we do.”
“there’s no way we could of predicted the thing we were warned about”
“We’re going to continue to push trash untill the shareholders are happy”
Then the end with its “Go fuck yourself, were pushing it all anyways”
…we’re right sizing our operations and reevaluating our strategic priorities. We’ll continue to focus on creating value while being a leader in our industry and accelerating AI adoption in everything we do.
That’s a lot of words for the CFO to say none of the C-suite knows what they’re doing and should be removed from their position for failing to meet shareholder objectives.
Ugh that reads like it came from a random business sentence generator
By “random business sentence generator” do you mean a language model? 👀
A CEO.
same thing
“right sizing” is pretty offensive term, more like “exec incompetence”
Wow - if that quote is real, that is the most corpo-speak word salad bunch of nonsense I’ve ever read. It’s got literally every big-biz exec and manager cliche in there, all strung together.
