Wanted to refactor a too large and hectic hacked together state machine into separate functions and reduce redundancy and increase code readability
Every model I’ve tried failed.
And that should be an easy task an actual AI should be able to handle.
Logic is done, just translate to a different structure.
That should be exactly what they are able to do at least.
None worked, because the project is just to big and interlinked.
I’ve got 24GB of VRAM, which helps a lot. I’m using the Q4_K_M version of this model, with the KV cache quantized to Q8, and that’s able to get me to 128000 tokens of context. I’m using the Qwen Code harness in VS Code.
If 128000 tokens still isn’t enough, break the task down into smaller chunks and help the harness out with MCP tools - I installed Codebase Memory to help it navigate large codebases more efficiently, for example. Ask the coding agent what’s causing problems and it may be able to help identify solutions too, I’ve had it troubleshoot itself several times now and it’s been quite good at inspecting its own settings and finding where the problems are with them. I even installed Firecrawl for it so it can do its own web searches.
When you add the additional codebase memory, that will also add all those mcp tools to each round-trip to the llm model. Do you still see improvements and less filling up if your context? Just curious how it works out; I can imagine that a text based memory that explains the architecture of the codebase would help as well. Not sure about the difference if using a graph to store the code context?
Depends on the framework you’re using. I’m using Qwen Code Companion (via a VS Code IDE plugin) and it’s got some kind of tiered system built into it for adding tool definitions to context, it’s got a standard built-in set that’s always mentioned but other stuff only gets added to the context under certain circumstances. I can’t recall all the details, I found out about it when I was chatting with Qwen Code to ask it why it hadn’t used a particular MCP I’d given it to solve a problem despite it seeming like a useful tool for the task and we got into a discussion about to configure it. As I recall it’s something along the lines of most tools only being present in context by default as little one-liner stubs and the agent has to tell the framework “that one looks like it might be useful for what I’m doing” for the full definition to get loaded.
Most of the time I’m working on small scripts and specialized applications these days, and codebase memory is overkill for those so I haven’t actually seen it in use much.
More Ram, and break stuff up into smaller pieces. I Have 12Gb of VRAM and 64 Gig System Ram, which - with CPU offloading - allows pretty large models to be used at acceptable speed (although probably not fast enough for someone who wants to make his living with it. But these also will invest in dedicated hardware). And i let a larger model sit there to coordinate, and let it use subagents for the rest. That keeps the Context for the main model a lot more manageable and keeps it on track for a pretty long time.
How you structure the Task is more important than what the task is at all. For example, i have a huge list of games to be catalogued, and just telling the model to search the web and fill in the blanks in the database makes it crap out pretty soon. But I tell it to let subagents handle everything regarding web search and data extraction and just let them hand back the resulting data. the main model then calls another subagent to sanity check the data and update the database. Each of these subagents do not run for long, and therefor never run out of context, while the main model can keep context size manageable because it does not have to work with “messy” data at all.
I’m always running into context limit problems
Wanted to refactor a too large and hectic hacked together state machine into separate functions and reduce redundancy and increase code readability
Every model I’ve tried failed.
And that should be an easy task an actual AI should be able to handle. Logic is done, just translate to a different structure.
That should be exactly what they are able to do at least.
None worked, because the project is just to big and interlinked.
I’ve got 24GB of VRAM, which helps a lot. I’m using the Q4_K_M version of this model, with the KV cache quantized to Q8, and that’s able to get me to 128000 tokens of context. I’m using the Qwen Code harness in VS Code.
If 128000 tokens still isn’t enough, break the task down into smaller chunks and help the harness out with MCP tools - I installed Codebase Memory to help it navigate large codebases more efficiently, for example. Ask the coding agent what’s causing problems and it may be able to help identify solutions too, I’ve had it troubleshoot itself several times now and it’s been quite good at inspecting its own settings and finding where the problems are with them. I even installed Firecrawl for it so it can do its own web searches.
@FaceDeer @eicker @floofloof @fonix232 @naeap
When you add the additional codebase memory, that will also add all those mcp tools to each round-trip to the llm model. Do you still see improvements and less filling up if your context? Just curious how it works out; I can imagine that a text based memory that explains the architecture of the codebase would help as well. Not sure about the difference if using a graph to store the code context?
Depends on the framework you’re using. I’m using Qwen Code Companion (via a VS Code IDE plugin) and it’s got some kind of tiered system built into it for adding tool definitions to context, it’s got a standard built-in set that’s always mentioned but other stuff only gets added to the context under certain circumstances. I can’t recall all the details, I found out about it when I was chatting with Qwen Code to ask it why it hadn’t used a particular MCP I’d given it to solve a problem despite it seeming like a useful tool for the task and we got into a discussion about to configure it. As I recall it’s something along the lines of most tools only being present in context by default as little one-liner stubs and the agent has to tell the framework “that one looks like it might be useful for what I’m doing” for the full definition to get loaded.
Most of the time I’m working on small scripts and specialized applications these days, and codebase memory is overkill for those so I haven’t actually seen it in use much.
More Ram, and break stuff up into smaller pieces. I Have 12Gb of VRAM and 64 Gig System Ram, which - with CPU offloading - allows pretty large models to be used at acceptable speed (although probably not fast enough for someone who wants to make his living with it. But these also will invest in dedicated hardware). And i let a larger model sit there to coordinate, and let it use subagents for the rest. That keeps the Context for the main model a lot more manageable and keeps it on track for a pretty long time.
How you structure the Task is more important than what the task is at all. For example, i have a huge list of games to be catalogued, and just telling the model to search the web and fill in the blanks in the database makes it crap out pretty soon. But I tell it to let subagents handle everything regarding web search and data extraction and just let them hand back the resulting data. the main model then calls another subagent to sanity check the data and update the database. Each of these subagents do not run for long, and therefor never run out of context, while the main model can keep context size manageable because it does not have to work with “messy” data at all.