How to solve the context problem in the agent
-
Prefer to watch? It's all in my YouTube video, it's available in English, Hindi and Turkish. The context breakdown starts from 1:39:34
All the materials for the video are on my Telegram channel, come join: t.me/mustangbas
Plan
- What context is
- What losing context looks like in practice
- How the agent compresses the dialog
- Why the agent gets dumber with long context
- How many tokens a big project eats
- How to reduce context loss
- Conclusion
What context is
Context is basically the whole set of info the model has access to at a given moment. It includes:
- the data of our project
- our requests and messages
- the documentation for all BAS modules
With all of that in hand, the model decides what to do with the project, hands it to BAS, and BAS builds it into the software
To put it really simply, it's like a backpack that fits six books. Put six in, and it's full. Want to add a seventh, then one of the old ones has to go, otherwise it physically won't fit. The agent in BAS works in a similar way: as soon as new info is pushed in, it drops the old
Every model has its own backpack size. It's measured in tokens, one token is roughly 4 characters of English text, and 100 thousand tokens comes to about 150 pages. Some models have a window of a million tokens, others 200 thousand or even less
What losing context looks like in practice
Someone sent me another developer's project with 6500 actions, and I wanted to completely rework it. I gave the agent tasks one at a time:
- Explain what this project even is and how it's built
- Sort the resources into tabs and translate them into English
- Make a plan for reworking the whole project
- Turn that plan into three tasks:
- rename functions and variables across the WHOLE project (161 functions)
- add a description to EVERY function (also 161 functions)
- add more logs across the WHOLE project (6438 actions)
It did the first two fine, and the most interesting part started on the third and fourth. I got the result there with far more than one message. Scrolling through the chat shows a huge pile of prompts, and many of them are basically about the same thing
The obvious question here is why write the same thing to the agent at all if it was all in the first prompt. It all comes down to that backpack, the agent's context is limited. The further it goes with the work, the higher the chance it loses both the project and the original prompt
It looked something like this. The agent made the plan itself, it laid out which variables to rename and how. Then at some point it writes that there's no saved plan in the project and asks me to send it the list. And I send it that same list it had sent me earlier
Plus it often stopped, and I had to start it again
At the end there was also a story with the logs: in the report the agent wrote that the logs task was 70% done, yet I didn't see new logs in the functions I checked. Most likely it counted as its own the logs a person had left before it
How the agent compresses the dialog
The agent doesn't send the whole dialog to the model. It keeps making a short summary of the old messages, and then the model works with that. What exact mechanism is behind it, I don't know, the server is closed to us. We can't influence it in any way either
Why not send everything as is? For the developer it would be very expensive in resources: handling that many requests and chats where each one drags the whole history along. So the summary is basically just an optimization
Because of this, some details get lost with every summary. On a long dialog the context gets lost really badly, and at some point something important drops out of the summary, like that same plan
By the way, there's nowhere to see how much context has already been lost. The Statistics tab under an answer shows the model and the tokens for the request, and that's it. What window the model has, how much of it is used and what the agent threw out in the summary isn't shown anywhere
Why the agent gets dumber with long context
There's one more thing here. Even if six books fit, carrying a backpack like that is heavy, and it's always tempting to put less in it. Same with the model: the fewer tokens loaded into it, the better it handles that info and the better decisions it makes
While the load is small, say around 5 thousand tokens, the model works fine. The more tasks get piled on it, the worse it thinks, and at some point it starts to hallucinate, meaning it just makes up things that aren't there
On that big project it was really easy to see. I loaded the model so much that it kind of dropped two books out of six, and those were exactly the ones with all the important stuff: what this project is and what it does. After that it lost the thread of the task and counted the logs a person had added earlier as its own
How many tokens a big project eats
The numbers for this dialog are in the downloaded log, and it shows well how every model call swells up
What the numbers say:
- in one of my requests the agent made 25 model calls in a row, and the call size grew from 145 all the way to 503 thousand tokens
- over the whole dialog it added up to 280 model calls, 732 tool calls, 372 project edits and 40 backups
- about 72 million tokens is basically the total over all 280 calls together
What these numbers make clear:
- even a single request basically swells more than three times, because every piece of code it read and every edit gets added to the call
- 503 thousand tokens is already more than the window of many models, so there's no way around the summary here, and some details get lost along the way
- a big project eats a truly huge number of tokens, that's really expensive for the developer, which is why they save on context
How to reduce context loss
Let me say right away, there's no way for us to boost the model's decisions several times over. The agent's context system is handled by the developer, and overall we have no influence on it at all
Some things are still under our control:
- one big task per chat
- split a big task into stages. This really is the right way to work with the agent, especially if the project is huge or the task is new and massive
- keep a short plan. If it's unclear how to even make one, we can ask the agent itself, and it'll lay out in detail what can be done. That's what I did on the big project, it made the plan for reworking the project by stages itself
- small and medium tasks can easily share one chat, overall it won't make much difference. Except that on a very long chat the model might start getting dumber. That's not 100% certain, how it's all optimized on the developers' server is unclear
Why a new chat helps so much is easy to explain with the same books. Say the backpack already holds math, chemistry and biology, and physics gets added with a task specifically about physics. To solve it, the agent will still read the old books and take them into account, and that confuses it
A new chat has no old books. We start from a clean slate with one book in it, and the agent works much better because it doesn't get tangled in things that have nothing to do with the task
Conclusion
The fact that context can get lost this badly is, in my view, one of the agent's main downsides. We don't control the dialog summary, and how much has already been lost isn't shown anywhere. Nobody really explains this to users, and here the developers, if they did it right, should tell people how to work with it
On the other developer's project, out of three tasks the agent handled the renaming and the function descriptions, and with the logs it already didn't. Across the whole project it came out, as I see it, at around 40%, maybe 50
If I really had the goal of getting it to 90% and had the time, like two days of sitting only with it, I'd split everything into stages with a plan, the right context and a decent prompt. I'm 100% sure that then the agent would have reworked the project completely, it's just a matter of time
