This bug took me longer to find than I’d like to admit, and I’m sharing it because it’s the kind of thing that costs you money without any error message.
Quick context for non-engineers: an AI model has a “context window” — think of it as working memory, how much conversation it can hold at once. When a conversation gets close to full, the system compresses it — summarizes the older parts to make room. My models had a 262k-token window. But compression kept kicking in at around 102k. Half the working memory — half of what I was paying for — was being thrown away.
The cause wasn’t one bug. It was two silent defaults stacked on top of each other. My agent’s config didn’t state the context size, so it fell back to asking the provider’s API. The API didn’t declare it either, so it fell back to a generic 128k default. Two quiet fallbacks, and compression was firing 160,000 tokens early.
The lesson that stuck with me: when the system compresses too early, don’t just lower the threshold. That was my first instinct, and it’s wrong — it hides the symptom while wasting even more memory. The right move is to fix the declared context size, explicitly, at every layer. Never let a default quietly speak for your setup.
The same kind of invisible-default bug showed up in a few other places:
Hitting the wall at the exact edge. Some models reject requests right at their maximum, because the prompt needs a little headroom. I now keep a small safety buffer on every model. Crude, but it works.
Smart routing was defeating caching. Caching saves money by reusing the fixed parts of a conversation, but the cache is tied to one specific model. My “smart” router kept switching models between requests, wiping the cache each time. Narrowing the router to just two or three models let it settle, and the savings came back.
Tool add-ons quietly eat memory. One integration I tried consumed a huge share of the context window just loading its tool definitions — before the conversation even started. Now I check what an add-on costs in memory before enabling it, not after.
The bigger point: a context window is a budget. And the failures aren’t dramatic crashes — they’re quiet defaults, each shaving off a chunk of something you paid for.