There’s a popular piece of open-source software that everyone recommends for the layer that sits between your AI tools and the AI providers — the gateway, the thing that routes every request. I used it for months. Then I moved my entire stack off it. Here’s why, and what I learned.
The wake-up call was a security incident. Attackers slipped backdoored versions of the project onto the public package registry, through a poisoned security scanner of all things. Anyone who installed during that window had their environment variables, SSH keys, and cloud credentials harvested. Tens of thousands of downloads before it was caught.
I wasn’t hit. But it made me look at my own setup with fresh eyes. My gateway is the single box holding every API key I own — the crown jewels. And it was built on a long chain of third-party software packages, each one a potential door. That’s a big attack surface for the most sensitive component I run.
So I switched to a gateway written in a different language with a much smaller footprint. The differences were not subtle:
Memory. The old stack used about 480MB plus swap. The new one uses around 107MB. On a small rented server, that’s the difference between “fits comfortably” and “doesn’t fit.”
A bug I’d been blaming on the wrong thing. My old setup had a helper script in the path that, without me realizing, waited for the entire AI response to finish before passing anything along. For long responses, that meant staring at nothing for 20–40 seconds, then everything arriving at once. I’d been blaming the AI providers for slowness. It was my own proxy. Fixing the path made responses appear in under a second, streaming properly.
One connection for everything. The new gateway also gathers up all my tools’ connections — search, GitHub, the memory bridge — behind a single endpoint, instead of each tool managing its own tangle of connections.
The move surfaced its own lessons. The gateway’s internal database turned out not to be the source of truth for what it was actually serving — it had 18 phantom entries quietly corrupting things, and the fix was to ask the live system instead of trusting the database. And my own firewall was blocking the official software’s default signature as a bot, so I had to route around that too.
The gateway is the least glamorous part of the whole stack, and it’s the one every single request passes through. Boring, small, and auditable turns out to be exactly what you want there. I learned that the expensive way.