I built an LLM API gateway. Between that and all the infrastructure modernisation I’ve been doing at work, the months have blurred. I’ve been really busy, and there was never a good moment to write. But this is really exciting. So here goes.
The project is called SoCLaaS. Nevermind the name — it means what it means. It’s our LLM infrastructure so that we can make LLM API service broadly available free-of-charge to our NUS School of Computing community, scalable for volume consumption without easily falling apart from overload or contenton.
SoCLaaS begun as an experiment with Codex. I was exploring some ideas and thought to give Codex some work. That experiment grew into what it is today. It’s not at all a small project as it stands now. If not for AI, I estimate it would have taken 3 experienced developers 3 months to complete. Codex itself estimates 18 man-months. It’s amazing what we can do with AI.
In a nutshell, SoCLaaS gives developers one stable OpenAI-compatible API while keeping operational complexities of local LLM infrastructure behind the scenes. Operators get control over access, routing, capacity, usage, and policy. It enables us to provide shared AI capability without committing to or depending on a public cloud-only strategy.
LiteLLM showed that a unified AI gateway is an enormously useful abstraction. SoCLaaS takes a more local-infrastructure-oriented view of that problem: how do we run a shared pool of self-hosted models fairly, safely, and predictably across many users and gateway replicas? LiteLLM is the right answer when your primary concern is supporting many different model providers through one interface. We’re solving slightly different problems — SoCLaaS is built for our specific needs, where the focus is on running a shared pool of local models, not on aggregating a hundred different APIs. That distinction shapes everything about the design.
The architecture has two halves, and this split was important from the start. The data plane is a Go gateway that handles every inference request — auth, routing, rate limiting, streaming, usage capture, the stuff that needs to happen quickly and consistently for every single request. The control plane is Django, for the things that don’t need to be in the hot path: managing users, policies, model definitions, usage reports, audit logs, and operational workflows.
There’s also a separate portal for end users. This separation was intentional. Operators need broad power to manage the platform; individual users should only manage their own keys and see their own usage and budget info. Keeping those concerns separate makes the system easier to reason about and safer to expose.
I did not choose Go. Codex recommended it because the gateway needed to stay lean and handle lots of concurrent long-lived requests — streaming inference calls are the opposite of quick-and-dirty HTTP requests. I had never written a line of Go. I was very resistant to the choice of Go. I pushed, but Codex pushed back with good reasons, and alas, I accepted with resignation that Codex’s decision was objectively correct.
I ended up letting Codex do all the work, still without much understanding of Go. Other parts of the project, such as the control plane, was written in Python/Django. I was the architect, and Codex was my senior developer.
The real surprise with my vibe-coding experience wasn’t the architecture or the tech choices. It was how much the whole thing got done without the usual procrastination drag. When someone else is doing the boilerplate, starting is almost nothing. I’d tell myself “I’ll set up the project structure tomorrow” in the old days. Now I’d describe the problem, Codex would generate code, I’d review it, fix what it missed, and move on. The feedback loop was tight enough that I was constantly in the work rather than circling it from a distance.
That’s not a minor benefit. It’s transformative for how productive I actually am.
There were rough patches, naturally. Codex isn’t perfect. It built a practically useless queuing system — it turned out to be merely a request sorter. But reviewing code is faster than writing it from scratch, and the review loop kept me in the zone. My job shifted from doing the work to thinking about the work that needed to get done. The things that matter are the design decisions — what the gateway should do, how it should handle failure — not the scaffolding around them.
Under the hood, the platform uses PostgreSQL as the durable source of truth — API keys, policies, model definitions, backend inventory, usage records, credit grants, audit trails. These need to survive restarts, incidents, and upgrades. Redis handles fast transient state: rate-limit counters, capacity leases, queue state. It’s not the system of record, which matters — if Redis restarts, the platform shouldn’t lose the durable history.
The queueing and fairness model is probably the most interesting part. LLM capacity isn’t like a normal stateless web API. Requests can take a long time, especially when streaming. If the platform just accepts every request and hopes for the best, noisy workloads crowd out everyone else. If it rejects aggressively, it wastes idle capacity.
SoCLaS tries to occupy the sensible middle ground. Each API key has a normal service level and a hard upper bound. Work within the normal level receives priority. Work above that level can still be accepted when capacity is available, allowing useful bursts. But when the system is busy, normal work takes precedence. It’s fair without being rigid. Queued work also preserves ordering per user — a later request shouldn’t leapfrog an earlier one. And queued work only starts when a real backend slot has been reserved, so you never get a request that appears admitted only to fail immediately because all capacity is busy.
The gateway also handles load balancing, failover when a model goes down, per-key rate limiting, and usage tracking. Public model definitions let operators swap out or upgrade underlying model versions without forcing every application to change config. For higher availability, multiple gateway instances can run behind a load balancer with Redis-backed coordination to prevent the common failure mode where every gateway independently thinks a backend is available and collectively overloads it.
Codex wrote the huge majority of the code. It’s been remarkably smooth. Nothing broke because an AI wrote it. Nothing magical appeared. It just worked, at a pace that would’ve felt unreasonable a year ago — and impossible if I’d had to learn Go properly first.
This isn’t my first vibe-coding of a non-trivial project. But it is the first that involved a programming language I had not every used before and required significant expertise that I truly lacked. Perhaps the most important skill, the one that I already have, is architecting.
So that’s the LLM gateway. More on the other stuff soon.
View Comment Policy