API Gateways

What an API gateway is

An API gateway is the controlled front door to a collection of backend services. Instead of every client knowing which internal service owns which endpoint, the client talks to the gateway and the gateway routes the request to the appropriate backend. This creates a centralized place for policies that should apply consistently across many APIs.

Responsibilities

A gateway may terminate TLS, authenticate requests, enforce rate limits, validate headers, perform request routing, attach correlation IDs, collect telemetry, transform protocols, and sometimes aggregate calls to several downstream services. It can also hide internal topology from clients, allowing services to move or scale without changing public URLs.

Gateway vs load balancer

The two roles overlap in many products but are conceptually different. A load balancer primarily distributes traffic across healthy backends. An API gateway usually understands API-level concerns such as routes, identities, quotas, authentication, transformations, and policies. Modern infrastructure products can perform both roles, but the architectural responsibilities remain distinct.

The downside of centralization

A gateway can simplify dozens of services while simultaneously becoming a critical failure point. If the gateway is overloaded, every service behind it appears unhealthy. Heavy business logic in the gateway also creates a central bottleneck and makes deployments risky. A good rule is to keep the gateway focused on transport and policy, while domain logic remains in the backend services that own it.

Streaming introduces another set of constraints. An API gateway that buffers responses or enforces an ordinary 30-second idle timeout can break SSE token streaming even though the backend works correctly. Gateways must therefore understand long-lived connections, cancellation, and flush semantics.

Example

A public AI platform might expose /v1/chat through a gateway. The gateway authenticates the API key, checks tenant quotas, attaches a trace ID, chooses a healthy inference cluster, and returns either a normal response or an SSE stream. The actual model workers remain private and can scale independently behind the gateway.

The core idea

An API gateway is a policy and routing boundary. It hides internal topology and gives clients one stable entry point while centralizing concerns that would otherwise be repeated across services.