Inference

Inference is the process of using a trained model to produce an output for new input. Training changes the model's parameters through an optimization loop; inference normally runs the learned weights forward without updating them. A classifier produces a label or score, a recommender ranks items, and a language model generates the next token repeatedly. The input and output can be simple, but the serving system around the model determines whether the result is affordable and reliable in production.

Online, batch, and streaming inference

Online inference handles a request interactively and usually has a latency target. A fraud score may need to return in milliseconds, while a chat assistant may optimize for time to first token and then for the rate at which remaining tokens arrive. Batch inference processes many records together on a schedule, such as nightly recommendations or document embeddings, so throughput and cost matter more than an individual request's latency. Streaming inference returns partial output as it is produced and is common for speech and LLM applications.

Inference generally needs less memory than training because it does not retain every intermediate activation for backpropagation. That does not mean it is automatically cheap. A high-traffic service may need multiple replicas, accelerator memory, networking, safety checks, retrieval, logging, and fallbacks. For autoregressive language models, the KV cache stores attention keys and values from earlier tokens so the system does not recompute the entire context at every generation step; long contexts and large concurrent batches can make this cache the dominant memory cost.

What to measure

Useful inference metrics include latency percentiles such as p50 and p95, time to first token, inter-token latency, requests or tokens per second, peak memory, cost per request, error rate, and output quality. A faster model that produces more retries or incorrect tool calls may be worse for the product. Measure the complete path from request arrival to usable answer, not only the model kernel.

Common optimization techniques

Quantization lowers numerical precision to reduce memory traffic. Batching shares accelerator work across requests, while continuous batching keeps a busy LLM server supplied with new sequences as others finish. Caching avoids repeating deterministic work. Pruning removes selected weights, and knowledge distillation trains a smaller model to imitate a larger one. Speculative decoding uses a smaller draft model to propose tokens that a larger model verifies. Hardware-specific kernels and runtimes can make the same model behave very differently, so benchmark on the hardware and concurrency level you will actually deploy.

Inference is still an application boundary

Production inference needs input validation, access control, rate limits, timeouts, observability, and a plan for model or data drift. For an agent, the model output may become a tool argument, so application code must validate permissions and side effects rather than trusting the generated text. The goal is not simply to run a forward pass; it is to deliver a measurable prediction or response within the quality, latency, and safety limits of the application.