How AI inference works

A documented token boundary example

The word antidisestablishmentarianism splits as ant / idis / establish / ment / arian / ism under cl100k_base. Its token IDs are 519, 85342, 34500, 479, 8997, 2191. This example is reproduced from OpenAI’s tiktoken cookbook, not assumed to apply to every model. Source: https://cookbook.openai.com/examples/how_to_count_tokens_with_tiktoken

The scope of this guide

We follow autoregressive text generation with a decoder-only transformer. Other AI systems can use different architectures, inputs, and generation methods.

Using learned weights

Inference runs an already trained model. Its learned weights ordinarily stay fixed while it answers, and its predictions can still be mistaken.

How to read the demo

The eight steps and probability tables are fixed teaching examples. They do not come from a live model or reproduce a specific tokenizer’s vocabulary. Other tokens groups many possible vocabulary entries.

Why chunks can look surprising

A real token can be a word, a fragment, punctuation, whitespace, or a special marker. Language, spelling, formatting, and the model-specific tokenizer all affect where boundaries fall.

Context changes the next choice

After each selected token is added, the updated sequence becomes the context for the next prediction. The model scores the vocabulary again rather than returning a complete answer in one operation.

Different work, different limits

Prefill often exposes more parallel arithmetic across prompt positions. Decode may be limited by reading weights or cache data, especially at small batch sizes. The actual bottleneck depends on sequence lengths, batching, model, kernels, and hardware.

Scheduling around both phases

A serving engine balances active requests and memory. Continuous batching lets requests join and leave as generation progresses; paged KV-cache systems organize cache storage into blocks to reduce waste.

Knowing when to stop

Generation ends when a configured stopping rule is met, such as an end token, a token limit, or a stop sequence. A finished sentence alone does not always end a request.

From IDs to representations

An embedding lookup maps each token ID to a learned vector. Position information lets the model distinguish where tokens occur in the sequence.

Attention and feed-forward blocks

Causal attention lets a position draw on its current and earlier positions, never future positions. A feed-forward network applies learned projections and a nonlinear transformation to each position. Residual connections and normalization help information flow through repeated blocks.

Logits become a choice

The output produces an unnormalized score, or logit, for every vocabulary entry. Greedy decoding chooses the highest score. Sampling selects from a distribution derived from the logits; temperature rescales scores, and top-p restricts sampling to a sufficiently likely set.

Settings do not add knowledge

Sampling settings change how tokens are selected. They do not add facts to the learned model or guarantee a correct answer.

A useful map

These eight layers organize the explanation. Real systems combine responsibilities and communicate in both directions; they do not always run as eight separate sequential programs.

The host around the GPU

A CPU coordinates application and server code. System RAM holds host data and buffers; the operating system manages processes, memory, and networking. The GPU driver connects the runtime and operating system to the device.

GPU memory and host RAM

The GPU performs highly parallel arithmetic and commonly holds weights, cache, and working data in local device memory. Host system RAM is separate on many systems, although shared-memory designs and offloading change how storage is arranged.

What makes the cache grow

For a fixed architecture and cache format, KV storage grows with cached sequence length and the number of active requests. Model layer count, KV-head count, head dimension, and bytes per value determine the cost per cached token.

Quantization changes representation

Weight quantization stores parameters with fewer bits. It can reduce raw weight memory, but metadata and packing add costs, and speed depends on compatible kernels and hardware. Changing weight precision does not automatically change the KV-cache format.

Bandwidth matters

An accelerator needs to move data as well as compute. Repeatedly reading model and cache data can constrain decode speed; compute capacity alone does not determine response latency.

Model weights

For a given loaded model and numeric format, raw weight storage is mostly fixed.

KV cache

The cache stores keys and values for token positions so later decode steps can reuse them.

Working memory

Activations, temporary buffers, runtime allocations, and other overhead require additional capacity.

Raw model weights

This example uses 7 billion parameters. FP16 uses 2 bytes per parameter: 7,000,000,000 × 2 bytes ÷ 2³⁰ = 13.04 GiB of raw weights. Ideal INT8 and INT4 raw weights are 6.52 GiB and 3.26 GiB before metadata and packing overhead.

The cache formula

32 layers × 8 KV heads × head dimension 128 × 2 bytes per FP16 value × 2 components (K and V) = 131,072 bytes, or 128 KiB, per cached token per active request.

The default example

At 4,096 cached tokens and one active request, cache storage is 0.50 GiB. Combined with FP16 weights, that is 13.54 GiB of raw weights plus cache.

What the estimate excludes

Activations, temporary buffers, quantization metadata, packing overhead, and runtime overhead are excluded. Weight precision changes only raw weights in this calculator; the KV cache remains FP16. These context values illustrate memory scaling and do not establish a real model’s supported context window.

Units

1 KiB is 1,024 bytes and 1 GiB is 1,073,741,824 bytes. The 7 billion parameter count is decimal, so 14 billion bytes is 13.04 GiB rather than 14 GiB.

The cost of talking

Devices communicate through PCIe, GPU-to-GPU links, or a network between machines. Link bandwidth, latency, topology, and the chosen partition determine how much communication costs.

Scale is not the same as latency

A model that does not fit on one GPU may require several. Adding devices can also improve throughput, but one request may see limited speedup because synchronization and data transfer take time.

From a machine to a fleet

Load balancers route requests across serving replicas. Health checks detect failures, and capacity management adjusts availability as demand changes. Queueing and routing remain part of the user’s response time.

Measure a representative workload

Compare the same model, input and output lengths, request concurrency, and numeric formats. Faster aggregate throughput does not guarantee lower queueing delay or a faster first token.

Look at the slow tail

Averages can hide intermittent stalls. A p95 latency is the value at or below which 95% of measured request latencies fall. Errors and rejected requests also matter.

Each independent request has a cache of the same size, so doubling the number doubles combined KV-cache storage. Shared raw weights stay fixed.

Keep the boundaries clear

A model, the serving system around it, and the application a person sees have different responsibilities. Locating a problem in the right layer is more useful than assigning every behavior to the model.

“A token is always a word.”

Tokens can be word fragments, punctuation, whitespace, whole words, or special markers; boundaries depend on the tokenizer.

“Every answer trains the model.”

Ordinary inference keeps learned weights fixed. Passing conversation history supplies context for prediction, rather than automatically updating those weights.

“The KV cache remembers facts forever.”

It stores attention keys and values for context positions. It is not permanent learned knowledge, and the server can release request cache when it is no longer needed.

“Smaller weights guarantee faster replies.”

Quantization reduces raw weight storage, but actual speed depends on kernels, hardware, batching, memory traffic, and scheduling.

“More GPUs always reduce latency.”

Partitioning a model adds communication and synchronization. Replicas can serve more independent requests without necessarily accelerating one request.

“Tokens per second tells the whole story.”

Aggregate throughput, a single request’s generation pace, queueing delay, time to first token, and memory limits describe different aspects of the service.

Words across the stack

Some terms name mathematical operations, some name software responsibilities, and others name hardware resources. Their definitions are scoped to the text-generation systems covered by this guide.

The smallest useful mental model

The system prepares context, computes scores, selects a token, appends it, and repeats until a stopping rule is met. Every repetition depends on both software decisions and physical resources.

A practical next question

When an answer starts slowly, examine overhead, queueing, and prefill. When it continues slowly, examine decode and data movement. When capacity runs short, separate weights, KV cache, and working memory before changing the system.