How LLM learning and fine-tuning work

What this guide covers

An LLM is a large language model. Parameters, often called weights, are learned numbers used to compute predictions. We follow next-token training for a causal language model, then plan supervised fine-tuning for one narrow task. This is a conceptual guide, not a command-by-command training recipe.

A prompt is not a parameter update

A model can respond differently to examples in its prompt without updating its weights. This guide uses training to mean optimization of trainable parameters. Source: Language Models are Few-Shot Learners (Brown et al., 2020), https://arxiv.org/abs/2005.14165 .

Two starting points

Pretraining builds a base model by learning from a large collection of text. Supervised fine-tuning continues training on examples of the behavior you want. For our small support-formatting project, adapting an appropriate existing model is a practical starting assumption; training from scratch is outside this guide.

Training has other stages

Supervised fine-tuning is one way to adapt a model. Preference-based training and reinforcement learning can pursue other objectives; they are not the loop we teach here. Source: Training language models to follow instructions with human feedback (Ouyang et al., 2022), https://arxiv.org/abs/2203.02155 .

What counts as a token?

A tokenizer maps text to units represented by integer IDs. A token may represent part of a word, punctuation, or another text unit. Boundaries depend on the tokenizer. Source: Tokenization algorithms, Hugging Face Transformers, https://huggingface.co/docs/transformers/tokenizer_summary .

The answer is already in the example

For an illustrative token sequence A, B, C, the model uses A to predict B and A, B to predict C. These are abstract token symbols, not a real tokenizer output. A causal mask prevents a prediction from using future tokens. Many positions can be scored in parallel during training. Source: Causal language modeling, Hugging Face Transformers, https://huggingface.co/docs/transformers/tasks/language_modeling .

Observed does not mean true

The target is the token observed in the training text. Its presence does not certify factual truth or quality. Selecting and reviewing examples is part of defining what the model learns.

What this toy model means

The context has observed target B rather than the alternative class. The scalar parameter w produces p(B)=sigmoid(w). Its negative log-likelihood loss is −ln(p(B)), with derivative p(B)−1. A gradient-descent update uses w′=w−learning_rate×gradient. This is a deliberately tiny differentiable model, not an LLM architecture or training result.

What actually changes

A forward pass computes predictions and loss. Backpropagation computes gradients: local sensitivities of loss to trainable parameters. The optimizer uses those gradients and its update rule to change parameters. The learning rate helps set the update scale. An update need not improve every example. Source: Optimizing Model Parameters, PyTorch, https://docs.pytorch.org/tutorials/beginner/basics/optimization_tutorial.html .

One batch, not one generated word

A batch is a group of examples processed together. This tour represents one optimization step; many token positions may contribute. Training does not need to sample each token in sequence. In supervised fine-tuning, configuration can restrict loss to completion or assistant tokens. Source: SFT Trainer, Hugging Face TRL, https://huggingface.co/docs/trl/sft_trainer .

Check the calculation

For one target token, cross-entropy loss is −ln(p), where p is the probability assigned to that target. Using natural logarithms gives nats: −ln(0.25) = 1.386294… and −ln(0.50) = 0.693147… . The displayed bars are rounded to two decimals: 1.39 and 0.69 nats. Source: SFT Trainer, Hugging Face TRL, https://huggingface.co/docs/trl/sft_trainer .

What the bars do not claim

These are two hypothetical predictions, not measured LLM results, a guaranteed training trajectory, or a before-and-after optimizer step. A training loss usually aggregates many scored tokens. Lower loss on the training examples does not by itself establish better task performance.

Why the top prediction stays B here

At p(B)=25%, the other probabilities are 20%,20%,20%,15%; B is already largest. At p(B)=50%, the others are 13.333…%,13.333…%,13.333…%,10%. B remains largest while target loss drops from about 1.39 to 0.69 nats. Small displayed rounding differences do not change the underlying sums.

An illustrative project

The visible sign-in request and two-field JSON response are an invented format example. They are not a recommended support policy. Prepare varied, reviewed requests and targets, define what counts as category accuracy and useful replies, and compare a clear prompt baseline before deciding to fine-tune.

Prepare the actual training input

Use a suitable licensed model and data you have permission to use; remove private information. Match the model’s tokenizer and chat template, inspect formatted examples, and check which response tokens contribute to loss. Source: SFT Trainer, Hugging Face TRL, https://huggingface.co/docs/trl/sft_trainer .

Choose a small, inspectable run

A batch groups examples; an epoch is one pass through the training set; the learning rate controls update scale. Choose settings supported by the model and hardware, inspect loss and held-out behavior, and revise from evidence. There is no universal batch size, memory requirement, or epoch count for an LLM. Source: Optimizing Model Parameters, PyTorch, https://docs.pytorch.org/tutorials/beginner/basics/optimization_tutorial.html .

What an adapter changes

LoRA represents selected weight updates with trainable low-rank matrices while keeping the original weights frozen. At use time, the adapter is applied with its compatible base model, or its updates may be merged where supported. Extra trainable modules depend on configuration. Source: LoRA: Low-Rank Adaptation of Large Language Models (Hu et al., 2021), https://arxiv.org/abs/2106.09685 .

Smaller does not mean free

Fewer trainable parameters can reduce training resource needs, but the base model and intermediate computations still consume memory. Model size, sequence length, batch size, precision, and implementation matter. This guide makes no universal savings or hardware claim. Source: LoRA, Hugging Face PEFT documentation, https://huggingface.co/docs/peft/package_reference/lora .

Why the split matters

A model can fit training examples without improving on new ones. Use validation results to choose settings and checkpoints; repeated decisions based on a test set turn it into another tuning set. Reserve a final test for the chosen approach. Keep duplicate or closely related cases in the same split to reduce leakage. Source: Cross-validation: evaluating estimator performance, scikit-learn, https://scikit-learn.org/stable/modules/cross_validation.html .

Measure the goal you chose

For the illustrative support task, compare the starting and adapted models on the same unseen requests. Check field validity and usefulness of replies, including difficult cases and regressions. Loss is one diagnostic; it is not a factuality, safety, or task-quality certificate. Keep evaluation conditions and prompts consistent.

The test cases did not drive updates or checkpoint choices. Compare both models consistently and inspect task quality and regressions, then report the result rather than tuning against it.

Why this is the key check

Training changes the model to fit its learning signal. The useful question is whether those changes help the intended task on examples that did not drive the updates or repeated tuning. Source: Cross-validation: evaluating estimator performance, scikit-learn, https://scikit-learn.org/stable/modules/cross_validation.html .

A practical next step

Write one narrow goal and a few reviewed examples. Test a prompt baseline before spending resources on a small fine-tuning run. Keep evaluation cases separate. Compare outputs, inspect failures, and save the model or adapter, compatible base revision, tokenizer, training settings, and evaluation results.

A checkpoint is learned state

A checkpoint saves parameter state; resuming training may also require optimizer and other training state. Source: Saving and Loading Models, PyTorch, https://docs.pytorch.org/tutorials/beginner/saving_loading_models.html . Learned parameters are not a dependable database of source documents, and low loss does not guarantee truthful answers. The useful mental model is a repeatable prediction-and-adjustment loop with a separate reality check. See also Training language models to follow instructions with human feedback (Ouyang et al., 2022), https://arxiv.org/abs/2203.02155 .