PEFT

PEFT stands for Parameter-Efficient Fine-Tuning. It is a category of techniques for adapting a pretrained model to a new task while training only a tiny fraction of the model's parameters. Instead of updating every weight in a large language model or foundation model, PEFT methods freeze most of the base model and learn small task-specific additions or modifications.

The motivation is practical. Full fine-tuning of large models is expensive in GPU memory, storage, and training time. It also creates a separate full model checkpoint for every task, which is wasteful when the base model remains mostly the same. PEFT methods reduce that cost by keeping the original model fixed and storing only the lightweight adaptation layers or vectors needed for the new behavior.

Common PEFT methods include LoRA (Low-Rank Adaptation), adapter layers, prefix tuning, prompt tuning, and IA3. These approaches make different architectural tradeoffs, but they share the same goal: preserve most of the pretrained model while learning a much smaller set of task-specific parameters.

PEFT vs LoRA

The simplest distinction is: PEFT is the umbrella category, and LoRA is one specific method inside that category. Saying "I used PEFT" is like saying "I used a compression method"; saying "I used LoRA" is naming the exact technique. Not every PEFT method is LoRA, but every LoRA setup is a form of PEFT.

In practice, people often blur the two because LoRA became the default PEFT method for many open-model workflows. But adapter tuning, prompt tuning, prefix tuning, and IA3 are also PEFT methods, even though they work differently under the hood.

In practice, PEFT is one of the main reasons smaller teams can customize large models at all. It lowers hardware requirements, shortens experimentation cycles, and makes it easier to maintain many specialized variants of the same base model. In the open-source model ecosystem, PEFT checkpoints are often small enough to distribute independently from the original model weights.

A useful mental model is that full fine-tuning rewrites the whole book, while PEFT adds an annotated layer on top of it. The base knowledge stays in place; the adaptation tells the model how to behave differently for a narrower job.