PEFT

PEFT stands for Parameter-Efficient Fine-Tuning. It is a category of techniques for adapting a pretrained model to a new task while training only a tiny fraction of the model's parameters. Instead of updating every weight in a large language model or foundation model, PEFT methods freeze most of the base model and learn small task-specific additions or modifications.

The motivation is practical. Full fine-tuning of large models is expensive in GPU memory, storage, and training time. It also creates a separate full model checkpoint for every task, which is wasteful when the base model remains mostly the same. PEFT methods reduce that cost by keeping the original model fixed and storing only the lightweight adaptation layers or vectors needed for the new behavior.

Common PEFT methods include LoRA (Low-Rank Adaptation), adapter layers, prefix tuning, prompt tuning, and IA3. These approaches make different architectural tradeoffs, but they share the same goal: preserve most of the pretrained model while learning a much smaller set of task-specific parameters.

PEFT vs LoRA

The simplest distinction is: PEFT is the umbrella category, and LoRA is one specific method inside that category. Saying "I used PEFT" is like saying "I used a compression method"; saying "I used LoRA" is naming the exact technique. Not every PEFT method is LoRA, but every LoRA setup is a form of PEFT.

People often blur the two because LoRA became the default PEFT method for many open-model workflows. But adapter tuning, prompt tuning, prefix tuning, and IA3 are also PEFT methods, even though they work differently under the hood.

PEFT vs Full Fine-Tuning vs Prompting/RAG

Approach	What changes	Best for	Main tradeoff
Prompting	The instructions sent to the model at runtime.	Fast behavior changes, experiments, and tasks the base model already understands.	No durable model adaptation; performance can be fragile across prompts.
RAG	The context retrieved and supplied to the model.	Adding fresh or private knowledge without retraining the model.	Depends on retrieval quality and does not deeply change model behavior.
PEFT	Small trainable adapters, vectors, or low-rank updates attached to a mostly frozen model.	Domain adaptation, style tuning, task specialization, and maintaining many lightweight variants.	Usually less flexible than full fine-tuning and adds adapter management complexity.
Full fine-tuning	Most or all model weights.	Deep behavioral changes, high-stakes specialization, or cases where PEFT underperforms.	Expensive to train, store, validate, and serve.

PEFT is one of the main reasons smaller teams can customize large models at all. It lowers hardware requirements, shortens experimentation cycles, and makes it easier to maintain many specialized variants of the same base model. In the open-source model ecosystem, PEFT checkpoints are often small enough to distribute independently from the original model weights.

When Not to Use PEFT

PEFT is not always the right tool. If the model only needs access to new facts, documents, or product data, retrieval-augmented generation is often safer and easier than training an adapter. PEFT changes behavior; RAG supplies information.

PEFT can also be the wrong choice when the desired change is very deep. If the base model is fundamentally bad at the target task, uses the wrong language or modality, or needs a major change in reasoning style, full fine-tuning or a different base model may work better. A small adapter cannot reliably compensate for a weak foundation.

Operationally, PEFT adds its own complexity. Teams have to track which adapter belongs to which base model version, validate adapter compatibility after model upgrades, decide whether to merge adapters for inference, and manage multi-adapter serving if many tasks share one base model. Those costs are much smaller than full fine-tuning, but they are not zero.

A useful mental model is that full fine-tuning rewrites the whole book, while PEFT adds an annotated layer on top of it. The base knowledge stays in place; the adaptation tells the model how to behave differently for a narrower job.