Adapter Layers

Adapter layers are one of the earliest and most influential Parameter-Efficient Fine-Tuning (PEFT) methods. Instead of changing the full weights of a large pretrained model, adapters insert small trainable modules between or inside existing layers while keeping the original model frozen.

In transformer models, adapter layers are usually added after attention or feed-forward blocks. They often use a bottleneck structure: project the hidden state down to a smaller dimension, apply a nonlinearity, then project back up. Because only these small inserted layers are trained, the number of trainable parameters is far lower than in full fine-tuning.

Advantages

Adapter layers are modular and easy to swap. You can keep one base model and load different adapters for sentiment analysis, legal drafting, medical Q&A, or code generation. They also work well when you need many task-specific variants of the same model because each adapter checkpoint is much smaller than a full model copy.

Disadvantages

Adapters add new layers to the network, which can increase inference latency and architectural complexity compared with methods that do not add extra computation at every forward pass. They can also be more cumbersome to integrate than LoRA in ecosystems where LoRA has become the default tooling standard.

Example

A company might keep one 7B base model for internal document work, then attach one adapter for finance summarization and another for customer support classification. The base model stays fixed; the adapters specialize it for each task.