IA3

IA3 stands for Infused Adapter by Inhibiting and Amplifying Inner Activations. It is a Parameter-Efficient Fine-Tuning (PEFT) method that adapts a frozen model by learning small vectors that scale internal activations, rather than inserting full new layers or training low-rank matrix updates.

In transformer models, IA3 typically learns multiplicative rescaling factors for selected attention and feed-forward components. Instead of changing the full weight matrices, it changes how strongly certain channels are amplified or suppressed during computation. This keeps the number of trainable parameters extremely small.

Advantages

IA3 is very lightweight, often even smaller than LoRA in trainable parameter count. Because it works through scaling rather than full inserted modules, it can be efficient to store and conceptually simple once implemented. It is attractive when you want stronger adaptation than prompt tuning but still want to keep the update footprint minimal.

Disadvantages

IA3 is less widely supported in tooling than LoRA, which makes it less common in practical open-model workflows. It can also be harder to explain intuitively to newcomers, and in some settings the extra simplicity of scaling-only updates may limit expressiveness compared with richer methods like LoRA or adapter layers.

Example

A research team might use IA3 to adapt one foundation model for biomedical question answering and another for software issue classification, storing only tiny learned scaling vectors for each domain-specific variant.