Parameter-Efficient Fine-Tuning

Word Embedding

Dense vector representations of words where semantically similar words are mapped to nearby points in vector space.

LoRA

Low-Rank Adaptation — a parameter-efficient fine-tuning technique that adds trainable low-rank matrices to frozen pretrained weights.

Adapter Layers

Small trainable modules inserted between frozen transformer layers that enable task-specific adaptation without modifying the original model weights.

Prefix Tuning

A parameter-efficient method that prepends trainable continuous vectors to the input of each transformer layer, guiding model behaviour without altering base parameters.

Pre-Training

The initial phase of training a deep learning model on a large unlabelled corpus using self-supervised objectives, establishing general-purpose representations for downstream adaptation.

Fine-Tuning

The process of adapting a pre-trained model to a specific task by continuing training on a smaller task-specific dataset, transferring learned representations to new domains.

A neural network component that learns to focus on relevant parts of the input when producing each element of the output.

The automatic discovery of data representations needed for feature detection or classification from raw data.

A problem in deep networks where gradients become extremely small during backpropagation, preventing earlier layers from learning.

A skip connection that adds a layer's input directly to its output, enabling gradient flow through deep networks and allowing training of architectures with hundreds of layers.

The process of taking a pretrained model and further training it on a smaller, task-specific dataset.

A neural network architecture that groups neurons into capsules to better capture spatial hierarchies and part-whole relationships.

A distributed training approach that partitions a model across multiple devices, enabling training of models too large to fit in a single accelerator's memory.

A transformer architecture adapted for image recognition that divides images into patches and processes them as sequences, rivalling convolutional networks in visual tasks.