Prefix Tuning

Word Embedding

Dense vector representations of words where semantically similar words are mapped to nearby points in vector space.

LoRA

Low-Rank Adaptation — a parameter-efficient fine-tuning technique that adds trainable low-rank matrices to frozen pretrained weights.

Parameter-Efficient Fine-Tuning

Methods for adapting large pretrained models to new tasks by only updating a small fraction of their parameters.

Adapter Layers

Small trainable modules inserted between frozen transformer layers that enable task-specific adaptation without modifying the original model weights.

Pre-Training

The initial phase of training a deep learning model on a large unlabelled corpus using self-supervised objectives, establishing general-purpose representations for downstream adaptation.

Fine-Tuning

The process of adapting a pre-trained model to a specific task by continuing training on a smaller task-specific dataset, transferring learned representations to new domains.

The process of taking a pretrained model and further training it on a smaller, task-specific dataset.

A mathematical function applied to neural network outputs to introduce non-linearity, enabling the learning of complex patterns.

An individual attention computation within a multi-head attention layer that learns to focus on different aspects of the input, with outputs concatenated for richer representations.

A technique that normalises layer inputs during training to stabilise and accelerate deep neural network learning.

A deep neural network architecture using skip connections that allow gradients to flow directly through layers, enabling very deep networks.

A subset of machine learning using neural networks with multiple layers to learn hierarchical representations of data.

A neural network trained to encode input data into a compressed representation and then decode it back to reconstruct the original.

A framework where two neural networks compete — a generator creates synthetic data while a discriminator evaluates its authenticity.