LoRA

Word Embedding

Dense vector representations of words where semantically similar words are mapped to nearby points in vector space.

Parameter-Efficient Fine-Tuning

Methods for adapting large pretrained models to new tasks by only updating a small fraction of their parameters.

Adapter Layers

Small trainable modules inserted between frozen transformer layers that enable task-specific adaptation without modifying the original model weights.

Prefix Tuning

A parameter-efficient method that prepends trainable continuous vectors to the input of each transformer layer, guiding model behaviour without altering base parameters.

Pre-Training

The initial phase of training a deep learning model on a large unlabelled corpus using self-supervised objectives, establishing general-purpose representations for downstream adaptation.

Fine-Tuning

The process of adapting a pre-trained model to a specific task by continuing training on a smaller task-specific dataset, transferring learned representations to new domains.

The process of taking a pretrained model and further training it on a smaller, task-specific dataset.

A deep learning architecture designed for processing structured grid data like images, using convolutional filters to detect features.

An architecture where different specialised sub-networks (experts) are selectively activated based on the input.

A neural network component that learns to focus on relevant parts of the input when producing each element of the output.

A position encoding method that encodes absolute position with a rotation matrix and naturally incorporates relative position information into attention computations.

A learned dense vector representation of discrete data (like words or categories) in a continuous vector space.

A problem where gradients grow exponentially during backpropagation, causing unstable weight updates and training failure.

A self-supervised learning approach that trains models by comparing similar and dissimilar pairs of data representations.