Sigmoid Function

Self-Attention

An attention mechanism where each element in a sequence attends to all other elements to compute its representation.

Multi-Head Attention

An attention mechanism that runs multiple attention operations in parallel, capturing different types of relationships.

Residual Network

A deep neural network architecture using skip connections that allow gradients to flow directly through layers, enabling very deep networks.

Layer Normalisation

A normalisation technique that normalises across the features of each individual sample rather than across the batch.

Dropout

A regularisation technique that randomly deactivates neurons during training to prevent co-adaptation and reduce overfitting.

Activation Function

A mathematical function applied to neural network outputs to introduce non-linearity, enabling the learning of complex patterns.

ReLU

Rectified Linear Unit — an activation function that outputs the input directly if positive, otherwise outputs zero.

Softmax Function

An activation function that converts a vector of numbers into a probability distribution, commonly used in multi-class classification.

Positional Encoding

A technique that injects information about the position of tokens in a sequence into transformer architectures.

Gradient Clipping

A technique that caps gradient values during training to prevent the exploding gradient problem.

Mixed Precision Training

Training neural networks using both 16-bit and 32-bit floating-point arithmetic to speed up computation while maintaining accuracy.

Rotary Positional Encoding

A position encoding method that encodes absolute position with a rotation matrix and naturally incorporates relative position information into attention computations.

An architecture where different specialised sub-networks (experts) are selectively activated based on the input.

A problem where gradients grow exponentially during backpropagation, causing unstable weight updates and training failure.

A computing system inspired by biological neural networks, consisting of interconnected nodes that process information in layers.

A skip connection that adds a layer's input directly to its output, enabling gradient flow through deep networks and allowing training of architectures with hundreds of layers.

A neural network architecture based entirely on attention mechanisms, eliminating recurrence and enabling parallel processing of sequences.

A neural network architecture where connections between nodes form directed cycles, enabling processing of sequential data.

A problem in deep networks where gradients become extremely small during backpropagation, preventing earlier layers from learning.

A distributed training approach that partitions a model across multiple devices, enabling training of models too large to fit in a single accelerator's memory.