Neural Scaling Laws

Tensor Processing Unit

Google's custom-designed application-specific integrated circuit for accelerating machine learning workloads.

Neural Processing Unit

A specialised processor designed to accelerate neural network computations in edge devices and mobile platforms.

Model Distillation

A technique where a smaller, simpler model is trained to replicate the behaviour of a larger, more complex model.

Model Pruning

The process of removing redundant or less important parameters from a neural network to reduce its size and computational cost.

Neural Architecture Search

An automated technique for designing optimal neural network architectures using search algorithms.

Model Quantisation

The process of reducing the numerical precision of a model's weights and activations from floating-point to lower-bit representations, decreasing memory usage and inference latency.

Sparse Attention

An attention mechanism that selectively computes relationships between a subset of input tokens rather than all pairs, reducing quadratic complexity in transformer models.

Model Collapse

A degradation phenomenon where AI models trained on AI-generated data progressively lose diversity and accuracy, converging toward a narrow distribution of outputs.

Speculative Decoding

An inference acceleration technique where a small draft model generates candidate token sequences that are verified in parallel by the larger target model.

The simulation of human intelligence processes by computer systems, including learning, reasoning, and self-correction.

The coordination and management of multiple AI models, services, and workflows to achieve complex end-to-end automation.

A machine learning approach where models learn to perform tasks from only a small number of labelled examples, often achieved through in-context learning in large language models.

A theoretical form of AI that would have consciousness, self-awareness, and the ability to truly understand rather than simulate understanding.

A thought experiment by John Searle arguing that executing a program cannot give a computer genuine understanding or consciousness.

A theoretical level of AI that surpasses human cognitive abilities across all domains, including creativity and social intelligence.

A structured representation of real-world entities and the relationships between them, used by AI for reasoning and inference.

A method of problem-solving that represents all possible states of a system and searches for a path from initial to goal state.