Part-of-Speech Tagging

Byte-Pair Encoding

A subword tokenisation algorithm that iteratively merges the most frequent character pairs to build a vocabulary.

Named Entity Recognition

An NLP task that identifies and classifies named entities in text into categories like person, organisation, and location.

Dependency Parsing

The syntactic analysis of a sentence to establish relationships between head words and words that modify them.

Coreference Resolution

The task of identifying all expressions in text that refer to the same real-world entity.

Information Extraction

The process of automatically extracting structured information from unstructured or semi-structured text sources.

Relation Extraction

Identifying semantic relationships between entities mentioned in text.

A neural network trained on massive text corpora that can generate, understand, and reason about natural language.

The generation of machine-readable formatted responses such as JSON, XML, or code from language models, enabling reliable integration with downstream software systems.

The component of conversational systems that tracks conversation state, determines the next system action, and maintains coherent multi-turn interactions with users.

A text generation strategy that restricts the model to sampling from the K most probable next tokens.

The method of dividing long documents into smaller segments for embedding and retrieval, balancing context preservation with optimal chunk sizes for vector search accuracy.

The process of breaking text into smaller units (tokens) such as words, subwords, or characters for processing by language models.

A generative probabilistic model for discovering topics in a collection of documents.

The field of AI focused on enabling computers to understand, interpret, and generate human language.