LLM Algorithm Evolution

The classic statistical language model. Kneser-Ney smoothing was the industry default for 30 years.

Mapping words into dense vector spaces — the origin point of modern NLP.

Mikolov's two simplified models (CBOW, Skip-gram) made word embeddings trainable at scale.

Replaces softmax with binary classification, reducing Skip-gram training from O(V) to O(k).

Global co-occurrence matrix factorization — a complementary philosophy to word2vec.

Subword embeddings via character n-grams. Strong for morphologically rich languages and OOV.

Older text representations that still have a role in information retrieval and document clustering.

The foundational paper of neural language modeling — it nailed down 'learn a word vector.'

First word2vec paper with CBOW and Skip-gram.

Second paper introducing negative sampling, hierarchical softmax, subsampling.

The Stanford GloVe paper.

The FastText paper.

The original C implementation — still fast on large corpora.

The easiest Python library for word2vec, LDA, LSI.

Industrial NLP pipelines with pretrained vectors and reliable tooling.

Teaching-oriented NLP toolbox — N-gram, HMM, PCFG are all there.

The NLP textbook. The first few chapters cover N-gram and word vectors best.

Stanford's NLP course dedicates its first three lectures to word vectors.

A single post that captures the core intuition of distributed representations.

The simplest recurrent structure, held back in practice by vanishing gradients.

Hochreiter & Schmidhuber 1997. Gates + cell state are the core insight.

Cho 2014. Merges the three LSTM gates into two — fewer parameters, faster training.

Sutskever 2014's encoder-decoder framework — the skeleton for modern generative models.

Lets the decoder re-attend to arbitrary encoder positions. Direct precursor to Transformer.

Facebook 2017 used CNNs for Seq2Seq — easier to parallelize than RNNs, quickly overtaken by Transformer.

Bi-LSTM contextual embeddings — an early industrial demonstration of pretraining in NLP.

The original paper on gated networks to combat vanishing gradients.

Google applied encoder-decoder to machine translation successfully.

The original attention paper.

Global vs local attention compared to Bahdanau-style.

The ELMo paper — the first glimpse of large-scale pretraining in NLP.

Early RNN implementations lived here and remain the teaching starting point.

Open-source neural MT framework, the industry choice during the RNN era.

Facebook's Seq2Seq toolkit, later extended to Transformers.

Data loading and modeling components around the PyTorch ecosystem.

The post that made millions of people get character-level RNNs.

The clearest visual explanation of LSTM gates.

Stanford's NLP course notes on RNN, LSTM, and attention.

Vaswani 2017 — the common ancestor of every modern LLM.

Google 2018 encoder-only pretrained model with MLM.

OpenAI 2018. Established the decoder-only pretrain + fine-tune recipe.

Google 2019 text-to-text framework unifying all NLP tasks as generation.

FAIR optimized BERT training: more data, longer, dropping NSP.

Permutation LM combining AR and AE — elegant but engineering-heavy.

FAIR's encoder-decoder pretrained model with noise → denoising objective.

Attention under several 'views' computed in parallel — the Transformer's compute core.

Sinusoidal (original), learned, relative, RoPE — the main ways to express sequence order.

Critical for stable Transformer training. Pre-LN vs Post-LN matters.

BERT's pretraining objective — randomly mask and predict.

GPT's pretraining objective and the foundation of every modern generative LLM.

The three mainstream subword algorithms. Determines vocab and encoding efficiency.

The de facto pretrained model library with nearly every Transformer ported.

Meta's Seq2Seq/Transformer training framework, popular in academia.

Reference implementation of the original Transformer, now mostly a historical archive.

~300 lines of PyTorch GPT. The best way to internalize Transformer details.

The most-read visual Transformer explainer.

BERT and GPT versions of the same series.

Lilian Weng's overview of Transformer variants.

Stanford's lectures on Transformer architecture.

Zero-shot showed the power of pretraining and caused the 'too dangerous to release' debate.

NVIDIA's large-model training systems paper. Foundation of tensor parallelism.

The largest model trained in the T5 paper.

Microsoft's then-largest LLM, trained with DeepSpeed.

Few-shot in-context learning arrived and shocked the NLP world.

AI21's contemporary to GPT-3.

Google's 540B Pathways model — briefly the largest.

DeepMind's compute-optimal experiment: a 70B model beat 280B at matched compute.

First systematic demonstration of zero-shot.

Original paper on compute/data/parameter power laws.

Classic paper on few-shot in-context learning.

Corrected Kaplan and introduced the new 20 tokens/param ratio.

Systematic record of capabilities that appear suddenly past a scale threshold.

NVIDIA's large-model training toolkit, reference implementation of tensor parallelism.

Microsoft's training optimization library and primary ZeRO implementation.

EleutherAI's open-source replication — the first 'community' big models.

The de facto hub for models and checkpoints.

OpenAI's introduction and early use-case showcase.

The original 'compute doubles every 3.4 months' analysis.

The most thorough external treatment of scaling — long but worth it.

Dwarkesh Patel's interviews with AI researchers on scaling.

Fine-tuning on instruction-response pairs. The first step of every subsequent alignment pipeline.

Takes answer pairs, emits preference scores. The referee in RLHF.

The RL algorithm InstructGPT used. Core idea: clipped objective to bound updates.

Replaces RL with a closed-form cross-entropy loss.

Anthropic's self-critique framework with AI feedback replacing most human labels.

2024 DPO variants that further simplify or correct the loss.

The landmark paper for industrial RLHF.

Anthropic's alignment methodology with explicit value principles.

Replaced RL with a supervised loss — 2023's most important alignment paper.

Simplifies DPO further by removing the reference model.

Preference learning grounded in Kahneman-Tversky prospect theory.

The turning point that turned GPT-3.5 into a conversational partner.

OpenAI's flagship — multimodal with sharply improved alignment.

Constitutional AI plus RLHF — helpful, harmless, honest as design goals.

Meta's open aligned models. The most detailed RLHF recipes in public papers.

The domestic Chinese open-aligned flagship. R1 pushed RL-on-reasoning to the open-source frontier.

OpenAI's alignment research portal.

First-hand source on Anthropic's alignment and interpretability work.

The most widely read RLHF explainer post.

Lil'Log's alignment coverage with the most complete technical detail.

Shazeer 2017's sparse gated expert layer — the foundation of modern large-model efficiency.

Google's simplified MoE scaling to trillions of parameters.

Storing weights in 8-bit or 4-bit can cut inference cost by an order of magnitude.

Low-rank adaptation lets fine-tuning touch <1% of parameters.

Tri Dao's IO-aware attention kernel, near O(n) on HBM in practice.

KV cache as paged virtual memory — memory efficiency from 20-40% up to 90%+.

Cut attention-head count or share KV to reduce cache and bandwidth pressure.

Small model drafts, large model verifies — typically 2-4× inference latency gain.

The first major MoE paper.

Google pushed MoE to 1.6T parameters in 2021.

The low-rank adaptation paper.

The landmark attention-kernel optimization paper.

The KV-cache management revolution.

Mistral's 8x7B — the open-source MoE milestone.

A complete case of MoE + MLA + FP8 done together.

The first trillion-parameter MoE.

Mistral's open-source MoE — ~13B active parameters, inference efficiency near Llama-70B.

The Chinese open-source flagship. MLA + MoE dropped training cost to a tenth of industry peers.

Google's flagship pushing context to 1M tokens.

Alibaba's open + closed MoE series with a strong efficiency focus.

The FlashAttention author's technical blog.

The authoritative reference for PagedAttention and continuous batching.

ZeRO, MoE, and inference engineering content.

Efficiency-related chapters capture many engineering trade-offs.

Image-text contrastive learning opened the multimodal era and set the zero-shot classification baseline.

Insert vision tokens into a frozen LLM for few-shot VQA.

Q-Former bridges the vision encoder and the LLM at low training cost.

OpenAI's multimodal flagship — GPT-4o delivers native text, audio, and vision.

Google's family of native multimodal models.

The Visual Instruction Tuning reference point for the open community.

Alibaba's open multimodal series chasing GPT-4V.

A single "let's think step by step" significantly boosts reasoning performance.

Sampling plus majority voting — a stronger CoT.

Expands CoT into a searchable reasoning tree.

Models trained with RL-on-reasoning. Expert-level on AIME and Codeforces.

The first open model to reproduce o1-class reasoning. GRPO became a new RL baseline.

Anthropic's explicit-thinking capability on Claude 3.7 / 4.

The seminal multimodal learning paper.

The reference few-shot visual language model.

The first systematic study of step-by-step reasoning.

The CoT → ToT expansion.

The open-source canonical example of GRPO and multi-stage RL training.

OpenAI's scaling laws for inference-time compute.

The official capability, safety, and benchmark report for o1.

All the details of GRPO and multi-stage RL training.

Ties CoT, ToT, and Agent together clearly.

Daily multimodal and reasoning evaluations from an engineering perspective.

Selective SSM with linear complexity and near-Transformer quality — the strongest current challenger.

RNN-style training with Transformer-style performance; a community-driven open project.

Microsoft's Retentive Network — another attention alternative.

Implicit long convolution replaces attention — part of the sub-quadratic wave.

Transformer + Mamba hybrid with an edge at long contexts.

Reasoning + acting interleaved — the skeleton behind most agent designs.

The paper that taught models to decide when to call APIs on their own.

Community projects that ignited agent imagination in 2023.

Anthropic-led open protocol to standardize tool connectivity.

Microsoft's and LangChain's multi-agent orchestration frameworks.

2025 products that turned 'models driving a computer' into real offerings.

Diffusion Transformer pushing minute-length video to convincing quality.

Generate interactive 2D / 3D worlds from a single image.

NVIDIA's World Foundation Model aimed at robotics and autonomous driving.

Google's high-quality video generation model.

Leading commercial video-generation products.

Chris Olah-led Transformer Circuits work — opening up LLM internals.

A new scaling curve where 'thinking longer' replaces 'bigger model.'

As public data approaches exhaustion, self-generated training data is a hot alternative.

Multiple specialized models / agents collaborating — already effective in code, research, and negotiation.

RT-2, OpenVLA, and Gemini Robotics bring LLMs into robotics.

Systems engineering detail behind Mamba.

The main site for mechanistic interpretability.

The most diligent blog tracking frontier products and models.

Anthropic co-founder's weekly with both policy and technical framing.

Annual reviews of AI industry evolution from major investors.