LLM Systems 01: Introduction

This article is the learning note of CMU 11868 LLM Systems course Lecture 1. For more details, please refer to the original slides.

0. Learning Objective of This Course

  • Understand key techniques for modern LLM systems
    • How much resources do you need to train a 100B model?
  • Engineering skills to implement key components for LLM systems
    • Fast CUDA kernel for LLM
    • Scalable training system
    • Efficient inference
  • Innovation: discover opportunities and solve new critical challenges in LLM system research

1. Capabilities of LLMs

  • Translation
  • Answer daily life questions
  • Summarize
  • Polish Email
  • Math Calculation
  • Write Code
  • ChatBot to Mimic a User
  • Rewrite Text with Style
  • Commonsense Reasoning
  • Math Reasoning
  • ...

2. Mathematical Foundations

Probability Model of LLM

The forward process of LLM is a probability problem:

Given a prompt with t tokens, what is the probability of output token \(t_{t}\) ?

In the formula of math equation: \[ P(\text{next word } y_{t} | \text{Prompt } {x}, \text{provious words } y_{1:t-1} ) \]

The LLM forward(inference) process just get the probability of all words in it's word dictionary(embedding), and you can just think we select the word with the biggest probability as the output token (In fact, there are different sampling strategies, which will be discussed in the following lectures, but just ignore it right now).

Based on the probability model of next token, we can induce the probability model of a full sequence:

Probability(“Pittsburgh is a city of bridges”) = 𝑃(“𝑃𝑖𝑡𝑡𝑠𝑏𝑢𝑟𝑔ℎ”) ∙ 𝑃(“𝑖𝑠”|“𝑃𝑖𝑡𝑡𝑠𝑏𝑢𝑟𝑔ℎ”) ∙ 𝑃(“𝑎”|”𝑃𝑖𝑡𝑡𝑠𝑏𝑢𝑟𝑔ℎ 𝑖𝑠”) ∙ 𝑃(“𝑐𝑖𝑡𝑦”| … ) ∙ 𝑃(“𝑜𝑓”| … ) ∙ 𝑃(“𝑏𝑟𝑖𝑑𝑔𝑒𝑠”| … )

Also in the formula of math equation: \[ \text{Prob.}\left( x_{1..T} \right) = \prod_{t=1}^{T}{P\left( x_{t+1} | x_{1..t} \right)} \]

The probability \(P\left( x_{t+1} | x_{1..t} \right)\) is gotten by predicting using Neural Networks(Transformer network, CNN, RNN).

Types of Language Models

There are 3 types of LLM classified by model architecture:

  • Encoder-only:

    • Key Features:
      • Masked LM
      • Non-autoregressive: it predicts masked tokens in parallel, not one-by-one from left to right.
      • Example: BERT.
    • Masked Prediction: \(P\left( x_{mask} | x \right)\) , which means: given unmasked sequence \(x\), predict the token(s) \(x_{mask}\) that are masked. This process is shown in the following picture:
    • The attention of BERT is bidirectional, since each [MASK] can see the contexts in front of and behind it, which is different from the GPT's unidirectional/causal attention (Each token can only see tokens in front of it).
    • Usually used in language understanding tasks, such as classification, NER(Named Entity Recognition), and extractive question answering.
    • Encoder-only LLMs can also be used to generate language. e.g. NAT, REDER (reversible duplex model)
      • NAT(Non-Autoregressive Translation): predicts many/all tokens in parallel, often with iterative refinement.
      • REDER(Reversible Duplex Model): A reversible, duplex model that can support generation by recovering or filling masked/noised tokens, rather than strictly decoding left-to-right.

  • Encode-Decoder:

    • Key Features
      • Models conditional probability \(P(Y|X)\): given input sequence \(X\), generate output sequence \(Y\).
      • Designed for sequence-to-sequence (seq2seq) tasks, e.g. machine translation, summarization.
      • Encoder encodes the input \(X\); decoder generates the output \(Y\) conditioned on the encoder’s representations.
    • Encoder choices: Bi-LSTM, Transformer, CNN.
    • Decoder choices: LSTM, Transformer, Non-autoregressive Transformer, CNN.
    • Decoder can be autoregressive (e.g. LSTM/Transformer, left-to-right) or non-autoregressive (parallel generation).

  • Decoder-only: Autoregressive

    • Key Features:
      • Causal Language Model: \(P(X)=\prod_{n=1}^{N}{P(x_{n}|x_{1..n-1})}\), i.e. predict the next token given all previous tokens.
      • Autoregressive generation: generate one token at a time from left to right; each predicted token is appended and fed back as context for the next prediction.
    • Model choice: Transformer or (uni-directional) LSTM.
    • Most popular choice of LLM architecture.

LLM Learning Framework

The training process of LLMs includes 3 main steps:

  • Pre-Training
    • Data: raw text corpus (e.g. Wikipedia, books, Reddit).
      • the larger the better, often crawled from the web
      • quality is important: Wikipedia, books, filtered Reddit posts
    • Design of Model Architecture
      • Sparse Attention, Mixture of Expert, etc.
    • Objective
      • cross-entropy loss for next token prediction(to learn general language and world knowledge).
      • \(CE=-\frac{1}{N}\sum_{n=1}^{N}{\log{P_{\theta}\left(x_{n} | x_{\lt n}\right)}}\)
  • SFT(Supervised Fine-Tuning)
    • Data: instruction–response pairs, e.g. question Why is the sky blue? → answer The sky appears blue because …
    • Objective: fine-tune the pretrained model to follow instructions and generate helpful answers.
  • RL(Post-Training)
    • Data: preference data, e.g. comparing The sky appears blue because … vs. The sky is not always blue …
    • Objective: align model outputs with human preferences (e.g. via RLHF), improving helpfulness, honesty, and harmlessness.

Overall: raw text → supervised demonstrations → human preferences, progressively making the model more capable and aligned.

Measuring Performance of Language Model

  • Perplexity (PPL): for open-ended generation.

    • Measures how well the LM predicts a sequence: \[ PPL = \exp\left(-\frac{1}{N}\sum_{n=1}^N \log{P_{LM}(x_n \mid x_{<n})}\right) \]

    • Lower PPL = better; it is the exponentiated average negative log-likelihood (cross-entropy).

    • Limitation: low PPL does not always mean high generation quality.

  • Reference-based metric: requires reference text(s).

    • Model-based score: SEScore2 / COMET / BLEURT.
      • Use pretrained models to score semantic similarity / quality, beyond surface n-gram overlap.
    • N-gram matching: BLEU (translation) / ROUGE (summarization).
      • old but classic; based on n-gram overlap between generated and reference text.
    • InstructScore: explainable score Xu et al. 2023.
      • Uses LLM to provide a score together with an explanation.

  • Using downstream task performance metrics: choose metric by task.

    • Named entity recognition: F1 — harmonic mean of precision and recall, \(F1 = 2 \cdot \frac{P \cdot R}{P + R}\).
    • Question Answering: accuracy, matching rate (e.g. exact match / overlap with reference answer).
    • Code generation: pass@k — fraction of problems solved by at least one of k generated samples.
    • Retrieval: NDCG (Normalized Discounted Cumulative Gain) — rewards relevant items ranked higher.
    • Summarization: ROUGE — n-gram overlap with reference summaries, recall-oriented.
    • Translation: COMET / SEScore2 / BLEU — model-based semantic scores plus classic n-gram overlap.
    • Takeaway: no single metric fits all; use task-specific metrics, often combined with human evaluation.

3. Challenges in LLM Systems

3.1 Key System Problem & Goal

  • Goal: compute (train/inference) larger LLMs on bigger datasets with fewer resources (GPU/memory/power) faster.
  • Key tension: scaling up (model, data, context) vs. limited resources.
  • Tradeoffs: every design choice involves fundamental constraints and different success metrics (latency, throughput, memory, cost, quality).

3.2 What Computation Actually Looks Like

  • High-level layers: Multi-head Attention, Layer Norm, Dropout, Linear, activation (Tanh / ReLU / GELU), Softmax.
  • Low-level operators: matmul / tensor multiply, reduction (sum, avg), map (element-wise), memory movement.
  • Key insight: computation is not the only cost — data movement (parameters, gradients, activations across devices) often dominates.
    • Large models → huge parameter transfer cost.
    • Long context → large working memory.

3.3 Where the Challenges Appear (by abstraction level)

Level Challenge
Model / data scaling to gigantic model, gigantic dataset, very long sequence / context
System / framework partitioning, scheduling, communication, distributed / parallel systems
Developer / framework easy dev & modify model, easy dev ML algorithm
Kernel / operator fast CUDA / TPU kernels
Compression data / model compression

3.4 How to Address Them

  • Model-Algorithm-System Co-design
    • Model architecture
    • Training / inference algorithms
    • Software optimization: partitioning, scheduling, data movement, latency hiding
    • Hardware acceleration: device-specific instructions
  • Right abstraction at each level:
    • Upper: integrate models into product system; track & improve quality over time.
    • Middle: training / inference software, streaming dataflow.
    • Lower: GPU kernels, compilers.
    • A good abstraction hides one or more concerns (performance, failures) while supporting many apps.

Takeaway: LLM systems are not just about fast computation. The real challenges are data movement, resource constraints, and abstraction design — solved by co-designing model, algorithm, software, and hardware.

4. Logistics

(omitted)

ML System
#CMU 11868 #LLM System
LLM Systems 01: Introduction
https://arcsin2.cloud/posts/4065116746/
作者
arcsin2
发布于
2026年9月12日
许可协议