LLM Systems 14: Distributed Training 0. Recap 之前课程主线: LLM 简介 CUDA 编程基础 深度学习框架(自动求导) Transformer/Pre-trained LLM LLM 前后两个辅助组件:Tokenizer 与 Decoding Transformer 加速:Kernel Fusion & Mix-precision & Memory Management Optimiza 2026-10-17 ML System #CMU 11868 #LLM System #Distributed Training
LLM Systems 10 & 11: Accelerating Transformer Training and Inference 0. Recap Transformer Architecture: 1. Accelerated GPU Computation for Transformer Training Overview: 对应下面将要介绍的 4 个优化手段: Kernel Fusion:计算图级别优化,通过算子融合减少计算图节点。 Reduce synchronization:算子内优化 2026-10-17 ML System #CMU 11868 #LLM System #kernel
LLM Systems 09: Decoding: Sampling, Beam Search and Speculative Decoding 0. Recap Subword tokenization: Byte-Pair-Encoding iteratively merging most frequent pairs of tokens Information-theoretic vocabulary (VOLT) solving entropy constrained optimal transport pr 2026-10-17 ML System #CMU 11868 #LLM System #Decoding
LLM Systems 08: Tokenization and Embedding 0. Recap Pretraining: Mask-labeled recovery of random spans + next token prediction Multitask supervised fine-tuning with instruction templates Relative position instead of absolute position: Ro 2026-10-17 ML System #CMU 11868 #LLM System #Tokenization
LLM Systems Assignment 3: Transformer Architecture 本次 homework 的任务是实现 Decoder-only 的 Transformer 模型,也就是 GPT-2,并在一个翻译任务数据集上进行模型训练。具体来说,包括以下几个子任务: 实现 CrossEntropy Loss 实现基础模块,包括 Linear/Dropout/LayerNorm1d/Embedding 实现 Transformer,包括:MultiHeadAtte 2026-10-05 ML System #CMU 11868 #LLM System #GPT #Transformer #minitorch
LLM Systems Assignment 2: Minitorch Framework 本次 homework 需要编写的代码量实际上非常少(100 行左右),完成作业的关键是理解 minitorch 设计和已有代码。因此,本文主要介绍 minitorch 代码结构,解释其 Tensor、Parameter、Operator、Module、History 等关键设计和实现。 课程官方对 minitorch 有个 Jupyter Notebook 做了详细介绍可参考。本文 2026-10-05 ML System #CMU 11868 #LLM System #minitorch
LLaMA: Open and Efficient Foundation Language Models LLaMA: Open and Efficient Foundation Language Models arXiv 提交日期:27 Feb 2023(ChatGPT 发布后约 3 个月) GitHub 链接:https://github.com/meta-llama/llama 作者: 摘要: 我们提出了 LLaMA,一组从 7B 到 65B 参数的基础语言模型。我们 2026-10-05 ML System #CMU 11868 #LLM System #LLaMA
GPT-3: Language Models are Few-Shot Learners Language Models are Few-Shot Learners arXiv 提交日期:28 May 2020 OpenAI blog:Language models are few-shot learners 作者: 摘要: 近期工作表明,通过在大规模文本语料上预训练,再针对特定任务进行微调,可以在许多 NLP 任务和基准上取得显著提升。虽然这种方法在架 2026-10-05 ML System #CMU 11868 #LLM System #GPT
GPT-2: Language Models are Unsupervised Multitask Learners Language Models are Unsupervised Multitask Learners OpenAI Technical Report,blog 链接:Better language models and their implications ,发布日期:February 14, 2019 官方 GitHub repo(TF 实现):gpt-2 ,仅公布了小模型权重 2026-10-05 ML System #CMU 11868 #LLM System #GPT
GPT-1: Improving Language Understanding by Generative Pre-Training Improving Language Understanding by Generative Pre-Training OpenAI Technical Report,blog 链接:Improving language understanding with unsupervised learning ,发布日期:June 11, 2018(Transformer paper 发布后的 2026-10-05 ML System #CMU 11868 #LLM System #GPT