arcsin2's blog
  • 首页
  • 归档
  • 分类
  • 标签
  • 关于
  • 友链
  • English

LLM Systems 14: Distributed Training

0. Recap 之前课程主线: LLM 简介 CUDA 编程基础 深度学习框架(自动求导) Transformer/Pre-trained LLM LLM 前后两个辅助组件:Tokenizer 与 Decoding Transformer 加速:Kernel Fusion & Mix-precision & Memory Management Optimiza
2026-10-17
ML System
#CMU 11868 #LLM System #Distributed Training

LLM Systems 10 & 11: Accelerating Transformer Training and Inference

0. Recap Transformer Architecture: 1. Accelerated GPU Computation for Transformer Training Overview: 对应下面将要介绍的 4 个优化手段: Kernel Fusion:计算图级别优化,通过算子融合减少计算图节点。 Reduce synchronization:算子内优化
2026-10-17
ML System
#CMU 11868 #LLM System #kernel

LLM Systems 09: Decoding: Sampling, Beam Search and Speculative Decoding

0. Recap Subword tokenization: Byte-Pair-Encoding iteratively merging most frequent pairs of tokens Information-theoretic vocabulary (VOLT) solving entropy constrained optimal transport pr
2026-10-17
ML System
#CMU 11868 #LLM System #Decoding

LLM Systems 08: Tokenization and Embedding

0. Recap Pretraining: Mask-labeled recovery of random spans + next token prediction Multitask supervised fine-tuning with instruction templates Relative position instead of absolute position: Ro
2026-10-17
ML System
#CMU 11868 #LLM System #Tokenization

LLM Systems Assignment 3: Transformer Architecture

本次 homework 的任务是实现 Decoder-only 的 Transformer 模型,也就是 GPT-2,并在一个翻译任务数据集上进行模型训练。具体来说,包括以下几个子任务: 实现 CrossEntropy Loss 实现基础模块,包括 Linear/Dropout/LayerNorm1d/Embedding 实现 Transformer,包括:MultiHeadAtte
2026-10-05
ML System
#CMU 11868 #LLM System #GPT #Transformer #minitorch

LLM Systems Assignment 2: Minitorch Framework

本次 homework 需要编写的代码量实际上非常少(100 行左右),完成作业的关键是理解 minitorch 设计和已有代码。因此,本文主要介绍 minitorch 代码结构,解释其 Tensor、Parameter、Operator、Module、History 等关键设计和实现。 课程官方对 minitorch 有个 Jupyter Notebook 做了详细介绍可参考。本文
2026-10-05
ML System
#CMU 11868 #LLM System #minitorch

LLaMA: Open and Efficient Foundation Language Models

LLaMA: Open and Efficient Foundation Language Models arXiv 提交日期:27 Feb 2023(ChatGPT 发布后约 3 个月) GitHub 链接:https://github.com/meta-llama/llama 作者: 摘要: 我们提出了 LLaMA,一组从 7B 到 65B 参数的基础语言模型。我们
2026-10-05
ML System
#CMU 11868 #LLM System #LLaMA

GPT-3: Language Models are Few-Shot Learners

Language Models are Few-Shot Learners arXiv 提交日期:28 May 2020 OpenAI blog:Language models are few-shot learners 作者: 摘要: 近期工作表明,通过在大规模文本语料上预训练,再针对特定任务进行微调,可以在许多 NLP 任务和基准上取得显著提升。虽然这种方法在架
2026-10-05
ML System
#CMU 11868 #LLM System #GPT

GPT-2: Language Models are Unsupervised Multitask Learners

Language Models are Unsupervised Multitask Learners OpenAI Technical Report,blog 链接:Better language models and their implications ,发布日期:February 14, 2019 官方 GitHub repo(TF 实现):gpt-2 ,仅公布了小模型权重
2026-10-05
ML System
#CMU 11868 #LLM System #GPT

GPT-1: Improving Language Understanding by Generative Pre-Training

Improving Language Understanding by Generative Pre-Training OpenAI Technical Report,blog 链接:Improving language understanding with unsupervised learning ,发布日期:June 11, 2018(Transformer paper 发布后的
2026-10-05
ML System
#CMU 11868 #LLM System #GPT
123456…11

搜索

Hexo Fluid
总访问量 次 总访客数 人
苏ICP备2021026429号-1