arcsin2's blog
  • 首页
  • 归档
  • 分类
  • 标签
  • 关于
  • 友链
  • English

Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism

Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism arXiv 提交日期:17 Sep 2019(GPT 2 blog 发布约半年后,即:February 14, 2019 -> 17 Sep 2019) 代码:Megatron-LM 作者: 摘要:
2026-10-31
ML System
#CMU 11868 #LLM System #Distributed Training #TP

GPipe: Easy Scaling with Micro-Batch Pipeline Parallelism

GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism arXiv 提交日期:6 Nov 2018 作者: 摘要: 扩大深度神经网络的容量,已被证明是提升多种不同机器学习任务中模型质量的有效方法。在许多情况下,当模型容量增长并超出单个加速器的显存限制时,往往需要开发专门的算
2026-10-31
ML System
#CMU 11868 #LLM System #Distributed Training #PP

LLM Systems 18: Memory Optimization in Distributed Training

1. Memory Consumption for LLM training DDP 与 MP 回顾: DP 优点: Good compute/communication efficiency 缺点: Poor memory efficiency(Every device has one copy of model) MP 优点:
2026-10-31
ML System
#CMU 11868 #LLM System #Distributed Training #ZeRO

LLM Systems 17: System for MoE Models

0. Recap Pipeline Parallelism split by layers (horizonal split) eliminate the bubbles (idle) interleaving forward/backward Tensor Parallelism split the matrix computation over multiple G
2026-10-31
ML System
#CMU 11868 #LLM System #Distributed Training #MoE

LLM Systems 16: Model Parallel Training

0.Recap Data Parallel via All Reduce Distributed Data Parallel Training gradient bucketing overlay backward and AllReduce communication 1. Model Parallel Training Motivation: 模型规模呈指数增长,单
2026-10-31
ML System
#CMU 11868 #LLM System #Distributed Training

LLM Systems Assignment 4: Transformer CUDA Acceleration

本次 homework 主要内容是:实现 softmax 与 LayerNorm kernel fusion 形式的 fwd 与 bwd,完成 CUDA kernel 编写;接入 Python 层;接入 Transformer 模型并验证训练加速。 下面对本次 homework 实现时一些注意点进行介绍。 1. Softmax 1.1 forward 课程已经提供了 block_dim
2026-10-17
ML System
#CMU 11868 #LLM System #kernel

DDP: PyTorch Distributed-Experiences on Accelerating Data Parallel Training

PyTorch Distributed: Experiences on Accelerating Data Parallel Training 代码:PyTorch DDP arXiv 提交日期:28 Jun 2020 (发表于 VLDB 2020) 作者: 摘要: 本文介绍了 PyTorch 分布式数据并行(Distributed Data Parallel, D
2026-10-17
ML System
#CMU 11868 #LLM System #DDP

LightSeq2: Accelerated Training for Transformer-based Models on GPUs

LightSeq2: Accelerated Training for Transformer-based Models on GPUs arXiv 提交日期:12 Oct 2021 GitHub:https://github.com/bytedance/lightseq 作者: 摘要 基于 Transformer 的神经网络模型已被广泛应用于许多人工智能领域。训练这
2026-10-17
ML System
#CMU 11868 #LLM System #kernel

LightSeq: A High Performance Inference Library for Transformers

LightSeq: A High Performance Inference Library for Transformers arXiv 提交日期:23 Oct 2020 GitHub:https://github.com/bytedance/lightseq 作者: 摘要 Transformer、BERT 及其变体在自然语言处理领域取得了巨大的成功。由于 Tran
2026-10-17
ML System
#CMU 11868 #LLM System #kernel

LLM Systems 15: Distributed Data Parallel Training

0. Recap Overall idea: partition the data, distribute the forward/backward Parameter Server server to update and distribute parameters, worker to get local grad NCCL Multi-GPU communicatio
2026-10-17
ML System
#CMU 11868 #LLM System #Distributed Training
12345…11

搜索

Hexo Fluid
总访问量 次 总访客数 人
苏ICP备2021026429号-1