arcsin2's blog
  • 首页
  • 归档
  • 分类
  • 标签
  • 关于
  • 友链
  • English

The Annotated Transformer

The Annotated Transformer code 本文仅摘录原始 blog 部分代码并添加部分注释,不涉及原文全部内容,推荐阅读原文。 Model Architecture 1234567891011121314151617181920212223class EncoderDecoder(nn.Module): """
2026-10-05
ML System
#CMU 11868 #LLM System #Transformer

Transformer: Attention Is All You Need

Attention Is All You Need 发表于 NIPS 2017(arXiv 提交日期:12 Jun 2017) 作者: Abstract 主流的序列转换模型基于复杂的循环或卷积神经网络,这些网络包含一个编码器和一个解码器。性能最佳的模型还通过注意力机制连接编码器和解码器。我们提出了一种新的简单网络架构——Transformer,它完全基于注意力机制,彻底摒弃
2026-10-05
ML System
#CMU 11868 #LLM System #Transformer

TensorFlow: A system for large-scale machine learning

TensorFlow: A system for large-scale machine learning OSDI 2016 paper link 阅读该 paper 主要要了解: TensorFlow 诞生背景 DistBelief 存在的问题 设计原则 关键概念&实现(TensorFlow execution model) 如何支持大规模训练(E
2026-10-05
ML System
#CMU 11868 #LLM System #TensorFlow

LLM Systems 07: Pre-trained LLMs

0. Recap Sequence-to-sequence encoder-decoder framework for conditional generation, including Machine Translation Key components in Transformer Positional Embedding (to distinguish tokens at d
2026-10-05
ML System
#CMU 11868 #LLM System #Transformer

LLM Systems 06: Transformer

0. Recap Learning parameters of an NN needs gradient calculation Computation Graph to perform computation: topological traversal along the DAG Auto Differentiation building backward compu
2026-10-05
ML System
#CMU 11868 #LLM System #Transformer

LLM Systems 05: Deep Learning Framework and Auto Differentiation

0. Recap Tiling for efficient matrix computation Use shared memory to reduce global memory access Roofline Model & Computational Intensity Coalesced memory access Understanding the ch
2026-10-05
ML System
#CMU 11868 #LLM System

LLM Systems Assignment 1: CUDA Programming

本次作业主要涉及两个文件: llmsys_hw1/minitorch/cuda_kernel_ops.py:Python 侧调用 kernel 编译成的 .so 接口 llmsys_hw1/src/combine.cu:CUDA kernel 实现 完成本次作业的主要预备知识要求是掌握基本的 CUDA 编程能力(如:按照课程 readings 要求完成 PMPP 这本书前 6
2026-09-19
ML System
#CMU 11868 #LLM System #CUDA

PMPP Chapter 06: Performance considerations

本章主要内容 present the off-chip memory (DRAM) architecture and discuss related performance considerations such as memory coalescing and memory latency hiding. discuss an important type of optimizati
2026-09-19
ML System
#CMU 11868 #LLM System #CUDA #PMPP

PMPP Chapter 05: Memory architecture and data locality

本章主要关注 GPU 的 on-chip memory,重点是利用 shared memory 减少 global memory 访问,提升 kernel 性能。 Importance of memory access efficiency Compute to Global Memory Access Ratio(算术强度 / 计算强度) 定义: 指程序某区域内,每
2026-09-19
ML System
#CMU 11868 #LLM System #CUDA #PMPP

PMPP Chapter 04: Compute architecture and scheduling

Architecture of a modern GPU GPU 计算单元由若干个 SM 组成(通常几十到一百多) 每个 SM 有若干个处理单元,称为 CUDA Cores(简称 cores) Memory(on-chip memory) Example:A100 GPU,108 SMs,每个 SM 64 个 cores global memory(off-chip
2026-09-19
ML System
#CMU 11868 #LLM System #CUDA #PMPP
1…34567…11

搜索

Hexo Fluid
总访问量 次 总访客数 人
苏ICP备2021026429号-1