Logo
目录

LLM Paper Reading List

MathewShen MathewShen
January 4, 2025
2 min read
AI 论文

Step0: Basic Concepts

如果之前对 DL 的基本概念不是很熟悉，可以先看一些基础的 DL 论文。 如一些介绍性的论文，见Deep Learning Papers。 此外，DL 的一个核心是 BP 算法，相关的论文可以参考Backpropagation。

Step1: NMT, Attention and Transformer

Note (Reference Blogs)

TODO: 2014-Google-Sequence to Sequence

Sequence to Sequence Learning with Neural Networks

2014-Bengio-Attention

Neural Machine Translation by Jointly Learning to Align and Translate1. 本文首次提出了 Attention 的概念，首先介绍了什么是 Neural machine translation(NMT): attempts to build and train a single, large neural network that reads a sentence and outputs a correct translation.

之后指出当时现有的 NMT model 大多属于 encoder-decoder 类型的网络： An encoder neural network reads and encodes a source sentence into a fixed-length vector. A decoder then outputs a translation from the encoded vector. 在这种框架下，A neural network needs to be able to compress all necessary information of a source sentence into a fixed-length vector. 那么就有一个潜在的问题，在 source sentence 过长的时候， 这里得到的 fixed-length vector 不足以表达全部的信息， 所以后面基于此 vector 的翻译任务将很难有好的表现， 如下图所示（参考 Learning phrase representations using RNN encoder-decoder for statistical machine translation）：

Encoder-decoder fixed-length vector bottleneck

在T较大的时候，C很难将全部的信息综合起来，损失了较多的信息，所以使得 Decoder 很难有好的表现。

接着就是文章提出方法的具体描述了： The most important distinguishing feature of this approach from the basic encoder–decoder is that it does not attempt to encode a whole input sentence into a single fixed-length vector. Instead, it en-codes the input sentence into a sequence of vectors and chooses a subset of these vectors adaptively while decoding the translation. This frees a neural translation model from having to squash all the information of a source sentence, regardless of its length, into a fixed-length vector.

Attention-based neural machine translation architecture

上图 (来自本文) 进一步地补充了上面的解释。这里的 Encoder 是一个双向 RNN, Decode 仍然是一个 RNN。 不同的是，我们不再将 source sentence 的全部信息强行压缩到一个 fixed-length vector, 而是加上了一个 alignment model，其在每次预测yty_t的时候都会汇总来自 Encoder 隐藏层的信息 (annotations, 双向 RNN 拼接而成) 作为输入。

接下来解释了为什么这样方法是有效的：Intuitively,this implements a mechanism of attention in the decoder. The decoder decides parts of the source sentence to pay attention to. By letting the decoder have an attention mechanism, we relieve the encoder from the burden of having to encode all information in the source sentence into a fixed-length vector. With this new approach the information can be spread throughout the sequence of annotations, which can be selectively retrieved by the decoder accordingly.

关于 alignment model 的实现：We parametrize the alignment model as a feedforward neural network which is jointly trained with all the other components of the proposed system.

综上可知，这里的 Attention 就是说在预测的时候持续“关注”source sentence(输入) 以获得更好的预测， 而完成这个“关注”任务的，协调“关注”工作的任务就交给了 Attention Mechanism 的实现，如这里的 alignment model.

Note (PyTorch Practice)

TODO: 2017-Transformers

Attention Is All You Need

Step2: GPT Series

2018-OpenAI-GPT1.0

Improving Language Understanding by Generative Pre-Training

2018-Google-BERT

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

2019-OpenAI-GPT2.0

Language Models are Unsupervised Multitask Learners

2020-OpenAI-GPT3.0

Language models are few-shot learners

Reference

Footnotes

  1. Dzmitry Bahdanau et al., “Neural Machine Translation by Jointly Learning to Align and Translate,” arXiv Preprint arXiv:1409.0473, 2014.↩

LLM Engineering

MathewShen MathewShen
January 4, 2025
2 min read
AI 编程

LLM Serving

Importance metrics

Note (Databricks: [LLM Inference Performance Engineering: Best Practices](https://www.databricks.com/blog/llm-inference-performance-engineering-best-practices))
  • Time To First Token (TTFT): How quickly users start seeing the model’s output after entering their query. Low waiting times for a response are essential in real-time interactions, but less important in offline workloads. This metric is driven by the time required to process the prompt and then generate the first output token.
  • Time Per Output Token (TPOT): ime to generate an output token for each user that is querying our system. This metric corresponds with how each user will perceive the “speed” of the model. For example, a TPOT of 100 milliseconds/tok would be 10 tokens per second per user, or ~450 words per minute, which is faster than a typical person can read.
  • Latency: The overall time it takes for the model to generate the full response for a user. Overall response latency can be calculated using the previous two metrics: latency = (TTFT) + (TPOT) * (the number of tokens to be generated).
  • Throughput: The number of output tokens per second an inference server can generate across all users and requests.

Optimization

NVIDIA: Mastering LLM Techniques: Inference Optimization

Speculative Sampling

  • DeepMind: Accelerating large language model decoding with speculative sampling1
  • Google: Fast inference from transformers via speculative decoding2

DeepMind 和 Google 前后分别发了一篇 Speculative Sampling 的文章，内容比较相似 (还是有些许不同)。

KV Cache

图解大模型推理优化之 KV Cache 给出了非常详细且简洁的解释，文章中参考的 HuggingFace 代码来自 transformers/models/decision_transformer/modeling_decision_transformer.py

query = self._split_heads(query, self.num_heads, self.head_dim)
key = self._split_heads(key, self.num_heads, self.head_dim)
value = self._split_heads(value, self.num_heads, self.head_dim)


if layer_past is not None:
    past_key, past_value = layer_past
    key = torch.cat((past_key, key), dim=-2)
    value = torch.cat((past_value, value), dim=-2)

核心的逻辑就是如果有历史的 key 和 value，就把当前的 key 和 value 拼接到历史的 key 和 value 上，以此减少计算量。

MQA

Multi-Query Attention(MQA)

GQA

Grouped-query Attention(GQA)

Flash Attention

PagedAttention

Footnotes

  1. Charlie Chen et al., “Accelerating Large Language Model Decoding with Speculative Sampling,” arXiv Preprint arXiv:2302.01318, 2023.↩

  2. Yaniv Leviathan et al., “Fast Inference from Transformers via Speculative Decoding,” International Conference on Machine Learning, 2023, 19274–86.↩