用 NumPy 60行代码实现GPT

Table of Contents

What is a GPT?

• Input / Output

  • Input

  • Output

• Generating Text

  • Autoregressive

  • Sampling

• Training

• Prompting

Setup

• Encoder

• Hyperparameters

• Parameters

Basic Layers

• GELU

• Softmax

• Layer Normalization

• Linear

GPT Architecture

• Embeddings

  • Token Embeddings

  • Positional Embeddings

  • Combined

• Decoder Stack

• Projection to Vocab

• Decoder Block

• Position-wise Feed Forward Network

• Multi-Head Causal Self Attention

  • Attention

  • Self

  • Causal

  • Multi-Head

Putting it All Together

What Next?

• GPU/TPU Support

• Backpropagation

• Batching

• Inference Optimization

• Training

• Evaluation

• Architecture Improvements