What is a Transformer Model?

Transformers are also moving beyond text, being adapted for audio, vision, and even robotics, pushing the limits of what AI can achieve.

Transformer Models Code Examples

The Transformer architecture, introduced by Vaswani et al. in 2017, has become the foundation of modern NLP models like BERT, GPT, and T5. I’ll explain the core ideas and walk through minimal code examples using PyTorch to build your own Transformer blocks.

🔧 Prerequisites

• Python 3.8+
• PyTorch
• NumPy

Install dependencies:

pip install torch numpy

⚙️ Step-by-Step Code

1. Positional Encoding

import torch
import math
def positional_encoding(seq_len, d_model):
    pos = torch.arange(0, seq_len).unsqueeze(1)
    i = torch.arange(0, d_model, 2)
    angle_rates = 1 / torch.pow(10000, (i / d_model))
    pos_enc = torch.zeros(seq_len, d_model)
    pos_enc[:, 0::2] = torch.sin(pos * angle_rates)
    pos_enc[:, 1::2] = torch.cos(pos * angle_rates)
    return pos_enc

2. Scaled Dot-Product Attention

def scaled_dot_attention(q, k, v, mask=None):
    d_k = q.size(-1)
    scores = torch.matmul(q, k.transpose(-2, -1)) / math.sqrt(d_k)
    if mask is not None:
        scores = scores.masked_fill(mask == 0, float('-inf'))
    attn = torch.nn.functional.softmax(scores, dim=-1)
    return torch.matmul(attn, v), attn

3. Multi-Head Attention

import torch.nn as nn
class MultiHeadAttention(nn.Module):
    def __init__(self, heads, d_model):
        super().__init__()
        assert d_model % heads == 0
        self.d_k = d_model // heads
        self.heads = heads
        self.qkv_proj = nn.Linear(d_model, d_model * 3)
        self.out_proj = nn.Linear(d_model, d_model)
    def forward(self, x):
        B, T, C = x.shape
        qkv = self.qkv_proj(x).reshape(B, T, 3, self.heads, self.d_k).permute(2, 0, 3, 1, 4)
        q, k, v = qkv[0], qkv[1], qkv[2]
        out, _ = scaled_dot_attention(q, k, v)
        out = out.transpose(1, 2).contiguous().view(B, T, C)
        return self.out_proj(out)

4. Feed-Forward Network

class FeedForward(nn.Module):
    def __init__(self, d_model, d_ff):
        super().__init__()
        self.linear1 = nn.Linear(d_model, d_ff)
        self.relu = nn.ReLU()
        self.linear2 = nn.Linear(d_ff, d_model)
    def forward(self, x):
        return self.linear2(self.relu(self.linear1(x)))

5. Transformer Block

class TransformerBlock(nn.Module):
    def __init__(self, d_model, heads, d_ff):
        super().__init__()
        self.attn = MultiHeadAttention(heads, d_model)
        self.ff = FeedForward(d_model, d_ff)
        self.norm1 = nn.LayerNorm(d_model)
        self.norm2 = nn.LayerNorm(d_model)
    def forward(self, x):
        attn_out = self.attn(x)
        x = self.norm1(x + attn_out)
        ff_out = self.ff(x)
        return self.norm2(x + ff_out)

6. Putting It All Together

class SimpleTransformer(nn.Module):
    def __init__(self, vocab_size, d_model=512, num_layers=6, heads=8, d_ff=2048, max_len=100):
        super().__init__()
        self.embed = nn.Embedding(vocab_size, d_model)
        self.pos_enc = positional_encoding(max_len, d_model)
        self.layers = nn.ModuleList([
            TransformerBlock(d_model, heads, d_ff) for _ in range(num_layers)
        ])
        self.out = nn.Linear(d_model, vocab_size)
    def forward(self, x):
        x = self.embed(x) + self.pos_enc[:x.size(1), :].to(x.device)
        for layer in self.layers:
            x = layer(x)
        return self.out(x)

🧪 Training with Dummy Data

model = SimpleTransformer(vocab_size=10000)
dummy_input = torch.randint(0, 10000, (2, 20))  # (batch, seq_len)
output = model(dummy_input)
print(output.shape)  # torch.Size([2, 20, 10000])

📦 Bonus: Use Hugging Face Transformers in 2 Lines

from transformers import AutoTokenizer, AutoModelForSequenceClassification
model = AutoModelForSequenceClassification.from_pretrained("bert-base-uncased")
tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")

🧩 Final Thoughts

Understanding Transformers from the ground up gives you a deep appreciation of the architecture behind ChatGPT, BERT, and many other models. You don’t need to reinvent the wheel every time, but knowing what’s under the hood helps with:

• Fine-tuning models
• Debugging
• Building custom variations for research or production

🔍 Resources

• Attention Is All You Need (original paper)
• Annotated Transformer (Harvard NLP)
• The Illustrated Transformer

Conclusion

Transformers have fundamentally changed how machines understand and generate human language. With constant innovation, they continue to unlock new possibilities in AI. Whether you’re building a conversational agent or analyzing financial documents, Transformer models offer the tools to tackle complex language tasks at scale.