Tokens, Context Windows, and Embeddings
Learning Objectives
By the end of this session, you will be able to:
Understand what tokens are
Learn how AI models process text using tokens
Understand context windows and their importance
Learn the limitations of context windows
Understand what embeddings are
Learn how embeddings convert language into numbers
Discover why embeddings are the foundation of modern RAG systems
Understand how semantic similarity works
Introduction
In previous sessions, we learned about Large Language Models (LLMs) and the Transformer architecture that powers them.
Now we will explore three fundamental concepts that every AI engineer must understand:
Tokens
Context Windows
Embeddings
These concepts are the foundation of modern AI systems.
Whether you are building:
Chatbots
AI Agents
RAG Applications
Enterprise Knowledge Systems
Semantic Search Engines
you will constantly work with tokens, context management, and embeddings.
Many AI development challenges can be traced back to these three concepts.
Understanding them now will make advanced topics much easier later in this series.
Why This Topic Matters
Imagine reading a book.
To understand the story, you must:
Break the text into readable pieces.
Remember what you previously read.
Understand the meaning of words and relationships.
AI models do something similar.
They:
Convert text into tokens.
Use context windows to remember information.
Convert language into embeddings for understanding meaning.
Without these mechanisms, modern AI systems would not be possible.
What Are Tokens?
A token is the smallest unit of text processed by an AI model.
Many beginners assume AI models read words directly.
They do not.
Instead, models process tokens.
Example
Sentence:
Artificial Intelligence is transforming software development.
Possible tokenization:
Artificial
Intelligence
is
transforming
software
development
.
Each piece becomes a token.
Depending on the model, a token may represent:
A complete word
Part of a word
A punctuation mark
A number
A symbol
Why Tokenization Is Necessary
Computers cannot understand language directly.
They understand numbers.
Before processing text, an AI model must convert language into a machine-readable format.
The process looks like:
Text
?
Tokenization
?
Token IDs
?
Model Processing
Example
Text:
Hello World
Might become:
[15496, 2159]
These numbers represent tokens inside the model.
Word Count vs Token Count
A common misconception is that words and tokens are the same.
They are not.
Example:
Artificial Intelligence
May become:
Artificial
Intelligence
Two words and two tokens.
However:
unbelievable
Could become:
un
believ
able
One word but multiple tokens.
This is why AI pricing and limits are usually measured in tokens rather than words.
Real-World Example
Suppose a company uploads a 100-page document into an AI system.
The AI does not process:
100 pages
It processes:
Thousands of tokens
Understanding token counts becomes important when designing AI applications.
What Is a Context Window?
A context window refers to the amount of information an AI model can consider at one time.
Think of it as the model's working memory.
Human Example
When reading a paragraph, you remember previous sentences.
That memory helps you understand the current sentence.
AI models behave similarly.
The context window stores information that the model can reference while generating responses.
Context Window Visualization
Conversation History
?
Current Question
?
Context Window
?
Generated Response
The larger the context window, the more information the model can consider.
Example of Context
User:
My favorite programming language is C#.
Later:
Why do I like it?
The model understands:
it = C#
because that information exists within the context window.
Without context, the model would not know what "it" refers to.
Why Context Windows Matter
Modern applications often involve:
Long conversations
Large documents
Research papers
Enterprise knowledge bases
The model must remember relevant information.
A larger context window allows:
Better document understanding
More accurate responses
Longer conversations
Improved reasoning
Context Window Limitations
Context windows are not unlimited.
Every model has a maximum limit.
Example:
Model Context Limit
?
128K Tokens
Once the limit is exceeded:
Older information may be removed
Important context may be lost
Response quality may decrease
This challenge becomes particularly important when working with large documents.
Later in the series, we will learn how RAG solves this problem.
Context Window Example
Imagine asking:
Summarize a 1,000-page book.
The entire book may not fit inside the context window.
Possible solution:
Book
?
Chunking
?
Retrieval
?
Relevant Sections
?
LLM
This approach forms the foundation of Retrieval-Augmented Generation (RAG).
What Are Embeddings?
Embeddings are one of the most important concepts in modern AI.
An embedding is a numerical representation of data that captures meaning and relationships.
Simply put:
Embeddings allow computers to understand similarity between concepts.
Why Embeddings Are Needed
Consider these sentences:
I love programming.
I enjoy coding.
Humans recognize that these sentences have similar meanings.
A computer sees different words.
Embeddings help AI recognize semantic similarity.
Embedding Transformation
Text
?
Embedding Model
?
Vector Representation
Example:
I love programming
Might become:
[0.12, 0.87, 0.44, ...]
This vector contains hundreds or thousands of numerical values.
These values capture meaning.
Simplified Embedding Example
Imagine a three-dimensional space.
Programming
*
/
/
Coding *
These concepts appear close together because they are semantically related.
Meanwhile:
Programming
*
*
Banana
The concepts are far apart because they have little relationship.
This distance measurement allows AI systems to find relevant information.
How Embeddings Understand Meaning
Embeddings learn relationships from massive datasets.
Example:
Doctor
Nurse
Hospital
Medicine
These words frequently appear together.
The embedding model learns that they are related concepts.
Similarly:
Football
Player
Coach
Stadium
form another related group.
The resulting vector space captures these relationships mathematically.
Embeddings and Similarity Search
One of the most powerful uses of embeddings is similarity search.
Example question:
How do I build a web API?
The system converts the question into an embedding.
Then it searches for documents with similar embeddings.
Result:
ASP.NET Core API Guide
instead of matching only exact keywords.
This is called semantic search.
Architecture of Embedding-Based Search
User Question
?
Embedding Model
?
Vector
?
Vector Database
?
Similar Vectors
?
Relevant Documents
This process is at the heart of modern RAG systems.
Real-World Applications of Embeddings
Search Engines
Understand intent instead of exact keywords.
Recommendation Systems
Recommend relevant products or content.
Chatbots
Retrieve relevant knowledge.
Enterprise Knowledge Bases
Search internal documentation intelligently.
Document Retrieval
Find relevant information across thousands of documents.
AI Agents
Provide contextual information to LLMs.
Why Embeddings Matter for RAG
RAG systems rely heavily on embeddings.
Workflow:
Documents
?
Embeddings
?
Vector Database
?
User Question
?
Similarity Search
?
Relevant Context
?
LLM Response
Without embeddings, modern RAG would not be possible.
This is why understanding embeddings is critical for every AI engineer.
Comparing Tokens, Context Windows, and Embeddings
| Concept | Purpose |
|---|---|
| Tokens | Break text into processable units |
| Context Window | Store information currently available to the model |
| Embeddings | Represent meaning as numerical vectors |
Together, these three concepts form the foundation of modern AI systems.
.NET Perspective
.NET developers frequently work with embeddings when building:
Enterprise search systems
AI assistants
Knowledge management platforms
RAG applications
Popular technologies include:
Semantic Kernel
Azure AI Search
Azure OpenAI Embeddings
Vector search integrations
Understanding embeddings is becoming an essential skill for modern .NET developers.
Python Perspective
Python provides extensive support for embeddings.
Popular tools include:
OpenAI SDK
Sentence Transformers
LangChain
LlamaIndex
ChromaDB
Pinecone
Weaviate
Example:
from openai import OpenAI
client = OpenAI()
embedding = client.embeddings.create(
model="text-embedding-3-small",
input="Artificial Intelligence"
)
print(embedding.data[0].embedding)
This generates a vector representation of the input text.
Assignment
Practical Exercise
Use an AI chatbot and test:
Short prompt
Long prompt
Multi-step conversation
Observe how context affects responses.
Research Activity
Compare:
Tokens
Context Windows
Embeddings
Create a one-page summary explaining how they work together in modern AI systems.
Key Takeaways
AI models process tokens rather than words.
Tokens are converted into numerical representations.
Context windows act as the model's working memory.
Context limitations create challenges for large documents.
Embeddings convert meaning into vectors.
Similar concepts produce similar embeddings.
Embeddings power semantic search and RAG systems.
Tokens, context windows, and embeddings are fundamental building blocks of modern AI.
What's Next?
In Session 6, we will explore:
Prompt Engineering Fundamentals
You will learn how to communicate effectively with Large Language Models, design high-quality prompts, improve response quality, and build reliable AI-powered applications.