Understanding Embeddings
Learning Objectives
By the end of this session, you will be able to:
Understand what embeddings are
Learn why embeddings are important in AI systems
Understand how text is converted into vectors
Learn how semantic similarity works
Understand embeddings in RAG systems
Explore real-world embedding use cases
Build a strong foundation for vector databases and semantic search
Introduction
In the previous module, we learned how RAG systems retrieve information from large collections of documents.
One important question remains:
How does a computer know that:
"Vacation Policy"
and
"Annual Leave Policy"
are related?
Traditional keyword search struggles with this problem.
For example:
Search:
Vacation
Document:
Annual Leave
Although humans understand these phrases are closely related, a traditional keyword search engine may not.
Modern AI systems solve this challenge using:
Embeddings
Embeddings are one of the most important concepts in Generative AI, RAG, semantic search, recommendation systems, and vector databases.
Without embeddings, modern RAG systems would not exist.
Why This Topic Matters
Imagine a student asks:
How can I apply for a scholarship?
The university document contains:
Financial Assistance Program
There is no exact keyword match.
Traditional search may fail.
However, embeddings recognize that:
Scholarship
˜
Financial Assistance
This allows the system to retrieve relevant information even when exact words differ.
This capability is the foundation of semantic search.
What Is an Embedding?
An embedding is a numerical representation of information that captures its meaning.
Think of an embedding as:
Text
?
Mathematical Representation
Example:
Text:
Artificial Intelligence
Embedding:
[0.24, 0.81, -0.17, 0.56, ...]
The numbers themselves are not important.
What matters is that similar meanings produce similar vectors.
Simple Analogy
Imagine a city map.
Locations that are close together are related geographically.
Example:
Delhi
Noida
Gurugram
are relatively close.
Similarly, embeddings place related concepts close together in a mathematical space.
Example:
Car
Vehicle
Automobile
will appear near each other.
Whereas:
Car
Banana
will be far apart.
Why Traditional Search Is Limited
Traditional search relies heavily on keywords.
Example:
Document:
Employees receive annual leave benefits.
Search:
Vacation policy
Traditional keyword search:
No Match
because the exact word "vacation" does not appear.
Humans understand:
Vacation
=
Annual Leave
Keyword systems often do not.
This is where embeddings become valuable.
Semantic Search vs Keyword Search
Keyword Search
Looks for exact words.
Example:
Vacation
Matches:
Vacation Policy
May miss:
Annual Leave Policy
Semantic Search
Looks for meaning.
Example:
Vacation
Can retrieve:
Annual Leave Policy
Paid Time Off Policy
Leave Guidelines
This significantly improves retrieval quality.
How Embeddings Work
At a high level:
Text
?
Embedding Model
?
Vector
The embedding model converts text into a list of numbers.
Example:
Remote Work Policy
might become:
[0.15, 0.84, -0.22, ...]
Another phrase:
Work From Home Policy
might become:
[0.17, 0.82, -0.19, ...]
The vectors are very similar because the meanings are similar.
Understanding Vector Space
Embeddings exist inside something called a vector space.
Think of this space as a giant map.
Example:
Animal
+- Dog
+- Cat
+- Lion
Vehicle
+- Car
+- Bus
+- Truck
Related concepts cluster together.
In reality, embedding spaces may contain:
Hundreds
Thousands
or
Thousands of Dimensions
Humans cannot visualize them directly, but computers can calculate distances efficiently.
Similarity in Embeddings
The key idea:
Similar Meaning
?
Similar Vectors
Example:
Phrase A:
Machine Learning
Phrase B:
Artificial Intelligence
Distance:
Small
Phrase A:
Machine Learning
Phrase B:
Pizza Recipe
Distance:
Large
This distance helps determine relevance.
Visual Representation
Imagine:
AI
?
Machine Learning
?
Deep Learning
?
Banana
?
Football
?
Related concepts appear close together.
Unrelated concepts appear far apart.
This is the basic principle behind embeddings.
How Embeddings Are Generated
Modern embedding models are trained on massive datasets.
Examples:
Books
Websites
Articles
Documentation
The model learns:
Language patterns
Context relationships
Semantic meaning
When text is provided, the model generates a vector that represents meaning rather than exact words.
Embedding Workflow
Text
?
Embedding Model
?
Vector
?
Vector Database
This process occurs during data ingestion.
Example
Document:
Employees receive 24 annual leave days.
Embedding:
[0.45, -0.22, 0.91, ...]
Stored in:
Vector Database
Later, when a user asks:
How much vacation time do employees get?
The question is also converted into an embedding.
The vectors are compared.
The document is retrieved because the meanings are similar.
Why Embeddings Matter in RAG
Recall the RAG workflow:
Documents
?
Embeddings
?
Vector Database
?
Search
?
Retrieved Context
?
LLM
Without embeddings:
Keyword Matching Only
With embeddings:
Meaning-Based Retrieval
This dramatically improves answer quality.
Embeddings and Similarity Search
Suppose we store:
Scholarship Information
Admission Requirements
Hostel Rules
A student asks:
Financial aid opportunities
Keyword search may fail.
Embeddings understand:
Financial Aid
˜
Scholarship
The correct information is retrieved.
This capability powers semantic search.
Real-World Example: E-Commerce
Customer searches:
Running Shoes
Product description:
Athletic Footwear
Keyword search:
May Miss Result
Embedding search:
Finds Product
because meanings are related.
This improves user experience.
Real-World Example: Customer Support
Customer asks:
How do I reset my password?
Knowledge base contains:
Account Credential Recovery
Embeddings identify the relationship and retrieve the relevant article.
Embeddings in Recommendation Systems
Streaming platforms often use embeddings.
Example:
User watches:
Science Fiction Movies
Embedding systems identify similar content and recommend:
Space Adventure Movies
Future Technology Movies
rather than relying only on exact categories.
Types of Data That Can Be Embedded
Embeddings are not limited to text.
Modern AI systems create embeddings for:
Text
Examples:
Documents
Emails
Articles
Images
Examples:
Photographs
Product Images
Audio
Examples:
Speech
Music
Video
Examples:
Educational videos
Tutorials
This enables multimodal AI systems.
Embedding Dimensions
Embeddings contain multiple numerical values.
Example:
128 Dimensions
512 Dimensions
1024 Dimensions
1536 Dimensions
3072 Dimensions
Generally:
More dimensions can capture more information
More dimensions require more storage
The optimal choice depends on the use case.
Embedding Models
Popular embedding providers include:
OpenAI
Commonly used in RAG systems.
Provides embedding capabilities for AI applications.
Cohere
Known for retrieval-focused embeddings.
Open-Source Models
Examples:
BGE
E5
Instructor
GTE
Organizations choose models based on:
Performance
Cost
Infrastructure requirements
Challenges with Embeddings
Storage Requirements
Millions of embeddings require significant storage.
Search Performance
Large vector collections require optimized retrieval systems.
Model Selection
Different models perform differently.
Data Quality
Poor input data produces poor embeddings.
Embedding quality directly affects retrieval quality.
Embeddings in a Production RAG System
Architecture:
Documents
?
Chunking
?
Embeddings
?
Vector Database
User Query
?
Embedding
?
Similarity Search
?
Relevant Chunks
?
LLM
?
Answer
Embeddings act as the bridge between documents and retrieval.
Real-World Enterprise Example
Company documents:
HR Policies
Benefits Guide
Travel Rules
Security Procedures
Employee asks:
Can I work remotely?
The system:
Convert Question to Embedding
?
Find Similar Policy Chunks
?
Retrieve Context
?
Generate Answer
The answer is grounded in actual company policies.
Why Embeddings Are Revolutionary
Before embeddings:
Search
?
Keywords
?
Limited Results
After embeddings:
Search
?
Meaning
?
Relevant Results
This shift enabled:
Semantic search
Modern RAG systems
Intelligent recommendation systems
Enterprise knowledge assistants
.NET Perspective
Popular .NET technologies include:
Semantic Kernel
Azure OpenAI Embeddings
Azure AI Search
Enterprise systems often use embeddings to power internal search and knowledge retrieval applications.
Python Perspective
Popular Python tools include:
LangChain
LlamaIndex
Sentence Transformers
OpenAI SDK
Hugging Face Transformers
Python remains the most popular ecosystem for embedding experimentation and development.
Assignment
Practical Exercise
Choose 10 sentences about different topics.
Group them by meaning.
Observe how humans naturally identify semantic similarity.
Explain how embeddings would perform a similar task.
Research Activity
Compare three embedding models and identify:
Features
Advantages
Limitations
Ideal use cases
Key Takeaways
Embeddings convert meaning into numerical vectors.
Similar concepts produce similar vectors.
Embeddings enable semantic search.
Modern RAG systems rely heavily on embeddings.
Embeddings allow retrieval based on meaning rather than exact keywords.
They are foundational to vector databases and retrieval systems.
Understanding embeddings is essential for building advanced AI applications.
What's Next?
In Session 20, we will explore:
Creating Embeddings Using Modern Models
You will learn how embeddings are generated in practice, compare popular embedding models, understand embedding dimensions, evaluate embedding quality, and create embeddings using modern AI platforms and frameworks.