Embedding Dimensions

Modern models use different embedding sizes:

| Model | Embedding dimensions | |-------|---------------------| | Word2Vec | 100–300 | | BERT | 768 | | GPT-3 | 12,288 | | OpenAI text-embedding-3-large | 3,072 |

More dimensions capture finer distinctions but require more memory and compute. Think of it like describing a person: 3 dimensions (height, weight, age) give a rough sketch; 768 dimensions paint a detailed portrait.

From Words to Sentences

Word embeddings represent individual words, but we often need to compare entire sentences or documents. Sentence embeddings (from models like Sentence-BERT or OpenAI's embedding API) compress a whole passage into a single vector.

"How do I reset my password?" and "I forgot my login credentials" would have very similar sentence embeddings, even though they share almost no words. The embedding captures intent, not just vocabulary.

Measuring Similarity - Cosine Similarity

To compare two embeddings, we use cosine similarity - the cosine of the angle between two vectors. It ranges from −1 (opposite) to +1 (identical direction).

• "Happy" and "joyful": cosine ≈ 0.85 (very similar).
• "Happy" and "table": cosine ≈ 0.10 (unrelated).
• "Love" and "hate": cosine might be ≈ 0.40 (related but opposite).

Cosine similarity ignores vector magnitude, focusing purely on direction - which is where meaning lives.

Vector Databases - Search by Meaning

A vector database stores millions of embeddings and retrieves the most similar ones blazingly fast. Instead of keyword matching ("find documents containing 'machine learning'"), you search by meaning ("find documents about AI education").

Popular vector databases include:

• Pinecone - fully managed, scales effortlessly.
• Weaviate - open-source with hybrid search (vectors + keywords).
• ChromaDB - lightweight, great for prototyping.
• pgvector - adds vector search to PostgreSQL.

These databases use algorithms like HNSW (Hierarchical Navigable Small World) to search billions of vectors in milliseconds.

RAG - Retrieval-Augmented Generation

RAG is one of the most important patterns in modern AI. It combines vector search with language models:

1. Embed your documents and store them in a vector database.
2. When a user asks a question, embed the query.
3. Retrieve the most similar document chunks via vector search.
4. Feed those chunks to the language model as context.
5. The model generates an answer grounded in your data.

RAG lets language models answer questions about your specific data - company documents, product catalogues, research papers - without retraining. It dramatically reduces hallucination because the model has real sources to reference.

Practical Applications

Embeddings power countless real-world systems:

• Semantic search - find relevant results regardless of exact wording.
• Recommendations - "users who liked this also liked..." via embedding similarity.
• Clustering - group similar support tickets, reviews, or documents automatically.
• Anomaly detection - spot outliers that are far from any cluster.
• Duplicate detection - find near-identical content across large corpora.

Key Takeaways

• Embeddings are dense vector representations where meaning becomes geometry.
• Similar concepts cluster together; relationships appear as directions.
• Cosine similarity measures how close two meanings are.
• Vector databases enable search by meaning at massive scale.
• RAG combines vector search with language models to answer questions from your own data.

📚 Further Reading

• Jay Alammar - The Illustrated Word2Vec - Visual, intuitive walkthrough of how word embeddings work
• Pinecone Learning Centre - What Are Embeddings? - Practical guide to embeddings and vector search
• OpenAI Embeddings Guide - How to generate and use embeddings with the OpenAI API