Vectors, dot products, and cosine similarity: the geometry behind embeddings

Embeddings are one of the simplest ways to understand modern AI systems: represent information as numbers, then use geometry to compare those numbers.

A sentence, image, product, or piece of code can become a vector — a list of numbers describing patterns learned from data. The individual values usually do not have a human-readable meaning. The useful information appears in the relationships between vectors.

This lesson builds the mathematical foundation behind embeddings:

• vectors and dimensions

• vector spaces

• DotProduct calculations

• vector norms

• cosine similarity

• semantic search

• real sentence embeddings with Python

The core idea:

Similar concepts should produce vectors that are close together in an embedding space.

1. Vectors: turning information into coordinates

A vector is an ordered collection of numbers.

In mathematics, vectors can describe:

• positions in space

• directions

• physical quantities

In machine learning, vectors represent features learned from examples.

A product embedding might capture patterns related to:

• category

• style

• description

• user behavior

• related products

A sentence embedding attempts to capture meaning.

For example:

• “A dog is running through a park.”

• “A puppy is playing outside.”

The wording is different, but a good embedding model should place these sentences near each other because their meanings overlap.

In Python, a vector is simply an array:

import numpy as np

vector = np.array([2.0, 3.0, 1.0])

print(vector)

Output:

[2. 3. 1.]

Real embedding models usually create vectors with hundreds or thousands of dimensions. Humans cannot visualize these spaces directly, but computers can measure relationships inside them.

2. Vector spaces: where embeddings live

Imagine a map.

A physical location can be represented with two coordinates:

• latitude

• longitude

An embedding space works similarly, but the dimensions are learned from data instead of representing physical locations.

A simplified example:

          |
      A   |
----------+----------
          |    B
          |

Real embeddings exist in much larger spaces. A model may represent text using hundreds of dimensions, where each direction captures statistical patterns learned during training.

The important questions are:

• Which vectors are close together?

• Which vectors point in similar directions?

• Which relationships appear as movement through the space?

These questions are answered using Linear Algebra.

3. DotProduct: measuring vector alignment

The DotProduct combines two vectors into one number.

For vectors A and B:

A · B = A₁B₁ + A₂B₂ + A₃B₃ + … + AₙBₙ

The process is simple:

1. Multiply matching positions.

2. Add the results.

Example:

import numpy as np

a = np.array([1, 2, 3])
b = np.array([4, 5, 6])

result = np.dot(a, b)

print(result)

Output:

32

The calculation is:

(1 × 4) + (2 × 5) + (3 × 6) = 32

A larger positive dot product usually means vectors point in a similar direction. A negative value indicates opposite directions.

Neural networks rely heavily on this operation. Modern AI models perform enormous numbers of vector and matrix calculations during training and inference.

4. Vector norms: measuring size

The DotProduct has an important limitation: it depends on both direction and length.

Consider:

A = [1, 1]

B = [100, 100]

Both vectors point in the same direction, but B is much larger.

A vector’s length is called its norm.

The most common measurement is the L2 norm:

L2 norm = square root of the sum of every value squared

For example:

[3, 4]

length = √(3² + 4²)

length = 5

NumPy can calculate norms directly:

import numpy as np

vector = np.array([3, 4])

length = np.linalg.norm(vector)

print(length)

Output:

5.0

Norms allow us to compare vectors while accounting for their magnitude.

5. Cosine similarity: comparing direction

For embeddings, direction is often more useful than magnitude.

Cosine similarity measures the angle between two vectors.

The equation:

cosine similarity = (A · B) / (||A|| × ||B||)

The result is usually between:

• 1: vectors point in the same direction

• 0: vectors are unrelated

• -1: vectors point in opposite directions

A simple implementation:

import numpy as np

def cosine_similarity(a, b):
    return np.dot(a, b) / (
        np.linalg.norm(a) * np.linalg.norm(b)
    )


first = np.array([1, 2, 3])
second = np.array([2, 4, 6])

score = cosine_similarity(first, second)

print(score)

Output:

1.0

The vectors have different lengths but exactly the same direction.

Cosine similarity powers many applications:

• semantic search

• recommendation systems

• document retrieval

• clustering

• duplicate detection

6. Building a tiny semantic search engine

Traditional keyword search asks:

Do these documents contain the same words?

Embedding search asks:

Are these documents about similar concepts?

The workflow:

1. Convert documents into embeddings.

2. Convert a user query into an embedding.

3. Calculate similarity scores.

4. Return the closest matches.

A small example with artificial vectors:

import numpy as np

documents = {
    "deep learning guide": np.array([0.8, 0.2, 0.1]),
    "cooking recipes": np.array([-0.3, 0.9, 0.4]),
    "neural network tutorial": np.array([0.7, 0.3, 0.2]),
}


def cosine_similarity(a, b):
    return np.dot(a, b) / (
        np.linalg.norm(a) * np.linalg.norm(b)
    )


query = np.array([0.75, 0.25, 0.15])

results = []

for name, embedding in documents.items():
    score = cosine_similarity(query, embedding)
    results.append((name, score))

results.sort(
    key=lambda item: item[1],
    reverse=True,
)

for name, score in results:
    print(name, round(score, 3))

The example ranks the closest vectors first.

Real systems replace the artificial vectors with embeddings generated by machine learning models and often use vector indexes for fast nearest-neighbor search at large scale.

7. The surprising geometry: king − man + woman ≈ queen

One of the most famous discoveries in embedding research came from word-vector models.

Researchers found that some relationships appeared as directions in vector space:

king − man + woman ≈ queen

The surprising part was not that a model contained a programmed dictionary rule. Nobody manually told the system this equation.

Instead, training on large text collections caused the model to discover statistical patterns:

• words appearing in similar contexts develop similar locations

• relationships between words can appear as consistent directions

The model learned that the relationship between “king” and “man” was similar to the relationship between “queen” and “woman.”

Modern transformer-based embeddings are much more powerful than early word vectors, but the underlying lesson remains:

Meaning can emerge as geometry.

8. From word vectors to sentence embeddings

Early embedding research focused heavily on individual words.

Modern AI applications usually need larger concepts:

• sentences

• paragraphs

• documents

• images

• source code

A sentence embedding represents the meaning of a complete text.

For example:

• “The company released a new phone.”

• “A smartphone was launched by the company.”

A strong embedding model should understand that these sentences describe a similar event even though they use different words.

This ability powers retrieval systems used in search, AI assistants, and retrieval-augmented generation (RAG).

9. Real sentence embeddings with Python

For a production embedding workflow, you need:

• an embedding model

• a client library

• similarity calculations

The following example uses the current OpenAI Python SDK style with the embeddings API and the supported text-embedding-3-large model.

Install dependencies:

pip install openai numpy

Generate embeddings:

import os

import numpy as np
from openai import OpenAI


client = OpenAI(
    api_key=os.environ["OPENAI_API_KEY"]
)

sentences = [
    "A dog is playing in the garden.",
    "A puppy is running outside.",
    "A person is cooking dinner.",
]


response = client.embeddings.create(
    model="text-embedding-3-large",
    input=sentences,
)


embeddings = [
    np.array(item.embedding)
    for item in response.data
]


def cosine_similarity(a, b):
    return np.dot(a, b) / (
        np.linalg.norm(a) * np.linalg.norm(b)
    )


dog_similarity = cosine_similarity(
    embeddings[0],
    embeddings[1],
)

unrelated_similarity = cosine_similarity(
    embeddings[0],
    embeddings[2],
)


print("Dog and puppy:", dog_similarity)
print("Dog and cooking:", unrelated_similarity)

The exact numbers depend on the model and input, but the expected pattern is:

• the dog and puppy sentences should have higher similarity

• the dog and cooking sentences should have lower similarity

That simple comparison is the foundation behind semantic search.

10. Why simple geometry powers modern AI

Dot products, norms, and cosine similarity are basic mathematical tools.

Their impact comes from combining them with:

• large neural networks

• massive training datasets

• specialized hardware

• efficient nearest-neighbor search algorithms

These ideas power:

• retrieval-augmented generation

• recommendation systems

• image search

• document discovery

• AI assistants

Whenever an AI system finds information related to a question, vector geometry is often part of the process.

The mathematics is simple.

The scale is what makes it powerful.

Your next step

You now understand the building blocks behind embeddings:

• vectors

• Linear Algebra

• DotProduct operations

• vector norms

• cosine similarity

Continue by creating a small semantic search project:

• embed your own documents

• compare vectors with cosine similarity

• experiment with different queries

• inspect how changing data changes the geometry

Build something with embeddings and explore what the numbers reveal about meaning.