Just asked Gemini to create an image for this blog

Vector Databases: What They Do and Why They Matter

Vector DB: The General Definition

A vector database is a specialized kind of database designed to store, index, and query vector embeddings with high computational efficiency. Vector databases organize and optimize high-dimensional semantic data, enabling fast querying capabilities.

The primary goal of a vector database is to find vector embeddings most similar to a given query.

What are Vectors?

Vectors are arrays of numbers (e.g., [0.23, -0.45, 0.67, …]). Each number in the vector represents a particular point(dimension) in a high-dimensional space, and collectively, the vector represents the characteristics of the data in that space.

A point (dimension) in a high-dimensional space is like one specific feature or characteristic of the data. Here is an example:

Imagine describing a cup of coffee ☕:

• One dimension could represent its temperature (e.g., hot, warm, or cold).
• Another dimension might represent its bitterness (e.g., mild, medium, or strong).
• A third dimension could represent its sweetness (e.g., unsweetened, slightly sweet, or very sweet).

When you combine these dimensions, you can describe a specific type of coffee:

• A hot, mildly bitter, slightly sweet coffee might be represented as [0.9, 0.3, 0.5].
• A cold, very bitter, unsweetened coffee might be represented as [0.1, 0.9, 0.0].

How are these embeddings created?

To create embeddings, we use machine learning models that transform raw data (such as text, images, or audio) into high-dimensional vectors. These models are trained on large datasets to capture the underlying patterns and semantic meanings of the data, representing them as vectors.

For example:

• Text: A sentence like “I love cakes” might be converted into a vector such as [0.5, -0.3, 0.7] using language models like BERT or Word2Vec.
• Images: An image of a cat could be encoded as [1.2, 3.5, 2.1,1.7] using computer vision models like ResNet or CLIP.

Indexing

Before understanding how a vector database works in an application, it’s essential to understand the concept of indexing.

Indexing is an important step in vector DBs that allows efficient search for vectors that are most similar to the input query vector.
Vectors often have hundreds or thousands of dimensions, which makes finding the closest match computationally expensive. Real-world applications involve searching massive datasets with millions or billions of vectors.

Indexing optimizes the search process, reducing the time it takes to find similar vectors, even in large, high-dimensional spaces.

Indexing Techniques

Vector databases use specialized Approximate Nearest Neighbor search algorithms to efficiently find vectors that are closest to the query vector in terms of similarity(cosine similarity, Euclidean distance, etc.)

• HNSW (Hierarchical Navigable Small World Graphs)
  HNSW builds a graph-based index where vectors are represented as nodes connected by edges based on their similarity, and the search begins from an entry point in the graph, progressing toward the query vector by following the most similar neighbors.
• FAISS (Facebook AI Similarity Search)
  FAISS is a library developed by Meta for fast and efficient nearest neighbor searches. It achieves high speeds by combining techniques like clustering, quantization, and optimized processing on CPUs and GPUs. Efficient in both exact and approximate search methods.
• Annoy (Approximate Nearest Neighbors Oh Yeah) 😂
  Annoy is a lightweight, tree-based algorithm designed for fast and approximate searches. It divides the vector space into subsets, making queries quick and efficient. It is ideal for smaller datasets or applications where ease of implementation is important.

How Vector Databases Fit Into the AI Workflow

Source: Writer — Vector Database vs. Graph Database

Let’s understand how vector databases fit into the AI workflow. The following diagram illustrates a typical pipeline where raw content is transformed into vector embeddings and stored in a vector database, enabling applications to efficiently retrieve and utilize relevant information.

Here’s a step-by-step breakdown of the process:

1. Content: This is the raw data (text, images, or any other input) that is processed for further use in applications.
2. Embedding Model: The raw content is passed through an embedding model, such as BERT (for text) or ResNet (for images). These models convert data into high-dimensional vector embeddings. These embeddings numerically represent the semantic meaning of the content.
3. Vector Embedding: The embeddings are structured arrays of numbers (vectors) that map the data into a high-dimensional space. These embeddings capture the semantic meaning of the data, encoding its context, relationships, and key characteristics into numerical form.
4. Vector Database: The vector embeddings are stored in the vector database. The database allows for efficient querying of the most similar embeddings using algorithms like HNSW, FAISS, etc. Indexing techniques are employed to organize and structure the data, enabling faster similarity searches.
5. Application: The application layer interacts with the vector database to retrieve relevant data based on similarity. For instance, in a recommendation system, the application might fetch products or content most similar to a user’s preferences.

Vector databases have emerged as a critical component of AI workflows, providing an efficient means to store, index, and query vector embeddings. By integrating advanced indexing techniques and embedding models, vector databases empower applications to handle vast amounts of data. Whether you’re building a semantic search engine, a recommendation system, or a content retrieval application, vector databases offer unparalleled speed and scalability.

Hope you find this guide helpful!