What is Milvus?
Milvus is an open-source, cloud-native vector database designed for storing, indexing, and querying high-dimensional embedding vectors at massive scale. Originally developed by Zilliz and donated to the LF AI & Data Foundation in 2019, it has become one of the de-facto standards for production-grade vector search infrastructure.
Unlike traditional databases optimized for structured, tabular data, Milvus is purpose-built for the fundamental operation of AI workloads: finding semantically similar items among billions of high-dimensional vectors — in milliseconds.
The Role of Vector Databases in Modern AI
The AI stack has fundamentally shifted. Large Language Models (LLMs) like GPT-4, Llama, and Claude encode knowledge into dense vector representations. The challenge: these models have fixed context windows and static training data. Vector databases solve the knowledge retrieval problem.
LLM Augmentation (RAG)
Inject fresh, domain-specific knowledge into LLMs at inference time without fine-tuning. Retrieve the top-k most relevant chunks, pass as context.
Semantic Search
Go beyond keyword matching. "Running shoes for flat feet" matches "athletic footwear for overpronation" via embedding similarity.
Multi-modal AI
Unify text, image, audio, and video in a shared embedding space. Search across modalities (text query → similar images).
Why Vector Search Matters: The Embedding Pipeline
Text/Image/Audio
BERT/CLIP/Ada
[0.21, -0.87, ...]
HNSW/IVF
Nearest Neighbors
Vectors, Indexes & Collections
Vectors & Embeddings
An embedding is a function that maps high-dimensional, complex data to a point in a lower-dimensional continuous vector space, such that semantically similar inputs map to geometrically nearby points. Common embedding models:
| Model | Modality | Dimensions | Use Case |
|---|---|---|---|
| text-embedding-3-large (OpenAI) | Text | 3072 | RAG, semantic search |
| all-MiniLM-L6-v2 (Sentence-Transformers) | Text | 384 | Lightweight similarity |
| CLIP (OpenAI) | Image + Text | 512 / 768 | Multi-modal search |
| Nomic-embed-vision | Image | 768 | Image similarity |
| BGE-M3 | Text (multi-lingual) | 1024 | Cross-lingual RAG |
Similarity Metrics
L2 Distance
d(a,b) = √Σ(aᵢ - bᵢ)²Measures straight-line distance in vector space. Best for image embeddings, normalized float vectors. Sensitive to vector magnitude.
Cosine Similarity
cos(θ) = (a·b) / (|a||b|)Measures angle between vectors regardless of magnitude. Best for text embeddings. Range [-1, 1]. Standard for NLP tasks.
Inner Product (IP)
IP(a,b) = Σ(aᵢ × bᵢ)Dot product of two vectors. Equivalent to cosine on unit-normalized vectors. Used in recommendation systems and MIPS problems.
Approximate Nearest Neighbor (ANN)
Exact nearest neighbor search over billions of vectors is O(n·d) per query — computationally infeasible. ANN algorithms trade a small accuracy loss (recall) for dramatically faster search times through pre-computed indexes.
nprobe/ef = higher recall (closer to exact) but more latency and CPU. Milvus exposes these params per-query, enabling dynamic trade-off at runtime.
Index Types in Milvus
| Index Type | Algorithm | Best For | Memory | Build Speed | Query Speed |
|---|---|---|---|---|---|
| FLAT | Brute-force | Small datasets (<1M), 100% recall | High | Instant | Slow |
| IVF_FLAT | Inverted File | Balanced recall/speed, medium scale | Medium | Medium | Fast |
| IVF_SQ8 | IVF + Scalar Quantization | Reduced memory, slight accuracy loss | Low | Medium | Fast |
| IVF_PQ | IVF + Product Quantization | Very large scale, aggressive compression | Very Low | Slow | Medium |
| HNSW | Hierarchical NSW Graph | High recall + speed, default choice | High | Slow | Very Fast |
| SCANN | Anisotropic quantization | High-dimensional, high throughput | Medium | Medium | Very Fast |
| DiskANN | Graph on NVMe SSD | Billion-scale, memory-constrained | Very Low (disk) | Slow | Moderate |
| GPU_CAGRA | GPU graph-based | GPU-accelerated, ultra-low latency | GPU VRAM | Very Fast | Extremely Fast |
| SPARSE_INVERTED_INDEX | Sparse IVF | BM25-style sparse vectors | Low | Fast | Fast |
Collections, Partitions & Segments
Collection
The top-level logical grouping. Equivalent to a table in RDBMS. Defines the schema: vector field(s), dimension, metric type, and scalar fields. A collection can hold billions of entities.
Partition
Logical sub-division within a collection (e.g., by date, category, tenant). Allows scoped searches to reduce scan space. A collection has a default partition; you can create up to 4096 named partitions.
Segment
The physical storage unit. Each partition is split into segments of configurable size (default 512MB). Segments progress through a lifecycle: Growing → Sealed → Indexed → Compacted.
# Python SDK — Creating a collection with schema from pymilvus import MilvusClient, DataType client = MilvusClient("http://localhost:19530") schema = client.create_schema(auto_id=True, enable_dynamic_field=True) schema.add_field("id", DataType.INT64, is_primary=True) schema.add_field("embedding", DataType.FLOAT_VECTOR, dim=1536) schema.add_field("content", DataType.VARCHAR, max_length=8192) schema.add_field("source", DataType.VARCHAR, max_length=256) schema.add_field("created_at",DataType.INT64) index_params = client.prepare_index_params() index_params.add_index( field_name="embedding", index_type="HNSW", metric_type="COSINE", params={"M": 16, "efConstruction": 256} ) client.create_collection( collection_name="rag_chunks", schema=schema, index_params=index_params )Python
Milvus 2.x Distributed Architecture
Milvus 2.x adopts a cloud-native, disaggregated architecture with complete separation of compute and storage. Every component is stateless (except storage backends) and scales independently. This design enables elastic horizontal scaling and zero-downtime upgrades.
┌─────────────────────────────────────────────────────────────────────────────────┐
│ MILVUS 2.x DISTRIBUTED ARCHITECTURE │
└─────────────────────────────────────────────────────────────────────────────────┘
CLIENT LAYER
┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐
│ Python │ │ Node.js │ │ Java │ │ REST │
│ SDK │ │ SDK │ │ SDK │ │ API │
└────┬─────┘ └────┬─────┘ └────┬─────┘ └────┬─────┘
└─────────────┴─────────────┴─────────────┘
│
════════════╪════════════
ACCESS LAYER (Stateless)
┌──────────────────────────────────────────────────┐
│ PROXY (×N) │
│ • Request routing & load balancing │
│ • Auth, rate limiting, schema validation │
│ • TSO (Timestamp Oracle) client │
│ • Result aggregation & re-ranking │
└──────────────────────────────────────────────────┘
│
┌───────────────────────┼────────────────────────┐
│ │ │
═════╪══════════ ══════════╪═══════════ ════════════╪══════
COORDINATOR LAYER │ │
┌────────────┐ ┌──────────────┐ ┌─────────────┐ ┌──────────────┐
│ Root Coord │ │ Query Coord │ │ Data Coord │ │ Index Coord │
│ │ │ │ │ │ │ │
│• Collection│ │• Query node │ │• Segment │ │• Index task │
│ lifecycle │ │ management │ │ assignment │ │ scheduling │
│• Schema │ │• Segment │ │• Flush │ │• Node health │
│ registry │ │ distribution│ │ management │ │ monitoring │
│• TSO │ │• Result merge│ │• Import │ │ │
└────────────┘ └──────────────┘ └─────────────┘ └──────────────┘
│ │ │ │
═════╪══════════════════╪═════════════════╪═════════════════╪════
WORKER LAYER
┌────────────────┐ ┌────────────────┐ ┌────────────────┐
│ QUERY NODES │ │ DATA NODES │ │ INDEX NODES │
│ (×N, scale) │ │ (×N, scale) │ │ (×N, scale) │
│ │ │ │ │ │
│• Load segments │ │• Receive DML │ │• Build indexes │
│• Execute ANN │ │• Write WAL to │ │• Store to │
│ vector search │ │ message queue │ │ object store │
│• Scalar filter │ │• Seal growing │ │• GPU accel │
│• Serve results │ │ segments │ │ (optional) │
│• Cache hot │ │• Flush to S3 │ │ │
│ segments │ │ │ │ │
└───────┬────────┘ └───────┬────────┘ └───────┬────────┘
│ │ │
════════╪══════════════════╪═══════════════════╪═════════
INFRASTRUCTURE LAYER
┌───────────────┐ ┌────────────────────┐ ┌───────────────┐
│ ETCD │ │ MESSAGE QUEUE │ │ OBJECT STORE │
│ │ │ Pulsar / Kafka │ │ MinIO / S3 │
│• Service disc.│ │ │ │ │
│• Metadata │ │• Write-Ahead Log │ │• Raw vectors │
│• Config store │ │• Binlog streaming │ │• Index files │
│• Coord leader │ │• Exactly-once dlv │ │• Delta logs │
│ election │ │• Replay on failure │ │• Snapshots │
└───────────────┘ └────────────────────┘ └───────────────┘
Component Deep Dive
Proxy
The single entry point for all client traffic. Stateless — can be scaled horizontally behind any load balancer. Responsibilities:
- Route DML (insert/delete) to Data Nodes via message queue
- Route DQL (search/query) to Query Nodes
- Validate schemas against etcd metadata
- Enforce Timestamp Oracle for consistent reads
- Aggregate and merge results from multiple Query Nodes
- Rate limiting and auth enforcement
Four Coordinators (Active-Standby HA)
- Root Coord: Manages collection/partition lifecycle, DDL ops, global timestamp service (TSO), and schema storage in etcd.
- Query Coord: Manages the "query cluster" — which segments live on which Query Nodes. Handles load balancing of segments across QNodes.
- Data Coord: Tracks segment lifecycle (growing → sealed). Triggers flushes. Monitors Data Node health. Manages binlog metadata.
- Index Coord: Assigns index-building tasks to Index Nodes. Tracks index build status. Ensures every sealed segment eventually gets indexed.
Search Execution Engine
Stateless workers that load vector segments into memory and execute ANN search. Key characteristics:
- Load sealed segments from object storage (S3/MinIO)
- Subscribe to message queue for growing segment data (streaming)
- Execute SIMD/GPU-accelerated vector similarity computations
- Apply scalar filters before/after vector search (pre/post filtering)
- Return partial results to Proxy for global top-K merge
- Cache frequently-accessed segments in memory
Ingestion & Persistence Engine
Consume the write-ahead log and persist data to object storage:
- Subscribe to Pulsar topics, consume DML (insert/delete) messages
- Buffer incoming vectors into growing segments (in-memory)
- When growing segment reaches capacity, seal it and flush to S3
- Write binlogs (vector data), delta logs (deletes), and stats logs
- Report segment metadata to Data Coordinator
- Stateless — can be replaced without data loss (log replay)
Offline Index Builder
Dedicated workers for CPU/GPU-intensive index construction:
- Receive index-build tasks from Index Coordinator
- Load raw vector data from object storage
- Build HNSW, IVF, DiskANN, GPU_CAGRA graphs
- Write finished index files back to object storage
- Can be GPU-equipped for accelerated CAGRA/IVF_GPU builds
- Scales independently — add more nodes during bulk ingestion
Object Store + Metadata + Messaging
- MinIO / S3: Stores raw vector binlogs, sealed segments, index files, delta logs, and checkpoints. Durable, replicated, infinitely scalable.
- etcd: Stores cluster metadata, collection schemas, segment info, coordinator leader state, service discovery records. Small, critical, backed up.
- Pulsar / Kafka: The WAL (write-ahead log). All DML flows through here. Enables exactly-once delivery, log replay on node failure, and pub-sub fan-out to multiple consumers.
Stateless Design & Separation of Compute/Storage
🔄 Control Flow
Coordinator services manage the cluster control plane via etcd. They make decisions (e.g., "seal this segment," "assign this index task to Node X") and write those decisions to etcd. Worker nodes watch etcd for task assignments and act accordingly.
📡 Data Flow
Actual vector data flows through Pulsar (write path) and S3 (persistence). The Proxy writes DML events to Pulsar. Data Nodes consume from Pulsar and flush to S3. Query Nodes read from S3 and serve searches. Data never passes through coordinator services.
Write Path, Read Path & Segment Lifecycle
Write Path (Ingestion Pipeline)
- Client calls insert() via SDK → hits Proxy over gRPC/REST.
- Proxy validates schema against etcd metadata, assigns row IDs if auto_id=True, gets a monotonic timestamp from Root Coord (TSO).
- Proxy publishes DML message (vector payload + timestamp) to a Pulsar topic partitioned by collection/shard.
- Data Node subscribes to the Pulsar topic. Receives messages and buffers them into a growing segment in memory.
- Growing segment reaches threshold (size limit or time window). Data Coord triggers a seal operation.
- Data Node flushes sealed segment to object storage: writes binlog files (raw vectors), stats log (min/max, bloom filter), and delta log (deletes). Notifies Data Coord.
- Data Coord notifies Index Coord. Index Coord schedules an index-building task and assigns it to an available Index Node.
- Index Node reads raw binlog from S3, builds the ANN index (HNSW graph, IVF clusters, etc.), writes index file back to S3.
- Segment becomes queryable. Query Coord loads it onto Query Nodes. Searches now include this segment.
WRITE PATH — INGESTION PIPELINE
Client
│
│ insert(vectors, metadata)
▼
┌─────────────┐
│ Proxy │ ── validates schema, assigns timestamps
└──────┬──────┘
│ publishes DML event
▼
┌──────────────────┐
│ Pulsar Topic │ ── WAL / durable message log
│ (per-shard) │
└──────┬───────────┘
│ subscribes & consumes
▼
┌─────────────┐
│ Data Node │ ── buffers in growing segment (RAM)
└──────┬──────┘
│ seal triggered → flush
▼
┌─────────────────────────────────┐
│ Object Store (S3) │
│ binlog/ ── raw vectors │
│ stats/ ── bloom filter, etc. │
│ delta/ ── deletes │
└──────────────┬──────────────────┘
│ index build task
▼
┌─────────────┐
│ Index Node │ ── builds HNSW/IVF from binlog
└──────┬──────┘
│ writes index file
▼
┌─────────────────────────────────┐
│ Object Store (S3) │
│ index/ ── HNSW graph file │
└──────────────────────────────────┘
│ Query Coord loads segment
▼
┌─────────────┐
│ Query Node │ ── segment ready for ANN search
└─────────────┘
Read Path (Search/Query Execution)
- Client calls search() with query vector, top-K, filter expression, and search params.
- Proxy receives request. Determines which shards/partitions to query based on collection metadata. Generates a "guaranteed timestamp" to ensure consistent reads.
- Proxy fans out the search request to all relevant Query Nodes in parallel (each holds different segments).
- Each Query Node executes locally: (a) applies scalar pre-filter if expr is provided, (b) runs ANN search on vector index in memory, (c) returns local top-K results with distances.
- Proxy collects partial results from all Query Nodes. Performs global merge/re-rank to produce final top-K.
- Proxy streams response back to client with entity IDs, distances, and any requested output fields.
# Read path — vector search with scalar filter results = client.search( collection_name="rag_chunks", data=[query_embedding], # [1536-dim float list] limit=10, # top-K output_fields=["content", "source"], filter='created_at > 1700000000 AND source == "docs"', search_params={ "metric_type": "COSINE", "params": {"ef": 200} # HNSW ef: higher = better recall }, consistency_level="Bounded" # eventual/bounded/strong ) for hit in results[0]: print(f"score={hit.distance:.4f} | {hit.entity['content'][:100]}")Python
Segment Lifecycle
In-memory buffer
on Data Node
Immutable, flushed
to S3 (raw)
ANN index built
stored in S3
On Query Node
serving searches
Merged small segs
deletes applied
Consistency Levels
| Level | Guarantee | Latency Impact | Use Case |
|---|---|---|---|
Strong | Read-your-writes. Waits for all writes to be visible. | High (+10–50ms) | Financial, compliance-critical |
Bounded | Reads data within a bounded staleness window (e.g., 5s). | Low–Medium | Most production workloads ✓ |
Session | Within a session, reads are monotonic. | Low | User-specific consistency |
Eventually | No guarantees. Fastest possible read. | Minimal | Analytics, batch workloads |
Performance & Scalability at Billion Scale
Horizontal Scaling Strategy
Every worker tier scales independently. This means you can tune resource allocation precisely to your workload profile:
| Component | When to Scale Out | Bottleneck Signal |
|---|---|---|
| Proxy | High inbound QPS, connection limits | CPU saturation, gRPC queue depth |
| Query Nodes | High search QPS, high latency | Search latency P99 > SLA, CPU > 80% |
| Data Nodes | High ingestion throughput, slow flushes | Pulsar lag, flush queue depth |
| Index Nodes | Slow index building during bulk load | Index build queue depth > threshold |
Hardware Acceleration
SIMD / AVX-512
Milvus uses CPU SIMD intrinsics (SSE4, AVX2, AVX-512) for vectorized distance computations. AVX-512 processes 16 float32 values per clock cycle. Auto-detected at runtime, no config needed.
GPU Acceleration
GPU_CAGRA (RAPIDS cuVS) and GPU_IVF_FLAT indexes run on NVIDIA GPUs. Index build is 10–100× faster than CPU. Search throughput increases dramatically for high-dim vectors.
DiskANN (NVMe)
For memory-constrained environments, DiskANN stores the graph on NVMe SSDs. Enables billion-scale search on commodity hardware with 10–20× less memory than HNSW.
Columnar Storage & Memory Management
Milvus stores data in a columnar format (Apache Arrow-compatible). Benefits for vector workloads:
- Load only the vector field for search (skip scalar fields) — reduces I/O from S3
- SIMD-friendly memory layout for batch distance computations
- Efficient compression per-column (quantization, delta encoding)
- Memory-mapped segment loading with OS page cache for hot segments
Quantization & Compression
| Technique | Memory Reduction | Recall Loss | Method |
|---|---|---|---|
| FP32 (raw) | 1× baseline | 0% | Full precision float |
| FP16 / BF16 | 2× | <0.1% | Half-precision float |
| INT8 (SQ8) | 4× | 0.5–1% | Scalar quantization |
| Product Quantization (PQ) | 8–32× | 2–5% | Sub-vector codebook |
| Binary Vectors | 32× | Depends on task | Hash-based encoding |
Benchmarks (Reference Figures)
Real-World Applications
RAG (Retrieval-Augmented Generation)
Pattern: Chunk documents → embed with OpenAI/BGE → store in Milvus. At query time: embed question → search Milvus for top-5 chunks → inject into LLM prompt.
Example: Enterprise knowledge base chatbot. 500k internal documents indexed. Employees query in natural language. Milvus returns relevant policy/process docs in <50ms. LLM synthesizes a grounded answer, eliminating hallucinations.
E-Commerce Semantic Search
Pattern: Embed product titles + descriptions. At search time: embed user query → ANN search → re-rank by business rules (price, inventory, margin).
Example: Fashion retailer with 50M SKUs. Query "casual summer dress for petite women" retrieves semantically matching products even if no keyword overlap. Hybrid search combines vector similarity + BM25 for optimal results. 25% CTR uplift vs keyword search.
Recommendation Systems
Pattern: Embed user interaction history (clicks, purchases) + item features into a shared space. Use MIPS (Maximum Inner Product Search) to find items closest to user embedding.
Example: Video streaming platform. 10M user embeddings + 50M content embeddings. Real-time personalized recommendations at login. Batch-update user embeddings daily. Candidate generation via Milvus → scoring/filtering → final recommendations.
Fraud Detection
Pattern: Embed transaction behavior (merchant category, amount distribution, geo, time) into feature vector. Real-time search for similar historical fraudulent transactions.
Example: Payment processor. Each transaction becomes a 256-dim behavioral embedding. Milvus searches 500M historical transactions in <5ms. If top-5 neighbors are flagged as fraud with high similarity, escalate for review. Catches novel fraud patterns that rule-based systems miss.
Image & Video Similarity
Pattern: Embed images via CLIP/ResNet/DINOv2. Store embeddings + metadata in Milvus. Query by image upload or text description (CLIP multi-modal).
Example: Stock photo agency. 500M images indexed. "Find photos similar to this mood board" — CLIP embeds the mood board image, Milvus returns 50 visually similar candidates in <30ms. Also supports text-to-image: "golden hour mountain sunset."
Drug Discovery / Bioinformatics
Pattern: Embed molecular fingerprints (ECFP, Morgan) or protein sequences (ESM-2). Search for structurally or functionally similar compounds at scale.
Example: Pharma R&D. 50M compound library. Researcher uploads a candidate molecule → Milvus retrieves 100 similar compounds from the library in <100ms. Dramatically accelerates hit-finding and scaffold-hopping. Used in combination with Tanimoto similarity via binary vector indexes.
Vector Database Landscape
Choosing a vector database is a system design decision driven by scale, operational model, team expertise, and budget. Here's an objective comparison of the major options.
| Dimension | Milvus | Pinecone | Weaviate | FAISS | pgvector |
|---|---|---|---|---|---|
| Type | Purpose-built DB | Managed SaaS | Purpose-built DB | Library | Extension |
| Open Source | ✓ Apache 2.0 | ✗ Proprietary | ✓ BSD-3 | ✓ MIT | ✓ PostgreSQL License |
| Deployment | Self-hosted / Zilliz Cloud | Fully managed (AWS/GCP) | Self-hosted / WCS Cloud | In-process library only | Any PostgreSQL host |
| Scalability | 🟢 Billion-scale, horizontal | 🟡 Large scale, serverless | 🟡 Multi-node cluster | 🔴 Single machine only | 🔴 PostgreSQL limits (~100M) |
| Max Vectors | 10B+ (tested) | ~1B+ (managed) | ~1B (self-hosted) | RAM/disk limited | ~100M practical |
| Hybrid Search | ✓ Dense + Sparse + Scalar | ⚠ Sparse + Dense | ✓ BM25 + vector | ✗ Vector only | ⚠ Vector + SQL (manual) |
| Multi-vector | ✓ ColBERT-style | ✓ (sparse) | ⚠ Named vectors | ✗ | ✗ |
| GPU Support | ✓ CAGRA, IVF_GPU | ✗ (managed) | ✗ | ✓ FAISS-GPU | ✗ |
| Index Types | IVF, HNSW, DiskANN, CAGRA, Sparse, Binary | Proprietary (HNSW-based) | HNSW | IVF, HNSW, PQ, LSH | IVFFlat, HNSW, IVFPQ |
| Persistence | ✓ S3 / MinIO | ✓ Managed | ✓ Self-managed disk | ✗ In-memory (ext. needed) | ✓ PostgreSQL storage |
| ACID Transactions | ⚠ Eventual (tunable) | ✗ | ✗ | ✗ | ✓ Full ACID (Postgres) |
| Multi-tenancy | ✓ Partitions, RBAC | ✓ Namespaces | ✓ Classes | ✗ | ⚠ Schema-level isolation |
| Operational Complexity | High (etcd, Pulsar, S3, 4 coordinators) | Zero (fully managed) | Medium | Zero (library) | Low (existing Postgres) |
| Cost Model | Self-host (infra cost) / Zilliz (usage-based) | Per-unit pricing (expensive at scale) | Self-host (free) / WCS (usage-based) | Free (compute cost only) | Free (Postgres infra cost) |
| Ecosystem | LangChain, LlamaIndex, Haystack, Spark | LangChain, LlamaIndex | LangChain, LlamaIndex | LangChain (low-level) | All Postgres tooling |
| Best For | Large-scale production, billion+ vectors | Fast start, managed ops, mid-scale | GraphQL API, semantic layer | Research, custom systems | Existing Postgres users, <10M vecs |
When to Use Milvus vs Alternatives
Decision Framework
Trade-offs Summary
🏢 Enterprise / Large Scale → Milvus
- Need to store and search 100M–10B+ vectors
- Require GPU-accelerated index builds
- Multi-tenancy with namespace isolation
- Hybrid search (sparse + dense + scalar filters)
- Data sovereignty / on-premises requirements
- Cost optimization at scale (vs. Pinecone per-unit pricing)
🚀 Startup / Prototype → Consider Alternatives
- Team has no Kubernetes/distributed systems expertise
- Dataset is < 5M vectors (pgvector is simpler)
- Need zero-ops infrastructure immediately
- Budget constraints favor managed SaaS until scale demands otherwise
- Already using PostgreSQL as primary datastore
Milvus in the AI Stack
Complete RAG Pipeline Architecture
┌─────────────────────────────────────────────────────────────────────────┐
│ PRODUCTION RAG SYSTEM ARCHITECTURE │
└─────────────────────────────────────────────────────────────────────────┘
┌────────────────────────────────────────────────────────────────────────┐
│ INGESTION PIPELINE (offline / async) │
│ │
│ Document Sources │
│ [PDFs, Web, DBs, APIs, Confluence, SharePoint, S3] │
│ │ │
│ ▼ │
│ ┌──────────────────┐ │
│ │ Document Parser │ ── PDF extract, HTML clean, Markdown parse │
│ └────────┬─────────┘ │
│ │ │
│ ▼ │
│ ┌──────────────────┐ │
│ │ Chunking Engine │ ── Recursive, semantic, or fixed-size chunking │
│ │ (LangChain/ │ Overlap: 10–20% for context continuity │
│ │ LlamaIndex) │ │
│ └────────┬─────────┘ │
│ │ text chunks │
│ ▼ │
│ ┌──────────────────────┐ │
│ │ Embedding Model │ ← OpenAI text-embedding-3-large │
│ │ (Async Batch API) │ OR BGE-M3 (self-hosted, ONNX) │
│ └────────┬─────────────┘ │
│ │ [1536-dim float32 vectors] │
│ ▼ │
│ ┌──────────────────────┐ │
│ │ MILVUS │ ── collection: rag_chunks │
│ │ (Vector Store) │ index: HNSW (M=16, efConstruction=256) │
│ │ │ metric: COSINE │
│ └──────────────────────┘ │
└────────────────────────────────────────────────────────────────────────┘
┌────────────────────────────────────────────────────────────────────────┐
│ QUERY PIPELINE (real-time, per user request) │
│ │
│ User Query: "What is our refund policy for international orders?" │
│ │ │
│ ▼ │
│ ┌──────────────────────┐ │
│ │ API Gateway / │ ── Auth, rate limit, logging │
│ │ FastAPI │ │
│ └────────┬─────────────┘ │
│ │ │
│ ┌──────┴──────┐ │
│ │ │ (optional: hybrid search) │
│ ▼ ▼ │
│ ┌──────────┐ ┌──────────┐ │
│ │Embedding │ │ BM25 │ │
│ │ Model │ │ Sparse │ │
│ │ (query) │ │ Encoder │ │
│ └────┬─────┘ └────┬─────┘ │
│ └──────┬───────┘ │
│ │ vector(s) │
│ ▼ │
│ ┌──────────────────────┐ │
│ │ MILVUS │ ── ANN search (ef=200, top-20) │
│ │ .search() call │ scalar filter: source IN ["policy", ...] │
│ │ │ returns: [(chunk_id, score, content)] │
│ └────────┬─────────────┘ │
│ │ top-20 candidates │
│ ▼ │
│ ┌──────────────────────┐ │
│ │ Re-ranker │ ── Cross-encoder (BGE-Reranker / Cohere) │
│ │ (optional) │ Reduce 20 → top-5 for context window │
│ └────────┬─────────────┘ │
│ │ top-5 chunks │
│ ▼ │
│ ┌──────────────────────┐ │
│ │ Prompt Builder │ ── System prompt + retrieved context + query │
│ └────────┬─────────────┘ │
│ │ full prompt (~4000 tokens) │
│ ▼ │
│ ┌──────────────────────┐ │
│ │ LLM │ ← GPT-4o / Claude 3.5 / Llama-3.1-70B │
│ │ (Completion API) │ │
│ └────────┬─────────────┘ │
│ │ grounded response + citations │
│ ▼ │
│ User Response │
└────────────────────────────────────────────────────────────────────────┘
Complete RAG Code Example
import os from openai import OpenAI from pymilvus import MilvusClient # ── Setup ────────────────────────────────────────────────────────────── openai_client = OpenAI(api_key=os.environ["OPENAI_API_KEY"]) milvus_client = MilvusClient("http://milvus:19530") COLLECTION = "rag_chunks" EMBED_MODEL = "text-embedding-3-large" LLM_MODEL = "gpt-4o" # ── Embedding helper ─────────────────────────────────────────────────── def embed(text: str) -> list[float]: resp = openai_client.embeddings.create(input=text, model=EMBED_MODEL) return resp.data[0].embedding # 3072-dim vector # ── Ingestion ────────────────────────────────────────────────────────── def ingest_documents(chunks: list[dict]): """ chunks: [{"content": str, "source": str, "created_at": int}, ...] """ embeddings = [embed(c["content"]) for c in chunks] data = [ { "embedding": embeddings[i], "content": chunks[i]["content"], "source": chunks[i]["source"], "created_at": chunks[i]["created_at"], } for i in range(len(chunks)) ] milvus_client.insert(collection_name=COLLECTION, data=data) # ── RAG Query ────────────────────────────────────────────────────────── def rag_query(user_question: str, top_k: int = 5) -> str: # Step 1: Embed the question q_vec = embed(user_question) # Step 2: Retrieve relevant chunks from Milvus results = milvus_client.search( collection_name=COLLECTION, data=[q_vec], limit=top_k, output_fields=["content", "source"], search_params={"metric_type": "COSINE", "params": {"ef": 200}}, ) # Step 3: Build context string context_parts = [] for hit in results[0]: src = hit["entity"]["source"] body = hit["entity"]["content"] context_parts.append(f"[Source: {src}]\n{body}") context = "\n\n---\n\n".join(context_parts) # Step 4: Call LLM with RAG prompt prompt = f"""You are a helpful assistant. Answer based ONLY on the provided context. If the answer is not in the context, say "I don't have that information." CONTEXT: {context} QUESTION: {user_question} ANSWER:""" resp = openai_client.chat.completions.create( model=LLM_MODEL, messages=[{"role": "user", "content": prompt}], temperature=0.1, ) return resp.choices[0].message.content # ── Usage ────────────────────────────────────────────────────────────── answer = rag_query("What is our refund policy for international orders?") print(answer)Python — Full RAG Pipeline
Hybrid Search (Dense + Sparse)
from pymilvus import AnnSearchRequest, RRFRanker, WeightedRanker # Dense vector search request dense_req = AnnSearchRequest( data=[dense_embedding], anns_field="dense_vector", param={"metric_type": "COSINE", "params": {"ef": 100}}, limit=20 ) # Sparse vector search request (BM25-style) sparse_req = AnnSearchRequest( data=[sparse_embedding], anns_field="sparse_vector", param={"metric_type": "IP", "params": {}}, limit=20 ) # Merge results using Reciprocal Rank Fusion results = client.hybrid_search( collection_name="rag_chunks", reqs=[dense_req, sparse_req], ranker=RRFRanker(k=60), # or WeightedRanker([0.7, 0.3]) limit=10, output_fields=["content", "source"] )Python — Hybrid Search
Pros & Cons
✅ Strengths
- Billion-scale proven: The only open-source vector DB with documented 10B+ vector deployments in production.
- Index diversity: HNSW, IVF, DiskANN, GPU_CAGRA, sparse — tune for any cost/performance profile.
- Hybrid search: Native dense + sparse + scalar filter in a single query. First-class RRF reranking.
- GPU acceleration: CAGRA index builds and searches on NVIDIA GPUs. 10–100× speedup for large batches.
- Cloud-native: Kubernetes-native, horizontally scalable, cloud-agnostic (AWS/GCP/Azure/on-prem).
- Active ecosystem: LangChain, LlamaIndex, Haystack integrations. Strong community (30k+ GitHub stars).
- DiskANN: NVMe-based billion-scale search with dramatically lower RAM requirements.
- Multi-vector fields: Support for ColBERT-style late interaction (multiple embeddings per document).
- Milvus Lite: Zero-infra dev mode — prototype locally, deploy to cluster unchanged.
- Open source: No vendor lock-in. Apache 2.0 license. Zilliz Cloud as optional managed path.
❌ Limitations
- Operational complexity: Full distributed mode requires etcd, Pulsar, MinIO, plus 4 coordinator types. Significant K8s expertise needed.
- No ACID transactions: Eventual consistency by default. Not suitable as a source of truth for financial/transactional data without careful design.
- Memory-heavy HNSW: HNSW index holds entire graph in RAM. 1B vectors @ 768-dim ≈ 3TB RAM with HNSW. DiskANN mitigates but adds latency.
- No native joins: Cannot join with external relational data. Must denormalize metadata into Milvus or handle joins in application layer.
- Learning curve: Concept of coordinators, segments, WAL, and TSO is unfamiliar to engineers from RDBMS backgrounds.
- Compaction overhead: Background compaction can spike I/O and CPU. Must size resources with compaction headroom.
- Pulsar/Kafka dependency: Adds operational overhead and a potential failure domain. New Woodpecker WAL (in development) aims to replace this.
- Slow index builds: CPU-based HNSW builds on 100M+ vectors can take hours. GPU nodes or DiskANN are workarounds.
- Limited analytical queries: Not designed for aggregations, GROUP BY, or complex analytical SQL. Use alongside a data warehouse for analytics.
Operational Complexity Breakdown
| Concern | Details | Mitigation |
|---|---|---|
| etcd management | Must be backed up. Leader elections. 3-node HA minimum. | Use managed etcd (etcd-operator, cloud provider) |
| Pulsar complexity | Requires BookKeeper + ZooKeeper. Topic retention tuning. | Use Kafka alternative or wait for Woodpecker WAL |
| Index rebuild on schema change | Changing index type requires full rebuild. Downtime risk. | Blue/green deployment with dual collections |
| Memory sizing | HNSW must fit in Query Node RAM. Undersizing = OOM crashes. | DiskANN, quantization, or partition-based loading |
| Monitoring | Many internal metrics (segment count, WAL lag, query latency). | Prometheus + Grafana dashboards (provided by Milvus) |
Summary & Future of Vector Databases
Engineering Summary
Milvus is the most feature-complete, production-battle-tested open-source vector database available today. Its disaggregated, cloud-native architecture — separating access, coordination, compute, and storage — enables genuinely elastic scaling from millions to billions of vectors without redesigning your system.
The key differentiators vs alternatives are: (1) billion-scale proven, (2) GPU-accelerated index building/search, (3) native hybrid search (dense + sparse + scalar), (4) DiskANN for memory-constrained scale, and (5) the richest index type selection in any vector DB.
The trade-off is operational complexity. Running Milvus Distributed requires mature Kubernetes operations, monitoring discipline, and capacity planning expertise. For teams without this, Milvus Lite → Standalone → Zilliz Cloud is a viable progression that defers operational burden until scale demands it.
When Milvus is the Right Call
- You are building AI-native products where semantic search is a first-class feature, not an afterthought
- Your dataset exceeds 50M vectors or is projected to reach that within 12 months
- You need hybrid search (combining dense, sparse, and structured filters) in a single query
- You require data sovereignty, on-premises deployment, or multi-cloud portability
- Your team has (or is building) Kubernetes operational capability
- Cost optimization matters — self-hosted Milvus is dramatically cheaper than Pinecone at billions of vectors
The Future of Vector Databases
Convergence with Traditional DBs
PostgreSQL (pgvector), SingleStore, Oracle, and MongoDB are all adding vector capabilities. The future likely involves multi-model databases that handle relational + vector + document in one system. Milvus responds with richer scalar query support.
Hardware-Accelerated Search
GPU-native indexes (CAGRA), custom ASIC accelerators (e.g., FPGA-based ANN), and NVMe-optimized DiskANN will push billion-scale search latencies below 1ms. GPU memory bandwidth is the new CPU cache for AI workloads.
Serverless & Edge Deployment
Milvus Lite and on-device embedding models enable vector search at the edge. Serverless vector DBs (scaling to zero) will reduce costs for intermittent workloads. Expect WASM-compiled vector indexes in browsers.
Multi-modal & Learned Indexes
Universal embedding models (text, image, audio, video in one space) will simplify schemas. Learned index structures (using neural nets to predict data distribution) will surpass handcrafted ANN algorithms for specific domains.