In the first article of this series, we walked through Retrieval-Augmented Generation (RAG) using the metaphor of a librarian who finds pages that are semantically similar to a question. It is a strong tool, but the librarian cannot tell you how the pages relate to each other. That is why vector RAG often stumbles on multi-hop reasoning, disambiguating overlapping names, and synthesizing answers from multiple sources.
Three terms now keep coming up in the data and AI industry: ontology, knowledge graph, and GraphRAG, the pattern that combines the first two with RAG. Much of social media frames this as “RAG is dead, ontology and GraphRAG are the answer.” When you actually follow the source material, the story is closer to complement than replace.
Four things are worth taking in order: what an ontology is, how a knowledge graph differs from it, how GraphRAG works when both are combined with RAG, and what the operational cost of running one looks like in practice.
What is Ontology? Making the Meaning of Data Explicit
An ontology is a model that writes down, in advance, what kinds of things exist in a domain and how they relate. In 1993, Tom Gruber’s classic definition framed it as “an explicit specification of a conceptualization.” In plainer terms, an ontology defines the objects in a domain and the relationships that hold between them, so machines and people read the data the same way.
Ontology is not a new idea. It traces back to the Semantic Web vision Tim Berners-Lee promoted in the late 1990s and made famous in a 2001 Scientific American article — the goal being to layer machine-readable meaning on top of web data. The field went quiet for years, and the LLM era has pulled it back into focus.
The pull is practical. An AI agent that makes decisions needs to understand how things relate and what they mean. Vector embeddings capture meaning only loosely — close enough to find similar passages, but not precise enough to state that this person manages that team. An ontology supplies that explicit layer. Year of the Graph Vol. 29, “The Ontology issue,” documents how Microsoft, Palantir, GenAI adopters, and RAG application builders have all started talking about ontologies again going into 2026.
Example
For example, an ontology for a company domain looks like this:
[Ontology Example — Company Domain]
┌────────────────────────┐
│ Person │
│ - name │
│ - hire_date │
│ - email │
└───────────┬────────────┘
│ belongs_to
▼
┌────────────────────────┐
│ Department │
│ - name │
│ - cost_center │
│ - head_count │
└───────────┬────────────┘
│ owns
▼
┌────────────────────────┐
│ Project │
│ - name │
│ - start_date │
│ - status │
└────────────────────────┘Three things are written down explicitly: the entities (Person, Department, Project), their attributes (name, hire date, cost center), and the relations between them (belongs_to, owns). Once a relation like “a person belongs to a department” is named and defined, two people looking at the same data are less likely to interpret it in different ways.
A common follow-up question is how this differs from a database schema. A schema asks “how should we store this data?” An ontology asks “what does this data mean?” Two tables can look identical and still describe entirely different things. The ontology’s job is to write down what they mean. Palantir Foundry’s ontology documentation follows the same shape: it models data as object, link, and action types, separating the structure of data from its meaning.
Knowledge Graph: Filling the Ontology with Real Data
If an ontology is an empty dictionary, a knowledge graph is the encyclopedia filled in along those definitions. Nodes are the actual entities, edges are the relationships between them, and both nodes and edges carry attributes. Facts are usually stored as triples — (subject, predicate, object) — so a single row like “Park Ji-sung — plays for — Manchester United” becomes one fact.
Here is a small knowledge graph filled in along the ontology above:
[Knowledge Graph Example — Company Domain]
┌────────────────┐ belongs_to ┌──────────────┐
│ Ji-young Kim │─────────────→│ Data Team │
│ (Person) │ │ (Department) │
└─┬──────────┬───┘ └──┬───────────┘
│ │ │ collaborates
│ owns │ mentor_of │ _with
▼ ▼ ▼
┌────────────┐ ┌──────────────┐ ┌───────────────┐
│Recommend- │ │ Seo-jun Park │ │ Platform Team │
│ation Sys │ │ (Person) │ │ (Department) │
│ (Project) │ └──────────────┘ └───────────────┘
└────────────┘Because every node follows the shared ontology, its type is unambiguous: “Ji-young Kim” is a Person, “Data Team” is a Department. The belongs_to edge has a pre-defined meaning, so two readers will not argue about what it implies. This is the line between a knowledge graph and a plain graph database. A general graph database stores data in a graph shape; a knowledge graph adds the meaning of that data on top, so the graph can be reasoned over.
Note: This is not the “Graph” in GraphQL.
The two terms share a name, but they live at different layers. GraphQL is an API query language — whatever sits underneath (a relational database or anything else), GraphQL lets the API return its response in a graph shape. A knowledge graph is a store where the data itself lives as nodes and edges, and inference runs on top of it. One is about the shape of a response. The other is about the shape of the storage.
GraphRAG: Combining RAG with Knowledge Graphs
Vector RAG finds similar pages well, but it does not know how those pages relate. A knowledge graph holds exactly that — the relationships, as data. GraphRAG is the hybrid that combines them. During retrieval, it runs vector similarity search and graph traversal in parallel, then hands a richer context to the LLM.
How GraphRAG Works: A 3-Step Flow
The GoodData blog describes the flow in three steps:
- When a user query arrives, GraphRAG extracts the key entities from it.
- In parallel, it runs vector search to find semantically similar text chunks and traverses the knowledge graph to pull the neighbors and relationships of those entities.
- It merges the two results into a single context and passes that to the LLM.
In other words, the librarian who finds books by meaning now works alongside a second librarian who also knows how the things in those books relate to each other.
[A vector RAG vs GraphRAG flow diagram — vector RAG runs a single linear pipeline (query → embedding → vector DB similarity → top-K chunks → LLM), while GraphRAG splits after entity extraction into two parallel paths (vector similarity for related chunks, and knowledge graph traversal for related subgraphs) that merge before the LLM. Caption: “Vector RAG retrieves similar pages; GraphRAG retrieves similar pages plus the relationship network between the entities in them.”]
Multi-Hop Reasoning: Where the Numbers Diverge
The accuracy gap shows up on multi-hop questions. In Microsoft Research’s BenchmarkQED and follow-up comparisons, multi-hop question accuracy widened from 32% for vector RAG to 86% for GraphRAG (summarized by TianPan, April 2026). On the University of Leeds ORAN benchmark over telecom standards documents, Hybrid GraphRAG improved factual correctness by 8% and GraphRAG improved context relevance by 11% over vector RAG.
These numbers do not mean GraphRAG is always better. Simple fact lookup is fast and accurate with vector RAG. The gap opens up only when the question requires multiple hops or reasoning about relationships. The two are not substitutes — they are tools you pick based on the question type.
GraphRAG Tradeoffs and Operational Costs
GraphRAG starts to look like the obvious answer at this point, but the cases get messier in practice. A knowledge graph is expensive to build and expensive to keep alive. Both sides have to stay in view to keep the picture honest.
The Cost of Building a Knowledge Graph
Standing up a knowledge graph means doing all of the following before the system answers a single question:
- Designing the ontology by hand.
- Extracting entities and relationships from raw text.
- Running entity resolution to fold every variation of the same person — “Ji-young Kim,” “Kim Ji-young,” an employee ID — into a single node.
None of these steps are light. One estimate puts the cost of indexing 5 GB of legal documents with Microsoft’s original GraphRAG approach and GPT-4o-mini at around $33,000. Later variants like LazyGraphRAG and HippoRAG cut the bill substantially, but a knowledge graph is still not a tool you can drop in and try out lightly. CIO Magazine reports that knowledge graphs have been pursued in the data industry for nearly twenty years, yet production deployments remain rare. Building and operationalizing one is genuinely hard.
The Cost of Maintaining a Knowledge Graph
Maintenance is where the gap with vector RAG widens further. When new data arrives, vector RAG chunks it, recomputes the embeddings, and is done. A knowledge graph adds:
- Graph consistency checks.
- Conflict resolution with existing nodes.
- Re-running entity resolution when needed.
Converting unstructured text into graph nodes also requires additional schema design and entity cleanup, and the initial build slows down further when the source data is not already structured (Meilisearch’s comparison). If the environment is mostly documents with very little structured relational data, the graph you would draw is often too sparse to be worth modeling.
Vector RAG vs GraphRAG vs Hybrid: A Comparison
The trap many teams fall into is picking GraphRAG first because it looks impressive. DEV.to’s analysis makes the same point: start with vector RAG, hit its limits in real workloads, then layer a graph on top once the gap is clear. Match the tool to the weight of the problem you have, not the other way around — forcing the problem to fit a heavier tool usually shows up later as a large bill.
| Approach | Strength | Weakness | Best Fit |
|---|---|---|---|
| Vector RAG | Fast, simple to set up, strong on semantic similarity | Weak on relationships and multi-hop reasoning | Single-fact lookup, FAQs, document Q&A |
| GraphRAG | Strong on multi-hop reasoning and relationship questions | Expensive to build and maintain, slower to iterate | Domains with rich, structured relationships |
| Hybrid (Vector + Graph) | Combines semantic recall with relational precision | Operational complexity is highest | Mixed domains, enterprise knowledge with both documents and structured data |
Choosing the Right RAG Approach for Your Domain
GraphRAG is a striking piece of technology. What matters more is the question behind the technology choice.
What question types does your domain actually ask, and what retrieval method does it take to answer them?
The decision starts at the same place every time. Look at how users query the system:
- Are most queries single-fact lookups?
- Do they require relational reasoning?
- Do they need to synthesize across multiple sources?
- Do they frequently involve disambiguating overlapping names?
Then look at the shape of the data:
- Is it mostly unstructured text?
- Does it include structured relational data?
- Is it a mix?
Once those two axes are mapped, the right tool tends to become clear.
The industry direction points the same way. Gartner’s 2026 Data & Analytics Top Trends names GraphRAG as a solution for complex use cases while also flagging that adoption is hard. Year of the Graph Vol. 30 tracks the same trend and notes that vendors like Neo4j are moving ontology up to a primary modeling unit in their products. The conversation is shifting from “should we adopt a knowledge graph?” to “how do we build ontology and a semantic layer as a long-term asset?”
Conclusion
Whether the answer is RAG, GraphRAG, or a knowledge graph, what makes it work is a retrieval and reasoning structure that fits the questions your domain actually asks. Vector RAG handles single-fact lookup with little setup. GraphRAG earns its keep on multi-hop and relational questions. The choice comes from the question type and the data shape, not from which architecture looks newest.
For a foundation on how vector RAG itself works, see the first article in this series.
RAG Series
(1) What is RAG? Retrieval-Augmented Generation Explained
(2) GraphRAG Explained: Ontology, Knowledge Graphs, and How They Extend RAG
Leave a Reply