Context Efficiency: Why AI Shouldn't See Your Whole Repository

The easiest way to build an AI feature into an architecture tool is to take whatever the user asks, staple the entire model onto it as context, and let the AI sort out what matters. It works in a demo. It works on a model with forty elements. It stops working, in ways that are expensive rather than merely annoying, somewhere around the size of model that real organizations actually have.

This isn't a niche implementation detail. It's a default that most tools reach for because it's the least amount of engineering effort, and it happens to be wrong for architecture specifically, for three separate reasons that compound rather than cancel out: cost and latency scale with the size of the repository instead of the size of the question, more irrelevant context measurably increases the odds the AI answers about the wrong thing, and an enterprise architecture repository is sensitive enough that shipping all of it to a third-party AI provider on every keystroke is a decision that deserves to be made deliberately, not by default. Mooodels is built around a different ordering: turn the question into a graph query first, run that query against the model deterministically, and only then hand the AI the resulting slice — not the repository it came from.

The default that feels safe until the model grows

"Just give the AI everything and let it figure out what's relevant" is an appealing design because it sidesteps the hard part. You don't have to decide what's relevant to a given question — you outsource that decision to the model, which is, after all, pretty good at reading. For a small number of elements this is fine. A dozen applications and their relationships fit comfortably into a prompt with room to spare, and the AI has no trouble finding the two or three that matter to whatever was asked.

Enterprise architecture repositories don't stay that size. A mid-sized organization's application landscape runs to hundreds or low thousands of elements — applications, services, databases, interfaces, technology components, business capabilities — connected by several times that many relationships. Serialize that into text and it's not a prompt anymore, it's a document. Every question asked against it, however narrow, now drags the whole document along, whether the question was "what depends on the payment service" or "is there a cycle between these two integration layers" or "rename this application everywhere it appears." None of those questions need the whole repository to answer. All of them get it anyway, because the tool never built the machinery to give the AI anything smaller.

Cost and latency scale with the wrong variable

The practical consequence shows up first as a bill and a stopwatch. Token-based AI pricing means the cost of a request tracks the size of what you send it, not the size of the answer you need back. A question that has a precise, three-element answer still gets billed for however many thousand tokens the entire model serialized to, every single time it's asked — and architects ask a lot of small, precise questions in the course of a working session, not one big one. Multiply a full-repository payload by the number of questions in a normal review session and the cost curve looks nothing like the value being delivered.

Latency follows the same shape. Larger inputs take longer to process before the AI produces a first token of output, which is the part of the wait a user actually feels. A tool that reloads the entire model into context on every question is choosing, structurally, to make every interaction slower as the model grows — which is exactly backwards, because a larger model is where fast, precise answers matter more, not less. Nobody wants an architecture assistant that gets sluggish in proportion to how seriously the organization has actually used it.

There's a second cost that doesn't show up on an invoice: context windows are finite, and a repository large enough eventually stops fitting at all. A tool architected around "send everything" has a hidden ceiling — some model size beyond which the whole approach silently breaks, forcing an emergency retrofit of exactly the scoping logic that should have been there from the start. Building the scoping in from day one avoids ever hitting that wall.

More context is not free even when it fits

The cost and latency argument is the easy one to make because it's just arithmetic. The more important argument is about correctness, and it's less intuitive: giving an AI more context than it needs doesn't just cost more, it makes the answer less reliable.

This runs against the instinct that more information can only help. In practice, long, mostly-irrelevant context creates two distinct failure modes. The first is attention dilution — models weight different parts of a long input unevenly, and material buried in the middle of a large document gets less effective attention than material near the edges, so a request to "look for anything relevant" across a thousand-element model is quietly harder for the AI than the same request against fifteen elements, even though nothing about the underlying question changed. The second, more specific to architecture, is false pattern-matching: when a model is asked about the payment service and the payment service's actual neighbors are surrounded by hundreds of unrelated elements, some of which share naming patterns, similar tags, or superficially similar relationship types, there is a real chance the AI's answer references one of those unrelated elements instead of — or alongside — the one that actually matters. An architecture repository is full of exactly this kind of near-miss material: three different "Gateway" components, five things tagged Tier-1, a dozen relationships labelled "uses." Handing all of it over and asking the AI to find the right needle is asking for a mistake that a scoped query would never have made possible in the first place, because the wrong needle wouldn't have been in the haystack to begin with.

This is the part of the tradeoff that's easy to underrate, because a hallucinated dependency in an AI-generated explanation doesn't look wrong. It looks exactly as confident and well-formatted as a correct one. The failure mode isn't an error message — it's a plausible-sounding answer about the wrong part of the architecture, which is a much worse outcome in a tool people use to make retirement and migration decisions.

Sending the whole repository is a decision, not a default

Separate from cost and correctness, there's a question that has nothing to do with the AI's capability at all: an architecture repository is one of the more sensitive documents an organization has. It's a map of every system, what talks to what, which integrations cross a security boundary, which vendors are embedded where, and often enough properties and tags that record ownership, criticality, and known weak points. That's not incidental information — for anyone thinking about attack surface, vendor exposure, or competitive systems, the architecture repository is close to the whole picture in one place.

Shipping all of that to an external AI provider, on every question, as an invisible implementation detail, is a real decision about where sensitive data goes — and it's one that should be made explicitly by the organization running the tool, not embedded silently in how a vendor chose to wire up a feature. This is a big part of why Mooodels treats AI access as provider-neutral and bring-your-own-key: the organization decides which AI provider gets used and under what terms, and separately, the tool minimizes what any given question actually needs to expose to that provider by scoping the payload before it ever leaves the model. Those are two different controls working together — who the AI is, and how much it gets to see — and neither one substitutes for the other. A provider-neutral setup that still ships the entire repository on every question has only solved half the exposure problem.

Turning the question into a graph query first

The alternative isn't "give the AI less information and hope for the best." It's inserting a deterministic step before the AI ever gets involved. Take the question as it's actually asked — in plain language — and turn it into a graph query against the model, run that query, and only then hand the AI the result, scoped to exactly what's relevant, for it to explain or act on.

"What depends on the payment service" is the clearest example, because it's really two different tasks wearing one sentence. Answering it correctly is a graph traversal: start at the payment service element, follow incoming relationships, and follow them again from whatever you find, out to some bounded depth. That's the part with a factual right answer, and it doesn't need an AI at all — a graph traversal engine answers it exhaustively and deterministically, the same way every time, with no chance of missing a dependency because it happened to be described in an unusual way. The second task is explaining that result in useful language, flagging what looks risky, or drafting a change based on it — and that's where an AI assistant genuinely earns its place, working from the traversal's output rather than trying to discover the dependency structure itself by reading a document.

// "what depends on the payment service" becomes:
MATCH upstream(PaymentService, direction: incoming, maxHops: 3)
RETURN elements, relationships, hopDistance

// result: 11 elements, 14 relationships, 3 hops
// this scoped result -- not the 1,400-element model -- goes to the AI

The traversal returns eleven elements and fourteen relationships. That's what gets serialized and handed to the AI for explanation — not the fourteen hundred elements the rest of the model happens to contain. The AI's job shrinks to something it's actually well-suited for: turning a small, precise, already-correct graph result into a clear explanation, a risk flag, or a proposed patch. It is no longer being asked to first find the needle and then describe it — just to describe a needle that's already been handed to it.

The graph operations doing the real work

"Turn the question into a query" only holds up if the underlying graph engine can actually answer the range of questions architects ask. In practice, a fairly small set of traversal primitives covers most of what comes up in day-to-day architecture work, and each one maps cleanly onto a recognizable question:

• Incoming and outgoing relationships. The base case — what does this element connect to, in either direction, one hop out. Answers "what does this call" or "what calls this."
• Upstream and downstream impact. The multi-hop version of the same idea, following relationships transitively to some bounded depth. Answers "what depends on this" and "what does this depend on," which is most of what a retirement, migration, or outage conversation actually needs.
• Shortest path. Given two elements, how are they connected at all, and through what. Useful for "how does this front-end reach that database" when the answer isn't obvious and might run through two or three intermediate services.
• Reachability. A yes/no version of shortest path — can element A reach element B at all, through any number of hops. Cheap to compute, and often the first thing worth checking before running a more expensive traversal.
• Cycle detection. Finds circular dependency chains, which are exactly the kind of structural problem that's nearly invisible on a hand-drawn diagram and mechanically obvious to a graph algorithm.
• Connected components. Groups the model into clusters that are internally linked but disconnected from each other — useful for spotting orphaned subsystems, or confirming that a supposedly isolated domain actually is.
• Cross-domain dependency detection. Flags relationships that cross a declared boundary — between business domains, security zones, or ownership groups — which is often the exact thing a governance review is trying to catch.

None of these need an AI to execute correctly, and that's the point. They're deterministic algorithms over a graph — the same query against the same model produces the same answer every time, which is not a property any AI-based reasoning process can promise over a large text dump. Reaching for graph traversal before reaching for AI reasoning isn't a stylistic preference; it's using the tool that's actually guaranteed to be right for the part of the problem that has a factually right answer, and saving the AI for the part of the problem — language, judgment, drafting — that genuinely needs it.

What actually reaches the AI

Once the graph query has run, what gets handed to the AI is the result set: the relevant elements, their relationships, and whatever properties matter to the question — not the query mechanics, and not the rest of the model. For an explanation request, that's enough context to produce a grounded answer in plain language, because everything in the payload is already known to be relevant; there's no need for the AI to first sift the relevant from the irrelevant. For a change request — "add a rate limiter in front of the payment service" — the same scoped slice becomes the working set the AI proposes a patch against, and that patch is generated with full knowledge of the payment service's actual, current neighbors rather than a guess based on naming conventions or general assumptions about what payment services usually look like.

That patch is still a proposal, not a write. Scoping the context tightens what the AI reasons over; it doesn't extend how much it's trusted to act on its own. The two are separate safeguards — a small, accurate blast radius for what the AI sees, and a human approval step for what changes as a result — and a tool needs both, because a perfectly scoped context can still produce a patch someone should look at before it lands.

Why this ordering generalizes past architecture tools

The instinct behind "query first, then AI" isn't specific to architecture repositories. It's a general answer to a question that comes up any time an AI feature is bolted onto a system that already holds structured data: should the AI search for the relevant information itself, or should the system hand it information that's already known to be relevant?

The popular default for unstructured data is retrieval-augmented generation — embed a large corpus of text, run a similarity search against the embeddings for whatever the user asked, and feed the AI whatever comes back. That's a genuinely good solution to a genuinely hard problem: how do you find relevant material inside a pile of text that has no inherent structure connecting one document to another. Similarity search is doing real work there, because there's no graph to traverse — a folder of PDFs doesn't know which of its own documents relate to which.

An architecture model is not that kind of data. It already has structure — elements and typed relationships between them, which is a graph by construction, not something that needs to be inferred from vector proximity. Running embeddings and similarity search over a graph is solving a problem the data doesn't have, at a cost the data doesn't need to pay. "What depends on the payment service" has one exact, computable answer sitting in the relationships that already exist; approximating it through semantic similarity between text chunks is strictly worse on every axis that matters — slower to compute for equivalent accuracy, prone to returning things that merely sound related instead of things that actually are connected, and unable to guarantee exhaustiveness the way a traversal can.

The general principle sits above both cases: when the underlying data is already structured — a graph, a relational schema, a well-defined API — query it directly, and reach for retrieval-style approaches only for the parts of the problem that are genuinely unstructured text with no other way in. This applies well outside architecture tools. A codebase has a call graph and an import graph; asking an AI to find every caller of a function by embedding the whole repository and searching by similarity is a worse tool than just walking the graph the compiler already understands. A support system has a structured ticket schema with fields, statuses, and relationships between tickets; a query against that schema beats a semantic search over ticket text for most operational questions. The pattern holds any time structure already exists: try the query first, and only reach for something heavier when there's genuinely nothing to query.

Where full context genuinely earns its place

None of this makes graph traversal a universal substitute for giving an AI a broad view. Some questions are legitimately open-ended in a way no query can pre-scope. "Does anything about this architecture look unusual" or "what's the weakest part of this domain" doesn't reduce to a traversal, because there's no starting element and no fixed hop count — the question is asking for judgment across a wide surface, which is exactly the kind of task a broader context window is suited for. In those cases the honest move is to widen the slice deliberately — a whole domain, a tagged subset, a named view — rather than pretend a narrow query can answer a question that was never narrow to begin with.

Similarly, a very small model doesn't need any of this machinery. If the whole repository is forty elements, scoping the context is solving a problem that doesn't exist yet, and it's fine to hand the AI everything. The design point isn't "always scope, no matter what" — it's that scoping should be available and should be the default for the large majority of narrow, well-formed questions, so that cost, latency, and exposure don't quietly get worse as the thing being modelled grows into something worth protecting.

A short checklist

For anyone evaluating an AI feature bolted onto an architecture tool — this one or any other — a few direct questions tend to surface which side of this design choice it landed on:

• Does the size of the model change how long a single AI question takes to answer, or does it stay roughly constant?
• For a question with a factual answer — "what depends on this" — is the answer computed deterministically, or is it something the AI inferred from a text dump?
• Can you see, or configure, how much of the model actually leaves the system on a given request — or is "everything, always" the only mode available?
• Does the tool let you choose your own AI provider, separately from how much context gets sent to it?
• When you ask the same narrow question twice, do you get the same answer both times?

A tool that scopes context by default, computes what it can compute deterministically, and only calls on an AI for language and judgment tends to answer all five comfortably. A tool built around "paste the whole thing in and let the AI figure it out" usually can't, and the gap between those two answers is exactly the gap this article has been describing — not a difference in how capable the underlying AI is, but a difference in how much of the problem was solved before the AI was ever asked to help.