The false choice most vendors sell
The AI conversation in most boardrooms gets framed as a binary: send your data to a frontier API and get the best models, or keep everything in-house and settle for less. Both halves of that framing are outdated.
Open-weight models have closed most of the gap for the workloads that dominate business AI — classification, extraction, summarization, retrieval — and they run entirely inside your VPC or on-prem. Meanwhile frontier APIs remain genuinely better for hard reasoning and complex generation. The correct architecture uses each where it wins.
The privacy gateway: classification before any model sees a token
The load-bearing component is not a model at all. It's a gateway that sits between your systems and every LLM, and does three jobs on every request:
- Classify — is this content sensitive (PII, financials, contracts, health data) or not? Policy is defined once, enforced everywhere.
- Redact — non-sensitive requests get sensitive fragments stripped or tokenized before leaving the boundary.
- Audit — every call, every destination, every payload class is logged. When compliance asks 'what did we send where,' the answer is a query, not a shrug.
With the gateway in place, the sensitive/public question stops being a philosophical debate and becomes routing policy — mechanical, testable, and provable.
The private layer: open-weight models inside your boundary
Sensitive workloads route to models you host: Llama, Mistral, Qwen, DeepSeek and their fine-tuned descendants, running in your cloud VPC or on your own hardware. Your contracts, customer records, and financials are processed by infrastructure you control, under your existing security perimeter and compliance posture.
For the high-volume workloads that dominate enterprise AI usage, small open-weight models are not a compromise — they're frequently faster and an order of magnitude cheaper per call than frontier APIs, while being fully private by construction.
The router: never boxed into OpenAI, Anthropic, or anyone
The second load-bearing component is a model router behind a single abstraction layer. Every workflow calls 'the model'; the router decides which model that is — per task, on three axes: privacy class, quality requirement, and cost.
The leaderboard reshuffles quarterly. Claude leads some reasoning tasks this quarter; GPT or Gemini may lead next; an open-weight release may collapse the price of a workload overnight. With a router, capturing those shifts is a configuration change. Without one, it's a re-platforming project — which is why single-vendor architectures quietly become expensive architectures.
What this means for your AI roadmap
Sequence the architecture before the use cases: gateway and router first, then workflows. Teams that start with a single vendor's SDK wired directly into production code spend the next two years paying an invisible tax — in switching costs, in privacy reviews that block use cases, and in inference bills that routing would have cut.
The encouraging part: none of this requires exotic engineering. The pattern is well-established, the open-weight ecosystem is mature, and a privacy-first multi-LLM foundation can be stood up inside a quarter — usually alongside the first two or three workflow pilots it will serve.
Read the failure data before the vendor deck
In 2025, MIT's NANDA initiative published the most-cited enterprise AI study of the cycle — The GenAI Divide: State of AI in Business 2025 — built on 300 public deployments, 150+ executive interviews, and $30–40 billion of tracked enterprise spend. Its headline finding: roughly 95% of enterprise GenAI pilots deliver no measurable P&L impact. Only about 5% of integrated systems create significant value, and the study's authors are explicit that the divide is organizational, not technological: data readiness, integration depth, and the capacity to learn from deployment separate the winners.
Two secondary findings matter even more for architecture decisions. First: deployments built through purchasing and partnership succeed roughly 67% of the time, while solo internal builds succeed at about a third of that rate — a direct verdict on the build-it-all-ourselves instinct. Second: MIT found a 'shadow AI economy' in over 90% of firms — employees quietly using personal AI tools while the official pilot stalls — which means your privacy posture is already being tested daily, whether or not you have written one.
Why architecture is the survivorship trait
Layer Gartner's projection on top: over 40% of agentic AI projects will be cancelled by the end of 2027 — escalating costs, unclear business value, inadequate risk controls. Put the two studies together and the anatomy of failure is consistent: single-vendor SDKs wired directly into production code, no privacy boundary, no baselines, no owner. Every quarterly model-market reshuffle becomes a re-platforming project; every consumption invoice arrives without a competitor; every compliance question gets answered after the fact.
The gateway-and-router architecture in this guide is, precisely, the survivorship pattern: sensitive data served by open-weight models inside your boundary, redacted tasks routed to whichever frontier API wins the task this quarter, every call classified and logged. It converts the three failure causes into managed variables — costs (in our implementations, task-fit routing has typically cut inference spend 40–70% versus single-vendor defaults), value (baselined pilots make impact arithmetic), and risk (the boundary is enforced policy, not a memo).
A decision rule you can steal
If you take one operating rule from the research, take this composite of what MIT's successful 5% actually do:
- Pick workflows by measured value and data readiness — never by demo appeal; back-office and operations workflows outperform the sales-and-marketing pilots most budgets chase
- Buy or partner for the platform layer; build only the thin layer that encodes your proprietary process
- Put a privacy gateway in front of everything before the first pilot — shadow AI means the traffic already exists
- Baseline before deployment; measure at 30 and 90 days; kill what does not clear the bar without sentiment
- Route per task and review the model market quarterly — vendor swaps should be configuration changes, not projects
The Corelynx gateway pattern — our take on doing this well
The two-lane architecture is increasingly standard advice; the difference is operating detail. Ours, refined across implementations:
- Classification before routing, always: every request is labeled — sensitivity, task type, cost tier — before any model sees it. The gateway is a policy engine, not a proxy.
- Two lanes, one log: private in-boundary models and redacted public calls share a single audit trail, so 'where did our data go' is a query with a timestamp, not a meeting.
- Per-task evaluation harnesses before model selection: models are chosen on your measured tasks, not leaderboard reputation — and the harness re-runs at a quarterly model-market review that sits on the governance calendar like a board meeting.
- Cost budgets per workflow, with alerts. Consumption surprises are an architecture failure, so we design them out rather than apologize for them.
- The exit is written before the entrance: every vendor integration ships with its replacement path documented. If swapping a model is not a configuration change, we consider the integration unfinished.
That last rule is our sharpest departure from common practice — and, judging by MIT's data on who survives the GenAI Divide, probably the most valuable one.