AI Operating Model Design
A practical framework for structuring roles, governance, and tooling so AI adoption scales beyond isolated pilots.
A practical framework for structuring roles, governance, and tooling so AI adoption scales beyond isolated pilots.
Who in your organization decides which AI use case gets built next? Who is accountable when an AI-assisted decision turns out to be wrong? Who makes sure legal, marketing, and customer service aren't quietly running three contradictory AI tools at once? If you can't answer at least one of these on the spot, that's not a gap in your knowledge — it's the default state of most companies before they've defined an operating model for AI.
A single pilot survives without these answers. One department launches a customer-service chatbot, a second team hears about it and builds something similar with a different vendor, a third quietly signs up for a consumer AI tool because IT approval takes six weeks. Eighteen months later the company has a dozen disconnected initiatives, no shared risk standard, and still no way to answer a simple board question: how much of our AI spend is actually governed?
The operating model is the answer to these questions before they get asked. It defines who decides which use cases get built, who is accountable when an AI-assisted decision goes wrong, which tools are sanctioned, and how the organization learns from one department's mistake instead of repeating it five times. Strategy documents and use-case backlogs get most of the attention in AI transformation work, but without an operating model behind them, the backlog becomes a wish list and the strategy becomes a slide that nobody operationalizes.
This matters more, not less, as AI adoption becomes routine rather than exceptional. Early pilots can survive on enthusiasm and a motivated individual contributor. Scaling past ten or twenty active use cases requires the same discipline you'd apply to any other capability that touches customer data, employee decisions, or regulated processes: clear roles, a governance cadence, and a shared toolset. This page lays out a concrete design for that structure, built for mid-size companies as well as large enterprises, not just organizations with a dedicated AI division.
Three archetypes cover almost every operating model in practice, and it's worth naming them precisely because the tradeoffs are structural, not cosmetic.
Centralized Center of Excellence (CoE). A single team owns AI strategy, builds solutions, and controls the tooling and risk standards for the whole organization. Business units submit requests and the CoE delivers. This model gives you consistency: one risk framework, one approved-tools list, one place where institutional knowledge about what works accumulates. The failure mode is equally consistent. A central team sized for the first five use cases cannot handle the fiftieth without becoming an intake bottleneck. Business units that need a solution this quarter, not next year, start finding workarounds, and workarounds are exactly how shadow AI usage takes root.
Federated / embedded model. Each business unit builds and runs its own AI capability, hiring or training its own prompt engineers and data specialists, choosing its own tools. This produces speed and strong domain fit because the people building the solution sit next to the people who understand the business problem. The cost shows up elsewhere: five departments independently license overlapping SaaS tools, none of them apply the same risk classification to a similar decision, and the company has no aggregate view of its AI exposure. When a regulator, auditor, or customer asks "which systems make automated decisions about individuals," a purely federated organization often cannot answer quickly.
Hybrid hub-and-spoke. A central hub owns the platform, the approved-tools list, risk standards, and cross-functional governance. Business units ("spokes") own their use-case backlog and run day-to-day delivery through embedded roles who report a dotted line into the hub for standards and a solid line into the business for priorities. This is the model that most mid-size and enterprise organizations converge on once they've moved past their first few pilots, and for good reason: it keeps consistency where consistency has real value (risk, tooling, vendor management) while pushing speed and domain judgment to where they have real value (use-case selection, prompt design for a specific workflow, adoption inside a team).
The practical implication is not to design for the archetype you'd prefer in principle, but for the one that matches your current stage. Very few organizations should start with a hybrid model on day one — it requires roles and cadences that don't yet have anything to govern. Almost none should stay purely centralized or purely federated once usage crosses roughly ten active production use cases. Treat the choice as a starting position with a known next step, not a permanent architecture.
Generic role lists ("you need an AI lead") aren't useful for planning headcount or job descriptions. Here is what the work actually requires, regardless of which archetype you're running:
AI Product Owner. Owns the use-case backlog and the business case for a department or function. This person decides what gets built next based on value and feasibility, not the most senior person who wants their pet idea done first. Without this role, backlogs get prioritized by whoever complains loudest.
Prompt Engineer / AI Solutions Engineer. Builds and maintains prompts, evaluation sets, and guardrails for production use cases. This is closer to a quality-engineering discipline than a one-time writing task: prompts drift in effectiveness as models update, and someone needs to own the evaluation harness that catches regressions before users do.
AI Governance/Risk Lead. Owns risk tiering, compliance sign-off, and model documentation. This role is what keeps a hybrid model from collapsing back into either extreme — it's the person business units call when they're unsure which tier a new use case falls into, and the person who can say yes quickly to a low-risk request instead of routing everything through a full committee.
ML/Data Engineer. Handles integration, data pipelines, and any fine-tuning or retrieval infrastructure. Distinct from the prompt engineer role because the skill set is different — this is systems and data work, not language-model behavior work.
Embedded Business Translator / AI Champion per department. The bridge between a business problem and a technical solution, sitting inside the department rather than in the central team. This role is what makes a hub-and-spoke model actually work day to day: they know both the workflow and enough about what the technology can and can't do to scope a use case realistically before it ever reaches the governance process.
Executive Sponsor on an AI steering committee. Provides budget authority and resolves cross-departmental conflicts over priority. Without a named executive sponsor, governance decisions get relitigated every time a new department head arrives.
Change/Enablement Lead. Owns training and adoption. Technically sound AI tools fail constantly for the mundane reason that nobody taught the people who were supposed to use them, or the rollout ignored the existing workflow it was meant to replace.
The credible caveat, especially for mid-size companies: in year one, most of these are part-time or rotational assignments layered onto existing jobs, not dedicated full-time hires. A department's most technically curious analyst becomes the AI Champion at 20% of their time. The compliance manager adds risk-tiering to their existing remit. This is appropriate and sustainable at low use-case volume. The threshold where these roles typically need to become dedicated FTEs is somewhere around ten or more active production use cases running concurrently — below that, the coordination overhead of full-time roles usually exceeds the value they protect. Design the role list now, staff it lightly now, and revisit the staffing model as volume grows rather than over-hiring against pilots that may not scale.
The single mechanism that keeps a Center of Excellence from becoming a bottleneck is applying different levels of scrutiny to different levels of risk, rather than running every use case through the same approval depth. A low-risk internal productivity tool — summarizing meeting notes, drafting internal documentation — should not wait behind a customer-facing or regulated decision tool in the same queue.
A workable structure classifies each use case at intake into tiers, roughly:
The critical design element is not the tiers themselves but the decision right: who is explicitly authorized to approve production deployment at each tier, and what happens when that's ambiguous. Many organizations discover their governance gap not in the tiering logic but in the absence of a named approver — a use case sits in limbo because three people each assume someone else has authority to say yes. Write the decision right down as an explicit RACI next to each tier, not as an implied assumption.
"Regular reviews" is not a cadence. A cadence that actually functions has distinct forums with distinct purposes and distinct attendees:
Escalation paths deserve explicit definition too: what happens when the working group disagrees on a tier, or a department wants an exception to the approved-tools list. Route disagreements up to the steering committee rather than leaving them to be resolved informally — informal resolution is where inconsistent precedent creeps in.
Tooling sprawl is the most common, most avoidable cost problem in mid-size AI adoption. When procurement is decentralized, it is entirely ordinary to find three or four departments independently paying for overlapping AI SaaS subscriptions that do substantially the same thing, discovered only once someone maps total spend. A standardized tooling layer is as much a cost lever as a governance one.
Four concrete elements make this work:
An approved-tools and approved-models list, maintained by the vendor review board, published somewhere every employee can actually find it rather than buried in a wiki nobody visits.
A central LLM gateway or abstraction layer sitting between departments and model providers. This avoids hard-coding a single vendor's API into every internal tool, gives you usage and cost visibility in one place, and means a future model swap is a configuration change rather than a rebuild.
A shared prompt and evaluation library across departments. Without it, five teams independently solve the same summarization or classification problem, each with its own untested prompt and no shared sense of what "good" looks like. A library turns each solved problem into a reusable asset instead of a one-off.
A shadow AI registration and amnesty process. Ask departments to register any AI tool they're already using, no penalty attached, in exchange for getting it evaluated and either approved, replaced, or sunset. This is close to universal as a finding on any first assessment: uncovering existing, ungoverned tool use is normal, not a sign of organizational failure, and treating it as an amnesty rather than an audit gets you honest disclosure instead of concealment.
How the model is funded shapes behavior as much as how it's governed, and the choice interacts directly with which archetype you're running.
Fully central budget. The CoE or hub funds everything. This produces the strongest governance — nobody can spend around the rules because nobody controls their own budget line — but it slows business buy-in. Departments treat centrally funded initiatives as someone else's project rather than their own, and prioritization fights move to the steering committee instead of being resolved locally.
Department chargeback / cost allocation. Business units pay for their own use cases, typically against a shared platform. This produces stronger accountability and faster local decision-making, but it requires a level of financial and organizational maturity that not every company has yet — chargeback models built too early become an argument about internal billing rather than a discipline on spend.
Hybrid funding. Central platform, infrastructure, and governance overhead are funded centrally; department-specific use-case development is funded by the business unit that benefits. This tends to match the hybrid hub-and-spoke operating model well for the same reason: shared infrastructure benefits from shared funding, while local judgment about which use case is worth the money benefits from local budget ownership.
Individual use-case ROI answers whether a specific tool is worth what it costs. It doesn't answer whether the operating model itself is functioning. Track these separately:
Review these alongside the quarterly portfolio review, not as a separate exercise — they're the health check on the machine that produces the use cases, not a duplicate of use-case-level reporting.
A handful of failure patterns recur often enough to name directly:
A pure centralized CoE that becomes an intake bottleneck. Business units start routing around it, which is precisely how shadow AI gets fueled rather than prevented — the well-intentioned control creates the problem it was meant to stop.
A pure federated model with duplicated tool spend and inconsistent risk handling. Two departments make materially different risk judgments about similar use cases because nobody is holding a consistent standard.
Governance designed on paper before any real use case exists. Elaborate committees and forms built in the abstract, then either ignored once actual use cases arrive because they don't fit the paperwork, or so over-engineered that the first real request stalls for months.
No clear decision rights for production approval. As covered above — ambiguity about who can say yes is often worse than a strict rule, because it produces stalled use cases rather than wrong ones.
Treating AI governance as a bolt-on IT security review. AI risk overlaps heavily with existing data-governance and model-risk structures many companies already run for other purposes. Bolting on a separate, parallel AI review process duplicates effort and creates two answers to the same question. Integrate it with what already exists rather than standing up a parallel structure.
For organizations operating in Germany, Austria, or Switzerland's EU-adjacent regulatory environment, two considerations should be built into the tiering framework from the start rather than layered on afterward.
The EU AI Act's risk categories — unacceptable, high, limited, minimal — map naturally onto an internal risk-tiering framework, and mapping them explicitly is the efficient way to build this in. Design your tiers so that a use case landing in the AI Act's "high-risk" category automatically triggers the corresponding obligations inside your own governance process: conformity assessment, defined human oversight, and model/system documentation. Done this way, regulatory compliance becomes a natural consequence of your internal tier, not a separate compliance exercise run in parallel.
Second, in Germany and Austria specifically, AI systems that touch employee monitoring or performance evaluation typically trigger Betriebsrat (works council) co-determination rights (Mitbestimmung) under German works constitution law. This needs an explicit role in the governance cadence — a defined point where the works council is consulted before deployment, not something discovered after a system is already live. Build works council consultation into the high-risk approval path for any use case touching employee-facing decisions, rather than treating it as a legal afterthought bolted on when a complaint arises.
Training that treats every employee identically wastes time for people who need depth and loses the people who need only awareness. Structure capability-building in three tracks tied directly to the roles above:
Executive-awareness track for the steering committee and department heads: what the technology can and can't do, what the risk tiers mean for decisions they'll be asked to approve, and what questions to ask before greenlighting a use case.
Deep-dive track for department AI champions and prompt engineers: hands-on evaluation methods, prompt design and testing discipline, and the specifics of the approved-tools stack.
General employee AI-literacy track: what tools are approved, how to register a tool they're already using informally, and basic judgment about what shouldn't be pasted into an external AI tool.
Each track should be short and role-specific rather than a single generic course everyone sits through once and forgets.
Resist the urge to design the target operating model before you know what's actually happening in the organization today. Start with a rapid assessment: what shadow AI usage already exists, what pilots are already running and who's funding them, and where decision rights currently sit even if nobody has written them down. This assessment typically takes two to four weeks and should answer three questions directly — how much ungoverned spend exists, which pilots have real business traction worth formalizing, and where the biggest gaps in decision rights currently sit.
Design the target model against that reality, not against a theoretical ideal. Most organizations should expect to start closer to organic and federated than they'd like to admit, consolidate into a lightweight Center of Excellence once several use cases need to scale together, and mature into the hybrid hub-and-spoke model with embedded roles as volume grows. Anticipate that path when you design the initial structure rather than building for the end state on day one — a model sized for where you are now, with a clear next step already defined, will outperform an elaborate structure with nothing yet to govern.