Back to the AI Leadership Lab
PillarPhase 04 · OperateFocus: Organisationales Lernen / KI im Betrieb

Operate: capture results so the next round builds on them

Role of the Company Brain: Capture results and new insights. Review and approve changes so the next round builds on them.

Who it's for

For CEOs and leadership teams who want to learn from results instead of starting every round from zero. The deep dive is for product and tech leaders who keep AI products running.

Company Brain and the four phases
Abschluss der Reparatur

Operate is the fourth phase: it records what became of the decisions. Results, customer signals, support cases and metrics go into the Company Brain, and changes are reviewed and approved. That knowledge flows back into Discover, and the next round starts with better context.

Operate is the fourth phase. It records what became of the decisions.

Input is results from operations: customer signals, support cases, metrics. The Company Brain records them, and changes are reviewed and approved. Output is knowledge that flows back into Discover, and the next round starts with better context.

The deep dive below shows how product and tech teams measure and maintain AI products in operation.

Phase 04 · Operate
Today

Results fizzle out and the next round starts from zero.

What happens

Capture results, customer signals and lessons; review and approve changes.

Result

The next round builds on what the company already knows.

At the centre: the Company Brain

Every phase draws on the Company Brain and adds new knowledge. The guide explains what it is, who owns it and how quality holds.

Leadership questions in this phase

Answered in the Company Brain guide.

Read the guide

Deep dive for product and tech teams

An AI product is never finished

Classic software stays stable after deployment until someone changes the code. An AI product does not: it interacts with end users, processes new data and changes its behaviour – continuous learning is a property, not a feature. That demands a highly specialised operations phase with its own monitoring mechanisms, cost control and escalation paths.

Product ops takes the role of the strategic link between product development, engineering and customer success: it keeps quality measurable and costs visible – and makes sure insights from operations flow back into product decisions.

Model drift: data drift, concept drift, AI ageing

91 per cent of machine-learning models degrade over time – without a single changed line of code. Three mechanisms drive this: data drift, when the statistical distribution of the input data shifts. Concept drift, when the relationship between input and target variable changes. AI ageing, the creeping decay against a world that keeps moving.

Monitoring this degradation is the ultimate product ops task: eval sets, thresholds and automated alerts when error rates rise or prediction uncertainty grows – plus defined rollback paths. “Model drift: when your product quietly degrades” reviews the evidence and shows the monitoring setup for life after launch.

RAG and data quality SLAs: the intelligence depends on the context

Retrieval-augmented generation (RAG) supplies models with company-specific knowledge without expensive fine-tuning – but answer quality then hangs directly on the integrity and freshness of the RAG context. Product ops therefore monitors the data pipelines like a product: schema validations, ongoing checks of data distributions, testing for systematic bias.

Data quality needs metrics tied to the use case: for dynamic pricing, freshness and accuracy matter more than historical depth; for compliance reports, completeness is mandatory. And it needs service level agreements: when a critical data stream fails, it must be clear who is responsible and how escalation runs – before the incident, not during it.

KPIs for AI products: edit distance beats MAU

MAU and churn are lagging indicators: they signal failure only once trust is already destroyed. AI products need leading indicators – metrics that measure whether the AI output actually creates value: edit distance (how much a user manually changes in the AI draft), resolution rate (how many requests get fully resolved) and acceptance rates per suggestion.

Then comes the financial side: an AI investment model continuously weighs token, latency and infrastructure costs against the value created. Without these unit economics, a successful feature can be a loss-maker. “KPIs for AI products: prompt success beats MAU” describes the four measurement layers and the feedback loops that don't lie; the table below summarises the key metrics.

MetricWhat it measuresWhy it matters
Edit distanceHow heavily users rework the AI draft manuallyA direct quality indicator, long before churn shows
Resolution rateShare of requests resolved completelyMeasures value, not usage
Acceptance rateShare of AI suggestions acceptedLeading indicator of trust in the output
Unit economics per interactionToken, latency and infrastructure cost against value createdPrevents adoption from eroding the margin
MAU & churnActive users and attritionLagging indicators – necessary, but too late as a warning signal
Measurement layers for AI products – leading indicators before lagging metrics.

Support with code context

After every release, support often binds the most expensive people in the company: senior engineers triaging routine cases instead of building. AI agents with code context invert this – incoming support requests and error reports are triaged with full code context: the system compares the report with source code and spec, generates fix proposals and resolves routine errors on its own.

The decisive part is the backflow: insights from real errors and customer problems feed back into the Company Brain and the Discover phase. That closes the development cycle – the system learns from production, and the product team works on new value instead of old noise.

Model-agnostic workflows: dependency is a board-level issue

Dependency on a single model provider manifests in the operations phase: if a critical API fails or data protection requirements change overnight, the product stands still. The architecture must allow the switch – to a local model via Ollama, say, or to an alternative cloud provider – without rebuilding workflows.

Regulatory intervention is a frequent trigger – the link to the EU AI Act from the discover phase is direct. “Model dependency: why model-agnostic workflows are becoming a boardroom issue” shows the routing, abstraction layers and hosting decisions that defuse the risk.

Product-led growth: adoption that survives the margin

In product-led growth, the product itself is the primary sales channel. AI features can fuel this – they shorten time-to-value when onboarding shows the value in minutes rather than weeks. They can also ruin it: heavy AI usage costs money on every request, and a freemium model without dosing lets inference costs eat the margin.

The mechanics against it: dosed quotas, upgrade triggers tied to experienced value, and monetisation gates continuously calibrated in the operate phase. “Product-led growth for AI SaaS: adoption that carries itself” describes them in detail.

Go-to-market: convincing sceptical B2B buyers

The AI hype undermines the trust of exactly the buyers who procure B2B software. Positioning, pricing and market entry must therefore be built on trust and measurable ROI – not on feature fireworks. That includes understanding B2B search intent: the research phase wants guides and concepts, the decision phase wants case studies and architectural depth, the purchase phase wants action-oriented pages with regional signals.

B2B search volumes are small; the buying intent behind them is high – a term with thirty searches a month can lead to multi-year contracts. Precision beats reach. “GTM strategy for AI products: convincing sceptical B2B buyers” walks through positioning, pricing and launch – the last link in the chain, carrying the product into the market.

Deep dives for this phase

Each article answers a concrete question from the practice of product and tech teams.

Matching Activities from the Playbook

From the article straight into practice: these Activities put the concepts to work with Teklens.

Frequently asked questions

What is product operations?

Product operations (product ops) is the discipline connecting product development, engineering and customer success to orchestrate scaling and quality assurance: metrics, processes, tooling and data flows. For AI products, AI lifecycle management is added – drift monitoring, data quality SLAs and continuous control of inference costs.

What is model drift and how do I detect it?

Model drift is the creeping degradation of a model in production – through data drift (input data shifts), concept drift (the relationship between input and target changes) or AI ageing. It can only be detected through monitoring: eval sets, thresholds and automated alerts when error rates or uncertainty rise. 91 per cent of ML models are affected over time.

Which KPIs make sense for AI products?

Leading indicators that measure value instead of usage: edit distance (how much users change in the AI draft), resolution rate (share of fully resolved requests), acceptance rates – plus unit economics per interaction, weighing token and infrastructure costs against the value created. MAU and churn remain necessary but are lagging indicators.

Why are model-agnostic workflows a board-level issue?

Because dependency on a single provider is an operational and compliance risk: API outages, price jumps or regulatory intervention can shut a product down overnight. An architecture with an abstraction layer and routing allows switching to local models or alternative providers without rebuilding workflows – a strategic decision, not a purely technical one.

Next phase in the cycle

From Operate back to Discover: the cycle closes. Results, resolved cases and metrics flow into the Company Brain and trigger the next round, with better context than last time.

Phase 01 · Discover – Discover
Simon ScheurerAmr AbulseoudMarc Gasser
The lab letter

No new piece without you.

New articles, new interactive tools, new evidence – in your inbox first. And when you reply, we reply: you write directly with the authors, not with a no-reply.

No spam, no sharing, unsubscribe any time.

Bring one important decision.

Discovery Call: 30 minutes with a founder. Bring an important decision or a concrete context problem.

With a founder. No obligation.