Cisco's FAPO System Automates LLM Pipeline Repair Using Claude Code

Cisco AI has introduced FAPO (Fully Automated Prompt Optimization), a Claude Code-driven system that automatically diagnoses and repairs failures in multi-step AI pipelines. The open-source tool addresses a persistent bottleneck in enterprise AI deployment: when a chained sequence of language models produces incorrect output, identifying which step caused the failure typically requires manual inspection of intermediate outputs.

The Core Problem FAPO Solves

FAPO's central innovation is step-level failure attribution, a method that traces errors back to the exact prompt in a multi-step sequence that produced bad output rather than treating the pipeline as a black box. When a pipeline consisting of multiple chained models returns a wrong answer, FAPO evaluates which stage caused the failure by classifying it into one of four root causes: retrieval failures (when document search returns empty or irrelevant content), cascading failures (when an early step produces empty output that corrupts downstream processing), format failures (when the correct answer is hidden inside text the scoring system cannot parse), or reasoning failures (when good inputs still produce a wrong conclusion).

This diagnostic precision matters because multi-step pipelines compound error rates in ways single-model deployments do not. As autonomous enterprise AI systems chain together multiple reasoning steps, a model operating at 95% accuracy at each stage produces a combined output accuracy of roughly 77% across five sequential steps without error correction between stages. Finding where in that sequence reliability breaks down is essential to fixing it efficiently.

How FAPO Works: Three Levels of Escalation

FAPO operates through a closed optimization loop that runs six stages per iteration: evaluate (run the pipeline on the dataset and collect per-case scores), attribute (classify failures by root cause using rule-based heuristics plus LLM analysis), propose (generate a variant targeting the dominant failure type), review (validate the proposal for scope compliance and data leakage), compare (accept the variant only if it improves the previous best), and iterate until target accuracy is reached or the optimization budget is exhausted.

The system works across three escalating levels. Prompt edits are lowest cost and tried first, targeting format and reasoning failures. Parameter changes adjust configuration values like retrieval_k or temperature. Structural changes alter chain topology—such as adding a self-reflection node or switching to a ReAct pattern—when attribution finds that prompt-level fixes cannot address retrieval or cascade bottlenecks. FAPO exhausts one level before escalating to the next.

Claude Code serves as the orchestration agent, autonomously managing the optimization loop. The system also supports Codex as an alternative optimization agent. This design choice reflects a broader shift in enterprise infrastructure: rather than building custom rule-based repair logic, organizations are delegating the "reasoning" layer to capable frontier models that can understand task descriptions, generate variants, and validate them independently.

Benchmark Performance Against State-of-the-Art

Cisco evaluated FAPO against GEPA (Generalized Evolutionary Prompt Architecture), a state-of-the-art prompt optimization method that uses evolutionary search with genetic operators. Both systems started from identical baseline pipelines, but FAPO could escalate to structural changes while GEPA was limited to prompt-level optimization. The comparison spanned six benchmarks (HoVer, IFBench, LiveBench-Math, HotpotQA, Papillon, and AIME) across three task models: GPT-4.1-mini, GPT-5.4-mini, and Gemma 3-12B, with Claude Opus 4.6 serving as both FAPO's orchestrator and GEPA's reflector.

FAPO won 15 of 18 model-benchmark comparisons with a mean gain of +14.1pp over GEPA. On HoVer and IFBench—the two benchmarks where FAPO escalated to pipeline changes—it won all six model-benchmark pairs with a mean gain of +33.8pp. On the four benchmarks without structural changes, FAPO still won 9 of 12 through prompt optimization alone. AIME was the only benchmark where GEPA led, by 3.1pp, a gap smaller than standard deviation across stochastic trials.

The magnitude of improvement on benchmarks where structural changes were needed demonstrates the value of escalation. On HoVer, the baseline scored 35.9%, GEPA reached 48.5%, and FAPO achieved 83.8%—a +35.3pp gain. On IFBench, baseline was 35.7%, GEPA 48.5%, and FAPO 80.7%—a +32.2pp gain. These benchmarks involved multi-hop question answering and instruction-following tasks where retrieval quality or chain structure was the limiting factor.

Guardrails Against Overfitting

FAPO includes three guardrails against overfitting: it inspects only training-split cases during optimization while validation and test sets expose only aggregate scores; every variant is written to disk as an immutable file, never edited in place; and an independent reviewer agent checks each proposal before it runs. This architectural approach reflects lessons learned from gradient descent training, where neural networks iteratively adjust weights based on error signals, making careful tracking of which data influences which updates critical to preventing overfitting.

Practical Use Cases and Concrete Results

FAPO targets multi-step LLM pipelines across several domains. In a documented multi-hop question answering walkthrough, a chain rose from 39.3% to 70.3% validation exact match across two iterations, after which attribution flagged remaining failures as retrieval-limited, signaling a structural fix was needed. On the HotpotQA benchmark, FAPO reached 68.3% test accuracy versus GEPA's 61.8%.

The system also handles instruction-following tasks like those in IFBench, where format-constraint failures pushed FAPO to escalate beyond prompts to reach 80.7% test accuracy; classification pipelines such as software-name-to-category tasks that can be scaffolded by Claude Code then optimized to exact-match targets; and ReAct agents that use trajectory scoring and LLM-as-Judge scoring. These use cases span question answering, constraint satisfaction, and tool-use workflows—the breadth of enterprise AI deployment patterns.

Architecture and Availability

FAPO is a multi-tenant evaluation and optimization framework where each tenant is a self-contained optimization project containing one task's prompts, dataset, chain definition, scorer, and configuration. The core engine, named hephaestus, is domain-agnostic and handles evaluation, chain execution, and scoring. Chains are LangGraph state graphs that process each test case. Out of the box, FAPO supports three providers: OpenAI, Baseten, and SageMaker.

The system ships open source under Apache 2.0. From a task description, Claude can scaffold the rest: the initial prompt, the chain, and the scorer. Once the pieces exist, users supply a dataset of paired inputs and expected outputs that define success, which FAPO splits into a validation set and a held-out test set—the validation set drives iteration while the test set is used only for final one-shot evaluation.

Implications for Enterprise AI Deployment

FAPO signals a structural shift in how AI research outputs reach production scale. Building a capable model is no longer the binding constraint—keeping it functioning reliably inside a real enterprise pipeline is. The decision to delegate orchestration to Claude Code rather than build proprietary repair logic suggests that frontier model APIs are now capable enough to manage infrastructure-level tasks, not just user-facing applications.

The problem FAPO addresses is rooted in how neural networks are trained and chained together. Modern AI systems learn by iteratively adjusting connection weights based on error signals through backpropagation and gradient descent. When multiple trained models are chained into a pipeline, errors do not simply propagate linearly—they interact and amplify across stages, making the source of a final output failure difficult to isolate using standard debugging methods. FAPO's step-level attribution applies a version of the same causal tracing logic that gradient descent uses during training, but at inference time and across pipeline boundaries rather than within a single model's weight updates.

For organizations running complex agentic workflows, the reliability implications are significant. As autonomous systems introduce more sequential reasoning steps, the compounding error problem becomes acute. FAPO's architecture is designed specifically to arrest that cascade by automatically diagnosing which step failed and what type of fix—prompt, parameter, or structural—can address it.