Skip to content

DevOps AI Agent: Autonomous CI/CD Pipeline Monitoring

A real devops ai agent is not a chat window you paste stack traces into — it's a long-lived process that wakes up when your pipeline emits an event, investigates on its own, fixes what it safely can, and only pages a human for the genuine criticals. That daemon shape is where most tutorials stop and where production teams actually start. By the end of this post you'll understand the pattern SRE teams use, see a webhook-triggered agent process wired with budget, health, and mission governance, and know how the journal replays state after a crash so a restart doesn't lose the incident it was working.

From chat demo to daemon: what a devops ai agent really needs

The gap between a demo and an on-call replacement is not the model — it's the lifecycle around the model. A CI/CD monitoring agent has to survive things a chat script never faces:

  • It runs for weeks. Events arrive at 3 a.m. whether or not anyone is watching the terminal.
  • It costs money on every call. An agent stuck in a retry loop can burn your budget before morning.
  • It takes real actions. Restarting a stage or paging on-call has consequences, so blast radius matters.
  • It crashes. Deploys, OOM kills, and node evictions happen. A restart must not drop the incident mid-investigation.

Promptise Foundry's Agent Runtime is built for exactly this. Instead of calling agent.ainvoke() in a loop yourself, you register an AgentProcess that owns a trigger queue, a heartbeat, governance, and a journal. The runtime turns a stateless LLM into a governed, crash-recoverable service. The full build-along lives in the Pipeline Observer lab; this post distills the architecture and the survival mechanics.

Build the pipeline observer: a webhook-triggered agent process

The core of a pipeline observer agent is a ProcessConfig that binds a model, its MCP tools, and one or more triggers. A WebhookTrigger stands up an HTTP endpoint; your monitoring stack (Datadog, GitHub Actions, Argo, whatever emits alerts) POSTs a JSON event, and the runtime converts each request into a queued TriggerEvent that invokes the agent.

Here's a minimal but runnable observer daemon. Swap the stub MCP server for your real monitoring and remediation tools; everything else is production shape.

"""Pipeline observer daemon — a webhook-triggered AgentProcess."""
import asyncio
import signal
import sys

from promptise.runtime import AgentRuntime
from promptise.runtime.config import (
    ProcessConfig, TriggerConfig,
    BudgetConfig, HealthConfig, MissionConfig, JournalConfig,
)

INSTRUCTIONS = """
You are an autonomous CI/CD pipeline monitoring agent.
- INFO events: acknowledge briefly, call no tools.
- WARNING events: investigate with metrics, retry failed jobs, open a WARNING incident.
- CRITICAL events: check health, attempt a restart, open a CRITICAL incident, page on-call.
Prefer safe remediation before escalation. Be concise and cite specific metrics.
"""


async def main():
    config = ProcessConfig(
        model="openai:gpt-5-mini",
        instructions=INSTRUCTIONS,
        servers={
            "pipeline": {
                "command": sys.executable,
                "args": ["pipeline_tools_server.py"],
                "transport": "stdio",
            },
        },
        triggers=[
            TriggerConfig(type="webhook", webhook_path="/alerts", webhook_port=9090),
        ],
        budget=BudgetConfig(
            enabled=True,
            max_tool_calls_per_run=20,
            max_cost_per_day=5.0,
            on_exceeded="pause",
        ),
        health=HealthConfig(
            enabled=True,
            stuck_threshold=3,
            loop_window=20,
            on_anomaly="escalate",
        ),
        mission=MissionConfig(
            enabled=True,
            objective="Keep the pipeline above 99.9% uptime by fixing recoverable issues.",
            success_criteria="No unresolved P1 incidents for more than 15 minutes.",
            eval_every=10,
        ),
        journal=JournalConfig(backend="file", path="./pipeline_journal"),
        concurrency=1,
    )

    async with AgentRuntime() as runtime:
        await runtime.add_process("pipeline-observer", config)
        await runtime.start_all()
        print("[observer] Listening on http://localhost:9090/alerts")

        stop = asyncio.Event()
        loop = asyncio.get_running_loop()
        for sig in (signal.SIGINT, signal.SIGTERM):
            loop.add_signal_handler(sig, stop.set)
        await stop.wait()


if __name__ == "__main__":
    asyncio.run(main())

Prefer to watch a directory instead of an HTTP endpoint? Swap the trigger for TriggerConfig(type="filewatch", ...) and the same process reacts to new log or artifact files with glob patterns — useful when your CI writes JUnit reports or crash dumps to a mounted volume. You can also attach several triggers to one process, so a single observer handles both webhooks and file drops.

Governance that keeps an autonomous SRE agent from running wild

An autonomous SRE agent with tools is only as safe as its guardrails. Promptise's runtime governance is the difference between "automation you trust overnight" and "a bot with a corporate credit card." Three subsystems run alongside every invocation, and each is one config object in the block above.

Budget counts tool calls and cost per run and per day. When a run exceeds max_tool_calls_per_run or the daily max_cost_per_day, the on_exceeded policy fires — "pause" self-suspends the process instead of grinding through your budget. The agent even sees its remaining budget in context, so it can prioritize.

Health watches behavior for anomalies competitors ignore: stuck_threshold catches the same tool call repeated N times, loop_window catches repeating call patterns, and empty-response detection catches a model that's given up. On a trip, on_anomaly="escalate" fires a webhook and an EventBus event so a human learns the agent is spinning — without you writing loop-detection code yourself.

Mission turns the process from "respond to events" into "hold a target." You give it an objective and success_criteria, and an LLM-as-judge evaluates progress every eval_every invocations against a bundle of the conversation, state, and tool log. For CI/CD that objective is usually uptime.

Together these give you the survival mechanics real on-call automation needs. If you want the full escalation-and-remediation tool set spelled out, the Pipeline Observer lab walks through nine MCP tools (health checks, retries, restarts, incident creation, Slack, PagerDuty) end to end.

Crash recovery: journals and the ReplayEngine

The mechanic that makes this daemon deployable is crash recovery. With JournalConfig(backend="file"), the runtime records every state transition, trigger event, and invocation result to an append-only journal, and periodically writes a full-state checkpoint. When the process dies mid-incident and restarts, the ReplayEngine loads the last checkpoint and replays the entries after it to rebuild the process's context and lifecycle state.

Under AgentRuntime this happens automatically before the process accepts new triggers. You can also drive it directly to inspect what a restart would restore:

from promptise.runtime.journal import FileJournal, ReplayEngine

async def inspect_recovery():
    journal = FileJournal("./pipeline_journal")
    recovered = await ReplayEngine(journal).recover("pipeline-observer")
    print("lifecycle:", recovered["lifecycle_state"])
    print("entries replayed:", recovered["entries_replayed"])
    print("context:", recovered["context_state"])

One honest caveat: replay reconstructs state, not side effects. It applies recorded state mutations in order, so the same journal always rebuilds the same context — but it never re-executes tool calls. A resume will not re-restart a stage or re-page an engineer, which is exactly what you want. For the deeper mechanics, see the sibling walkthrough on AI agent crash recovery.

Classify, remediate, escalate — and when to hand off to a team

The observer's decision loop is deliberately narrow: classify the event, attempt safe remediation, and escalate only what it can't handle. INFO gets an acknowledgment and zero tool calls. WARNING gets an investigation and a retry. CRITICAL gets a restart attempt, a formal incident, and a page. That tiered policy is what keeps AI on-call automation from either doing too much or waking humans for noise.

When an incident spans services — a bad deploy that needs a rollback, a triage lead, and a comms owner — a single observer is the wrong shape. That's where you compose specialists that delegate over HTTP and coordinate through an event bus, the topology covered in the multi-agent coordination guide and the broader multi-agent systems in Python guide. Real deployments of this daemon pattern, including the pipeline observer, are collected in the showcase gallery.

When another approach is the better fit

Be honest with yourself before you reach for an autonomous agent at all. If your "monitoring" is a fixed sequence — on failure, retry twice then alert Slack — a plain webhook handler or a GitHub Actions step is simpler, cheaper, and fully deterministic. You don't need an LLM to run an if statement, and a rules engine won't hallucinate a remediation.

An agent earns its keep when events are ambiguous, the right fix depends on correlating several signals, and the failure modes are too varied to enumerate up front. If you're already deep in a managed platform like Datadog Workflow Automation or a mature runbook engine and it covers your cases, staying there is reasonable. Promptise is the better fit when you want to own the process, run models locally or across providers, and keep budget, health, and crash recovery as first-class code rather than a vendor's black box.

Frequently asked questions

How is a devops ai agent different from a chatbot?

A chatbot is stateless and human-driven: you ask, it answers, the process ends. A devops ai agent is a long-lived AgentProcess driven by triggers (webhooks, file watches, cron) with governance and a journal. It runs unattended, takes actions through MCP tools, self-pauses on budget limits, and recovers its state after a crash.

Will the agent take dangerous actions on its own?

Only within the guardrails you set. Budget caps limit how many tool calls and how much cost a run can incur, health detection catches stuck or looping behavior, and the instructions gate destructive tools behind severity. For actions that always need a person, add a server-side approval gate so remediation waits for human sign-off before it runs.

What happens if the observer crashes mid-incident?

With a file journal enabled, nothing is lost. On restart the ReplayEngine loads the last checkpoint and replays subsequent entries to rebuild the process's context and lifecycle state before it accepts new events. Because replay reconstructs state without re-running tools, a resume never re-triggers side effects like a second page or a duplicate restart.

Next steps

Deploy the pipeline observer daemon from the lab and fire test events at its webhook to watch it triage, remediate, and escalate in real time. Start with the Quick Start to stand up your first agent, then follow the Pipeline Observer lab to wire triggers, governance, and journals into a process you can actually run on-call.