Back to blog Product

Building the Operations Agent: What We Learned in Beta

By Ege Celik ·

Whiteboard session with early product planning notes

The Operations agent was the first role template we built, and it took about eight months of beta testing to get right. Not because the underlying technology is complicated, but because "operations" is one of the least well-defined functions in a growing company. Ask 10 ops directors at 40-person companies what their job involves and you get 10 different answers. Building an agent template that works across that variation required us to understand what the common denominator actually was, and our first few attempts got it wrong in ways that were instructive.

This post is an honest account of what we built, what failed, and what the current template looks like as a result. It is written primarily for teams evaluating Atlantic who want to understand how the Operations agent was designed. It is also a partial record of what we learned about building agent systems for non-technical users during our beta period.

What we got wrong in version 1

The first version of the Operations agent was designed around a task list. We had interviewed about 20 ops directors before building it, and we compiled the most common tasks they mentioned: vendor follow-ups, meeting scheduling, expense routing, request triage, status reporting. We built a template that handled all five, connected to the most common tools (email, calendar, Slack, project management), and let users configure which tasks to enable.

The problem was that this was a task catalog, not a role definition. When our beta users deployed it, they immediately started asking questions like: when should the agent send a vendor follow-up on its own versus putting it in my queue? What makes a request high-priority versus normal? Who does the escalation go to if I'm not available?

These were not questions about the agent's capabilities. They were questions about the agent's judgment boundaries, and the first version of the template had no good answers for them. Each user was left to figure out the judgment layer from scratch, which required them to understand the system deeply enough to configure it correctly. Most beta users didn't have that depth yet. The template had lots of features and was underused because the entry cost was too high.

What we changed

The pivot happened after about three months of beta when we noticed that the users who were getting the most value from the Operations agent had all done something similar in their configuration: they had picked one workflow, gotten it right, and left everything else disabled.

We redesigned the template around this observation. Instead of a task catalog, we structured the Operations agent around a primary workflow with an explicit "start here" default. The default configuration handles vendor follow-up and request routing only, with conservative escalation thresholds and a review queue for everything that isn't clearly routine. Users who want to expand to meeting scheduling, expense routing, and status reporting can do so incrementally after they've calibrated the core workflow.

We also added what we call the scope narrative: a plain-language description of what the agent is allowed to do, what it is not allowed to do, and what conditions trigger escalation. This is generated automatically from the configuration settings and is shown to the user at the end of setup so they can verify that the configuration matches their intent. In beta, we found that about 30 percent of misconfiguration issues were cases where users had set options they didn't fully understand, and the scope narrative caught most of those before the agent went live.

The integration problem

The second major learning from beta was about integrations. We had assumed that connecting the agent to email and Slack would be sufficient for most ops workflows, with additional integrations (project management tools, CRM, accounting software) as add-ons. What we found instead was that the email and Slack connections were necessary but insufficient because they had no context about the company's specific relationships and processes.

An email-connected operations agent can see that a vendor sent a message. But without context about that vendor's relationship history, payment terms, and communication norms, it cannot make good decisions about how to respond. The same is true for internal requests: an agent that can read Slack can see a request come in, but without context about the requester's role, the request type's priority classification, and the relevant owner, it cannot route it effectively.

We added a context layer to the Operations agent template that lets users provide this background: a vendor registry with communication preferences, an employee directory with role-based routing rules, and a request category taxonomy with associated owners and SLAs. This took more setup time but dramatically improved the agent's first-run performance. Users who provided this context in beta saw measurably better results from day one compared to users who skipped it and relied on the agent to figure out context from interaction history alone.

What stayed the same

A few decisions from the initial design survived the redesign intact. The escalation path as a required configuration, not an optional one: users cannot deploy the Operations agent without specifying who receives escalations and in what channel. We made this hard to skip after watching too many beta deployments fail because escalations went nowhere and backlogs built up invisibly.

The review queue as a default for edge cases: anything the agent cannot confidently classify goes into a human review queue rather than being processed with low confidence or dropped. This produces a slightly higher review burden in the first weeks but prevents the silent failure mode where the agent handles ambiguous items incorrectly and nobody notices for a month.

The audit log showing every action the agent took: this was in version 1 and has gotten more detailed with each iteration. Teams that review the audit log regularly consistently get more value from the agent and calibrate their configuration faster than teams that only look at the log when something goes wrong.

Where it still falls short

The current Operations agent template is not a good fit for all types of operations work. It handles high-frequency, structured workflows well: vendor follow-ups, request routing, expense categorization. It handles poorly-defined or highly-contextual work poorly: strategic vendor negotiations, internal relationship-sensitive communications, anything that requires reading organizational politics or personal history.

We say this clearly in the onboarding materials, but it is worth repeating here. The Operations agent is a high-volume workflow executor, not a general operations advisor. The boundary between what it does well and what it does not is meaningful, and teams that try to use it outside its effective range without appropriate oversight tend to get poor results. Knowing that boundary is part of using the tool correctly.

More from the blog

Role-based agent model explained
Product

The Role-Based Agent Model, Explained

Why AI agent projects fail
Operations

Why Most AI Agent Projects Never Make It to Production

Agent monitoring 101
Engineering

Agent Monitoring 101: What to Watch When Your Agents Go Live