Jira AI Agent: How We Built Compass for Backlog Refinement

Introduction

A Jira AI agent is a program that reads your Jira project, decides which tools to call, and proposes changes to tasks, labels, statuses and releases on your behalf. I built Compass, the one we use at Mobile Reality, and this article is the engineering view: the stack, the agent loop, retrieval, memory, and what failed along the way. It is written for engineers and technical leads who are thinking about building an agent on top of Jira instead of buying another tool.

The business case is covered in our article on AI project management and why we built Compass. Here we stay with the code. Every number below comes from the Compass repository, not from a benchmark slide.

What Our Jira AI Agent Does (and What It Refuses to Do)

Compass syncs a project's Jira issues (stories, tasks, bugs and epics, plus their comments) and the project's repositories from Bitbucket, GitHub or GitLab. On top of that it offers a shared chat where PMs and developers ask questions, and a module view that groups tasks into product areas. The agent can draft tasks, edit descriptions, change labels and statuses, and prepare Jira releases.

What it refuses to do is write to Jira on its own. Every write tool is a "propose" tool: it draws a form, and a person clicks Apply. The system prompt states it plainly: nothing changes in Jira until the user clicks Apply. That single rule shaped most of the architecture below.

The prompt also tells the agent to ground every answer in tool results and to quote task keys and file paths. Personal data such as email addresses, phone numbers or pay is banned from facts, tasks and releases.

The Architecture: Convex, Two Models and One Hand-Rolled Loop

The frontend is React 19 with Vite, Tailwind CSS and reactflow for the module graph, hosted on Firebase Hosting. Everything else runs on Convex: queries, mutations, long-running actions, cron jobs, file storage, full-text search and vector indexes. There is no separate queue service; chat turns and repository syncs are queued with the Convex scheduler and plain tables.

Sign-in is Google OAuth restricted to our company domain, and access is invite-only with two roles, PM and teammate. Each user connects their own Atlassian and Bitbucket accounts over OAuth, so Jira writes happen under the name of the person who approved them.

Why we route models by role

Compass uses two chat models, chosen per job rather than one model for everything:

Pinning a provider on OpenRouter is a deliberate choice. We fix the host and keep a single fallback instead of letting the router pick, so latency and behaviour do not change from one request to the next. Inference code also puts a failing provider on a cooldown before retrying it.

A tool-calling loop capped at 12 calls

The agent is a plain while loop, not a framework. Each round sends the conversation and the tool definitions to the model, validates the returned arguments with zod, runs the tools, and appends the results. The chat agent may make at most 12 tool calls per turn, which is enough for "find the release, list its tasks, read two files, propose labels" and short enough to stop a runaway loop.

The loop handles truncated output, supports a stop button, and records per-round token usage, time to first token and the provider that served the request. Each thread shows its running cost in the UI. We describe the same pattern in more depth in our guide on how to build an AI agent.

Grounding the Agent in Jira Tasks, Code and Documents

The chat agent has more than 25 tools, in four groups: task tools (semantic and keyword search, filters by release or date, task details and live Jira comments), code tools (directory tree, file search, grep, read file), document tools (search, read, create, edit), and proposal tools. A Jira re-sync tool exists too, but it refuses to run if the project was synced in the last two hours.

Hybrid retrieval with reciprocal rank fusion

Requirement documents are chunked and indexed twice: as vectors and as full text. A search runs four legs (semantic, keyword, heading and title) and merges them with reciprocal rank fusion, using weights of 1.0, 0.9, 0.35 and 0.15 and the standard constant k = 60. Documents the user attached to the thread get a 1.5x boost.

The result is capped at five hits, with no more than two chunks from any one document. Without that cap, one long specification crowds out everything else and the agent answers from a single source.

Reading the repository instead of guessing

Repository files are synced into Convex, so the agent can browse, grep and read them without calling the Git host on every turn. Repositories above 50 MB use a staged batch sync, and cron jobs fail stuck syncs and purge staging data. When a PM asks whether a feature exists, the agent is expected to open the code and cite the path, not infer it from the ticket title.

The Project Brain: Why We Skipped Agent Memory Frameworks

Teams need the agent to remember agreed facts: what "partner" means in this product, who owns billing, which library was dropped and why. We call this the project brain. It stores five kinds of facts (glossary, convention, decision, ownership and fact), and each moves through a lifecycle of proposed, active, superseded or rejected.

Before building it, we evaluated the agent memory libraries: mem0, Zep with Graphiti, Letta and LangMem. Our verdict was not to adopt any of them. None runs inside Convex's V8 isolate (mem0 imports Node's fs and crypto at module load; Graphiti needs Python and a graph database), and their published benchmarks contradict each other, so there was no quality basis for choosing one.

The design we built has two tiers. "Canon" is a small set of facts, about one to two thousand tokens, injected into every turn. "Recall" is everything else, retrieved on demand. Every fact records who asserted it, when, from which source, and which fact it supersedes. The extractor proposes; a human promotes.

Each project and each thread can set the brain to full, read-only or off. When the agent proposes a fact that resembles an existing one, the user sees a warning that names the similar fact, which stops duplicates before they reach canon.

Backlog Refinement Through Proposals, Not Writes

Backlog refinement is where a Jira AI agent earns its keep: dozens of small edits that a PM would otherwise make one ticket at a time. In Compass, a request like "label all open survey tasks and add them to next month's release" becomes two bulk forms, one for labels and one for the release, with up to 50 tasks per form.

The forms are not hard-coded React screens. The server builds them as MDMA blocks, our open format for interactive components inside Markdown, and the client renders them with @mobile-reality/mdma-renderer-react. The agent fills in the values, the PM edits any field, and Apply sends the change to the Jira REST API, converting Markdown to Jira's ADF format on the way. You can read about the format itself in markdown for AI agents.

New tasks follow a project skill, an instruction pack the team attaches to a thread. The default Jira issue template asks for business-level wording, a summary, context and at most five acceptance criteria. Teams change the skill instead of editing the system prompt.

Matt Sadowski

CEO of Mobile Reality

Transform Your Business with Custom AI Agent Solutions!

Leverage our expertise in AI agent development to enhance efficiency, scalability, and innovation within your organization.

• Expert development of modular and scalable AI software solutions.
• Integration of Large Language Models (LLMs) for advanced capabilities.
• Customized AI agents tailored to your business needs.
• End-to-end support from design to deployment.
• Enhance decision-making and operational efficiency with AI.

Book a consultationEstimate project

What Failed First: The Code-Graph Detour

The first version of Compass, in March 2026, looked very different. It parsed repositories into a Neo4j graph and ran a multi-phase "enhanced analysis" agent with worker pools to match code against tasks. The commit that first made it work end to end admitted, in its own message, that the results were poor.

In June 2026 we removed Neo4j and the static analysis layer and replaced them with the chat agent and MDMA forms. In October we deleted the last of the old analysis backend. The lesson was that a heavy offline pipeline produced confident, stale answers; an agent that reads the current files and tasks at question time, and shows its sources, produced answers the PM could check.

The module extraction prompt kept one idea from that period. It asks the agent to act as "a staff engineer joining a product team", to derive modules from both tasks and code, and to keep assigning tasks until at least 80% belong to a module. Generic names such as "Core", "General" or "Utilities" are rejected by the tool itself, not only discouraged in the prompt.

Testing Tool Routing with PromptFoo

An agent with more than 25 tools fails quietly: it calls a slower or broader tool and still gets an answer. We test routing with PromptFoo suites that run against the same chat preset the app uses. One case checks that a question about a release calls gettasksbyrelease instead of pulling every task with getall_tasks.

Other suites cover task-to-module assignment. We run them by hand before changing a model or a prompt, not in CI, and the codebase also carries a large Vitest suite and Playwright end-to-end tests. Model changes are frequent: the chat model moved to GLM 5.3 Flash in September 2026 and task assignment moved to Claude Haiku 5.5 in October. Evals are what make those switches safe to try.

Jira AI Agent vs Atlassian Rovo: When Each Fits

If you are on Jira Cloud, Atlassian Rovo is the first option to look at. It is Atlassian's own AI layer, it needs no infrastructure from you, and for search and summaries across Atlassian products it is the shortest path.

A custom Jira AI agent makes sense when you need things a vendor feature will not give you: your own approval flow, your own task templates and project brain, a model you choose and can swap, code from repositories outside Atlassian, and cost reporting per thread. It also makes sense when the agent is part of a delivery process you sell, as it is for us. The price is that you own the evals, the prompts and the upkeep.

Conclusion

Compass shows that a useful Jira AI agent does not need an agent framework, a graph database or a memory vendor. It needs good context, a small loop, strict proposals, and tests on tool choice.

• A Jira AI agent should read code and documents as well as tickets, and cite task keys and file paths.
• A hand-rolled tool-calling loop with a hard cap (12 calls per turn in Compass) is easier to debug than a framework.
• Hybrid retrieval with reciprocal rank fusion and per-document caps keeps answers from collapsing onto one source.
• Agent memory works best as curated, attributed facts that a human promotes, not as automatic extraction.
• Backlog refinement belongs in bulk approval forms; nothing should reach Jira without a click.
• Evaluate tool routing with real presets before every model switch.

I would start with the read-only half: sync tasks and code, and let your PMs ask questions for a few weeks before you add a single write tool. If you want help building an agent like this for your own delivery process, our team can set up Compass on your Jira or build one around your stack. For the broader picture of how we build agents, see our AI agent development guide.

Frequently Asked Questions

Does the Jira AI agent modify issues automatically, or does it require human approval?

Which LLM models power Compass, and how are they routed?

How does Compass retrieve context from code repositories and project documentation?

When should a team build a custom Jira AI agent instead of using Atlassian Rovo?