ML & Data ScienceAI Agent Development: A Practical Guide to Building Agents

Introduction

AI agent development is the discipline of building software that decides its own next step instead of following a fixed script. At Mobile Reality we have shipped agents into production across fintech and proptech, and this practical guide shares how we actually do it: the architecture, the tools, and the tradeoffs. It is written for engineers and product leaders who want a comprehensive guide grounded in real code, not a shopping list of frameworks.

The market has moved past "should we build agents" to "how do we build ones we can maintain." Most teams start by picking a framework and regret it within a year. We take the opposite route: a small, boring stack that stays debuggable as the system grows. This guide walks that route step by step.

What Is an AI Agent?

An AI agent is a program that takes a goal, decides which action to take, executes it with a tool, observes the result, and repeats until the goal is met. That loop is the whole idea. A plain script runs fixed steps; an agent chooses its steps at runtime based on what each result tells it.

The confusion usually starts with the model underneath. A large language model generates text. An agent wraps that model in a loop and gives it tools, so the model's output becomes actions, not just words. The model is the engine; the agent is the car built around it.

Types of Agents You Will Actually Build

Most agents fall into a few practical shapes. Reflex agents apply fixed rules and never plan. Goal-based agents decompose an objective and pick their next step at runtime. Utility-based agents weigh tradeoffs like cost against quality. In production, the large majority of agents we ship are goal-based agents driving a tool-calling loop. Learning agents that adapt from feedback are rarer, because most businesses lack the clean feedback loop those agents need. Pick the simplest type that solves the task; overbuilt agents are the most common and most expensive mistake we see.

Is ChatGPT an Agent or an LLM?

ChatGPT is primarily an interface over an LLM, and it becomes agentic only when it can call tools and act on the results. Asked a question, the base model answers in text: that is LLM behavior. When ChatGPT browses, runs code, or calls a plugin and then uses what came back, it is acting as an agent. So the honest answer is: both, depending on whether tools and a loop are in play. The same model powers an ai chatbot and a research agent; what separates them is the orchestration around it.

The Big 4 AI Agent Providers

Four companies supply most of the models teams build agents on today:

• OpenAI, whose GPT models and function-calling format became the de facto standard for tool use.
• Anthropic, whose Claude models lead on long-context reasoning and are widely used in regulated work.
• Google, whose Gemini models bring strong multimodal and long-context capability.
• Meta, whose open source Llama models let teams self-host when data cannot leave their infrastructure.

You do not have to marry one. We route across providers per task, which we cover below. The point of naming the big four is to show that the model layer is a commodity you should keep swappable, not a lock-in decision.

Most business automation built on agents mixes these providers anyway. One agent might reason with Claude, write with an open source Llama model, and classify with a small OpenAI model, all inside the same workflow. Keeping the provider swappable is what makes that automation cheap to evolve as prices and capabilities shift.

How Is an AI Agent Developed? Our Architecture

An AI agent is developed by wiring a model to a set of tools through an orchestration loop, then hardening that loop for production. Here is the architecture we have converged on after 100+ projects delivered since 2016. It is deliberately simple, which is exactly why it holds up.

• Model access: OpenRouter, one API across providers, with per-request model selection.
• Orchestration: a hand-rolled tool-calling loop in TypeScript, no heavyweight framework.
• Memory: the conversation array during a run, plus vector retrieval for knowledge that persists between runs.
• Vector store: Convex native vector search, or PostgreSQL with pgvector, whichever the project already runs.
• Embeddings: openai/text-embedding-3-small via OpenRouter for the Mobile Reality knowledge base.
• Observability: Langfuse for prompt versioning and full call tracing.

This is the same agent architecture behind our internal CMS agent, the one that drafts and edits SEO articles on our own blog. We describe the full pattern in our guide on how to build an AI agent, the companion piece to this one.

Why We Skip LangChain and Dedicated Vector Databases

Every one of the agent systems we inherited from another agency was built on LangChain or a similar framework, and each of those agent systems was a rewrite candidate within eighteen months. The failure mode repeats: the framework hides the prompt, the state, and the control flow behind its own abstractions, right up until a production bug forces you to trace why the model picked one tool over another, and the abstraction is in your way.

We also skip dedicated vector databases like Pinecone or Weaviate for most work. If the project already runs Convex or PostgreSQL, the built-in vector support covers retrieval with one fewer system to operate. The marginal gain from a separate vector DB does not pay for its operational cost until you reach tens of millions of vectors.

The Agent Development Process, Step by Step

Building ai agents follows a repeatable process. Each step below is one decision that shapes the agent's cost, reliability, and how hard it is to maintain.

Step 1: Define One Narrow Task

Every agent we ship starts as a single sentence: this agent will take one specific action, on one trigger, to reach one measurable outcome. Narrowness is the feature. A well-scoped agent beats a general-purpose assistant on cost, latency, and reliability, because there is far less room for it to wander. Resist the pull to build a do-everything ai system on the first pass.

Step 2: Choose Models by Role, Not by Vendor

There is no single best ai model. Different roles in one agent need different things: the orchestrator needs strong tool-calling, the writer needs fluent prose, the analyst needs deep reasoning, and lightweight calls just need speed. We map each role to a model and swap freely, which is where role-based routing saves budget with no quality loss.

export const AGENT_ROLES = {
    orchestrator: { default: 'z-ai/glm-5', requiresTools: true }, // cheap, strong tool-calling
    contentWriter: { default: 'moonshotai/kimi-k2' },             // long-form prose
    analyst:       { default: 'openai/gpt-5.2' },                 // deep reasoning
    lightweight:   { default: 'openai/gpt-4.1-mini' },            // quick tasks
    vision:        { default: 'openai/gpt-4.1-mini', requiresVision: true }, // alt text, image description
};

Because every call goes through OpenRouter, switching a role to a different model or provider is a one-line change. The openai function-calling format is accepted across models, so your tool code does not change when the model does.

Step 3: Build the Tool-Calling Loop

The orchestration core is a plain loop over a messages array. The model responds, you run any tools it asked for, you append the results, and you go around again until it stops calling tools. This is the entire pattern behind our agent workflows.

const messages = [
    { role: 'system', content: systemPrompt },
    { role: 'user', content: userRequest },
];

let iteration = 0;
while (iteration < MAX_ITERATIONS) {
    iteration++;
    const result = await callModel({ model: getModel('orchestrator'), messages, tools });
    messages.push({ role: 'assistant', content: result.text, tool_calls: result.toolCalls });
    if (!result.toolCalls?.length) break;          // no tools -> the agent is done
    for (const tc of result.toolCalls) {
        const out = await runTool(tc.name, tc.arguments);
        messages.push({ role: 'tool', tool_call_id: tc.id, content: JSON.stringify(out) });
    }
}

Around that core goes the production hygiene: an iteration cap to bound cost, duplicate-call detection so a looping model breaks out, an abort signal so a user can stop a runaway run, and streaming so the user sees progress. No graph compiler, no state-machine DSL, just a loop and an array.

This loop is also where agent workflows get their flexibility. Because the model chooses the next tool from what the last one returned, the same loop handles a two-step task and a twenty-step one without new code. When you need richer automation, you add tools, not control-flow branches, and the agent composes them at runtime.

Step 4: Define Tools as Data, and Handle Failure

Tools are JSON schemas the model reads and your code executes. The schema is the easy part. The hard part is what happens when the model gets an argument slightly wrong, hallucinating whitespace or paraphrasing text it was meant to quote. A strict match fails often and the agent spirals into retries.

Our production edit tool tries six matching strategies before giving up, from an exact match down to a first-and-last-line anchor. That fuzzy layer moved one agent's tool-success rate from the high 60s to the high 90s with no change to the prompt. When you build tools for agents, budget as much effort for the failure path as for the happy path.

Step 5: Delegate Generation to a Specialized Model

A high-value pattern: the orchestrator should not write prose. When our agent needs a paragraph rewritten, it calls a tool that makes a separate model call to a writer model chosen for prose quality. That keeps the orchestrator cheap and fast on "what happens next," lets you change the writer independently, and gives you a clear place to look when output quality drops.

Essential Tools and Frameworks for Agents

Here is the toolkit we actually reach for, and the reasoning behind each choice.

Notable omissions are deliberate: LangChain, LlamaIndex, Haystack, Pinecone, Weaviate. We have tried them and run none in production today. They earn their keep only when you are doing something genuinely novel, such as complex multi-agent choreography across distinct processes. For the common pattern, an agent that uses tools to do a job, they are weight. We keep a small set of ai tools on purpose, because visual programming canvases and sprawling frameworks hide the one thing you need to debug: the exact sequence the model chose.

Working With Vision and Multimodal Models

Not every agent is text only. In our CMS, a vision role sends the actual image to a multimodal model to generate alt text and image descriptions, which keeps our published content accessible without manual effort. The same principle of role-based selection applies: pick a model that supports image input for that step, leave the rest of the pipeline untouched.

What Does an AI Agent Developer Do?

An AI agent developer designs the loop, the tools, and the guardrails that let a model act reliably in production. Day to day, that means scoping the task, mapping roles to models, writing tool schemas and their failure handling, wiring retrieval and memory, and instrumenting every call for observability. The job is less about prompt wording and more about software development discipline: bounded loops, clean interfaces, and audit trails.

The work also reaches into integration. Real agents connect to the systems a business already runs, from CRMs to internal APIs. We maintain production HubSpot and CRM integration code across several products, because an agent that cannot read and write the tools your team uses is a demo, not a product.

Where Agents Deliver Real Value

The agents that get adopted solve one severe, frequent problem rather than promising to do everything. Our internal CMS agent edits SEO articles; our vision agent generates alt text; other agents route data between business systems. Each is narrow, and together they cover complex workflows that no single monolithic agent could handle reliably. When several focused agents cooperate, each one stays simple enough to trust, and the system as a whole does far more than any one of them alone. That composition, many small agents over one large one, is the pattern we return to on almost every project.

Common Challenges in Agent Development

A few challenges show up on nearly every project, and planning for them separates a demo from a deployment:

• Tool failure handling. Models get arguments wrong. Without fuzzy matching and retries, complex tasks stall in loops.
• Cost control. An unbounded agent can burn budget fast. Role-based model selection and an iteration cap are the two biggest levers.
• Observability. If you cannot trace why the agent acted, you cannot fix it. Log every tool call and model response from day one.
• Scope creep. The broader the goal, the worse the agent performs. Keep each agent narrow and compose complex workflows from several of them.

Get these right and the rest is mostly the ordinary craft of shipping maintainable software with agents at its core.

Matt Sadowski

CEO of Mobile Reality

Transform Your Business with Custom AI Agent Solutions!

Leverage our expertise in AI agent development to enhance efficiency, scalability, and innovation within your organization.

• Expert development of modular and scalable AI software solutions.
• Integration of Large Language Models (LLMs) for advanced capabilities.
• Customized AI agents tailored to your business needs.
• End-to-end support from design to deployment.
• Enhance decision-making and operational efficiency with AI.

Book a consultationEstimate project

Summary

Building agents well is not about picking the trendiest framework. It is about a narrow goal, models chosen by role through OpenRouter, a hand-rolled orchestration loop you can read end to end, retrieval only where it earns its place, and relentless observability. That is the architecture we use to design, build, and operate agents for clients across fintech and proptech.

If you are starting this week, pick one narrow task, build one of these agents as a tool-calling loop against OpenRouter, and ship it behind a feature flag. Iterate from there. For deeper reading, see our companion guide to building an AI agent, our take on running an AI arbitrage agency, and the underlying AI arbitrage stack. If you want a team that has shipped this many times, our AI automation practice designs and operates these systems end to end.