Vision AI & Agentic Workflows for Heavy Machinery
Off-Highway Machine Agent
A vision system tells you what it saw. An agent decides what happens next, and records who approved it. This is an interactive walkthrough of that step, built on MDMA, the open-source framework for governed AI workflows.
From Detection to Decision
A vision model on an excavator or a haul truck produces detections: an object class, a confidence score, a distance, a frame. That is a reading, not a decision. The gap between "a person was detected in the swing radius" and "the hydraulics were released again, by this supervisor, at this time" is where the operational value sits, and it is the part that is usually built ad hoc.
An agentic layer closes that gap. It takes the machine event, assembles the context a human needs, walks the required checks, and stops at a human sign-off before anything actuates. Every step is written to an append-only, hash-chained audit record, so months later you can show not just what the machine did, but who authorised it and on what evidence. MDMA is the framework that makes that flow declarative rather than bespoke.
Interactive Machine Workflow Demo
Pick a scenario and walk it end to end. The agent collects the event context, shows what the machine reported, runs the pre-action checks, and routes the decision to a human. Nothing actuates until someone approves it.
Every machine ID, detection frame, sensor reading and threshold in this demo is synthetic sample data. No real machine, site or fleet is represented.
Audit record
Every step you take above is appended here as a hash-chained entry, and each one carries the hash of the entry before it. Press "Verify chain" to check the record end to end, then press "Tamper with an entry" to rewrite one entry's payload the way an attacker with log access would, and verify again.
Complete a step in the demo above and it will appear here.
How the Machine Agent Works
The same four stages carry both scenarios. Each stage is one MDMA component, and each transition is recorded.
Event Context
The machine event arrives, either a vision detection or a telemetry alert, and the agent collects the context a human needs to judge it.
Evidence
Detection frames with confidence and distance, or a sensor trend against its threshold, presented as structured evidence rather than a raw log.
Pre-Action Checks
The checks that must be cleared first: area cleared and spotter confirmed, or part availability and downtime window agreed.
Human Sign-Off
A site supervisor releases the lockout, or a fleet manager approves the work order. The decision, the person and the reason are all recorded.
What the Framework Provides
The governance parts of an agentic machine workflow are built into MDMA rather than written per project.
Human-in-the-Loop Approval
Nothing actuates on the agent's own authority. An approval gate names the roles that may sign off, requires a reason, and blocks the downstream action until a human decides. Denial is a first-class outcome, not an error path.
Hash-Chained Audit Record
Every action, from form submission to check completion to approval decision, is appended to a log where each entry carries the hash of the previous one. Editing history breaks the chain and verification says exactly where. The demo above uses SHA-256 in the browser, and the hash function is pluggable.
System Integration
Approved outcomes post to whatever comes next, whether that is a machine controller, a dealer service system, a maintenance backlog or a site safety system, through MDMA's webhook component, with dynamic bindings, retries and timeouts. The demo above simulates that call rather than making it.
Environment Policy Engine
Control which actions are permitted per environment. Block outbound calls in staging, restrict approval gates to production, require a minimum set of checks before sign-off. Policies are declared alongside the workflow, not buried in code.