Without hardware feedback, frontier agents had 0% deployment success in the benchmark. With a hardware-in-the-loop setup, agents achieved successful deployments and sometimes surpassed human expert results.
The tiny-computer problem
A model that runs beautifully on a laptop can fail completely on a microcontroller. The device may run out of memory, overheat, draw too much power, or simply fail to compile. These constraints matter for wildlife cameras, wearables, health devices, and privacy-preserving sensing systems that need local inference.
Today, expert engineers often tune these deployments by hand. They compress models, adjust firmware, measure real hardware behavior, and iterate through messy tradeoffs.
What Embedded Arena changes
Embedded Arena turns that engineering loop into a benchmark. An agent cannot merely write plausible code. It must compile, flash, run, measure, and respond to feedback from physical hardware.
That feedback is the key. In our experiments, hardware-in-the-loop optimization enabled large compressions: 250x for vision models with less than 3.3% accuracy loss, and 400x for audio with less than 6% feature error rate loss. We demonstrated the approach in an elk-detection camera trap and a phonetic-transcription wearable for child development research.
Why it matters
The world does not run only in cloud APIs. Many important AI systems need to work offline, cheaply, privately, and on a battery. Embedded Arena makes those constraints visible to agents, and it gives the research community a way to measure whether agentic coding systems can handle physical reality rather than just software benchmarks.
Interactive Demo
Could a tiny field camera ignore the wind?
The original motivation was practical: a farmer needed to know when crop-damaging animals entered large fields, without maintaining a power and WiFi grid. Try the compressed animal detector below with a preset image or your own upload. With hardware-in-the-loop feedback, agents can easily compress models like this to run battery-free in the field.
Model loading...
The model runs in your browser using ONNX Runtime Web. No image is uploaded.