|
Hi HN — we build atopile, an open-source language and compiler for designing circuit boards in code. People kept asking us "can AI just design the board?" — so we measured it. EEBench is a set of original EE tasks: design-from-spec, datasheet comprehension, design review, picking real parts that actually exist. Agents submit design source, not prose. The grader runs compiler checks and simulations, and scores measured behavior — transients, thresholds, tolerance corners — against the spec. No human graders, no LLM-as-judge. When a score says the rail sagged below 3.0 V during brown-out, that's a waveform, not an opinion. Things that surprised us: The frontier cluster spans roughly 45–72% (Claude Fable 5 on top at 71.7%).
Open-weight models that do well on coding benchmarks score 5–13% here.
Cost per task varies 15x between models with similar scores.
Honest limitations: the task set is curated from a larger pack to separate the frontier models, and it's growing. The pack is held out to keep it out of training data, so you can't self-run it yet. If you want a model on the board, email us and we'll run it. Happy to go deep on the grading harness — ask us anything. |