DeepMind SMART Treats Design Docs as ML Perf Source
Google DeepMind researchers, with co-authors from MIT, Stanford, and Google, posted arXiv paper 2609.05364 describing SMART, a symbolic performance-modeling library for machine-learning systems whose main branch contains almost
PromptCrates Editorial
Staff Writer
Google DeepMind researchers, with co-authors from MIT, Stanford, and Google, posted arXiv paper 2609.05364 describing SMART, a symbolic performance-modeling library for machine-learning systems whose main branch contains almost no code. Instead, the repository is a directed acyclic graph of roughly 50 natural-language design docs totaling about 9,000 lines of prose; coding sub-agents regenerate the implementation from those docs, and humans edit documentation only. Full regeneration takes 1.5 to 3 hours and costs about 100 US dollars in Claude Code API spend per rebuild, while regenerated implementations match hand-audited references—including DeepSeek V3 serving on a TPU pod slice—to round-off precision.
Why design docs become the durable artifact
Performance modeling for ML systems is hostile terrain for long-lived software. Model architectures and accelerator stacks both churn, so abstractions that assumed uniform transformer layers break when mixture-of-experts routing, latent attention, or heterogeneous inference phases arrive. Traditional frameworks absorb that churn as perpetual refactors. SMART’s wager is that regenerating an entire library from crisp specs is now cheaper than paying down incremental patch debt, especially once AI coding agents can rebuild modules in topological order.
The main branch therefore stores a DAG of self-contained markdown design docs spanning TPU topology, collectives, numerics, schedulers, and a frontier model catalog that covers dense, MoE, latent-attention, and robotics or vision-language-action variants. Dependency edges are machine-discovered by read-only agents rather than hand-maintained. An orchestrator walks the DAG, assigns a coding sub-agent per doc, logs where agents struggle, and only promotes builds that reconcile against hand-built references. Humans never patch generated code as the source of truth; they rewrite the prose that failed to pin down semantics.
Two ingredients make regeneration reliable, the authors argue. First, an opinionated design-doc style built around step-by-step worked examples that act as in-context demonstrations—exact intermediate shapes, values, and closed-form cost expressions. Second, a minimal recursively defined operator IR with SymPy cost expressions, a fast analytical roll-up mode for large sweeps, and a slow modulo-scheduling mode for fine-grained schedule studies. Together they attack both incremental-generation debt and context-window myopia that appear when agents edit fragments of a mature codebase without seeing global invariants.
How the symbolic IR prices models on TPUs
Models are authored in a thin Python-embedded tracing DSL that never constructs operator nodes by hand. Decorated blocks become named subgraphs; loops become interior nodes; builder calls emit system-priced leaves such as MXU matmul tiles, VMEM loads, or ICI collectives. Every dimension stays a SymPy symbol, so one trace serves a design space of batch sizes, sequence lengths, and mesh axes. Distribution uses sharding annotations; many collectives are inferred from operand and output shardings, while layout-moving collectives such as expert-parallel all-to-all remain explicit.
Fast mode scales leaf costs by enclosing trip counts and applies analytical overlap transforms in a roofline spirit, evaluating closed-form expressions quickly enough for thousands of design points. Slow mode modulo-schedules each loop into a resource reservation table and rolls initiation intervals up the tree for points that sweeps flag as interesting. Because costs propagate symbolically, numeric binding happens only at the edge—one substitution per design point—so a single symbolic build serves an entire sweep. Generated tests can assert exact expected cost expressions stated in the docs.
The validation claim is concrete: regenerated implementations reproduce hand-audited reference models, including DeepSeek V3 serving on a TPU pod slice, to round-off precision. That bar matters because a docs-as-source system fails if regenerated code only “looks right.” Round-off agreement against audited references is the authors’ evidence that prose plus a minimal IR can carry co-design tools without a sacred handwritten core.
What SMART means for ML systems teams
For accelerator and systems groups, SMART reframes documentation from after-the-fact explanation into the build product’s parent. If a full rebuild is 1.5–3 hours and about 100 dollars, continuous regeneration becomes a weekly habit rather than a crisis rewrite. Orchestrator logs that highlight which docs confuse agents become a map of underspecified intent. Dynamic model routing—stronger models on foundational docs, cheaper models downstream—keeps cost inside high-tier usage budgets the paper pegs near 20 percent of a weekly Claude Max allotment for a full rebuild.
Readers tracking lab tooling can pair this paper with PromptCrates coverage of Anthropic’s automated alignment researcher paper, Google DeepMind’s double-blind benchmark enclave, and agent evaluation threads such as T1 terminal agent RL for long horizons. SMART is not a chatbot feature; it is an argument that co-design libraries should treat natural-language specs as durable and code as disposable.
Documented facts stay anchored to arXiv:2609.05364. SMART’s main branch is almost code-free; about 50 design docs and 9,000 prose lines cover TPU topology, collectives, numerics, schedulers, and frontier model families; regeneration takes 1.5–3 hours at roughly 100 dollars Claude Code cost; worked-example docs plus a recursive Op IR with SymPy costs enable fast analytical and slow modulo-scheduling modes; regenerated impls match hand-audited refs including DeepSeek V3 on a TPU pod slice to round-off precision; authors claim design docs—not code—are the durable artifact for ML-systems co-design tools.
Primary source: arXiv 2609.05364 Design Docs Are All You Need.
