AI Meets the Deadline
AI can plan the next move. The system still has to meet the next deadline. As robot models gain new capabilities, an architecture decision becomes more important: where should inference run, when may its answer be used, and what happens while it is unavailable? This film and executable experiment explore those choices around a 60 Hz controller.
The decision: let intelligence improve without blocking execution
In this design, inference runs in a separate worker. The execution loop consumes an admitted proposal or uses a defined baseline controller. A proposal carries the observation time and context version it was computed from. Admission and expiry checks apply before use. That preserves a place for richer planning while making availability, freshness and timing explicit responsibilities.
The tradeoff is real: the execution loop can meet every deadline while spending most of its time without useful AI input. A freshness limit can reject outdated proposals, but cannot establish that an action is correct. Resource isolation, action constraints, state estimation and the baseline controller require their own evidence.
Try the architecture under stress
Change the delay pattern, disconnect inference, or remove resource isolation. Both designs use the same latency rule. Their request times differ because blocking execution queues jobs, while the asynchronous worker skips requests when busy.
- Blocking: missed deadlines
- Loading
- Async: missed deadlines
- Loading
- Async: fresh-AI coverage
- Loading
- Peak response time: blocking / async
- Loading
Cyan: on time. Red: missed deadline. Amber: on time using fallback. Fresh-AI coverage counts jobs with an admitted proposal, including any late jobs.
Model assumptions and what these numbers mean
360 controller jobs are released at 60 Hz over six simulated seconds. Each has a deadline one period after release: 16.67 ms. The controller costs 4 ms; the asynchronous design adds a fixed 1 ms admission cost. These are chosen inputs, not measured hardware timings. The model completes queued jobs even if they finish after the release window.
Inference takes 8 ms, with the selected extra delay on every fourth call. The blocking design calls inference every sixth job. The asynchronous design offers a request every 100 ms to one worker, skips busy opportunities and transfers completed proposals atomically. The worker's first result becomes available after startup. An outage abandons requests that begin in its window after a 400 ms timeout.
Context changes every two seconds. A proposal is usable only if its observation age is within the selected limit and its context matches at execution time. Expired proposals trigger fallback. Shared contention adds 20 ms to one controller job per second in both designs. The model assumes no other operating-system jitter and does not simulate a plant, action quality, a model's accuracy or the safety of fallback.
A zero miss count here follows from the chosen execution costs and isolation assumptions. It is not a worst-case timing proof. All traces are deterministic and use the same model as the animation. Inspect the model source and its executable checks.
What recent AI work makes worth exploring
Keep moving while inference runs. Physical Intelligence's Real-Time Action Chunking studies asynchronous generation of action sequences and continuity between them. It addresses the harder question of compatible actions during motion. The simple freshness checks in this lab do not implement that algorithm.
Move inference closer to execution. Gemini Robotics On-Device explores local execution to address latency and connectivity, and describes integration with low-level safety controllers. Local inference can remove a network dependency; it still needs timing and resource analysis.
Generate more demanding test candidates. NVIDIA Cosmos provides world-model tools for physical AI. My proposed next experiment is to use generated scenario variations to search for failures, then reproduce relevant cases in an independently checked simulator. A plausible generated video is not a validated physical model.
What I would measure before changing the design
| Decision | Potential gain | Evidence needed |
|---|---|---|
| Larger or remote model | More capable planning | Successful task completion per unit time, inference cost, delay distribution and outage behaviour |
| Local model with isolated resources | Fewer network dependencies | Timing under competing load, memory limits, thermal throttling and update procedures |
| Longer proposal lifetime | More AI availability | Task outcomes when observations age or the environment changes |
| AI-generated test scenarios | Broader failure search | Reproducible failures, scenario validity and comparison with a fixed regression set |
An initial experiment could replay identical observation traces across candidate designs, inject delay and context changes, and report task success alongside timing and fallback use. Version the model, test inputs and admission rules together so an apparent improvement can be reproduced.
The timing mathematics, with assumptions stated
At frequency f, the period is 1/f. At 60 Hz this is 16.67 ms. A job misses its deadline when completion time exceeds release time plus that period. The frequency here is a teaching choice, not a qualification requirement.
For a count of missed deadlines, E[K] = Σ pi. If every frame has the same marginal miss probability p, this becomes Np. Independence is not required for that expectation. Correlation affects clustering and the distribution of the count. Four hours at 60 Hz gives 864,000 frames; at p = 10−5, the expected count is 8.64.
For an idealised parallel barrier, completion time is the maximum participant time, plus any omitted communication and coordination overhead. The probability that all participants meet their budgets equals a product only when their success events are independent. Shared resource stalls violate that assumption.
Explore the engineering projects · Read the published research