What you will learn
Specify interfaces, timing, and arbitration; revisit learned planning after the RL and world-model parts.
Before you begin: Robotics: estimation, planning & control; Drones: simulation before flight; Spiking neural networks
Use different timescales deliberately
A fast collision response and a long-term task planner solve different problems. A layered system can combine a reflex detector, visual perception, memory, a predictive model and planning. The important design work is in communication and arbitration: every component needs a clear input, output, timestamp and failure behavior.
An architecture is a set of contracts
A perception layer might report object positions and uncertainty. A planner proposes an intention. A controller accepts bounded, physically meaningful commands. Language-model output can be useful for task reasoning, but the diagram does not grant it unrestricted actuator authority. A stale plan should not overrule fresh evidence of an obstacle simply because its text sounds confident.
Neural-network perspective
Each module is a hypothesis about what information a task needs. Adding an SNN, VLM or world model only helps if it improves measurable outcomes after accounting for latency, compute and integration failures. Start with a working simple pipeline, add one component, and compare. A modular architecture makes these ablations easier to interpret.
Budget the complete path
For serial processing, stage delays add. For parallel processing, queueing and synchronization complicate the budget. An average below a deadline is insufficient if occasional long stalls cause failures. Measure distributions and deadline misses; the small sum below is an illustrative deterministic budget.
SEE THE RELATIONSHIPS
From structure to behavior.
MAKE THE IDEA PRECISE
The mathematics.
| Symbol | Meaning |
|---|---|
| L | Stage or total latency, milliseconds |
| D | Task deadline, milliseconds |
| slack | Remaining time budget; negative means the deadline is missed |
Build the equation, step by step
- If stages execute serially with no overlap, add their latencies to estimate end-to-end processing delay.
- Subtract the total from deadline D to get slack. Negative slack means that sequence cannot meet the deadline even before considering unmodelled overhead.
- Pipelines and parallel branches need a dependency-based critical-path calculation. Average latency also hides jitter, queueing and rare long delays that matter to control.
WORK THROUGH THE NUMBERS
Worked example
Chosen stage costs of 3, 5, 2 and 4 ms total 14 ms. Against a 20 ms deadline, deterministic slack is 6 ms. Real tail latency needs measurement.
A small Python experiment.
Python 3 · Standard library only · Synthetic educational example. Run this in your own notebook or a local file.
stages_ms = [3, 5, 2, 4]
deadline_ms = 20
total = sum(stages_ms)
print(f"latency = {total} ms; slack = {deadline_ms-total} ms")Expected output
latency = 14 ms; slack = 6 ms
Change one quantity, predict the result, and then run the code again. Explain any difference before adding complexity.
Test your understanding.
Try it: Add a 12 ms queueing delay. Which design change would you test first, and why?
Self-check: Does positive average slack guarantee a deadline?
No. A deadline guarantee concerns the relevant worst case or a specified probabilistic bound, including communication and scheduling. A positive mean can coexist with frequent overruns. Record the latency distribution and define what the system does when a result arrives late.
Where this model stops: A layered diagram does not demonstrate intelligence or reliability. Interface tests and system evaluation are still required.
A research question: Can a simpler perception/controller pair match the full architecture under the same deadline?
Record evidence in your mastery worksheet →THE NEXT LAYER
Chapter depth map.
This introduction provides a first model. The following chapters are outlines for deeper future editions; their full treatments have not yet been published.
Five interacting levels
15.1 Reflex and perception · Outline
- 15.1.1 Event camera and SNN immediate-response layer
- 15.1.2 CNN/YOLO, depth, tracking and SLAM alternatives
- 15.1.3 Timestamped observations and confidence
- 15.1.4 Resource sharing and backpressure
- 15.1.5 Compare reflex response with conventional detectors
15.2 Cognition, planning and control · Outline
- 15.2.1 VLM/LLM interfaces and grounded observations
- 15.2.2 World-model and memory concepts preview
- 15.2.3 Task planning, reinforcement learning and reasoning
- 15.2.4 Robot/drone controller executes bounded commands
- 15.2.5 Missing capabilities must remain explicit
System contracts
15.3 Timing and arbitration · Outline
- 15.3.1 Fast and slow loops with explicit budgets
- 15.3.2 Message schemas and unit conventions
- 15.3.3 Priorities, conflicts and stale plans
- 15.3.4 Runtime monitors and safe fallback controllers
- 15.3.5 Avoid direct unconstrained language-to-actuator control
15.4 Evaluate the whole system · Outline
- 15.4.1 End-to-end tasks and baseline architecture
- 15.4.2 Ablate each layer independently
- 15.4.3 Failure propagation and recovery
- 15.4.4 Revisit after Parts XVI, XVIII and XX
- 15.4.5 Measure whether added complexity earns its cost
Read with the original sources.
These readings support the concepts and deeper investigation. The worked numbers and illustrations above are teaching examples.
Primary paper / author manuscript
Vaswani et al. (2017): Attention Is All You Need ↗
Read after matrix multiplication and optimization. Reconstruct query-key-value shapes and distinguish sequence modeling from embodied control.
Official engineering documentation
PX4 — official simulation guide ↗
Begin with simulation and versioned interfaces. Simulation success alone does not establish hardware flight readiness.