Taming Linux Jitter: Achieving Sub-50µs Cycle Times with PREEMPT_RT and Xenomai

1. The Challenge: Soft PLC on a $30 SoC

Tainuo’s next-generation motion controller needed to run a soft-PLC runtime doing 20 kHz closed-loop PID control on six servo axes simultaneously. The target hardware: a $30 Allwinner A40i quad-core Cortex-A7. No FPGA, no dedicated DSP — just plain ARM cores running Linux.

The problem was obvious: standard Linux, even with SCHED_FIFO, routinely exhibits 200–500µs scheduling jitter on this class of hardware. For a 20 kHz control loop (50µs period), that’s a non-starter. Missing the deadline means a phase error that accumulates and destabilizes the mechanical system.

2. The PREEMPT_RT Foundation

The first step was straightforward: apply the PREEMPT_RT patchset and configure the kernel with CONFIG_PREEMPT_RT_FULL=y. This converts essentially all Spinlock sections into preemptible mutexes, making the kernel nearly fully preemptible.

Key kernel config changes:

With PREEMPT_RT alone, we brought worst-case scheduling latency from ~480µs down to ~80µs. Better, but still not meeting the 50µs budget.

3. Adding Xenomai 4 (EVL Core)

Xenomai 4’s EVL core runs a separate real-time nucleus alongside Linux, intercepting interrupts before they reach the Linux kernel. Real-time threads are scheduled by the EVL core directly, bypassing the Linux scheduler entirely.

We ran cyclictest for 72 hours straight under a synthetic load (hackbench + Ethernet IRQ storm + memory pressure). The results:

With Xenomai, the worst-case latency over 72 hours was 31µs — comfortably within the 50µs budget. Even with all four cores under heavy IRQ load, the control thread never missed its 20 kHz cadence.

4. Hardware Considerations

Not every SoC is suitable for real-time work. We tested three candidates before settling on the A40i:

• RK3399: powerful but aggressively power-gated — deep C-states added 200µs wakeup latency that we couldn’t reliably suppress.
• i.MX6ULL: excellent for RT but only single-core Cortex-A7, insufficient for concurrent vision + control.
• A40i: quad-core, decent clock gating granularity, no power-gated idle states, and solid mainline PREEMPT_RT support.

The lesson: choose your SoC for deterministic wake-up behavior, not peak performance. A slow core that wakes in 2µs is better than a fast core that occasionally takes 200µs to come out of a deep sleep.

5. Production Results

The motion controller shipped in Q2 2026. Running six 20 kHz PID loops, the system uses approximately 35% of one Cortex-A7 core — leaving headroom for the PLC runtime, Modbus TCP bridge, and a local HMI served over HTTP. Total BOM cost reduction vs. the previous FPGA+DSP architecture: roughly 60%.

What made this possible wasn’t just kernel configuration — it was the combination of hardware-aware SoC selection, PREEMPT_RT for soft real-time paths, and Xenomai for the hard real-time control thread. Each layer handles the latency class it’s best suited for.