Why can low priority processes affect the latency of an high priority process?

Question

I have tested the rt process's latency by cyclictest. And found that when I launch a few lower priority rt processes(priority is 40) the high priority rt process's latency will get bigger. I can't explain this.

cyclictest is one of the programs of rt-tests. https://github.com/jlelli/rt-tests/tree/master/src/cyclictest

Cyclictest measures the latency between when a timer expires and when the thread which set the timer actually runs. It does this by taking a time snapshot just prior to waiting for a specific time interval (t1), then taking another time snapshot after the timer finishes (t2), then comparing the theoretical wakeup time with the actual wakeup time (t2 -(t1 + sleep_time)). This value is the latency for that timer wakeup.

The cyclictest's fake code:

clock_gettime(CLOCK_MONOTONIC, &t1);
next = t1 + 1000;
clock_nanosleep(CLOCK_MONOTONIC, TIMER_ABSTIME, &next, NULL);
clock_gettime(CLOCK_MONOTONIC, &t2);
latency = t2 - next;

The test command is(-p80 means the test thread's priority is 80, the -n means useing clock_nanosleep to waiting for a specific time interval): ./cyclictest -p80 -n

The disturbing process is simple:

struct sched_param schedp;
schedp.sched_priority = 40;
sched_set_scheduler(0, SHED_FIFO, &schedp);
while(1) {
    srand(time(0));
    while(j++ < rand()%1000000);
    usleep(10);
}

In Linux kernel, no matter how many lower priority processes are there, as long as the high priority process is runnable, it will be executed immediately.

Can anyone explain why launching a few low priority processes can affect high priority process's latency?

Answer 1

1. At first, I would ensure more RT-ability by applying the Preempt_RT patch to your Linux kernel.
2. Next, if I am right, all timers are controlled by the softIRQ daemon (see ps -e | grep soft) which has a low default priority. That's why increase it for your core e.g. by chrt --fifo --pid 90 9 for cpu 0. Also switch off the disturbing printk as much as you can (e.g. sysctl kernel.printk='3 3 3 3') because it heavily breaks realtime.
3. Third, expired timers run their call with the inherited priority of the task which created the timer. That's why ensure, you use a schedulling mode other than SCHED_OTHER for it (e.g. SCHED_RR or SCHED_FIFO) and set a sufficiently high priority. SCHED_OTHER threads don't have a realtime priority, because for those threads you can just use the nice level which is between Linux kernel priority 20 and -20 AFAIRC.