A pthread with SCHED_RR and higher real time priority failed to preempt a kthread in kernel module with lower priority

Question

Preface:

I have two threads: one kernel thread and one userspace pthread. I assume pthread set to SCHED_RR with higher rt priority should preempt a kthread with SCHED_RR and lower priority while both of them running one the same cpu.

However my test failed. kthread keeps running and pthread did not progress.

NOTE: I add both C and C++ tag for this post since my example are one with C++ and one with C. If any tag suits this post, please edit it and remove C and C++ tag.

Experiment prepare:

1. Ubuntu 18.04. ( I assume it's workable whether on normal linux and RT linux )

2. kthread is created with SCHED_RR and rt priority 50.

   • kthread is just a infinite loop with udelay() which should not disable preeption.
   • kthread would check kthread_should_stop() before calling udelay

3. pthread is created with SCHED_RR and rt priority 70.

   • pthread is a busy infinite loop
   • userprogram created pthread would call pthread_join to wait until pthread end by ctrl+c

    while( true ){
        sleep( 1 );
        std:: cout << "wake\n";
    }

result

If kthread run first, and userprogram would stuck after pthread_join called. Output string from pthread cannot be received until kernel module removed ( calling kthread to stop ).

My assumption is pthread never start since no output printed on console. But I'm not sure if this assumption correct.

Reference

• pthread failed to preempt kthread

  • Got this post stating RT linux is required.

• How “real-time” are the FIFO/RR schedulers on non-RT Linux kernel?

  • A comment in the post mentioned even CONFIG_PREEMPT is disabled, it should work.

Code to reproduce the experiment

==== This is the wall of text with test code ====

kthread.c - a kernel module

#include <uapi/linux/sched/types.h> // For sched_param
#include <linux/kernel.h>
#include <linux/module.h>
#include <linux/init.h>
#include <linux/kthread.h> // kthread_create(), kthread_should_stop(), kthread_stop()
#include <linux/delay.h> // msleep()
#include <linux/sched.h> // For struct sched_param, sched_setcheduler()

#define MY_THREADNAME "kthread example"
#define MY_MODULENAME "kthread occupy cpu1 module"

#pragma GCC diagnostic ignored "-Wdeclaration-after-statement" // Ignore C90 declaration convention since gcc has extension for it.

MODULE_LICENSE( "GPL v2" );
MODULE_AUTHOR( "TEST" );
MODULE_DESCRIPTION( MY_MODULENAME );
struct task_struct *m_task1;
static int m_t1id = 1;

int thread_fn(void *data)
{
    int id = *((int*)data);

    int ret = 0;
    int cpu = -1;
    cpu = get_cpu();
    put_cpu();
    pr_info( "IN THREAD FUNCTION %d, CPU is %d \n", id, cpu);
    while(!kthread_should_stop()){
        // try to busy blocking cpu, but udelay doesn't disable preemption.
        udelay(1000);
    }
    pr_info( "EXIT from thread function 1\n");
    return 0;
}

static int __init init_my_module(void) {

    pr_info( "Hello, %s!\n", MY_MODULENAME );

    // Get cpu will disable preemption, so must put_cpu to enable preemption
    int cpu = get_cpu();
    put_cpu();
    pr_info( "Current cpu for initializing is %d\n", cpu );
    pr_info( "Current pid is %d\n", current->pid );

    // threadfn, data, and printf-style name. Created thread would be suspended, need to wake up.
    m_task1 = kthread_create(&thread_fn,(void *)&m_t1id,"testing kt%d",m_t1id);
    pr_info( "T1 pid is %d", m_task1->pid );
    kthread_bind(m_task1,1 );

    // Set Realtime priority 
    struct sched_param param = {50}; 
    sched_setscheduler( m_task1, SCHED_RR, &param );
    pr_info( "T1 effective prio AFTER set policy= %d", m_task1->prio ); 
    wake_up_process(m_task1);
    return 0;
}

static void __exit exit_my_module(void) {
    kthread_stop( m_task1 );
    pr_info( "Bye, %s!\n", MY_MODULENAME );
}

module_init( init_my_module);
module_exit( exit_my_module);

Makefile for kthread.c

#Test on local ubuntu
KVERSION := $(shell uname -r)
KERNEL_DIR = /lib/modules/$(KVERSION)/build

PWD := $(shell pwd)
MODULE_NAME = kthread

obj-m  = $(MODULE_NAME).o

all:
        make -C $(KERNEL_DIR) M=$(PWD) modules

clean:
        make -C $(KERNEL_DIR) M=$(PWD) clean

pthread.cpp

Compile commandline for pthread.cpp is g++ pthread.cpp -pthread

#include <pthread.h>
#include <sched.h>
#include <iostream>
#include <sys/resource.h> // to use getrusage
#include <unistd.h>

void* helloworld ( void *arg ){

    int cpuid = sched_getcpu();
    int sum = 0;

    std::cout<< "Hello pthread on cpu " << cpuid << std::endl;
    pthread_t pself = pthread_self();
    int policy = 0;
    struct sched_param sparam;
    pthread_getschedparam( pself, &policy, &sparam );
    std::cout << "Current thread Policy: " << policy <<  "  prio:" << sparam.sched_priority << "\n"; 
    int err =  pthread_setschedprio( pself, 99 );
    std::cout<< "set prio ret:"<< err <<"\n";
    // keep sleep to check process priority
    while( true ){
        sleep( 1 );
        std:: cout << "wake\n";
    }
    return NULL;
}

int main(void){
    std::cout<<" pthread create\n";
    pthread_t handle;
    pthread_attr_t attr;
    cpu_set_t cpus;
    struct sched_param sparam;
   
    int error = 0;

    // Setting SCHED_FIFO must run program in sudo
    int policy = SCHED_RR;
    CPU_ZERO( &cpus );

    // Set bit cpu0 to enable in mask.
    CPU_SET( 1,  &cpus );

    pthread_attr_init( &attr );
    
    // Let pthread using specified attribute explicitly
    error = pthread_attr_setinheritsched( &attr, PTHREAD_EXPLICIT_SCHED );
    std::cout << "set inherit sched result: "<< error << "\n";

    error = pthread_attr_setschedpolicy( &attr, policy );
    std::cout << "set policy result: "<< error << "\n";

    // _np means "non-portable"
    pthread_attr_setaffinity_np( &attr,sizeof(cpu_set_t), &cpus );

    // Set priority
    int max = 70;//sched_get_priority_max(policy);
    sparam.sched_priority = max;// ( 4* min + 6*max )/10; 

    std::cout << "Set priority: " << sparam.sched_priority << "\n";

    error = pthread_attr_setschedparam(&attr ,&sparam);
    std::cout << "setschedparam: " << error << std::endl;

    // before create thread, raise main thread as higher priority
    error = pthread_setschedparam(pthread_self(), policy, &sparam);
    std::cout<< "main thread set prio " << error << std::endl;

    error = pthread_create( &handle, &attr , &helloworld, NULL );
    std::cout<< "pthread_create " << error << std::endl;
    void* retvalue;
    pthread_join( handle, &retvalue );

    pthread_attr_destroy( &attr );
    return 0;
}

Answer 1

The short answer: CONFIG_PREEMPT enabled is required.

I tried on CONFIG_PREEMPT linux and pthread does preempt the kernel thread with higher priority.

If you're interested in build a kernel, following links are for ubuntu 18.04.

• How to enable CONFIG_PREEMPT option in linux kernel

I choose to my own question but would not accept it since it's just the tested behavior but not why. I'd like to receive an anwer explaining CONFIG_PREEPT.