How to set visible CPUs for (python) programs in Linux？

Question

When running (python) programs, I can use below code to set it only using No.1 and No.2 GPU:

CUDA_VISIBLE_DEVICES=1,2 python xxx.py

So How can I set CPU devices for this same (python) program? Something like:

CUDA_VISIBLE_DEVICES=1,2 CPU_VISIBLE_DEVICES=0-10 python xxx.py

---

Why I need this?

I have a server which has multi cpus and gpus. The gpus are divided as needed by multi user using CUDA_VISIBLE_DEVICES=0,1. But the cpus are not. So one user's program can take too much cpu power, slowing down other users.

I need some ways to divide the cpu as needed.