`SCHED_IDLE` processes vs KVM

I would like to run a computation on the lowest possible priority, by using SCHED_IDLE:

On this host, there is a VM (using KVM) that occasionally uses a lot of CPU. My expectation would be that the VM would get precedence over the idle task, but they appear to be sharing the CPU pretty much exactly 50:50.

Is there a way to run a task with a low priority if the machine is otherwise only running VMs?