Crossing the Hyper-Thread Boundary

Modern processors have multiple CPU cores, and the physical CPU cores in turn have two logical CPU cores. The latter permit the physical core to run two independent programs simultaneously, an approach known as hyper-threading. Hyper-threading uses CPU resources more efficiently, but it exposes transient execution vulnerabilities, in which a program running on one hyper-thread is able to extract data from the program running on the other hyper-thread. An example of such an attack is MDS.

exe.dev runs virtual machines on behalf of different users. We need to protect against the possibility of one user exploiting a vulnerability to extract data from another user running on a different hyper-thread of the same CPU core. Fortunately the Linux kernel supports core scheduling cookies to control which processes are permitted to share a CPU core.

It is straightforward to give each VM an independent cookie, meaning that two different VMs never run on the same CPU core. However, in practice this leads to measurably inefficient use of physical CPU cores. So we instead implemented a more efficient, but still secure, mechanism: the VMs of each team use an independent cookie. This means that two different VMs from a different team (or from a different user for users not on teams) never run on the same CPU core. To put it another way, we assume that different members of the same team trust each other.

Of course, for various reasons, some teams do not have trust among all their VMs. An admin of those teams can run ssh exe.dev team settings core-sharing off to prevent their VMs from sharing CPU cores. (Currently VMs that do not belong to a team never share cores with other VMs, even VMs from the same user; if that changes someday we will introduce a similar setting for individuals.)