QSOE 0.4 released

QSOE 0.4 is out today. In component terms:

0.3 gave you a disk that keeps what you wrote. 0.4 gives you a system you can listen to, and one you install rather than unpack. QSOE now plays sound on every board it runs on, on both kernels, and plays it cleanly while the rest of the machine is busy. The base system is a package, a package manager adds to it, and a C compiler installed that way has built and run its first program on QSOE's own filesystem. Under all that, the two kernels moved closer together than ever: one boot archive boots both, the C library is one binary for both, and QSOE/L now boots on every board, the K3 included. As always, QSOE/N on Skimmer and QSOE/L on seL4 carry one identical userspace, and much of what follows was found only because the same test ran against both.

Sound, on every board and both kernels

In 0.3 QSOE made no sound at all. In 0.4 it plays over HDMI from the Unmatched's GK208, over DisplayPort on the K3 and over HDMI on the VisionFive 2, with a fourth driver written for the VisionFive 2's 3.5 mm jack. deva-hda, deva-k3dp, deva-jh7110hdmi and deva-pwmdac are each their hardware over one engine, libaudio, behind a /dev/snd/pcmC0D0p node: frames arrive by write(), and everything else is a devcontrol() command. They are the same binaries on QSOE/N and QSOE/L.

Sound is in this release because it is the instrument for everything after it: a late buffer is heard long before it would be noticed any other way. So the engine had to count honestly. A dropped frame is defined in its header — a frame the hardware played from a period the client had not filled — and every counter is measured against that definition. A period the interrupt thread slept through is counted as a late wake, not mixed in with the drops. The engine is two threads and one ring with no lock, and the interrupt thread runs above the disk's priority. qplay plays WAV and Ogg Vorbis and takes every operand as a track, so qplay * plays a folder. When a device lacks a file's rate it converts through a polyphase windowed-sinc filter with an exact ratio: the VisionFive 2 takes only 48 kHz, and most music is 44.1. With -v it prints what the driver counted, down to a histogram of the interrupt thread's lateness.

What the instrument found first

It earned its place before the release was out. On QSOE/L, a song played beside the test suite tore: 407 underruns in one song on the VisionFive 2. Nothing in the player or the driver was slow. The player's writes were larger than seL4's IPC buffer, so taskman copied them; and the driver's interrupt thread woke its own dispatcher, 47 times a second, with a pulse that also went through taskman. A spawn off the NVMe holds taskman for tens of milliseconds, and the song tore every time. A third party on a data path is a party to every stall the data path can meet.

Both now bypass taskman. A player attaches a window at setup — pages taskman maps once into both ends of a connection — and its writes go straight to the driver. A process's pulses to itself go into a lock-free queue in the process, never a message. taskman's request rate during playback fell from about fifty a second to its background level, and the same song beside the same suite is clean on both kernels. Testing it by killing players in a loop found one more thing: the resource-server framework dropped a cancelled client's call unanswered, holding its window for good. It answers EINTR now, and windows are given back while the server lives. After twenty-one kills and twenty plays, the driver held nothing it should not.

Installed, not unpacked

In 0.3 the system volume was an image you wrote. In 0.4 it is an installed qsoe-base: /usr/bin, /usr/sbin, /usr/lib and /usr/share are links into its directory in the package store, and init selects the qsoe-base that matches the boot archive it booted. The test suite, the network tools, the GPIO and I2C drivers and the audio samples are packages beside it.

The package manager is qupkg, written in FreePascal and ported from the package manager of Serge Vakulenko's Braam system. The repository index is signed with Ed25519 by keys that an anchor shipped with the system vouches for, and it names every package by its SHA-256. Packages unpack once into an immutable store, and what is installed is a generation of links, switched in with one rename. A power cut leaves the old set or the new one, never half of each. A repository is any directory reachable by path, including one on another station over QSP.

And the first step towards self-hosting was taken through the package system rather than around it. qupkg install qjmcc brings a C compiler that runs on QSOE, with its assembler, the linker and the C library's SDK as dependencies, and qjmcc -o hello hello.c builds a program that runs: preprocessed, compiled, assembled and linked on QSOE's own filesystem. Finally, update-os fetches a new boot set — the archive and each board's kernels — from a development station over QSP into /usr/boot, all or nothing, keeping the previous set one step back. QSOE installs its own kernel.

Closer together: one C library, one boot archive

In 0.3, libc.so was built twice, once per kernel. In 0.4 the QNX-style native API — channels, messages, pulses, Sync*, threads — moved into libqsoe.so, built once per kernel and loaded by each taskman under the same name. libc.so above it is a single binary for QSOE/N and QSOE/L. Their interface is written down, one line per symbol in abi/qsoe-abi0.exports, and the build fails on a removed or unlisted export. There is no weak linkage anywhere: a program that needs a name nothing defines is refused at spawn, with the name, instead of running into a call at address zero.

The boot changed just as much. In 0.3 QSOE/L was one image nesting the elfloader, seL4, taskman and the userland, and the K3 ran QSOE/N only. In 0.4 the same modpkg.cpio boots both kernels. seL4 is patched to boot by mr-bml's multiboot3 command, the seL4 changes are a patch series, one change per patch, and taskman.elf went from 1.6 MB to 516 KB. The K3 runs QSOE/L on k3sel4, our port of seL4 to the SpacemiT SoC, booted the same way. Every board now runs both kernels.

Holding up under load

0.3 established that the disk keeps what it is given. 0.4 spent much of its time on the processes that write to it, and on what happens when hundreds of them come and go. Skimmer ran 270 passes of the suite on the K3 in one boot, after three races in thread teardown were found and fixed, each of them hit only once in a few hundred process ends. A write to a thread's flags from the wrong hart now panics, naming the file and line. QSOE/L ran 104 passes in one boot after two walls came down: a memory pool nobody counted, drained by 72 KiB per process, and a fault flag at bit 16 that made every request on a connection past the 65,536th look like a fault. A server is now told, by a disconnect pulse, of every connection a killed process left open.

On Skimmer a thread can be ended wherever it is: ThreadDestroy reaches a thread spinning in user mode on another hart, and every park is a cancellation point, so kill -9 of a runaway program means what it says. On seL4, a program placed on a different hart from the one taskman wrote it from used to fault on its first instruction, at random, because RISC-V keeps no coherence between data stores and instruction fetch. Every thread now starts through a fence.i.

It got faster, too. Program images arrive 64 KiB per message instead of 896 bytes, so a 500 KB spawn on QEMU went from 404 to 166 ms, and a whole suite pass on the K3 under QSOE/L from 19.9 to 11 s. On the network side, the Unmatched's Ethernet driver had been polling once a millisecond, locking every QSP transaction into one per poll. It receives on its interrupt now, and a 57.9 MB copy from a development station that took 73 s on QSOE/L takes 18 s there, and 12 s on QSOE/N.

And at the prompt

The shell runs jobs in the background, without a fork(): cmd & spawns the program itself, or a second shell for shell code, and jobs, fg, wait, kill %n and $! work. Ctrl-C reaches a command that a script runs. New utilities include tail -f, head, sha512sum, resize and workday; chown and chmod now follow the POSIX ownership rules on every server built on the framework. The shared suite grew from 502 checks to 793: all pass on QSOE/N, and 785 of 786 on QSOE/L. The one is a case QSOE/L cannot do yet, cancelling a blocked thread, reported rather than hidden.

Next

0.5 is the real-time package: priorities, interrupt service threads, priority inheritance and bounded paths end to end, with every step between an interrupt and the thread that services it accounted for. With it come the things 0.4 still lacks on that path: timed waits on both kernels, mutex priority inheritance, and a QSOE/L server that runs at its client's priority. Sound is how it will be heard.

Pre-built images are at github.com/qsoe-dev/dl, with a SHA512SUMS to check them by; the source is at gitlab.com/qsoe/os (make prepare checks out exactly the 0.4 set); the six manuals are at github.com/qsoe-dev/doc. The full release notes — with the exhaustive change list, and the known gaps stated rather than implied away — are at qsoe.net/qsoe_0.4.html.