QSOE/N v0.18: a second board, and a second compiler

Two things happen in this release, and they turn out to be the same thing seen from opposite ends.

QSOE/N boots to an interactive shell on the SpacemiT K3 — the second piece of silicon it has ever run on, after the SiFive Unmatched. And the entire operating system — kernel, taskman, libc.so, the runtime linker, every program under /bin and /sbin — now builds with a second compiler, jmcc, and boots on it too.

Neither was on the roadmap as "portability work". Both were the same lesson delivered twice: stop assuming the thing you were tuned for. The K3 broke every assumption baked into the hardware — one machine, one DRAM base, one interrupt geometry. jmcc broke every assumption baked into the toolchain — position-independent code, hard-float, GCC's particular generosity about relocations. Each found bugs the other never would have, and by the end the system is more honestly portable than it was before either one started.

The K3 does not resemble the machine we were tuned for

The Unmatched taught QSOE/N how to leave the emulator. The K3 taught it that leaving the emulator once is not the same as being portable.

It is a hostile target in all the interesting ways. Its DRAM starts at 4 GiB, not at zero. It numbers its harts 0..15. It is heterogeneous: eight X100 application cores and eight A100 AI cores, the latter carrying a quarter of the L2 and a narrower ISA. And it does not boot alone — it runs management firmware on companion cores, an OpenSBI and an esos and two RT24 remoteproc images, all of it described in a device tree that lists seventeen /reserved-memory children before the kernel has reached its first page table.

Getting from power-on to a shell prompt on that machine was a sequence of boot blockers, each found the same way: on the board, by a silence where output should have been.

A board is now a first-class concept. QSOE/N gained a Kconfig board choice — several boards buildable from one tree — and stopped hard-coding the machine it was first tuned for. The board turns out to be a purely link-time property: KERNEL_LOAD_PA arrives from a linker --defsym, and a missing one is a link error rather than an image quietly built for the wrong DRAM base. That mattered more than it sounds: the old identity-map loop started from a compile-time constant, and on a part with DRAM at 4 GiB it would have mapped two gigabytes of megapages straight over the taskman and initrd regions and panicked at the first user blob.

The load address was off by one hex digit, and the board went dark. The K3 image wanted 0x102200000 — 4 GiB plus 34 MiB, safely above the vendor carve-outs. It was written 0x100200000, one digit short: 4 GiB plus 2 MiB, which lands exactly on the reserved region the firmware gives to a companion core. head.S zeroes some 15 MB of BSS from the load address, so the firmware was overwritten before the first console call — and then the kernel tried to speak to the console through firmware that no longer existed. No output, no panic, no clue. Just a board that had gone quiet.

The interrupt geometry is genuinely different, not just relocated. The K3 is an AIA machine — APLIC plus IMSIC — and two assumptions that had never been wrong on QEMU virt were wrong here. The IMSIC file stride is not one page: the K3 declares riscv,guest-index-bits = 6, which makes the real stride 256 KiB, and every hart above hart 0 would have had its interrupts written into the wrong page — nothing faulting, the interrupts simply never arriving, which reads as a broken device rather than a broken address. And the S-domain APLIC used to be found by way of the console UART; now it is identified by its msi-parent, which is what actually defines it, so a board whose serial port we do not recognize no longer takes the interrupt controller down with it.

The console itself had to be learned. The K3's UART is the xscale lineage — spacemit,k1-uart — with its own register shift and width, all of it now published to the userspace driver so the driver stays board-independent.

The reserved-memory tables were sized for a simpler world. Eight exclusions was plenty for QEMU virt and the FU740. The K3's seventeen reserved children, plus the FDT and initrd and firmware span, needed three times that before the first page table exists. The tables now hold 64 exclusions, and overflow is a deliberate panic that names the region that did not fit — because with a table this size, the useful question is which node was being registered, not that a count was reached.

And the device tree lies about how much memory there is. The firmware hands over a static blob declaring 8 GiB; the board has 16 GiB starting at 4 GiB. A bootloader that allocates the initrd high, through EFI's AllocateMaxAddress, then puts it around 19 GiB — perfectly good memory that the old pmap flatly refused to boot on, because it insisted the initrd sit inside the RAM the kernel manages. It never needed to. That requirement is gone; the check that remains is the one that matters — an initrd overlapping the firmware or the kernel image is memory already in use.

Sixteen cores, and only half of them belong to real-time work

The heterogeneous split needed a policy, and the policy QSOE/N chose is to refuse to have one. The kernel discovers which harts are the AI cores — by the device tree's own cpu-ai property, not by matching a vendor model string, because a name is the wrong thing to hang a scheduler decision on — records the distinction, and publishes it three ways: to per-hart kernel code, to userspace through the sysmap, and to the operator as a single boot line. Then it enforces nothing. Placement of interrupt service and hard-real-time threads is a decision made where the latency requirements are actually known. The kernel's whole job here is to make the difference visible, because a machine that is quietly slower on half its cores is not something anyone should have to discover by benchmark.

Every line the post argues for is in that capture. gidxbits=6 stride=0x40000 is the IMSIC geometry, derived rather than assumed — 0x40000 is 256 KiB, the real per-hart file stride. The managed bank opens at 0x102000000, above the first 32 MiB of vendor firmware; the initrd sits near the top of RAM at 0x4f8cd0000, memory the old pmap would have refused. The seventeen reserved-memory nodes expand — with the firmware, initrd and kernel spans — into the 29 exclusion(s) the walk logs, comfortably under the raised ceiling and well past the old one of eight. mask 0xff00 marks harts 8–15 as the auxiliary A100 cores, held back from interrupt and hard-real-time work. And the last four lines are the whole of “Not in this release,” captured live: no ECAM, no mainfs=, no root filesystem — and then a shell, and a machine that answers back.

jmcc: the whole OS, built by a compiler that trusts nothing

Running the entire build through jmcc — the Johnson-Magnusson C Compiler, now a selectable toolchain alongside GCC — did to the software what the K3 did to the hardware. GCC had been quietly carrying the system on a set of conveniences: it built everything position-independent, it reached globals through the GOT, it papered over a whole class of relocation. jmcc does none of that, and every place the system had leaned on GCC's habits, it fell over — usefully.

The loader was silently skipping R_RISCV_COPY relocations. It never mattered while every binary was PIE and reached imported data through the GOT. A non-PIE executable importing a data object from a shared library — devb-nvme reaching stderr in libc.so — got a private copy that read as zero, and faulted on the first setvbuf. Now the loader binds COPY relocations eagerly, and the storage driver comes up.

Worker threads were running with the wrong global pointer. The thread trampoline lives in libc.so and set gp to its own __global_pointer$, but gp is process-global and must be the main executable's — invisible under PIE, fatal for a non-PIE qsh whose SIGCHLD handler then read its job list out of libc's data segment and crashed on the first child exit. The trampoline now carries the creating thread's gp forward instead of recomputing it.

There were smaller ones — taskman's stack grown to four pages because jmcc's frames spill everything to memory and overflow a single page; a per-ABI multilib query so the soft-float runtime links against the soft-float image; the discovery that jmcc writes no dependency files, which had let a raised CONFIG_MAX_CPU_COUNT rebuild one translation unit and not another, leaving a loop over sixteen harts reading an array still sized for eight. Every one of them was a latent bug that GCC's defaults had been hiding. The fully jmcc-built QSOE/N now boots the whole stack — slogger, the PCI server, the serial and NVMe drivers, the /usr mount, sysinit — to the login prompt, at parity with GCC.

Not in this release

The K3 boots to an interactive shell. It does not boot to login:.

There is no getty, no login, no shared command suite on the K3 yet, and the reason is simple and worth stating plainly: its storage is UFS, or NVMe behind a PCIe controller QSOE/N does not yet recognize. The Unmatched reaches its root filesystem; the K3 does not, so there is nowhere to mount /usr from and nothing to spawn a shell out of a filesystem for. That is the next piece of work, and it is named here rather than left for someone to find missing.