Seriously, what is the large code-model even for?

I have been working on making massive binaries possible at $DAYJOB$. One of the Hail Marys that you should be able to rely on is the large code-model (mcmodel=large) as it makes no assumptions about size and distance of relocations.

Large code model: The large code model makes no assumptions about addresses and sizes of sections. [cite]

In a previous post, I documented how I hit simple performance bottlenecks that made me believe that the code-model is largely theoretical in practice. In those cases, the fixes were evident and relatively small; however their omission was a hint at how no one uses the code-model because without them the performance penalty was a non-starter.

I hinted at some other ways I found the large code-model to still be lacking however I had not yet fully understood the pain of those failure modes. I left a small teaser near the bottom about Thread Local Storage (.tdata / .tbss) being one of the “fun failure modes”.

Turns out that it is worse than I thought. The instruction sequences the compiler emits for TLS are 32-bit by construction, and -mcmodel=large has nothing to swap them for. 🤦🏻‍♂️

To put that very bluntly, mcmodel=large is incapable of building complex large binaries despite its stated goal.

How do you even make a >2GiB binary?

The annoying part of this whole line of research is producing a test subject. Emitting 2GiB of real instructions into .text is genuinely painful, you’d have to generate and assemble billions of instructions, and the object file is enormous. Beyond that the cardinality of the problem space grows when you consider whether the code is position-independent, do we use a procedure-linkage-table (PLT), TLS, GOT and all the various nuances each compiler brings with how they layout and order sections in their default linker scripts.

But relocation overflow isn’t about how many bytes are on disk, it’s about virtual-address distance between a reference and its target. So we have a few options.

Uninitialized globals land in .bss, which is a NOBITS section: it occupies virtual address space but zero bytes on disk (the same sparse-file trick I wrote about in massively huge fake files). So a few thousand 1MiB arrays gives us gigabytes of address space essentially for free.

We can synthetically produce this pretty easily with the following script.

# N * 1-MiB globals, plus a function that touches each one
# so the compiler emits one relocation per array.
import sys
N, CHUNK = int(sys.argv[1]), 1024 * 1024
for i in range(N):
    print(f"char s{i}[{CHUNK}];")
print("long sum(void){ long a = 0;")
for i in range(N):
    print(f"  a += s{i}[0];")
print("  return a; }")
print("int main(void){ return (int)sum(); }")

$ python3 gen.py 4096 > huge.c
$ gcc -c huge.c -o huge.o
# 4 GiB of address space, 288 KiB on disk
$ du -h huge.o
288K    huge.o

Link it with the (default) small code-model and it falls over exactly where you’d expect, at the 2GiB signed-32-bit boundary:

$ gcc -no-pie huge.o -o huge
huge.c:(.text+0x780f): relocation truncated to fit: R_X86_64_PC32 against symbol `s2048' ...

s2048 sits at 2048 * 1MiB = precisely 2GiB.

Recompile the same source with -mcmodel=large and it links cleanly:

$ gcc -c -mcmodel=large huge.c -o huge_large.o
$ gcc -no-pie -mcmodel=large huge_large.o -o huge_large

The large model did its job: it replaced the 32-bit R_X86_64_PC32 references with 64-bit R_X86_64_GOTOFF64 sequences: a movabs of a 64-bit offset + an add.

The .bss trick is great for exercising the linker, but it’s a little unsatisfying and overly synthetic. I often want to mimic relocation failures as they traverse a large .text segment. We can leverage the assembler’s .fill directive to repeat instructions. We can generate a tiny source of N functions spaced 1MiB apart with a NOP sea between them, plus one dispatcher that calls every function. This requires each call to have a relocation and the calls to functions past the 2GiB mark overflow. 🔥

Generator: gen_realtext.py

# N tiny functions spaced 1 MiB apart, plus a dispatcher that calls each one.
# .fill is a memset, so the assembler emits the bytes without codegen.
import sys

N, CHUNK = int(sys.argv[1]), 1024 * 1024   # e.g. 4096 -> ~4 GiB of .text

print(".text")
print(".globl main")
print("main:")
for i in range(N):
    print(f"  call f{i}")               # one relocation per function
print("  ret")

for i in range(N):
    print(f".globl f{i}")
    print(f"f{i}:")
    print("  ret")
    print(f"  .fill {CHUNK}, 1, 0x90")   # 1 MiB of real NOP bytes -> real .text

A quick primer on TLS access models

Now let’s do the exact same thing, but using __thread.

When you access a thread-local variable, the compiler doesn’t just load an address, it emits one of four access models, from fastest/least-flexible to slowest/most-flexible:

Local Exec (LE) The variable lives in the main executable’s own TLS block. The offset from the thread pointer (%fs) is a link-time constant, baked directly into the instruction as a 32-bit immediate; uses R_X86_64_TPOFF32.Initial Exec (IE) The variable is in a module loaded at startup. The 64-bit thread-pointer offset is stored in aGOT slot, and the code loads it with a RIP-relative reference; uses R_X86_64_GOTTPOFF to reach the slot & R_X86_64_TPOFF64 to fill the slot.General Dynamic (GD) / Local Dynamic (LD) The fully general case fordlopen‘d modules, a call into __tls_get_addr; uses R_X86_64_TLSGD & R_X86_64_TLSLD.

Notice the pattern: the value that reaches the thread pointer can be 64-bit (i.e. it lives in a GOT slot), but every instruction that participates in a TLS access uses a 32-bit field. TPOFF32, GOTTPOFF, TLSGD, TLSLD are all 32-bit.

Generator: gen_tls.py

# Same as gen.py, but every array is thread-local (__thread) so it lands in
# .tbss and each access emits a TLS relocation instead of a plain data one.
import sys

N, CHUNK = int(sys.argv[1]), 1024 * 1024   # e.g. 4096 -> ~4 GiB of .tbss

for i in range(N):
    print(f"__thread char t{i}[{CHUNK}];")

print("long sum(void){ long a = 0;")
for i in range(N):
    print(f"  a += t{i}[0];")
print("  return a; }")
print("int main(void){ return (int)sum(); }")

$ python3 gen_tls.py 4096 > tls_huge.c   # like gen.py but each array is `__thread`
$ gcc -c -mcmodel=large tls_huge.c -o tls_huge.o
$ du -h tls_huge.o
288K    tls_huge.o

4GiB of .tbss, still tiny on disk. Now look at the relocations the large code-model generated:

$ readelf -r tls_huge.o | awk '{print $3}' | grep R_X86 | sort | uniq -c
      1 R_X86_64_GOTOFF64
      2 R_X86_64_GOTPC64
      2 R_X86_64_PC32
   4096 R_X86_64_TPOFF32

Every single TLS access is R_X86_64_TPOFF32 which is 32-bit, even under -mcmodel=large. 🫣

$ gcc -no-pie -mcmodel=large tls_huge.o -o tls_huge
tls_huge.c:(.text+0x25): relocation truncated to fit: R_X86_64_TPOFF32 against symbol `t0' defined in .tbss ...
tls_huge.c:(.text+0x36): relocation truncated to fit: R_X86_64_TPOFF32 against symbol `t1' ...

The exact same large code-model that just happily linked 4GiB of ordinary .bss fails on 4GiB of .tbss.

This isn’t a GNU quirk either. LLVM (clang & lld) does exactly the same thing, albeit with lld’s diagnostic being a bit friendly by printing the actual offset and the window it has to fit in:

$ clang -c -mcmodel=large tls_huge.c -o tls_huge.o
$ clang -no-pie -fuse-ld=lld -mcmodel=large tls_huge.o -o tls_huge
ld.lld: error: tls_huge.o:(function sum: .ltext+0x17): relocation R_X86_64_TPOFF32
  out of range: -4294967296 is not in [-2147483648, 2147483647]; references 't0'

-4294967296 is exactly -4GiB, t0 sitting at the far bottom of the TLS block, and [-2147483648, 2147483647] is the signed 32-bit window TPOFF32 has to live in.

lld even parks large-model code in a .ltext section.

Why the large model is powerless here

I was a little confused at first: R_X86_64_TPOFF64 exists. Why doesn’t the compiler use it, especially with mcmodel=large ?

A relocation type is glued to the specific field it patches and is determined…