Low_LvL_Blog

A Minimal Operating System Kernel with Runtime Instruction Translation for Low-Cost RISC-V Microcontrollers · part 9 of 9

QEMU virt, Part 1: The Kernel That Ran and Did Nothing

Bringing my Zig RISC-V kernel up on QEMU's virt board. It built, linked and ran with zero errors and zero output. Here's why, wrong turns included.

Keith Gangarahwe 20 min read

At the end of the last post I said the next step was booting the P4 with nothing but our own startup code. That’s still the plan, but I took a small detour first: before going anywhere near real silicon, the kernel now has to boot on QEMU’s generic RISC-V virt machine. If I can’t get two words out of a UART on a board that doesn’t physically exist and can’t be bricked, I have no business flashing anything.

Those two words were boot ok. Getting them took an embarrassingly long time.

All the code is on GitHub, if you want to follow along: KeithAGang/zig-esp32-riscv-kernel, for this exact state look at tag v0.1.

The Short Version (If You Came Here From Google)

If your bare-metal kernel on qemu-system-riscv32 -machine virt -bios none builds, links, “runs” with no errors and prints nothing, check this first:

QEMU’s virt boot ROM jumps to 0x8000_0000. Always. It does not read your ELF’s entry point.

My linker script put the kernel at 0x8020_0000. The ELF header proudly said the entry point was 0x8020_0000. None of that mattered. The CPU ran six instructions of QEMU’s reset ROM, jumped to 0x8000_0000, found nothing there, and that was the end of it. No crash message, no fault, no output. Just silence.

A linker script placing your code correctly, and an ELF reporting the right entry point, does not guarantee the CPU ever executes a single instruction of it. That’s the whole lesson. The rest of this post is how I found that out, including all the wrong turns, because honestly the wrong turns are the useful part. A post that only shows the final answer doesn’t help anyone debug their own board.

First, What Changed in the Project

A quick bit of background. The project used to be an ESP-IDF project: a CMake build, a main/idf_c/idf_c.zig file wrapping FreeRTOS, esp_log and esp_heap_caps through @cImport, and an app_main() that called vTaskDelay and heap_caps_malloc. As I explained last time, all of that is going away, so I restructured the whole thing into this:

src/
├── main.zig          # was main/app.zig, now a bare kmain with no IDF calls
├── hal.zig           # comptime chip selection, one file, chip-agnostic
├── kernel/
│   ├── boot.S        # the first instruction the CPU runs
│   ├── boot.zig
│   ├── trap.S / trap.zig   # not written yet
│   ├── context.zig   # TrapFrame + primeStack
│   ├── task.zig      # TCB + task table
│   └── sched.zig, ipc.zig, pmp.zig   # stubs for now
├── chip/
│   ├── virt/         # QEMU virt: the only one with real content so far
│   ├── c3/
│   └── p4/
├── servers/          # empty: filesystem/display servers later
├── guest/            # empty: ARM7TDMI guest execution later
└── lib/              # empty: shared no-alloc data structures later
attic/                # old IDF-era uart.zig and compositor.zig
  • context.zig and task.zig came from splitting the old main/task/task.zig in two. Nothing was lost from the context switcher work.
  • attic/ holds the old UART driver and the ANSI compositor. They’re outside the build, but I didn’t delete them, since bits of them will be useful for the C3 port and the display server later.
  • Deleted entirely: both CMakeLists.txt files, main/idf_c/idf_c.zig and generate_paths.py. The main/ directory doesn’t exist anymore. If you read the build pipeline post, yes, that entire pipeline is now one build.zig. I’m not crying, you’re crying.

Wait, I Never Needed the Zig Fork?

This one genuinely surprised me, so it gets a proper section.

Back in the ESP-IDF days I’d set up a forked Zig/LLVM toolchain, kassane/zig-espressif-bootstrap, because I assumed you needed it for ESP32 targets. When I finally sat down and actually read what the fork is for, it turns out it exists to add Xtensa code generation (the original ESP32, S2, S3 and ESP8266) using Espressif’s own LLVM fork. Xtensa is a proprietary Tensilica instruction set that upstream LLVM has never supported, so you need the fork for those chips.

But:

  • The ESP32-C3 is RV32IMC.
  • The ESP32-P4 is RV32IMAFC.
  • Both are standard RISC-V, and upstream LLVM has supported RISC-V for years.

The fork’s own README lists exactly two RISC-V targets (esp32p4 and esp32h4), and that looks like a completeness thing rather than a sign that upstream can’t handle them.

So, plainly: the fork was never necessary for this project. It was a leftover assumption from the ESP-IDF/CMake era, probably something I installed “just in case”. Now that build.zig is the entire build system, targeting bare RV32 needs nothing but stock Zig.

One caveat worth being clear about: IDF’s libraries (Wi-Fi, BLE and so on) are a separate question from the compiler’s code generation. If you needed those, you’d be linking against IDF’s precompiled .a files and headers, which is a linker and header problem, not a “you need a different compiler” problem. I’m avoiding that path completely anyway. On-chip Wi-Fi and BLE are out of scope, and networking, if it’s ever needed, goes to a companion controller over UART.

Choosing a Chip at Compile Time

Before the debugging story, one design decision that’s worth explaining, because it shapes everything that comes after: the hardware abstraction layer.

The chip is picked with a build option, -Dchip=virt|c3|p4, and resolved entirely at compile time in hal.zig:

const build_options = @import("build_options");

pub const chip = switch (build_options.chip) {
    .virt => @import("chip/virt/mod.zig"),
    .c3 => @panic("ESP32 C3 not yet implimented!"),
    .p4 => @panic("ESP32 P4 not yet implimented!"),
};

comptime {
    if (!@hasDecl(chip, "consoleInit")) @compileError("chip missing consoleInit");
    if (!@hasDecl(chip, "consolePutc")) @compileError("chip missing consolePutc");
}

pub const consoleInit = chip.consoleInit;
pub const consolePutc = chip.consolePutc;

A few things are going on here:

  • No vtables, no function pointers. The compiler knows which chip was selected, so hal.consolePutc is just a direct call into that chip’s function, which it’s free to inline.
  • The other chips don’t exist in the binary. Zig only compiles what’s reachable, so when you build for virt, the C3 and P4 code is never even compiled. When I say this is “zero-cost”, I mean something you can actually check, not a vague performance claim.
  • Half-finished ports fail at compile time. The comptime block checks that the selected chip module actually declares the functions the kernel needs. Forget one, and you get a clear compile error instead of a mystery on real hardware at 3am.
  • One rule, enforced from day one: no file outside src/chip/<name>/ is allowed to contain a literal hardware address. Everything goes through hal.zig. That’s what should make adding the C3 and P4 a matter of adding files rather than rewriting the kernel.

And the kernel’s entire job, for now, is this:

const hal = @import("hal.zig");

export fn kmain() noreturn {
    hal.consoleInit();
    const msg = "boot ok\n";
    for (msg) |c| hal.consolePutc(c);
    while (true) {}
}

Print eight bytes. How hard can it be?

The Errors, In the Order They Actually Happened

1. The Zig API moved again

The first error had nothing to do with RISC-V. On current Zig (I’m on 0.16.0), root_source_file, target and optimize aren’t options on addExecutable anymore. They live on a Module that you create with b.createModule(...) and then pass in as root_module. It’s a one-line fix, and not very interesting, but if you’re following along on 0.16 you’ll hit it in the first thirty seconds.

2. Overlapping ROM regions

With the kernel linked at 0x8000_0000, the first boot attempt died straight away:

qemu-system-riscv32: Some ROM regions are overlapping

Two things wanted the same memory. One was my kernel. The other was the device tree blob (the FDT, a description of the board’s hardware), which QEMU generates and loads into guest RAM for you when you boot virt with -kernel. I’d also given the machine a deliberately tiny -m 500K, to rehearse the memory pressure of the C3, which only has about 400KB of SRAM.

My reasoning at the time: -kernel on virt expects a Linux-style kernel, one that knows to go looking for the FDT and work around it. My from-scratch kernel does none of that, and nothing tells QEMU not to generate an FDT anyway. I’d also convinced myself that QEMU always puts the FDT right at the start of RAM, which is exactly where my kernel was. Hold on to that belief, because it’s about to cause trouble.

3. “Fine, I’ll move the kernel” (this was the mistake)

My fix was to shift the kernel’s origin up to 0x8020_0000, leaving the first 2MB of RAM as a gap for the FDT. I updated layout.zig and the MEMORY block in virt.ld together, and even wrote a nice comment explaining that the offset was a QEMU-only thing, since real C3 and P4 silicon boots straight into our image with nothing else fighting over the address space. It made perfect sense on paper.

It was wrong, and it caused a whole cascade of problems, because it was fighting a fixed property of the board. More on that in a minute.

4. “No enough memory to place DTB after kernel/initrd”

(Yes, “No enough”. That’s QEMU’s spelling, not mine.)

Same kind of problem, different message, and this one took several rounds to sort out, because it was two unrelated problems producing the same-looking error:

  1. I hadn’t updated -m. It was still -m 500K, so the total RAM I was telling QEMU about ended at 0x8007_A120, well before my kernel’s new origin at 0x8020_0000 even began. QEMU wasn’t wrong to refuse. There genuinely was no room.

  2. There were two QEMUs. This is the sneaky one. which qemu-system-riscv32 gave a different answer depending on whether a Python virtual environment was active:

    • outside the venv: /usr/bin/qemu-system-riscv32, vanilla QEMU 11.0.2 from my distro
    • inside the venv: ~/.espressif/tools/qemu-riscv32/.../qemu-system-riscv32, Espressif’s fork, QEMU 9.2.2, left over from the old ESP-IDF setup

    So the exact same zig build qemu could run a different QEMU depending on which terminal I typed it into. Leftover ESP-IDF tooling strikes again. Because both problems produced the same error text at different points, untangling them was genuinely confusing.

The fixes:

  • While debugging, I pinned the absolute path (/usr/bin/qemu-system-riscv32) in build.zig, so invisible shell state couldn’t pick the QEMU for me.
  • I stopped trying to calculate -m exactly. It went 500K → 2560K (still failed) → 8M (worked). How much space the device tree needs isn’t something worth working out to the byte. Just give QEMU plenty of headroom.

That second point needs one clarification, because it looks like I just threw away the C3 memory rehearsal. I didn’t. -m and the kernel’s memory budget are two different numbers. -m is how much RAM QEMU pretends the board has. The kernel’s budget is the stack and arena sizes in layout.zig and virt.ld, and the kernel never touches anything past _arena_end, no matter how much RAM QEMU claims exists.

If you take one tip from this section: run which -a qemu-system-riscv32 before you trust anything QEMU tells you.

5. -serial on:stdio

It should be mon:stdio. QEMU’s error message told me exactly that, immediately. Not every bug is deep. I’m including this one purely so you feel better about your own typos.

6. It boots! And says nothing.

Now QEMU started cleanly. No errors, no crash. And no output. Just a blinking cursor.

This is the worst kind of failure, because there’s nothing to point at. Nothing complains. Here’s everything I poked at, in order:

GDB said there were no symbols. Turns out this was about the tool, not the kernel. My plain gdb doesn’t understand riscv32 targets (and may also have been pointed at a stale path). Instead of hunting down a RISC-V-specific binutils build, I switched to LLVM’s tools (more on tooling below), and the symbols were all right there.

Disassembly failed because I got the name wrong. My first llvm-objdump --disassemble-symbols=consolePutc found nothing. llvm-nm explained why:

800188e8 t chip.virt.mod.consolePutc

Zig names functions after the module they live in, so the symbol is chip.virt.mod.consolePutc, not consolePutc. Right tool, wrong name. With the full name, the disassembly worked.

llvm-readelf -h found a completely different bug. While checking the ELF header, I noticed this:

Flags: 0x5, RVC, double-float ABI

Double-float ABI? On a kernel that should make zero assumptions about having an FPU? My target query didn’t list any CPU features explicitly, so Zig had picked a default profile that included hardware double-precision floating point. It had nothing to do with the silence, but it was a real bug that would have bitten me later, especially on the C3, which has no FPU at all. Fixed by being explicit:

const target = b.resolveTargetQuery(.{
    .cpu_arch = .riscv32,
    .os_tag = .freestanding,
    .abi = .none,
    .cpu_features_add = std.Target.riscv.featureSet(&.{.c}),
    .cpu_features_sub = std.Target.riscv.featureSet(&.{ .d, .f }),
});

Now the header just says Flags: 0x1, RVC. Lesson: if you don’t state your target’s features, the compiler will choose some for you, and it won’t tell you.

The polling loop had “vanished” from consolePutc. When I disassembled consolePutc, the loop that waits for the UART’s transmit register to be empty was completely gone. My first thought was that the compiler had eliminated it as dead code. I spent a while on that theory.

It wasn’t the compiler. I had commented the loop out myself, earlier on, as a diagnostic step, and then completely lost track of that fact:

pub fn consolePutc(c: u8) void {
    // while (LSR.* & LSR_THRE == 0) {} // poll until transmitter is ready
    THR.* = c;
}

I’m including this because it’s the most honest part of the whole session. I was debugging with an AI assistant, and when you work through lots of small edits together, “try this, revert that, comment this out for a second”, it gets really easy for both of you to lose track of which changes are actually live. We ended up diagnosing a non-bug that we’d created ourselves. A one-line // TEMP: polling removed for debugging comment would have saved a whole round of that. (And as you can see in the appendix, that line is still commented out as I write this. QEMU’s UART model is always ready to transmit, so it doesn’t matter here, but it’ll definitely matter on real hardware. Putting it back is on the list.)

The UART address was right. Definitely. With the loop mystery cleared up, the disassembly showed a plain store to 0x1000_0000, then straight to ret. No branch, no loop. That address matches QEMU’s own source, hw/riscv/virt.c:

[VIRT_UART0] = { 0x10000000, 0x100 },

As far back as I looked in QEMU’s history, that address hasn’t changed. On top of that, -d guest_errors came back empty: no fault, no warning about accessing unmapped memory. That’s about as cleanly as “wrong address” can be ruled out without tracing individual instructions.

So, still silent. At this point my notes listed three possibilities, none of them confirmed:

  • The CPU never actually reaches the store instruction. Something before it, consoleInit or the way kmain gets called, loops or goes off somewhere else first.
  • The guest writes the byte just fine, but QEMU’s serial output isn’t wired up to my terminal. The bug would be in how I was running QEMU, not in the kernel at all.
  • -bios none might behave differently across QEMU versions in how (or whether) it respects the entry point from -kernel. Floated as an idea, never tested.

The next step was going to tell these apart: trace execution and check whether the CPU ever gets to the store instruction’s address.

7. The actual root cause: it never got there

So I asked QEMU to log every block of code it executed, with -d exec. I was expecting to grep a long trace for the store instruction. Instead, the whole trace was five lines:

Trace 0: 0x7f5a88000100 [001411ad/0000000000001000/01c14003/ff020000]
Stopped execution of TB chain before 0x7f5a88000100 [0000000000001000]
Trace 0: 0x7f5a88000100 [001411ad/0000000000001000/01c14003/ff020000]
Trace 0: 0x7f5a88000240 [001411ad/000000000000100c/01c14003/ff020000]
Trace 0: 0x7f5a88000400 [001411ad/0000000080000000/01c14003/ff020000]

The second hex number on each line is the guest address being executed: 0x1000, 0x100c, and then 0x8000_0000. And that’s it. Nothing ever ran at 0x8020_0000, where my kernel was. Adding in_asm to the flags shows the actual instructions:

0x00001000:  00000297          auipc                   t0,0
0x00001004:  02828613          addi                    a2,t0,40
0x00001008:  f1402573          csrrs                   a0,mhartid,zero
0x0000100c:  0202a583          lw                      a1,32(t0)
0x00001010:  0182a283          lw                      t0,24(t0)
0x00001014:  00028067          jr                      t0
0x80000000:  0000              illegal

That’s QEMU’s reset ROM at 0x1000: it loads the hart ID into a0, a pointer to the FDT into a1, loads a jump address and jumps to it. The jump address is 0x8000_0000. Not 0x8020_0000, where my kernel was actually sitting. The CPU landed in empty RAM, hit an all-zeros “instruction” (which is illegal on RISC-V), and since no trap handler exists yet, that was the end of it. Nothing printed, because nothing I wrote ever ran.

QEMU’s virt boot ROM jumps to a fixed address, 0x8000_0000. It does not read the ELF’s entry point.

What I confirmed, and what I didn’t:

This isn’t a QEMU bug. It’s how the board model works, much like a real RISC-V boot ROM: it jumps to a fixed address. Loading an image with -kernel and where execution actually starts are two separate things, and nothing makes them agree. llvm-readelf -h had been telling me the truth all along (Entry point address: 0x80200000). It just didn’t matter, because the boot ROM never reads it.

What I confirmed, and what I didn’t:

  • Confirmed: I ran the exact same kernel on Espressif’s QEMU 9.2.2 with zero changes and got an identical failure, so this isn’t a quirk of one QEMU version. It pointed the finger back at my own memory layout.
  • Confirmed: the UART address, 0x1000_0000, three separate ways: from memory, from QEMU’s source, and from the device tree QEMU generates (dumped with dumpdtb, more on that below), which lists memory@80000000 and names serial@10000000 as its stdout-path.
  • Not claimed: that this holds for every boot setup. I’ve only tested -bios none with -kernel and an ELF file. With firmware like OpenSBI in the picture, the boot flow is different, and your mileage may vary.

Which means every detour in step 6 was chasing a symptom of this one problem. GDB, the disassembly, the UART address: none of it mattered, because the CPU never reached my code at any point. boot.S, an obvious early suspect, was innocent the entire time. So was the UART driver. They were simply never run.

8. The real fix

Put the kernel back where the boot ROM actually jumps: ram_origin = 0x8000_0000, in both layout.zig and the MEMORY block in virt.ld. No offset, and no gap. _start now sits exactly where the CPU lands.

That leaves the original problem from step 2, the FDT collision. And here I have to own up to something, because the debug log I kept during the session is wrong about it.

My notes from the time say the collision was solved with -machine virt,dumpdtb=/tmp/dumped.dtb, which supposedly sends the device tree to a file on the host so nothing sits in guest RAM, and that -m could then drop back down to 1M. When I went back to check this while writing the post, neither of those held up:

  • dumpdtb makes QEMU write the device tree to a file and then exit immediately. It never boots your kernel. It’s a great tool for looking at the FDT (it’s how I checked the addresses in step 7), but it doesn’t move the FDT anywhere.
  • -m 1M fails with the same “No enough memory to place DTB” error.

What actually made the collision go away is the other half of the fix: RAM headroom. My belief from step 2, that QEMU always puts the FDT at the very start of RAM, doesn’t match what I see now. With enough RAM, QEMU puts the FDT somewhere after the kernel. With too little, there’s nowhere for it to go:

$ qemu-system-riscv32 -machine virt -m 500K -bios none -nographic -kernel kernel
qemu-system-riscv32: No enough memory to place DTB after kernel/initrd

$ qemu-system-riscv32 -machine virt -m 1M -bios none -nographic -kernel kernel
qemu-system-riscv32: No enough memory to place DTB after kernel/initrd

$ qemu-system-riscv32 -machine virt -m 2560K -bios none -nographic -serial mon:stdio -kernel kernel
boot ok

Remember 2560K, which failed back in step 4? It works now, because the kernel no longer starts 2MB into RAM. The repo uses -m 8M for comfortable headroom. The error message was literally telling me all along: “place DTB after kernel”. I just didn’t read it carefully.

I’m leaving this correction in rather than quietly fixing the story, because it’s the same lesson as the commented-out polling loop: your notes from the middle of a debugging session are a record of what you believed, not of what was true. Re-check before you publish. (I nearly didn’t.)

The C3 memory rehearsal isn’t lost, by the way. The linker script still only gives the kernel 500K, so it will fail to link long before it outgrows the real chip. QEMU just gets extra room for its own bookkeeping.

9. boot ok

QEMU virt printing boot ok
QEMU virt printing boot ok

Eight bytes. Two words. The same kernel image works on both QEMU builds. I have never been so happy to see a lowercase string.

Tooling: What You Actually Need

This tripped me up enough that it’s worth writing down.

Required:

  • Zig 0.16.0 (I use zvm, but a direct install is fine). No fork, no patched LLVM.
  • qemu-system-riscv32, vanilla, from your distro’s package manager. I’m on 11.0.2 on openSUSE Tumbleweed. Espressif’s QEMU fork (the one that comes with ESP-IDF) behaves identically for virt, but it’s baggage if you aren’t otherwise using IDF.

For inspecting binaries, you have options:

  • The traditional way: a riscv32-unknown-elf-* GNU toolchain (readelf, nm, objdump, gdb). I don’t have one installed, and it turned out I didn’t need it.
  • LLVM’s tools: llvm-readelf, llvm-nm and llvm-objdump understand every architecture from a single binary. These are what I actually used. They came from my distro’s llvm package, not from Zig.
  • What about Zig itself? Zig does have a zig objdump command, and I’d hoped it would save you the install. On 0.16.0, though, pointing it at an ELF file just prints TODO dump elf file. So for now, install LLVM’s tools.
  • GDB: plain gdb often doesn’t have RISC-V support built in. Run gdb --configuration | grep -i riscv to check. If there’s nothing, gdb-multiarch or a riscv32-unknown-elf-gdb build will do the job.

And the QEMU flags that actually found the bug:

  • -d exec,in_asm -D trace.txt: log every block executed and its instructions. If your kernel is silent, start here, not where I started.
  • -d guest_errors: log invalid hardware accesses by the guest.
  • -machine virt,dumpdtb=file.dtb: dump the generated device tree and exit.

What’s Next

If you’ve arrived here from a search, eight bytes might make this look like a small project. It isn’t. This is the very first brick of a microkernel with PMP-based memory isolation between tasks, which will eventually run ARM7TDMI binaries through a dynamic binary translator, all on a microcontroller with no MMU. The restructuring post has the full plan and why it’s interesting. None of it works without a kernel that can actually boot, though.

Part 1 was about getting the CPU to run our code at all. Part 2 is about making it survive when things go wrong:

  • Putting the UART polling loop back. Yes, really.
  • A trap handler (trap.S/trap.zig), so that the next time the CPU lands on an illegal instruction, it tells me instead of quietly giving up.
  • boot.zig, installing the trap vector and setting up the kernel arena.
  • Tidying up layout.zig and virt.ld, which currently have to be kept in sync by hand.

And once all of that works on virt, the C3 and P4 ports should be a case of adding a chip/ directory. Should be. What could possibly go wrong?

Appendix: The Working Files

These are the files exactly as they were when the screenshot above was taken, so if you’re stuck on the same wall, you have something to diff against. They have a few warts, which I’ve listed at the end instead of quietly fixing.

build.zig

const std = @import("std");

pub fn build(b: *std.Build) void {
    const target = b.resolveTargetQuery(.{
        .cpu_arch = .riscv32,
        .os_tag = .freestanding,
        .abi = .none,
        .cpu_features_add = std.Target.riscv.featureSet(&.{.c}),
        .cpu_features_sub = std.Target.riscv.featureSet(&.{ .d, .f }),
    });
    const kernel_mod = b.createModule(.{
        .root_source_file = b.path("src/main.zig"),
        .target = target,
        .optimize = .Debug,
    });
    kernel_mod.addAssemblyFile(b.path("src/kernel/boot.S"));

    const Chip = enum { virt, c3, p4 };
    const chip = b.option(Chip, "chip", "target chip") orelse .virt;

    const opts = b.addOptions();
    opts.addOption(Chip, "chip", chip);
    kernel_mod.addOptions("build_options", opts);

    const exe = b.addExecutable(.{
        .name = "kernel",
        .root_module = kernel_mod,
    });
    exe.setLinkerScript(b.path("src/chip/virt/virt.ld"));
    exe.entry = .disabled;

    b.installArtifact(exe);

    const qemu = b.addSystemCommand(&.{
        "qemu-system-riscv32", "-machine", "virt",       "-m",      "8M",
        "-bios",               "none",     "-nographic", "-serial", "mon:stdio",
        "-kernel",
    });
    qemu.addArtifactArg(exe);
    const qemu_step = b.step("qemu", "Run kernel under QEMU virt");
    qemu_step.dependOn(&qemu.step);
}

src/main.zig

The whole point of everything else in this post: the function the CPU is supposed to reach.

// main.zig

const hal = @import("hal.zig");

export fn kmain() noreturn {
    hal.consoleInit();
    const msg = "boot ok\n";
    for (msg) |c| hal.consolePutc(c);
    while (true) {}
}

src/hal.zig

// src/hal.zig — comptime chip selection.
//
// Exactly one chip/*/mod.zig is selected here from the build option
// (-Dchip=virt|c3|p4) and re-exported as `chip`. Kernel code imports
// hal.zig and never a chip module directly.
//
// Also the place for the comptime interface check: assert the selected
// module exposes the full surface (init, consolePutc, layout, timer, ...)
// so a half-finished port fails at compile time, not at 3am on hardware.

const build_options = @import("build_options");

pub const chip = switch (build_options.chip) {
    .virt => @import("chip/virt/mod.zig"),
    .c3 => @panic("ESP32 C3 not yet implimented!"),
    .p4 => @panic("ESP32 P4 not yet implimented!"),
};

comptime {
    if (!@hasDecl(chip, "consoleInit")) @compileError("chip missing consoleInit");
    if (!@hasDecl(chip, "consolePutc")) @compileError("chip missing consolePutc");
}

pub const consoleInit = chip.consoleInit;
pub const consolePutc = chip.consolePutc;

src/chip/virt/layout.zig

Heads up: the comment on ram_origin below is out of date. The value (0x8000_0000) is correct, but the comment still argues for the 2MB offset I reverted in step 8. It’s left as-is on purpose; see the warts list at the end.

// chip/virt/layout.zig
//
// Memory map for QEMU's RISC-V "virt" board — a fixed board model defined
// in QEMU's own source (hw/riscv/virt.c), not a real chip. VIRT_DRAM base
// and VIRT_UART0 base are architectural constants for this board and are
// stable across QEMU versions. RAM *size* is whatever we pass via -m, so
// it's a constant we own — build.zig must pass exactly this value.

pub const ram_origin: usize = 0x8000_0000; // was 0x8000_0000; leaves first 2MB
// for QEMU's auto-generated FDT, which the "virt" machine places at the true
// RAM base (0x80000000) regardless of where our own code starts. This offset
// is a virt-tooling artifact — the C3 and P4 boot straight the built image
// with nothing else contending for RAM, so this line has no counterpart
// once the QEMU port is done.

pub const ran_length: usize = 500 * 1024; // must match `-m` in build.zig; deliberately small
// to rehearse the memory pressure of the eventual C3 target (~400KB SRAM)

pub const stack_size: usize = 16 * 1024; // 16-bit aligned memory per RV32 ABI
pub const arena_size: usize = 128 * 1024; // NAPOT-aligned

// ns16550a UART, VIRT_UART0 base, hw/riscv/virt.c. IRQ 10 (UART0_IRQ),
// unused until trap.zig/PLIC exist.
pub const uart_base: usize = 0x1000_0000;

src/chip/virt/virt.ld

/* chip/virt/virt.ld
 *
 * Linker script for QEMU's RISC-V "virt" board. Numbers here must match
 * chip/virt/layout.zig exactly — this file has no way to import that
 * constant, so if you change one, change both. (This duplication is
 * exactly what build.zig will generate away, once a second chip exists
 * and copy-pasted .ld files start actually drifting.)
 */

ENTRY(_start)

MEMORY
{
  RAM (rwx) : ORIGIN = 0x80000000, LENGTH = 500K
}

SECTIONS
{
  /* boot.S's _start MUST end up here, first, or QEMU jumps into garbage.
   * boot.S marks its entry code with .section .text.init to land here. */
  .text : {
    KEEP(*(.text.init))
    *(.text*)
  } > RAM

  .rodata : {
    *(.rodata*)
  } > RAM

  .data : {
    *(.data*)
  } > RAM

  /* zeroed globals: boot.S reads _bss_start/_bss_end to know what to clear */
  .bss : {
    _bss_start = .;
    *(.bss*)
    *(COMMON)
    _bss_end = .;
  } > RAM

  /* stack: grows DOWN, so _stack_top (high addr) is where sp starts,
   * _stack_bottom (low addr) is where it would overflow into the arena */
  . = ALIGN(16);
  _stack_bottom = .;
  . += 16K;
  _stack_top = .;

  /* kernel arena: ALIGN(128K) forces this address to be a multiple of
   * 128K, which is exactly the NAPOT requirement (power-of-two size,
   * aligned to its own size) — this region is legal as a single PMP
   * entry the moment pmp.zig exists */
  . = ALIGN(128K);
  _arena_start = .;
  . += 128K;
  _arena_end = .;
}

src/chip/virt/mod.zig

The UART driver. Innocent the whole time (well, apart from the loop I commented out myself).

// src/chip/virt/mod.zig — virt chip support.
//
// Implements the interface hal.zig checks for: console, timer, trap
// plumbing. Pairs with layout.zig in this directory for addresses.

const layout = @import("layout.zig");

const UART_BASE: usize = layout.uart_base;
const THR: *volatile u8 = @ptrFromInt(UART_BASE + 0x00); // transmit holding register
const LSR: *volatile u8 = @ptrFromInt(UART_BASE + 0x05); // line status register
const LSR_THRE: u8 = 0x20; // bit 5, transmit holding register empty

pub fn consoleInit() void {
    // QEMU's ns16550 model on "virt" comes pre-configured — no baud/divisor
    // setup needed. C3/P4 will need a real init definition here.
}

pub fn consolePutc(c: u8) void {
    // while (LSR.* & LSR_THRE == 0) {} // poll until transmitter is ready
    THR.* = c;
}

src/kernel/boot.S

Not changed once during this whole investigation, despite being the first thing I suspected.

# kernel/boot.S
#
# The first instruction QEMU executes. Nothing is initialized: no stack,
# .bss is whatever garbage was in RAM. This file's only job is to fix
# that, then jump into kmain and never come back.

.section .text.init      # placed first by virt.ld's KEEP(*(.text.init))
.global _start           # matches ENTRY(_start) in virt.ld

_start:
    # 1. Stack pointer. Nothing below this line may call a function or
    #    touch the stack — sp is garbage until this executes.
    la sp, _stack_top

    # 2. Zero .bss. Compiled code assumes uninitialized globals start
    #    at zero; nothing guarantees that's true in raw RAM.
    la t0, _bss_start
    la t1, _bss_end
clear_bss:
    bgeu t0, t1, bss_done
    sw   zero, 0(t0)
    addi t0, t0, 4
    j    clear_bss
bss_done:

    # 3. Jump into Zig. `call` sets ra, but kmain must never return —
    #    there is nothing to return TO.
    call kmain

    # If kmain ever returns, something is badly wrong. Trap loudly
    # rather than executing whatever garbage follows in memory.
hang:
    j hang

Known warts

  • The comment on ram_origin in layout.zig still describes the reverted 2MB offset. The value is correct; the comment is a leftover from step 3.
  • “implimented” in the @panic messages in hal.zig is a typo in the real file, not in this post.
  • ran_length should be ram_length, and its comment says it must match -m, which is no longer true: -m is now 8M on purpose, and the 500K limit is enforced by the linker script instead.
  • The UART polling loop in consolePutc is still commented out.
  • build.zig calls plain qemu-system-riscv32 again instead of the absolute path I pinned while debugging, so step 4’s venv trap is still possible. Check which -a before you run it.
  • .data is linked straight into RAM, which is fine on QEMU, where -kernel loads the whole ELF into RAM, but won’t be on real hardware, where it has to be copied from flash. That’s a problem for the C3/P4 ports.
  • layout.zig and virt.ld duplicate the same numbers by hand, which is exactly the kind of thing that ends up drifting.

© 2026 Keith Gangarahwe