Is There I/O After Death? What Happens to io_uring When a Process Dies

Most people (myself included) would presume that I/O dies with the process. Kill it, reap it with waitpid(), and surely the old process can no longer touch your storage.

That intuition holds for Linux native AIO, at least on the kernel I tested. But with ordinary io_uring, it is wrong: a process can be dead and reaped while writes it submitted earlier are still alive inside the kernel — and can reach the device afterwards. A very subtle difference.

This matters if a fail-fast application treats waitpid() as an I/O barrier. A replacement process may start recovery, inspect storage, or begin writing while requests from the previous incarnation are still on their way.

Need a barrier? Take a lock

There is a surprisingly simple way to turn this behavior into a barrier.

Let the writer open the device and take an exclusive lock:

int fd = open(path, O_RDWR | O_DIRECT);
flock(fd, LOCK_EX);

After killing and reaping the writer, the replacement process must acquire the same lock:

int fd = open(path, O_RDWR | O_DIRECT);

for (;;) {
 if (flock(fd, LOCK_EX | LOCK_NB) == 0) {
 break;
 }
 if (errno != EWOULDBLOCK && errno != EAGAIN) {
 perror("flock");
 abort();
 }
 usleep(1000); // retry in 1 ms
}

The separate open() matters. flock() locks are associated with an open file description. A file descriptor inherited across fork() or created with dup() refers to the same open file description, so it would not contend with the writer's lock in the way we need here.

Outstanding io_uring requests retain references to the writer’s struct file. The lock belongs to the open file description and is removed only when its final reference is released. So if the old requests are still alive, the new process cannot acquire the lock yet.

In fact, this is exactly how we open devices in YDB, regardless of whether we use native AIO or io_uring. My hunch is that native AIO may have had the same issue in the past, and the lock protocol simply survived the transition.

TL;DR

On Linux 6.6.79, I observed:

For ordinary io_uring, I could observe storage changing after the writer had already been killed and successfully reaped.

For SQPOLL and Linux native AIO, the same delayed writes instead made process teardown take longer: waitpid() did not return until they had drained.

These are measured results backed by the Linux source, not a portable guarantee for every kernel, filesystem, device, or API called “AIO”.

Catching I/O from the grave

I wrote a small test. The child continuously keeps roughly 32K direct 4 KiB writes in flight. It cycles over 1,024 known positions, storing a writer ID and generation number in every block.

The parent then:

1. sends SIGKILL;
2. waits for the child using waitpid();
3. reads all 1,024 positions;
4. reads them again;
5. compares every 4 KiB block.

If a block changes between the two snapshots, the supposedly dead writer has modified storage after waitpid() returned.

First attempt: nothing happens

On an idle NVMe device:

SIGKILL at +1000.061 ms
waitpid took 7.514 ms

first snapshot: 1024/1024 blocks from writer
second snapshot: 1024/1024 blocks from writer

all 1024 positions unchanged

At first glance, this looks reassuring. It isn’t.

An idle NVMe is simply too fast. Outstanding writes usually finish before, or while, the parent performs its first snapshot. The interesting race window is there, but hard to see. So I made it larger.

Slow the writes down

Linux has a convenient device-mapper target called dm-delay. It can delay writes by a fixed amount while reads remain fast. I configured a 3-second write delay. The child writes through the delayed mapping, while the parent verifies the underlying partition directly.

Then:

SIGKILL at +1000.064 ms
waitpid took 2.951 ms

first snapshot:
 0/1024 blocks from current writer

second snapshot:
 1024/1024 blocks from current writer

1024/1024 positions changed

I/O from the grave dettected

The process was gone. waitpid() had returned in about 3 milliseconds. And almost three seconds later, its writes arrived. Hooray, we caught I/O from the grave. That is the entire bug in one experiment. waitpid() told us that the task was dead. It did not tell us that its I/O was dead.

It also happens on a real NVMe queue

Of course, dm-delay is artificial. So I tried to reproduce the same thing on the native NVMe path.

To make the queue deep without introducing another writer, I ran a read-only fio workload:

fio --name=grave_readload \
 --filename=/dev/nvme2n1p2 \
 --readonly \
 --rw=randread \
 --bs=4096 \
 --direct=1 \
 --ioengine=io_uring \
 --iodepth=2048 \
 --numjobs=32

Then I killed the writer as before. One run produced:

SIGKILL at +1000.059 ms
waitpid took 8.028 ms

first snapshot took 252.735 ms
second snapshot took 177.544 ms

627/1024 positions changed
I/O from the grave dettected

Both snapshots contained blocks from the same writer ID. But generation counters advanced at 627 positions between the two reads. No delayed mapper. No second writer. The old process was already reaped, and its writes were still completing.

What exactly survives?

The lock experiment gives another way to observe this. Before writing, the child opens the device separately and takes LOCK_EX. After killing and reaping it, the parent immediately tries:

flock(fd, LOCK_EX | LOCK_NB);

With ordinary io_uring and the same 3-second delayed writes:

+1000.086 ms: parent sent SIGKILL

+1002.916 ms:
 waitpid returned

+1002.922 ms:
 LOCK_EX | LOCK_NB -> EWOULDBLOCK

+3189.371 ms:
 LOCK_EX | LOCK_NB -> acquired

The child cannot execute userspace after SIGKILL, so there is no explicit flock(LOCK_UN) happening later.

Instead, outstanding io_uring requests retain the child's struct file. Linux removes locks from __fput() via locks_remove_file() when the final reference disappears. So the failed lock attempt tells us something stronger than “a write happened later”: after waitpid() returned, kernel objects associated with the dead writer were still retaining its open file description.

Once the lock became available, verification saw no further changes. That is why the same mechanism can be used as a barrier.

SQPOLL behaves differently

Then I repeated the experiment with IORING_SETUP_SQPOLL. Again, writes were delayed by 3 seconds. This time:

SIGKILL at +1000.065 ms
waitpid took 2149.286 ms

first snapshot: unchanged
second snapshot: unchanged

The delay did not disappear. It moved into teardown. Instead of returning after a few milliseconds, waitpid() spent roughly another two seconds waiting.

The lock agreed:

waitpid returned
LOCK_EX | LOCK_NB -> acquired on first attempt

I observed the same behavior under native NVMe queue pressure. So on this kernel, SQPOLL teardown acted as a barrier for these writes.

And so does Linux native AIO

I also implemented the writer using Linux native AIO:

io_setup()
io_submit()
io_getevents()

This is the kernel AIO API, not POSIX aio_write(). With the same 3-second dm-delay setup:

SIGKILL at +1000.064 ms
waitpid took 2246.371 ms

first snapshot: unchanged
second snapshot: unchanged

Again, the remaining delay was paid inside process teardown. And again, the parent’s first lock attempt after waitpid() succeeded immediately. Under heavy read pressure, waitpid() stretched even further - to several seconds in some runs — but I did not observe native-AIO writes completing afterwards.

Why?

I’m not a kernel developer, so what follows is source-code archaeology rather than kernel expertise. I also used AI heavily for this part and only did a quick manual review of the relevant kernel code.

The kernel teardown paths are different.

For ordinary io_uring, do_exit() calls io_uring_files_cancel(), which in turn calls __io_uring_cancel(false). The interesting bit is false: compare that with the cancel_all=true path below. io_uring_cancel_generic() treats these cases differently when deciding which in-flight requests must be waited for. This matches the experiment: with ordinary io_uring, the raw-block writes can survive far enough into teardown to retain their file references and complete after the process becomes reapable.

SQPOLL is different. Its kernel submission thread exits through io_uring_cancel_generic(true, sqd). Here cancel_all is true, and the cancellation path keeps waiting for the relevant in-flight requests before the SQPOLL thread finishes. In my experiment, that wait happened before waitpid() returned.

Linux native AIO takes yet another path. When the last user of an address space goes away, __mmput() calls exit_aio() before exit_mmap(). And exit_aio() explicitly waits until all I/O for its AIO contexts is done. The kernel comment is refreshingly unambiguous: the function is called when the last user of the mm goes away, and outstanding I/O is waited for before teardown completes.

This also explains an important caveat: SIGKILL does not magically run an application's AIO destructor. The barrier comes from kernel teardown of the final mm reference. If another task still shares that mm, for example through CLONE_VM, reaping one task does not necessarily trigger exit_aio().

Conclusion

The lesson I learned from this is simple: process lifetime and I/O lifetime are not the same thing. With ordinary io_uring, waitpid() can tell you that the task is gone while its writes are still very much alive.

I focused on direct block-device I/O here. The broader lifetime lesson applies beyond block devices, but I would not assume the exact same teardown behavior for other kinds of I/O. Filesystems add page cache and writeback semantics; networking has its own socket and packet lifetimes. If waitpid() needs to be a barrier in your recovery protocol, verify the particular I/O path you rely on — or introduce an explicit barrier yourself.

Is There I/O After Death? What Happens to io_uring When a Process Dies was originally published in YDB.tech blog on Medium, where people are continuing the conversation by highlighting and responding to this story.