Jellyfin 10.11.10: ffmpeg processes sometimes never terminate after transcode, full s

Hi,

Problem:
When a client plays a file that requires transcoding, ffmpeg processes are sometimes left running and never terminate.
This is intermittent — most transcodes complete normally. When it does happen, restarting the Jellyfin container is not enough; only a full server reboot recovers the machine.
I tried to solve it with Claude but I didn't find the solution, hope you can help me =)


Hardware/software:
- HP ProLiant MicroServer Gen8, Xeon E3-1265L v2, 12 GB RAM
- OMV 7.7.24-7 (Debian 12), kernel 6.8.12-1-pve
- Jellyfin 10.11.10 (lscr.io/linuxserver/jellyfin) with hardware transcoding on
- NVIDIA Quadro P400 (Pascal, 2 GB), driver 580.173.02 (proprietary/closed module, official CUDA repo), nvidia-container-toolkit
- Previously on driver 535.261.03


Here is the last 2 times it happened:

1) July 16:
- Up to 3 concurrent ffmpeg processes
- Host CPU at 100%, OMV still reachable but very slow
- No GPU load at that moment

Between the 2 problems, I upgraded the nvidia driver.

2) August 8:
- Two ffmpeg jobs started 23 seconds apart on the same file
- Kernel thread nv_queue pegged at 99.7% CPU continuously
- nvidia-smi hangs / never returns
- ps shows the process as [ffmpeg] (bracketed), state R

In all cases:
- kill (with every options possible) on the ffmpeg PID: no effect
- docker stop / docker kill on the container: "tried to kill container, but did not receive an exit event"
- The only way I've found to get rid of these processes is a full reboot

Files involved are 4K HDR10 / Dolby Vision.

Clients seen triggering it: realme Pad (Android 11) and Jellyfin Web (Chrome).

What I've tried:
- Upgraded the driver from 535.261.03 to 580.173.02 (580 being the last branch supporting Pascal), via the official CUDA repo with --branch 580 --module-flavor closed. (Note for anyone doing the same: the dependency packages drifted to 610.x and had to be manually downgraded and pinned with apt-mark hold)
- Applied nvidia-patch from https://github.com/keylase/nvidia-patch. In case it could be an issue with too many transconding thread at the same time when clients try to change the quality multiple times in a short time. => didn't solve it
- Reboot = the only thing that recovers the server.

I joined 2 zips with ffmepg and jellyfin log of the 2 events
If you need more informations, I'm here !

Thanks a lot for your help

  jellyfin-logs-20260716.zip (Size: 82.69 KB / Downloads: 0)

  jellyfin-logs-20260808.zip (Size: 98.78 KB / Downloads: 0)