Debian kernel package 6.1.0-55: ext4 oops and rollback

On Sunday 15 November 2026, at 19:48 Italian time (18:48 UTC), I stepped in on some of the servers I manage after the monitoring alert. The problem had most likely started a few minutes earlier. The common denominator was the update to the Debian kernel package 6.1.0-55, installed a few hours before, and I rolled back to the previous kernel. The fault is insidious, so I am documenting it carefully: what I observed, what the sources say and what I still do not know.

Diagram of the Debian kernel package 6.1.0-55: ext4 oops and rollback

Account: the 19:48 intervention

The kernel had moved from the 6.1.0-54 series to 6.1.0-55. The packages at fault are linux-image-6.1.0-55-amd64 and linux-image-6.1.0-55-arm64, Debian release 6.1.192-1 (upstream kernel 6.1.192), which came from the bookworm-security repository of Debian 12. Before rolling out an update I test it on a monitoring host that I use for exactly this purpose. There the new kernel caused no problem at all, not even during the stress tests.

It was not a clean crash but a performance degradation: a progressive slowdown ending in a system freeze. Meanwhile the TCP layer kept working: ping answered and ports accepted connections. Until application-level monitoring fired, it was impossible to tell whether a server was down, and only by connecting to the individual ports was the timeout visible. That is why 19:48 is the time of the intervention, not the exact time the problem began.

The servers involved are few, but of two architectures: amd64 and arm64. At first I suspected a memory fault or a single disk. The same symptom on two different architectures quickly ruled out a hardware cause.

The problem then showed up on the other servers too. I checked the kernel logs on every machine: all of them showed an Oops, that is, an error the kernel detects in one of its own functions and records with a trace. I therefore decided to go back, and at 20:08 (19:08 UTC) I rolled back the kernel updates, returning the servers to 6.1.0-54.

Incident timeline

Times for Sunday 15 November 2026, in Italian time (CET) and UTC. Italy is on UTC+1.

  • Critical phase, 19:48-20:08 (18:48-19:08 UTC): intervention after the application alert, with the degradation having started a few minutes earlier. The servers slow down until they freeze; I analyse the logs, identify the Oops and decide on the rollback.
  • Observation, medium risk, 20:08-24:00 (19:08-23:00 UTC): on the servers back on 6.1.0-54 I check that no other problems have appeared.
  • Closure, at 01:00 on 16 November (00:00 UTC): checks and monitoring end and the activity is closed.

An insidious bug: why the tests did not catch it

This is the aspect that struck me most: the Debian kernel package 6.1.0-55 did not cause an immediate, widespread failure. On servers with little disk load it caused no problem at all. The monitoring host, which I use to test updates before rolling them out, passed the stress tests with no errors or alerts.

As I interpret it, the error does not depend on the kernel alone but on a condition: the page holding the block map must have been evicted from memory. A server with few block writes and releases can go for days without meeting it, and a stress test that loads CPU or network does not touch it. It takes disk traffic, with files being created and deleted, and memory under pressure.

I draw a practical lesson: an “all green” test on a lightly loaded machine proves little for a bug of this kind. I say so openly.

There is another reason it is insidious: it does not look like a failure. The system slows down, but ping and ports still answer, so a network-only check reports that everything is fine. Only a check that actually exercises the service notices.

What the Oops says

An Oops looks unreadable, but three pieces of information are enough to interpret it: the type of error, the function where it occurs and the call chain leading to it.

  • Type of error: on amd64 a NULL pointer dereference, meaning the kernel read an invalid address. On arm64 the picture is the same, with a trace that starts from a directory removal.
  • Function: ext4_mb_load_buddy_gfp, at offset +0xbc on amd64. It is the same function on both architectures.
  • Call chain (amd64): closing a file (close()) leads to ext4_release_file, then to ext4_discard_preallocations and finally to the function above. On arm64 the trace starts instead from vfs_rmdir.

In short, the crash does not depend on an application or a driver but on the ext4 code that handles block allocation on disk: a part of the kernel used by practically every machine with this filesystem. The servers’ filesystem was ext4, in one case with journalled quotas enabled. It remains to be clarified whether disk load is the only factor or whether RAM and quotas also play a part.

The mballoc buddy cache in brief

ext4 does not allocate blocks at random. Its allocator, called mballoc, keeps for each block group a map of the free ones, organised with the buddy system technique. These maps reside in memory pages, the so-called buddy cache.

When blocks have to be allocated or freed, ext4_mb_load_buddy_gfp loads the map of the relevant group. If the page is already in memory it takes it at once. If it has been discarded, for example because memory is scarce, it reads it back from disk.

The case in which the page is no longer in memory is where the two code paths diverge, and that is where, according to the sources, the problem lies.

What the sources say about the Debian kernel package 6.1.0-55

First, the important thing: the problem is already reported in the Debian bug tracker. #1154101 (system freeze after the update) is dated 15 November 2026, #1154108 (NULL pointer dereference in ext4_mb_load_buddy_gfp, “regression from 6.1.190-1”) is from 15 November at 23:09 UTC, #1154119 is from 16 November. All of them give version 6.1.192-1 as the one where it appears. They came after my incident: the first, #1154101, is about three hours later. I did not find a CVE.

The picture that emerges is this. Kernel 6.1.191, released on 30 October 2026, brought into the 6.1 branch the conversion of the mballoc buddy cache and bitmap from the old “page” to “folio”. You can read it in ChangeLog-6.1.191. It is the code where the Oops I observed occurs.

Then there is pull request 13 of the sporevm/kernels project, merged on 11 November 2026. The authors report an identical Oops, in the same function, on their 6.1.191 kernel. Their explanation is this: in the backport the code checks the result of __filemap_get_folio() with IS_ERR(), but in 6.1 that function, when it does not find the page, returns NULL. The check does not trigger and the NULL pointer is then used. The problem appears when the buddy cache pages are discarded, because of low memory or with drop_caches. With 6.1.155 they do not see it, with 6.1.191 they do. They write that the fix for the 6.1 branch is queued and that 6.1.192 does not contain it.

ChangeLog-6.1.192 gives indirect confirmation: the fix was not there. It contains three commits, all on Xen drivers, and none on ext4. 6.1.192, on which the Debian package 6.1.0-55 is based, therefore came out without the fix, a circumstance consistent with what I observed on the servers.

What is confirmed and what is hypothesis

  • Confirmed by my logs: Oops in ext4_mb_load_buddy_gfp with kernel 6.1.0-55 on amd64 and arm64, while 6.1.0-54 is stable.
  • Confirmed by the changelogs: 6.1.191 contains the conversion of mballoc to folio and 6.1.192 has nothing on ext4, so no fix.
  • Hypothesis, reported by external sources: the cause is the IS_ERR() check on a value that in 6.1 can be NULL. The sporevm pull request and report #1154108 both point to it; the latter also says a fix is queued for 6.1.193. These are claims by the reporters and I have not verified them against the source.
  • Not available: a fixed package. As of 16 November 2026 the Debian Package Tracker shows 6.1.192-1 as the latest bookworm-security version. I have not analysed the Debian kernel source line by line.

The hypothesis is plausible because it explains all the symptoms: the same function, both architectures, a NULL dereference and the link with memory pressure. It remains a hypothesis.

Why close() and rmdir

Two different paths lead to the same function. When a file is closed, ext4 releases the block preallocations it had reserved. Removing a directory frees real blocks. In both cases the block group map is needed, hence the buddy cache.

That is why the crash looks random: it only happens if, at that moment, the map page is not in memory. A machine with plenty of free RAM can hold out for a long time, while one with memory under pressure, or that often drops its cache, gets there sooner.

Rolling back the Debian kernel package 6.1.0-55, step by step

The method is to go back to the previous kernel without deleting anything and to stop the new one from reinstalling itself. The commands below are an example: adapt them to your package names and do not run them on a production machine without knowing what they do. I accept no liability for incorrect use.

First I check which kernel is running and which are installed:

uname -r
dpkg -l "linux-image-*" | grep ^ii

Then I set the previous kernel as the default boot entry in GRUB. Alternatively the new package can be removed after rebooting; I leave it installed but inactive:

# vedo le voci disponibili
grep -E "menuentry |submenu " /boot/grub/grub.cfg | cut -d"'" -f2
# scelgo il kernel 6.1.0-54 come predefinito (adatta la stringa)
sudo grub-set-default "Advanced options for Debian GNU/Linux>Debian GNU/Linux, with Linux 6.1.0-54-amd64"
sudo update-grub
sudo reboot

After the reboot I check that the correct kernel is running, then I hold the kernel packages. The apt-mark hold command prevents apt from upgrading them until I remove it:

uname -r
sudo apt-mark hold linux-image-amd64 linux-image-6.1.0-55-amd64   # amd64
sudo apt-mark hold linux-image-arm64 linux-image-6.1.0-55-arm64   # arm64 (6.1.192-1)
apt-mark showhold

On arm64 servers the package is called linux-image-6.1.0-55-arm64, again at release 6.1.192-1. The hold is temporary: when Debian publishes a fixed version I will remove it with apt-mark unhold. Keep in mind that a hold on the kernel also covers security updates of the same package, so it should be lifted as soon as possible.

Always check that the hold is active with apt-mark showhold: it must list linux-image-amd64 (or linux-image-arm64). A simulation with apt-get -s upgrade is not enough: apt-get upgrade does not install new packages and keeps the metapackage back even without a hold, whereas apt upgrade would install it. For a reliable check use apt -s upgrade | grep -i linux-image and make sure no Inst linux-image line appears.

How to tell whether a server is exposed

No special tools are needed: three checks on the machine are enough.

# versione del kernel in esecuzione
uname -r
# Oops recenti nei log del kernel
journalctl -k -b -1 --no-pager | grep -i -E "ext4_mb_load_buddy|Oops|BUG:" | head
# memoria e cache
free -h

If the Debian kernel package 6.1.0-55 is in use and the filesystem is ext4, you are in scope. The risk grows if the machine runs with little free memory. If ext4_mb_load_buddy_gfp appears in the logs of the previous boot, it is the same problem.

One further precaution: if your crons contain a periodic drop_caches, remove it while you use this kernel, because dropping the cache is the action that, according to the sources, triggers the error. It is not general advice: normally drop_caches is not needed.

What I am watching

Before putting the Debian kernel package 6.1.0-55 back into production I check the Debian Package Tracker for a new version of the linux package and read its changelog, looking in particular for an entry on ext4 and mballoc. If one appears, I test it on a single machine before unblocking the others, and this time with real disk load. I have already covered kernel monitoring, for example in the article on the TCP vulnerability from late October: that was a security flaw, this is a stability regression, but the method is the same.

I will update this article when Debian publishes the fix.

Sources

Debian is a registered trademark of Software in the Public Interest, Inc. Linux is a trademark of Linus Torvalds. Commands are shown for information only, without warranty.

Comments

Leave a Reply

Your email address will not be published. Required fields are marked *

WordPress Cookie Notice by Real Cookie Banner