# Mail Watch recurring server freezes — 17 November 2026

## Resolution status

The user approved the reboot. On 17 November 2026 at 00:35 (Africa/Casablanca, UTC+1), the server booted successfully into `6.1.0-56-cloud-amd64`, Debian package `6.1.192-2`, which explicitly fixes the matching ext4 NULL folio regression.

Post-reboot checks:

- Apache, PostgreSQL, PHP-FPM and all three Gmail filter workers are active.
- No failed systemd units; no kernel error-priority messages in the new boot.
- Mail Watch homepage and live polling return HTTP 200.
- Other hosted project homepages return HTTP 200.
- All 19 Gmail watches are unexpired.
- Real Gmail notification requests returned HTTP 200 after reboot; a message was persisted at approximately 00:36:46 local time.
- PostgreSQL inspection found zero lock-blocked backends and zero idle transactions.
- Approximately 20 GiB RAM available; root filesystem 13% occupied.

The custom hourly TRIM and cache-clearing scripts remain disabled. Standard weekly `fstrim.timer` was restored after checking the fixed kernel; its next scheduled run is 23 November. The temporary boot-verification service has been disabled after its successful run. Its script and verification log remain under `/root/incident-2026-11-17/`.

This verifies immediate recovery and activation of the specific kernel fix. It does not constitute a multi-hour soak test or a guarantee against unrelated future outages. No forced cache eviction or deliberate crash reproduction was performed against the production filesystem.

## Finding

Two preserved kernel crash records match a critical Debian ext4 regression in package `6.1.192-1`, kernel `6.1.0-55-cloud-amd64`:

- 16 November 2026, 19:17:10 (Africa/Casablanca, UTC+1)
- 17 November 2026, 00:17:03 (Africa/Casablanca, UTC+1)

Both show `BUG: kernel NULL pointer dereference`, process `fstrim`, and `ext4_mb_load_buddy_gfp+0xbc/0x380`. These are kernel/filesystem failures capable of blocking writes across the host. A forced reboot temporarily recovers the server but reloads the same defective kernel.

An unattended update installed the defective kernel on 16 November around 06:04; it became active on subsequent reboots. The previous `6.1.0-54-cloud-amd64` boot had lasted approximately 28 days. Logs do not establish that every reported outage had the same cause, but two kernel crashes are conclusive and match the repeated freeze/reboot symptom.

The hourly cron sequence at minute 17 ran:

1. `/etc/cron.hourly/free`: `echo 1 > /proc/sys/vm/drop_caches`
2. `/etc/cron.hourly/fstrim`: `fstrim /`

The crash followed that sequence in both preserved records.

Debian issue: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1154151
Parent issue: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1154101

## Mitigation applied

- Removed executable permission from both hourly maintenance scripts. They are retained, not deleted.
- Masked and stopped `fstrim.timer` and `fstrim.service` to prevent scheduled TRIM until the corrected kernel is activated and checked.
- Saved original scripts, previous timer enablement state, GRUB files, and the two crash log extracts under `/root/incident-2026-11-17/`.
- Preserved the previous stable `linux-image-6.1.0-54-cloud-amd64` package as manually installed, so automatic cleanup does not remove the fallback.

Before reboot, these steps reduced exposure but did not replace the defective running kernel. Other filesystem activity can also trigger the regression. The fixed kernel is now active as recorded above.

## Corrected kernel installation

After refreshing APT metadata, Debian security repository offered package `6.1.192-2` / kernel `6.1.0-56-cloud-amd64`.

The downloaded package changelog explicitly states:

> ext4: fix NULL folio dereference in mballoc (Closes: #1154101)

Installed only:

- `linux-image-6.1.0-56-cloud-amd64` version `6.1.192-2`
- `linux-image-cloud-amd64` version `6.1.192-2`

The initramfs and GRUB configuration were generated successfully. GRUB's default entry now selects `6.1.0-56-cloud-amd64`. The preceding stable kernel remains available in Advanced options.

The corrected kernel was activated after the user approved the controlled reboot. See the resolution status above.

## Verification before activation

- Root filesystem: 14% occupied; inode usage approximately 3%.
- Memory after reboot: approximately 20 GiB available of 23 GiB; no OOM events found in retained incident-day kernel logs.
- No currently failed systemd units after stopping the maintenance timers.
- Apache configuration syntax valid.
- PostgreSQL accessible; no database-wide blocked state during inspection.
- Mail Watch homepage and `/poll`: HTTP 200; latest sample approximately 0.21s and 0.59s respectively.
- Other hosted project homepage: HTTP 200.

One earlier Apache `MaxRequestWorkers` event was also recorded. It is a secondary capacity observation; limits were not raised while investigating the confirmed kernel failures.

## Post-reboot verification checklist (completed where noted above)

1. Confirm `uname -r` is `6.1.0-56-cloud-amd64`.
2. Check Apache, PostgreSQL, PHP/background workers, and failed units.
3. Verify public homepage, `/poll`, and Gmail notification processing.
4. Check kernel logs for ext4 errors and observe beyond the former hourly trigger.
5. Restore only standard periodic TRIM after the corrected kernel is verified; the custom hourly cache flush is unnecessary and should remain disabled.

Do not deliberately reproduce the bug by flushing caches or running TRIM on the defective running kernel. Do not run filesystem repairs against the mounted root filesystem.
