Recover from Linux update disaster

Disaster just stroke me today. My Arch Linux laptop suddenly shutdown for no reason during system upgrade. - I'm 100% sure it's not a battery issue, I have like 23% left when I started upgrade - In any case. When I rebooted the system and got pass GRUB's boot screen. It just says "can't find linux-zen" and "kernel needs to be loaded". Shouldn't be a big deas as all my data are backed up. But still, RIP.

Fixing Linux with broken system files

I installed multiple kernels. And of my kernels are missing. Properbally there's a lot of writes that aren't synced to disk. Fortunatelly the situation is only seeming scary. The solution is easy. Get an Arch Live USB. arch-chroot into it. Reinstall all packages and pray documents are intact.

So I got another computer and made myself a Live USB. Booted my laptop with it. And arch-chroot into it. pacman complains libzstd is missing. Ohhh... Fine. pacman has a -r parameter to specsify an alternative root direcotry for this exact situation. Repairing broken pacman in my case is a simple command from the live system.

pacman -S pacman zstd -r /mnt --overwrite="*"

Then arch-chroot again. Nice now pacman works again. Now I'm ready to resinstall everything, replace all broken system files.

pacman -Qqn | pacman -Sy - --overwrite="*"

Done and reboot. I saw GRUB successfully loading the kernel and... Display Manager loaded! Nice! My system is back to live!

This kind of control is exactly why I love Linux. There's no chance I ccan fix a similar problrm with Mac or Windows.

Author's profile
Martin Chang
Systems software, HPC, GPGPU and AI. I mostly write stupid C++ code. Sometimes does AI research. Chronic VRChat addict

I run TLGS, a major search engine on Gemini. Used by Buran by default.

  • marty1885 \at
  • GPG: 76D1 193D 93E9 6444
  • Jami: a72b62ac04a958ca57739247aa1ed4fe0d11d2df