grub + cryptsetup not booting after kernel update

24 respostas [Última entrada]
amrprznu
Desconectado
Joined: 08/12/2013

ello, my boot is broken. I installed the new kernel (3.0.36?) last night with aptitude, but I run torsocks with it. It was late and I cancelled the thing (WRONG!). Before shutting down the computer with 'shutdown now' (and then pressing the power button), I did 'dpkg --configure -a' '/usr/bin/aptitude install linux-image-' and had to come back from bed, because the laptop was not off. I pushed the power button, and went to sleep. This morning, grub is telling me that it can't find my root partition which is encrypted with dm-crypt (cryptsetup). For what I have read, this might have to do with initrd.img not loading the dm-crypt modules, but I don't know. I end up being in a busybox shell that tells me that root is not found (that is /dev/mapper/Lap-root).

I tried doing a chroot by first mounting the encrypted drive:
(credits go to
http://stephentanner.com/index.php/2011/05/restoring-grub-for-an-encrypted-lvm/
blog.lifebloodnetworks.com/?p=1348
)
modprobe dm-crypt;
cryptsetup luksOpen /dev/sda5 hd;
vgscan;
vgchange -a y system;
lvscan;
mount /dev/system/root /mnt && sudo mount /dev/sda1 /mnt/boot && sudo mount -o bind /dev /mnt/dev ; sudo mount -t proc proc /mnt/proc ; sudo mount -t sysfs sys /mnt/sys;
chroot /mnt;
grub-install /dev/sda;
update-grub

the blkid from liveCD shows:
/dev/loop0: TYPE="squashfs"
/dev/sda1: UUID="5a07711e-189b-441f-a84c-774924575bb4" TYPE="ext2"
/dev/sda5: UUID="23020fcc-61ae-47f0-9215-a77a446b78c1" TYPE="crypto_LUKS"
/dev/sr0: LABEL="trisquel-mini 6.0 amd64" TYPE="iso9660"

the grub.cfg shows
...
search --no-floppy --fs-uuid --set=root 5a07711e-189b-441f-a84c-774924575bb4
linux /vmlinuz-3.5.0-36-generic root=/dev/mapper/Lap-root ro splash quiet $vt_handoff
...

zcat /mnt/grub/ | cpio -t | grep crypt
lib/modules/3.5.0-36-generic/kernel/drivers/block/cryptoloop.ko
lib/modules/3.5.0-36-generic/kernel/crypto
lib/modules/3.5.0-36-generic/kernel/crypto/async_tx
lib/modules/3.5.0-36-generic/kernel/crypto/async_tx/async_memcpy.ko
lib/modules/3.5.0-36-generic/kernel/crypto/async_tx/async_pq.ko
lib/modules/3.5.0-36-generic/kernel/crypto/async_tx/async_xor.ko
lib/modules/3.5.0-36-generic/kernel/crypto/async_tx/async_tx.ko
lib/modules/3.5.0-36-generic/kernel/crypto/async_tx/async_raid6_recov.ko
lib/modules/3.5.0-36-generic/kernel/crypto/xor.ko

So... can someone help me? Should I give more info?

Thanks!

amrprznu
Desconectado
Joined: 08/12/2013

Well, it seems that I shot myself in the foot. I compiled cryptsetup 1.6.1 from a tar-ball in the official repo. I built it with checkinstall. What I never suspected is that my nice little thing does not provide the appropriate hooks for initramfs-tools, which you can find in:

/usr/share/initramfs-tools/hooks/
(credits to:
Initramfs - Ubuntu Wiki
https://wiki.ubuntu.com/Initramfs )

I haven't tested the solution, but I uninstalled my own cryptsetup package (which I donate under the advice of not using it very much) and re-installed the version in the repos. It automatically created the executables in the mentioned directory above.

Also, I managed to load the dm-crypt, aes and xts modules into my image by modifying /etc/initramfs-tools/modules
(inspiration from
Custom kernel, encrypt hook not working... help. [SOLVED]
https://bbs.archlinux.org/viewtopic.php?id=157246:)
echo -e "crypto\nxts\ndm-crypt\naes" >> /etc/initramfs-tools/modules

Which I think is not really needed after installing the official cryptsetup, but I think it would be useful for someone trying to debug their own initrd.img

Now, I will restart my computer, and see if it works! Wish me luck.

[ UPDATE-1: DOES NOT WORK ]
It didn't work. I got to busybox, and now I had cryptsetup. After doing a cat /proc/modules, I saw dm_crypt

If someone would be so kind in telling me how to add the required things to my deb package or modify the current package to update it to version 1.6, I would appreciate it.

[UPDATE-2: slightly easier way to get chroot; just copy-paste to a file (or download chroot.sh and make executable]
#!/bin/bash
# Script made for Trisquel to chroot to a LVM on top of a LUKS using cryptsetup
# requires access to repositories with aptitude, cryptsetup and lvm2

# If you run this like ./chroot.sh m it's because you already unlocked the encrypted partition and know the name of the lvm volume of the boot partition

# Skip if the first argument given to chroot is "m"
if [[ "$1" != "m" ]]; then
echo "We are going to ask for all the data";
# Make sure that aptitude, cryptsetup and lvm2 are installed
[[ -z $(which aptitude) ]] && apt-get update && apt-get install aptitude;
[[ -z $(which vgscan) || -z $(which cryptsetup) ]] && aptitude install -y lvm2 cryptsetup;
# Load required modules (dm-crypt, aes, xts)
for i in {dm-crypt,aes,xts}; do
modprobe "$i";
done;

# Print name of current (un-encrypted) partitions
blkid | awk '{print $1}' | tr -d ':';
echo "Type the name of your encrypted partition (/dev/sdxY) ";
read encpart;
# Open the partition
cryptsetup luksOpen "$encpart" root;
# Make sure that lvm2 is installed
for lvgr in "$(vgscan | sed -n '/group/{s-.*group "\([^"]*\)".*-\1-g;p}')"; do
vgchange -a y all "$lvgr";
for lvdev in "$(lvscan | awk '{print $2}' | tr -d "'")"; do
list_lvm="$(printf "%s\n%s\n" $list_lvm $lvdev; printf "\n")";
done;
done;
echo -e "$list_lvm\nType the name of your root LVM partition from the list above (/dev/group/part) "
read rootpart;
echo -e "Type the name of your boot partition, if any (/dev/sdxY or /dev/group/part) ";
read bootpart;
fi;

if [[ "$1" == "m" ]]; then
rootpart="$2";
bootpart="$3";
fi;

# Mount $rootpart, $bootpart, dev, proc, sys, devpts, run
mount "$rootpart" /mnt && ([[ -n "$bootpart" ]] && mount "$bootpart" /mnt/boot) || mount -o bind /dev /mnt/dev; mount -t proc /proc /mnt/proc ; mount -t sysfs /sys /mnt/sys; mount -t devpts /dev/pts /mnt/dev/pts; mount -o bind /run /mnt/run; # [UPDATE-3: added run and dev/pts to the mounts]
echo "Ready to chroot";

chroot /mnt;

echo "Closed chroot, unmounting";
# Unmount everything
for i in {proc,sys,dev/pts,dev,boot,run,""}; do
umount /mnt/"$i";
done;

AnexoTamaño
cryptsetup_1.6.1-20130331_amd64.deb 484.62 KB
chroot.sh 2.11 KB
Magic Banana

I am a member!

I am a translator!

Desconectado
Joined: 07/24/2010

I cannot really help you (I have never used an encrypted partition) but I would like to know why you chose not to use the package shipped by Trisquel. Is it unsafe for some reason?

amrprznu
Desconectado
Joined: 08/12/2013

I just wanted the latest developments. I began this journey reading ArchWiki dm-crypt webpage (dm-crypt with LUKS - ArchWiki https://wiki.archlinux.org/index.php/Dm-crypt_with_LUKS ), and it said something about version 1.6. I saw that the version in the repos was 1.4, so I compiled it myself. Which was also a pain in the ass at the beginning. That is also why I decided to use checkinstall. I think that I got confused about the available options, like xts. I still don't know if the version in the repos has it.

BlinkingArrow

I am a member!

Desconectado
Joined: 12/27/2011

What happened when you ran update-grub? Once you had mounted everything, but before chroot, does blkid show the encrypted partition mounted ("/dev/mapper/Lap-root")?

amrprznu
Desconectado
Joined: 08/12/2013

1) I hadn't done the update-grub, I only did the update-initramfs -u -k all. Now that I do it, it says:
Generating grub.cfg ...
cat: /boot/grub/video.lst: No such file or directory
grub-probe: error: no such disk.
done
[ UPDATE4 - I exited chroot and remounted boot ]
Generating grub.cfg ...
Found linux image: /boot/vmlinuz-3.5.0-36-generic
Found initrd image: /boot/initrd.img-3.5.0-36-generic
Found linux image: /boot/vmlinuz-3.5.0-34-generic
Found initrd image: /boot/initrd.img-3.5.0-34-generic
Found linux image: /boot/vmlinuz-3.5.0-32-generic
Found initrd image: /boot/initrd.img-3.5.0-32-generic
umount: /var/lib/os-prober/mount: not mounted
rmdir: failed to remove ‘/var/lib/os-prober/mount’: Device or resource busy
grub-probe: error: no such disk.
done
[ UPDATE5 - This had to do with me plugging external devices (it is not really an issue) ]

2) Yes, it is mounted and shown with blkid

BlinkingArrow

I am a member!

Desconectado
Joined: 12/27/2011

Okay, so what happened when you tried to boot?

amrprznu
Desconectado
Joined: 08/12/2013
Liberty
Desconectado
Joined: 08/05/2013

It is unclear to me if your problem is solved or not?

If your problem is not solved, then don't panic...

I would recommend you to go the safe way:

If you have not damaged your encrypted partition, you should be safe. The boot loader is not on the encrypted partition. As I read your post, you haven't done anything bad with the encrypted partition. So that is good.

So, if you have another PC you can just take out your harddisk and mount it in another Trisquel-PC and boot with the working Trisquel. You should be able to unlock the drive on the other system and then you can make a backup of all your important stuff to a safe location. After that you can just put it back in your old PC and do a complete reinstallation of Trisquel - and learn from your mistake:)

amrprznu
Desconectado
Joined: 08/12/2013

Yes, that is done already. Back-up ready to f**k everything else to the sky.

amrprznu
Desconectado
Joined: 08/12/2013

Well, I restarted, did the cat /proc/modules again in busybox, and dm_crypt is not there. It is in my initrd.img file (zcat /boot/initrd.img | cpio -t)

BlinkingArrow

I am a member!

Desconectado
Joined: 12/27/2011

Did you try installing cryptsetup from the chrooted environment using apt? If it's there, you should maybe try to purge it and then reinstall and see if that adds your modules.

amrprznu
Desconectado
Joined: 08/12/2013

I tried by adding the modules into the /etc/initramfs-tools/modules (man initramfs.conf)

echo -e "crypto\nxts\n\ndm-crypt\naes" >> /etc/initramfs-tools/modules

and by modifying the /etc/default/grub file a little ( https://wiki.archlinux.org/index.php/GRUB#Root_encryption
https://wiki.archlinux.org/index.php/Dm-crypt_with_LUKS#Kernel_parameter_configuration_of_the_bootloader ) like this:
[... snip ...]
GRUB_CMDLINE_LINUX="cryptdevice=/dev/sda5:cryptoroot"
[... snip ...]

so that I get /boot/grub/grub.cfg
[... snip ...]
linux /vmlinuz-3.5.0-36-generic cryptdevice=/dev/sda5:cryptoroot root=/dev/mapper/Lap-root ro splash quiet $vt_handoff
[... snip ...]

I also added echo "insmod lvm" to /etc/grub.d/00_header , just in case.

(I just read https://trisquel.info/en/forum/grub-cryptsetup-not-booting-after-kernel-update#comment-40625 . I will reboot after that. If it does not work. I will kill this thing after having lunch. I also need to sleep)

amrprznu
Desconectado
Joined: 08/12/2013

The modules are there: aes, xts, dm_crypt (after doing cat /proc/modules), but it doesn't work. My last try: tweaking grub.cfg, because it seems that the cryptdevice thing is before the root=

BlinkingArrow

I am a member!

Desconectado
Joined: 12/27/2011

Okay, here's a copy of my grub.cfg:

menuentry 'Trisquel GNU/Linux, with Linux-Libre 3.2.0-49-generic' --class trisquel --class gnu-linux --class gnu --class os {
recordfail
gfxmode $linux_gfx_mode
insmod gzio
insmod part_msdos
insmod ext2
set root='(hd0,msdos5)'
search --no-floppy --fs-uuid --set=root [UUID of /boot partition]
linux /vmlinuz-3.2.0-49-generic root=/dev/mapper/[encrypted partition] ro splash quiet $vt_handoff
initrd /initrd.img-3.2.0-49-generic
}

amrprznu
Desconectado
Joined: 08/12/2013

Nice! Thanks! Now I will take over your computer lulz!

Guess what? in busybox:

(initramfs) cryptsetup luksOpen /dev/sda5 root_disk
(initramfs) lvm
lvm> vgchange -a y Lap
lvm> CTRL+D
(initramfs) CTRL+D

...and I'm writing to you now! (I don't remember if I mounted something). Anyhow, I can work now, and I will continue to fix this a little bit later. May be it's the part_msdos and the set root='(hd0,msdos5)' that I'm missing.

amrprznu
Desconectado
Joined: 08/12/2013

The part_msdos and changing set root='(hd0,msdos1)' to set root='(hd0,msdos5)' didn't work. I guess that the root partition has to be the boot partition for the menuentry for some reason.

I installed the busybox package from repo and uninstalled busybox-static. It didn't help. Then, I uninstalled busybox again and put back busybox-static, and the error message was gone in the busybox shell, but I was still dropped into it.
I am pretty sure that this has something to do with initramfs, but I can't get what it is.

Any ideas on how to get the manual procedure from above (comment #15: https://trisquel.info/en/forum/grub-cryptsetup-not-booting-after-kernel-update#comment-40635) automatically run at boot?

BlinkingArrow

I am a member!

Desconectado
Joined: 12/27/2011

Looking back at your first post you want to keep the line saying root='(hd0, msdos1)' because it is supposed to be the partition number of your /boot partition. It also seems that update-grub does find the necessary files, so I agree that it is probably a problem with initramfs.

As to automatically running the commands on comment #15, I'm not familiar with busybox. If it is possible to run scripts then writing one would be a solution.

Take a look in /etc/crypttab. It should have a line reading something similar to:

Lap-root UUID=[UUID of encrypted volume, not UUID of sda5] none luks

If it isn't there, add it. To get the UUID of Lap-root you'll have to mount it first. If you are using a key file then the third value should indicate it.

I think those commands you want to run automatically should be run if your crypttab contains that line, but I'm not sure. Also make sure that your fstab is configured to mount the encrypted volume as root, not the partition (/dev/mapper/Lap-root as opposed to /dev/sda5 or the UUID of sda5).

What happened when you ran "update-initramfs -u"?

amrprznu
Desconectado
Joined: 08/12/2013

Following four things:
1) Your advice

2) Linux Encrypted Filesystem with dm-crypt - FrontPage - CentOS ...
wiki.centos.org/HowTos/EncryptedFilesystem

3) How to configure LVM & LUKS to autodecrypt partition? - AskUbuntu
www.askubuntu.com/ questions/ 59487/ how-to-configure-lvm-luks-to-autodecrypt-partition

4) Encrypt an existingDebianGNU/Linuxinstallation with LUKS
ada.adrianlang.de/existing-debian-luks

I changed /etc/crypttab to look something like this:
root UUID=«really long code» none luks

and my /etc/fstab now contains
/dev/mapper/root / ext4 errors=remount-ro 0 1

I should say that I tried another value in the crypttab (instead of root), but it didn't work when I tried update-initrafms -u -k a. I kept getting messages from cryptsetup not knowing how to deal with it:
update-initramfs: Generating /boot/initrd.img-3.5.0-36-generic
device-mapper: table ioctl failed: No such device or address
Command failed
cryptsetup: WARNING: failed to determine cipher modules to load for cryproot

I think that it might have to do with me doing this in busybox:
(initramfs) cryptsetup luksOpen /dev/sda5 root
(initramfs) lvm
lvm> vgchange -a y Lap
lvm> CTRL+D
(initramfs) CTRL+D

I have no clue

I also used the advice from (4) and installed qemu:
aptitude install qemu-system xtightvncviewer

And ran the image on a virtual machine:
sudo qemu-system-x86_64 -hda /dev/sda -vnc :0 &
sudo vncviewer localhost

I am now worried about my keyboard distribution, because the disk is not un-encrypted in the virtual machine, and it even complains about some keystrokes.

[UPDATE6 - YESS!!! IT WORKS!!!!]
I should say that before running update-initramfs -u -k all again, I purged my linux-image and installed it again with headers and everything.

amrprznu
Desconectado
Joined: 08/12/2013

Lessons to be learnt:
1) initramfs is in charge of loading modules, scripts and hooks into the initrd.img files. These are files containing the kernel.
2) if you get to busybox because your kernel was not loaded, don't panic, you just probably only need the appropriate modules, scripts and hooks (check How to solve boot problems with Ubuntu after kernel upgrade: www.dedoimedo.com/computers/ubuntu-initrd-bug.html).
3) you need to do a chroot environment with the required modules and decrypting your dm-crypt or LUKS partition (https://trisquel.info/en/forum/grub-cryptsetup-not-booting-after-kernel-update#comment-40597)
4) just to make sure, purge cryptsetup (aptitude purge cryptsetup cryptsetup-bin libcryptsetup4) and reinstall (aptitude install cryptsetup cryptsetup-bin+M libcryptsetup4+M)
5) just to make sure, try installing another kernel or purge and reinstall the one you have now (I'm not giving instructions for this in case you are a newbie, and want to take it easy, but if you really dare use aptitude search '~ilinux' --disable-columns -F '%p' | grep `image\|headers' to see which ones are installed)
6) use the name for the current unencrypted partition (I used "root" in step 3: cryptsetup luksOpen "$encpart" root) and modify your /etc/crypttab so that it says something like (le'ts say I used root):
root UUID=«uuid of your encrypted partition» none luks

(to get the UUID you can type blkid). You are free to check the man-pages of crypttab: man crypttab
7) make sure that you have an according entry in /etc/fstab
/dev/mapper/root «path to mount it» «type» «options» 0 1

«path to mount it», «type» and «options»: are usually already there, but if not, you need to know how you formatted and where installed the system in the first place. You are also free to check the man-pages of mount and fstab: man fstab && man mount.
8-optional) you can try to see if it is possible that the thing will work by installing qemu:
sudo qemu-system-x86_64 -hda /dev/sda -vnc :0 &
sudo vncviewer localhost

You will get a GUI showing the booting process of your computer. It may not have the nice looks of the original, but if it works to the point of asking you for your password again, you are probably fine.

If you still get to busybox (initramfs) shell, then you can avoid restarting if you unencrypt manually (see https://trisquel.info/en/forum/grub-cryptsetup-not-booting-after-kernel-update#comment-40635)

Thanks to BlinkingArrow (https://trisquel.info/en/users/blinkingarrow) and Liberty (https://trisquel.info/en/users/liberty).

BlinkingArrow

I am a member!

Desconectado
Joined: 12/27/2011

You're welcome! I'm glad to hear that your computer is back to normal.

amrprznu
Desconectado
Joined: 08/12/2013

I don't know why or how, but I go a strange initrd.img-a in /boot/. I tried removing it as root, but every time I run update-initramfs -u -k all it comes back from the dead. Any ideas? Should I open a new discussion about this?

BlinkingArrow

I am a member!

Desconectado
Joined: 12/27/2011

What files do you have in the /boot partition? Delete all the files for previous kernels (no folders). If the initrd.img-a still shows then check in /etc/initramfs-tools for anything suspicious. On my system, the hooks folder is empty, as is anything under scripts.

And yes, it would probably be better if you open a new thread.

amrprznu
Desconectado
Joined: 08/12/2013

turns out that if you run update-initramfs -d -k «version» (in my case "a"), it deletes it completely. Then, I just removed every other file. Thanks again!

BlinkingArrow

I am a member!

Desconectado
Joined: 12/27/2011

So the solution was really that simple. While looking through the manpage I saw that, but figured it was too simple so I didn't mention it. You're welcome!