NVMe Base refuses to boot

Support
1

Hi!

Got myself a very nice NVMe Base and a Kingston SKC3000D2048G disk.

I followed Geerling’s guide for setting up the Rpi and cloning the working RPi install to the NVMe drive (NVMe SSD boot with the Raspberry Pi 5 | Jeff Geerling) and it works. The partitions are there, as is the contents.

However, I cannot get the RPi to boot from the NVMe. I can’t seem to figure out what I’ve done wrong (I’ve done all the boot order stuff and such that Geerling documents and have browsed the forums here and elsewhere to try to figure out what is wrong.

I’m close to swapping strategy and try to use bablokb’s pi-boot-switch (GitHub - bablokb/pi-boot-switch: Boot-Switcher for Raspberry Pi multiboot systems), but I’d rather boot directly off the NVMe and skip the SD-card alltogether. So, could I get one more pair of eyes to look over what I’ve got:

e8johan@cloud:~ $ mount
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
proc on /proc type proc (rw,relatime)
udev on /dev type devtmpfs (rw,nosuid,relatime,size=3952000k,nr_inodes=247000,mode=755)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,nodev,noexec,relatime,size=824704k,mode=755)
/dev/mmcblk0p2 on / type ext4 (rw,noatime)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)
tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k)
cgroup2 on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot)
pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=11972)
mqueue on /dev/mqueue type mqueue (rw,nosuid,nodev,noexec,relatime)
debugfs on /sys/kernel/debug type debugfs (rw,nosuid,nodev,noexec,relatime)
tracefs on /sys/kernel/tracing type tracefs (rw,nosuid,nodev,noexec,relatime)
fusectl on /sys/fs/fuse/connections type fusectl (rw,nosuid,nodev,noexec,relatime)
configfs on /sys/kernel/config type configfs (rw,nosuid,nodev,noexec,relatime)
ramfs on /run/credentials/systemd-sysctl.service type ramfs (ro,nosuid,nodev,noexec,relatime,mode=700)
ramfs on /run/credentials/systemd-sysusers.service type ramfs (ro,nosuid,nodev,noexec,relatime,mode=700)
ramfs on /run/credentials/systemd-tmpfiles-setup-dev.service type ramfs (ro,nosuid,nodev,noexec,relatime,mode=700)
/dev/mmcblk0p1 on /boot/firmware type vfat (rw,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=ascii,shortname=mixed,errors=remount-ro)
ramfs on /run/credentials/systemd-tmpfiles-setup.service type ramfs (ro,nosuid,nodev,noexec,relatime,mode=700)
binfmt_misc on /proc/sys/fs/binfmt_misc type binfmt_misc (rw,nosuid,nodev,noexec,relatime)
sunrpc on /run/rpc_pipefs type rpc_pipefs (rw,relatime)
tmpfs on /run/user/1000 type tmpfs (rw,nosuid,nodev,relatime,size=824688k,nr_inodes=206172,mode=700,uid=1000,gid=1000)

e8johan@cloud:~ $ sudo journalctl -b -g pcie
Feb 03 18:49:32 cloud kernel: Kernel command line: reboot=w coherent_pool=1M 8250.nr_uarts=1 pci=pcie_bus_safe snd_bcm2835.enable_compat_alsa=0 snd_bcm2835.enable_hdmi=1  smsc95xx.macaddr=D8:3A:DD:A0:02:66 vc_mem.mem_base=0x3fc00000 vc>
Feb 03 18:49:32 cloud kernel: brcm-pcie 1000110000.pcie: host bridge /axi/pcie@110000 ranges:
Feb 03 18:49:32 cloud kernel: brcm-pcie 1000110000.pcie:   No bus range found for /axi/pcie@110000, using [bus 00-ff]
Feb 03 18:49:32 cloud kernel: brcm-pcie 1000110000.pcie:      MEM 0x1b00000000..0x1bfffffffb -> 0x0000000000
Feb 03 18:49:32 cloud kernel: brcm-pcie 1000110000.pcie:      MEM 0x1800000000..0x1affffffff -> 0x0400000000
Feb 03 18:49:32 cloud kernel: brcm-pcie 1000110000.pcie:   IB MEM 0x0000000000..0x0fffffffff -> 0x1000000000
Feb 03 18:49:32 cloud kernel: brcm-pcie 1000110000.pcie: setting SCB_ACCESS_EN, READ_UR_MODE, MAX_BURST_SIZE
Feb 03 18:49:32 cloud kernel: brcm-pcie 1000110000.pcie: Forcing gen 2
Feb 03 18:49:32 cloud kernel: brcm-pcie 1000110000.pcie: PCI host bridge to bus 0000:00
Feb 03 18:49:32 cloud kernel: brcm-pcie 1000110000.pcie: link up, 5.0 GT/s PCIe x1 (!SSC)
Feb 03 18:49:32 cloud kernel: pci 0000:01:00.0: 4.000 Gb/s available PCIe bandwidth, limited by 5.0 GT/s PCIe x1 link at 0000:00:00.0 (capable of 63.012 Gb/s with 16.0 GT/s PCIe x4 link)
Feb 03 18:49:32 cloud kernel: pcieport 0000:00:00.0: enabling device (0000 -> 0002)
Feb 03 18:49:32 cloud kernel: pcieport 0000:00:00.0: PME: Signaling with IRQ 39
Feb 03 18:49:32 cloud kernel: pcieport 0000:00:00.0: AER: enabled with IRQ 39
Feb 03 18:49:32 cloud kernel: brcm-pcie 1000110000.pcie: clkreq control enabled
Feb 03 18:49:32 cloud kernel: brcm-pcie 1000120000.pcie: host bridge /axi/pcie@120000 ranges:
Feb 03 18:49:32 cloud kernel: brcm-pcie 1000120000.pcie:   No bus range found for /axi/pcie@120000, using [bus 00-ff]
Feb 03 18:49:32 cloud kernel: brcm-pcie 1000120000.pcie:      MEM 0x1f00000000..0x1ffffffffb -> 0x0000000000
Feb 03 18:49:32 cloud kernel: brcm-pcie 1000120000.pcie:      MEM 0x1c00000000..0x1effffffff -> 0x0400000000
Feb 03 18:49:32 cloud kernel: brcm-pcie 1000120000.pcie:   IB MEM 0x1f00000000..0x1f003fffff -> 0x0000000000
Feb 03 18:49:32 cloud kernel: brcm-pcie 1000120000.pcie:   IB MEM 0x0000000000..0x0fffffffff -> 0x1000000000
Feb 03 18:49:32 cloud kernel: brcm-pcie 1000120000.pcie: setting SCB_ACCESS_EN, READ_UR_MODE, MAX_BURST_SIZE
Feb 03 18:49:32 cloud kernel: brcm-pcie 1000120000.pcie: Forcing gen 2
Feb 03 18:49:32 cloud kernel: brcm-pcie 1000120000.pcie: PCI host bridge to bus 0001:00
Feb 03 18:49:32 cloud kernel: brcm-pcie 1000120000.pcie: link up, 5.0 GT/s PCIe x4 (!SSC)
Feb 03 18:49:32 cloud kernel: pcieport 0001:00:00.0: enabling device (0000 -> 0002)
Feb 03 18:49:32 cloud kernel: pcieport 0001:00:00.0: PME: Signaling with IRQ 45
Feb 03 18:49:32 cloud kernel: pcieport 0001:00:00.0: AER: enabled with IRQ 45
Feb 03 18:49:37 cloud ModemManager[774]: <info>  [base-manager] couldn't check support for device '/sys/devices/platform/axi/1000120000.pcie/1f00100000.ethernet': not supported by any plugin
Feb 03 19:01:29 cloud sudo[976]:  e8johan : TTY=pts/0 ; PWD=/home/e8johan ; USER=root ; COMMAND=/usr/bin/journalctl -b -g pcie


e8johan@cloud:~ $ sudo fdisk -l
Disk /dev/ram0: 4 MiB, 4194304 bytes, 8192 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 16384 bytes
I/O size (minimum/optimal): 16384 bytes / 16384 bytes
 
 
...
 
 
Disk /dev/ram15: 4 MiB, 4194304 bytes, 8192 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 16384 bytes
I/O size (minimum/optimal): 16384 bytes / 16384 bytes
 
 
Disk /dev/nvme0n1: 1.86 TiB, 2048408248320 bytes, 4000797360 sectors
Disk model: KINGSTON SKC3000D2048G                  
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x95d84451
 
Device         Boot   Start        End    Sectors  Size Id Type
/dev/nvme0n1p1         8192    1056767    1048576  512M  c W95 FAT32 (LBA)
/dev/nvme0n1p2      1056768 4000797359 3999740592  1.9T 83 Linux
 
 
Disk /dev/mmcblk0: 29.72 GiB, 31914983424 bytes, 62333952 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x1ff30c49
 
Device         Boot   Start      End  Sectors  Size Id Type
/dev/mmcblk0p1         8192  1056767  1048576  512M  c W95 FAT32 (LBA)
/dev/mmcblk0p2      1056768 62333951 61277184 29.2G 83 Linux
2

I think? you may have to set the boot order twice? Once on the SD card to do the first time boot to the NVME ( I didn’t clone). Then again (sudo raspi-config) on that instance of PI OS (once you get through the first time boot setup) so it boots from the NVME?
The first time setting mine up it wouldn’t boot to the NVME unless my SD card was inserted. I’m pretty sure running raspi-config a second time sorted it out for me. I followed the getting started here,
NVMe Base for Raspberry Pi 5 – NVMe Base (pimoroni.com)

And had a look see at the video. I’m a linux noob, just so you know ;) .

1 Like
3

I’m not sure what you mean with setting the boot order on the SD card. I was under the impression that the boot order was stored in the eeprom on the RPi (so not on the SD card, nor the NVMe).

I tried booting without the SD card and nothing happens, so for some reason the bootloader does not pick up the NVMe (or the boot partition from the NVMe looks for an SD card for some odd reason…)

This is a headless system. Maybe I need to go and find a screen to connect…

4

Like I said, I’m a Linux Noob. I could have it all wrong? That was what I “seem to remember” happening to me. It’s not hard for me to get lost on what is where sometimes ;). Can’t hurt to do it again (sudo raspi-config), and see what happens? If you haven’t already.

5

Oh by the way, speaking of firmware
It’s not /boot/config.txt
It’s now /boot/firmware/config.txt

Editing /boot/config.txt will do nothing. ;)

6

Noob or not, you’re helping - thx!

Also, something is messed up with the initramfs. I bet it is because of the cloning…

IMG_20240203_203627485_HDR
IMG_20240203_203627485_HDR1920×2560 444 KB

1 Like
7

Just curious as to why you cloned?

8

Found this one: official rpios bookworm issue · Issue #168 · billw2/rpi-clone · GitHub

I’m going to try the suggested fix later tonight.

9

I run the pi headless, so I can’t run the imager as it needs a display. Also, Geerling provided instructions for how to do it, so I assumed it would work :-)

10

I can confirm that this was the issue.

e8johan@cloud:~ $ blkid
/dev/nvme0n1p1: UUID="A937-6E9E" BLOCK_SIZE="512" TYPE="vfat" PARTUUID="95d84451-01"
/dev/nvme0n1p2: LABEL="molnet" UUID="1f3d5dea-750d-4cde-be8a-b271dff9ccab" BLOCK_SIZE="4096" TYPE="ext4" PARTUUID="95d84451-02"
/dev/mmcblk0p1: LABEL_FATBOOT="bootfs" LABEL="bootfs" UUID="5DF9-E225" BLOCK_SIZE="512" TYPE="vfat" PARTUUID="1ff30c49-01"
/dev/mmcblk0p2: LABEL="rootfs" UUID="3b614a3f-4a65-4480-876a-8a998e01ac9b" BLOCK_SIZE="4096" TYPE="ext4" PARTUUID="1ff30c49-02"

Compared to the cmdline.txt on the boot partition on the NVMe disk:

e8johan@cloud:~ $ cat t/cmdline.txt 
console=serial0,115200 console=tty1 root=PARTUUID=1ff30c49-02 rootfstype=ext4 fsck.repair=yes rootwait cfg80211.ieee80211_regdom=SE

A quick edit later, and the PARTUUID is 95d84451-02 and the NVMe boots like a charm!

1 Like
11

Nice, blind luck or not ( on my part), its still nice when things get fixed. =)

1 Like