I thought to share some notes about the install & setup. I successfully installed & setup 1TB Crucial P3+ on my rpi5 with the nvme base for general purpose storage.
The ribbon IMO is delicate & it isn’t clear when inserting in the rpi5 side whether it bottomed or not, as maybe 0.5mm of the leads was still showing.
Pushing the clip down can potentially misalign the ribbon, care is required to ensure gentle force down is applied as pushing the clip down.
I was surprised that there wasn’t much dmseg nvme entries, attached is the entries I found since lastboot after having plugged the nvme drive (with some warning/errors)
I used nvme-cli tool to find out info about the drive, did nvme format followed by mkfs.ext4 to create a partition.
As discussed in another post, I was curious about nvme temperatures & how it may influence the rpi5. @bablokb it was an interesting journey as smartctl v7.4 blocks if there’s IO on the disk & I found out that the rpi5 kernel isn’t compiled with CONFIG_NVME_HWMON which I ventured & successfully did on another SD card.
I did a quick test, copied a 50GB file from a USB3 attached disk to the nvme drive. I measured an increase of 2-3C for the nvme drive and similar to the core CPU. Though the fan kicked in & was on for a fair bit of the copy duration. This isn’t a great test as it involves a cp from USB attached drive. In any case, @bablokb you were right, not a significant increase of temperature :)
Jan 28 14:03:37 rpi5 kernel: nvme nvme0: pci function 0000:01:00.0
Jan 28 14:03:37 rpi5 kernel: nvme 0000:01:00.0: enabling device (0000 -> 0002)
Jan 28 14:03:37 rpi5 kernel: nvme nvme0: allocated 64 MiB host memory buffer.
Jan 28 14:03:37 rpi5 kernel: nvme nvme0: 4/0/0 default/read/poll queues
Jan 28 14:03:38 rpi5 systemd[1]: nvmefc-boot-connections.service - Auto-connect to subsystems on FC-NVME devices found during boot was skipped because of an unmet condition check (ConditionPathExists=/sys/class/fc/fc_udev_device/nvme_discovery).
Jan 28 14:03:50 rpi5 systemd[1]: Starting nvmf-autoconnect.service - Connect NVMe-oF subsystems automatically during boot...
Jan 28 14:03:50 rpi5 modprobe[813]: modprobe: FATAL: Module nvme-fabrics not found in directory /lib/modules/6.1.0-rpi7-rpi-2712
Jan 28 14:03:50 rpi5 systemd[1]: Failed to start nvmf-autoconnect.service - Connect NVMe-oF subsystems automatically during boot.
PS: created this github issue to request enabling the kernel config for NVMR HWMON.
One reason there might be IO on the disk is the way mkfs.ext4 works: inode-tables and the journal are usually created in “lazy”-mode. The OS will create these entries in the background when it is not busy.
What are your timings for the USB-copy? I usually use time (cp source dest; sync) to time the copy including the buffer sync. BTW: this is the only use-case for me where PCIe-GEN3 makes a difference.
One last question: I also have a P3+, but it identifies itself as a “P2”. Could you please check? I am not sure if this is normal or if I have fake hardware.
@bablokb sorry, unsure if I follow. I meant to say that during the 50GB file copy via rsync, if I run smartctl to to get the temperature, it’d take a long time before it returns info. In my test, I ran sensors that uses the kernel NVME_HWMON. Did you want me to do a normal timed cp followed by sync just as another unrelated test?
As for Nvme ID, there’s no ID info or much info in dmesg, but if I run ‘sudo nvme list’ or ‘inxi -D’ I get the model:
Yes, but please put cp and sync in parenthesis: time (cp source dest; sync). I have with Gen3: 327,3 MB/s when I copy from an USB3-SATA SSD to the Crucial. cp won’t give you the throughput directly, but it is simple math :-) With rsync you might have lower throughput since it is more sophisticated and uses encryption.
Thanks, that was exactly what I was looking for. I will have to check my SSD again, but I remember seeing some string with P2.
I havent enabled Gen3 yet, so here’s my P3+ pibenchmark result & the 50GB (53467831999 bytes on disk) cp is running.
Oh my USB3 disk is not an SSD, its a WD Easystore HDD, so it won’t be a relevant comparison. In any case here’s the USB3 WD Easystore HDD to Crucial P3+ (~160MB/s)
time ( cp /mnt/USB3HDD/Justice_League.mkv /mnt/nvme/; sync)
real 5m18.146s
I also have a Samsung T7 shield (USB3-Gen2), 40GB file cp+sync to nvme ~ 346MB/s.
Your throughput is what to expect, especially the 346MB/s is near the practical limit of USB3, very nice. The Pi5 is limited to USB3.0 (now called USB 3.1 Gen1, but the docs from RPI still talk about USB3.0). And I think 160MB/s read speed is also good for a HDD.
Regarding rsync: you are right, rsync does not encrypt by itself, but for remote copies, it defaults to ssh which does the encryption.
With Gen2, I have 274,1 MB/s, so my 327,3 MB/s with Gen3 is 20% faster. Maybe this is just the limit of my USB-SSD, it is from 2012 but I don’t have anything newer available for tests.
It must be as my Samsung T7 ( usb3.1G2) with rpi5 (default pcie) got 346MB/s. Here’s its pibenchmark results.
I can try pciex1_gen=3 next for fun. Interesting same or tinybit slower (3s slower) than Gen2 but oc the biggest surprise is in the pibenchmark results 😄 now what to with all that speed given a measly 1G eth !