Issues with NVME base and Kingston NV2 2TB, PCIe 4.0

Support
1

I flashed Ubuntu 24.04 LTS on Kingston NV2 2TB using rpi-imager and I was able to boot from the SSD.
I did a hdparm read test on both Gen2 and Gen3. On Gen3 I was getting about 600-700 MB/sec. At this point I started using rsync to copy large files from a WD usb external hard drive. I was getting speeds of about 60-70 MB/sec then after about 100GB the speed started dropping to values under 3 MB/Sec. When I use rsync I start seeing these errors in journalctl:

May 27 10:15:32 pi-master kernel: nvme nvme0: I/O tag 0 (3000) opcode 0x1 (I/O Cmd) QID 1 timeout, aborting req_op:WRITE(1) size:4096
May 27 10:15:32 pi-master kernel: nvme nvme0: Abort status: 0x0
May 27 10:15:32 pi-master kernel: nvme nvme0: I/O tag 209 (10d1) opcode 0x1 (I/O Cmd) QID 2 timeout, aborting req_op:WRITE(1) size:131072

Other unexpected behaviours that I am seeing:

  • terminal history is not persisted between ssh sessions, last item in history is from the first time I ran rsync
  • changes in /boot/firmware/cmdline.txt are not persisted, I was able to edit it before using rsync (I added cgroup_enable=cpuset cgroup_memory=1 cgroup_enable=memory for k3s)
  • hdparm results dropped from 600-700 MB/sec to 30-50 MB/sec

lspci output:
0000:00:00.0 PCI bridge: Broadcom Inc. and subsidiaries BCM2712 PCIe Bridge (rev 21)
0000:01:00.0 Non-Volatile memory controller: Realtek Semiconductor Co., Ltd. RTS5765DL NVMe SSD Controller (DRAM-less) (rev 01)
0001:00:00.0 PCI bridge: Broadcom Inc. and subsidiaries BCM2712 PCIe Bridge (rev 21)
0001:01:00.0 Ethernet controller: Raspberry Pi Ltd RP1 PCIe 2.0 South Bridge

Could someone please help me solve this issue?
Let me know if I can provide other details.

Thank you!

1 Like
2

I’d go back to Gen2 mode and see what happens. PCIe 3 is experimental and not-officially-supported on the NVMe Base.

3

I flashed Raspberry Pi OS Lite and on Gen2 mode started copying using rsync. It went fine for about 1 hour. Then I started seeing these messages in journalctl:

May 27 13:07:07 pi-master kernel: nvme nvme0: I/O 153 QID 3 timeout, reset controller
May 27 13:07:07 pi-master kernel: nvme nvme0: Abort status: 0x0
May 27 13:07:07 pi-master kernel: nvme nvme0: Abort status: 0x0
May 27 13:07:07 pi-master kernel: nvme nvme0: Abort status: 0x0
May 27 13:07:07 pi-master kernel: nvme nvme0: Abort status: 0x0
May 27 13:07:07 pi-master kernel: nvme nvme0: Abort status: 0x0
May 27 13:07:20 pi-master kernel: nvme nvme0: 4/0/0 default/read/poll queues
May 27 13:07:50 pi-master kernel: nvme nvme0: I/O 435 QID 3 timeout, disable controller

At this point I can’t even execute commands like ls:

pi@pi-master:~ $ rsync -ah --progress /mnt/hdd/ /mnt/ssd/
-bash: /usr/bin/rsync: Input/output error
pi@pi-master:~ $ ls
-bash: ls: command not found
4

This is outside my area of expertise, best guess is maybe its a power issue? I’ve seen a few posts where users have had to wire up supplemental power to the two pads on the NVMe Base.