On Mon, May 22, 2023 at 05:11:39PM -0700, Dan Williams wrote:
> Yasunori Gotou (Fujitsu) wrote:
[...]
Hi,
There was some confusions about CXL device hotplug when I recently
tried to use Qemu to emulate CXL device hotplug and verify the relevant
functions of kernel.
> > Q1) Can PCIe hotplug driver detect and call CXL driver?
[...]
>
> Yes.
>
> The cxl_pci driver (drivers/cxl/pci.c) is just a typical PCI driver as
> far as the PCI hotplug driver is concerned. So add/remove events of a
> CXL card get turned into probe()/remove() events on the driver.
>
1. Can we divide steps of CXL device hotplug into two parts(PCI hotplug & Memory Hotplug)?
PCI Hotplug: the same as the native PCIe hotplug, including initializing cxl.io,
assigning PCIe BARs, allocating interrupts, etc. And the cxl_pci driver
is responsible for this part.
Memory Hotplug: focusing on enabling CXL memory including discovering and Configuring HDM,
extracting NUMA info from device, notifying memory management, etc.
> >
> > Q2) Can QEMU/KVM emulate CXL device hotplug?
> >
> > I heard that QEMU/KVM has PCIe device hotplug emulation, but I'm not sure
> > it can hotplug CXL device.
>
> It can, but as far as the driver is concerned you can achieve the same
> by:
>
> echo $devname > /sys/bus/pci/drivers/cxl_pci/unbind
>
> ...that exercises the same software flows as physical unplug.
>
2. What is the difference between "echo $devname > /sys/bus/pci/drivers/cxl_pci/unbind" and
"(qemu) device_del cxl-mem0" ?
According to the test, I found that "(qemu) device_del cxl-mem0" would directly
unplug the device and cause the interrupts on the cxl root port. It seems like this
operation would not only trigger cxl_pci driver but also pcieport driver.
The kernel dmesg is like below:
(qemu) device_del cxl-mem0
# dmesg
[ 699.057907] pcieport 0000:0c:00.0: pciehp: pending interrupts 0x0001 from Slot Status
[ 699.058929] pcieport 0000:0c:00.0: pciehp: Slot(0): Button press: will power off in 5 sec
[ 699.059986] pcieport 0000:0c:00.0: pciehp: pending interrupts 0x0010 from Slot Status
[ 699.060099] pcieport 0000:0c:00.0: pciehp: pciehp_set_indicators: SLOTCTRL 90 write cmd 2c0
Then I also tried "echo $devname > /sys/bus/pci/drivers/cxl_pci/unbind"
to check the behaviour of kernel. The kernel dmesg is like below:
# echo 0000:0d:00.0 > /sys/bus/pci/drivers/cxl_pci/unbind
# dmesg
[70387.978931] cxl_pci 0000:0d:00.0: vgaarb: pci_notify
[70388.021476] cxl_mem mem0: disconnect mem0 from port1
[70388.033099] pci 0000:0d:00.0: vgaarb: pci_notify
It seems like this operation would just unbind the cxl_pci driver from the cxl device.
Is my understanding about these two method correct?
3) Can I just use "ndctl/test/cxl-topology.sh" to test the cxl hotplug functions of kernel?
IIUC, cxl-topology.sh would utilize cxl_test (tools/testing/cxl) which is for regression
testing the kernel-user ABI.
PS: My qemu command line:
qemu-system-x86_64 \
-M q35,nvdimm=on,cxl=on \
-m 4G \
-smp 4 \
-object memory-backend-ram,size=2G,id=mem0 \
-numa node,nodeid=0,cpus=0-1,memdev=mem0 \
-object memory-backend-ram,size=2G,id=mem1 \
-numa node,nodeid=1,cpus=2-3,memdev=mem1 \
-object memory-backend-ram,size=256M,id=cxl-mem0 \
-device pxb-cxl,bus_nr=12,bus=pcie.0,id=cxl.1 \
-device cxl-rp,port=0,bus=cxl.1,id=root_port0,chassis=0,slot=0 \
-device cxl-type3,bus=root_port0,volatile-memdev=cxl-mem0,id=cxl-mem0 \
-M cxl-fmw.0.targets.0=cxl.1,cxl-fmw.0.size=4G,cxl-fmw.0.interleave-granularity=4k \
-hda ../disk/ubuntu_x86_test_new.qcow2 \
-nographic \
Qemu version: 8.2.50, the lastest commit of branch cxl-2024-03-05 in "https://gitlab.com/jic23/qemu"
Kernel version: 6.8.0-rc6
Many thanks
Yuquan