Received-SPF: pass (google.com: domain of linux-wireless+bounces-1969-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.48.161 as permitted sender) client-ip=147.75.48.161;
Message-ID: <543d9d2f-e132-4fdb-8454-68595cd20515@quicinc.com>
Date: Tue, 16 Jan 2024 18:08:35 +0800
Precedence: bulk
MIME-Version: 1.0
User-Agent: Mozilla Thunderbird
Subject: Re: ath11k and vfio-pci support
Content-Language: en-US
To: Alex Williamson <alex.williamson@redhat.com>,
        Kalle Valo
	<kvalo@kernel.org>
CC: James Prestwood <prestwoj@gmail.com>, <linux-wireless@vger.kernel.org>,
        <ath11k@lists.infradead.org>, David Woodhouse <dwmw2@infradead.org>,
        <iommu@lists.linux.dev>
References: <adcb785e-4dc7-4c4a-b341-d53b72e13467@gmail.com>
 <8734v5zhol.fsf@kernel.org> <87fa5220-6fd9-433d-879b-c55ac67a0748@gmail.com>
 <87r0ipcn7j.fsf@kernel.org> <356e0b05-f396-4ad7-9b29-c492b54af834@gmail.com>
 <26119c3f-9012-47bb-948e-7e976d4773a7@quicinc.com>
 <87mstccmk6.fsf@kernel.org> <df9fd970-5af3-468c-b1f1-18f91215cf44@gmail.com>
 <8734v4auc4.fsf@kernel.org> <e8878979-1f3f-4635-a716-9ac381c617d9@gmail.com>
 <285b84d0-229c-4c83-a7d6-4c3c23139597@quicinc.com>
 <4607fb37-8227-49a3-9e8c-10c9b117ec7b@gmail.com>
 <3d22a730-aee5-4f2a-9ddc-b4b5bd4d62fe@quicinc.com>
 <87il3w7zjh.fsf@kernel.org>
 <20240115104658.0b56bd35.alex.williamson@redhat.com>
From: Baochen Qiang <quic_bqiang@quicinc.com>
In-Reply-To: <20240115104658.0b56bd35.alex.williamson@redhat.com>
Content-Type: text/plain; charset="UTF-8"; format=flowed
Content-Transfer-Encoding: 7bit


On 1/16/2024 1:46 AM, Alex Williamson wrote:
> On Sun, 14 Jan 2024 16:36:02 +0200
> Kalle Valo <kvalo@kernel.org> wrote:
> 
>> Baochen Qiang <quic_bqiang@quicinc.com> writes:
>>
>>>>> Strange that still fails. Are you now seeing this error in your
>>>>> host or your Qemu? or both?
>>>>> Could you share your test steps? And if you can share please be as
>>>>> detailed as possible since I'm not familiar with passing WLAN
>>>>> hardware to a VM using vfio-pci.
>>>>
>>>> Just in Qemu, the hardware works fine on my host machine.
>>>> I basically follow this guide to set it up, its written in the
>>>> context of GPUs/libvirt but the host setup is exactly the same. By
>>>> no means do you need to read it all, once you set the vfio-pci.ids
>>>> and see your unclaimed adapter you can stop:
>>>> https://wiki.archlinux.org/title/PCI_passthrough_via_OVMF
>>>> In short you should be able to set the following host kernel options
>>>> and reboot (assuming your motherboard/hardware is compatible):
>>>> intel_iommu=on iommu=pt vfio-pci.ids=17cb:1103
>>>> Obviously change the device/vendor IDs to whatever ath11k hw you
>>>> have. Once the host is rebooted you should see your wlan adapter as
>>>> UNCLAIMED, showing the driver in use as vfio-pci. If not, its likely
>>>> your motherboard just isn't compatible, the device has to be in its
>>>> own IOMMU group (you could try switching PCI ports if this is the
>>>> case).
>>>> I then build a "kvm_guest.config" kernel with the driver/firmware
>>>> for ath11k and boot into that with the following Qemu options:
>>>> -enable-kvm -device -vfio-pci,host=<PCI address>
>>>> If it seems easier you could also utilize IWD's test-runner which
>>>> handles launching the Qemu kernel automatically, detecting any
>>>> vfio-devices and passes them through and mounts some useful host
>>>> folders into the VM. Its actually a very good general purpose tool
>>>> for kernel testing, not just for IWD:
>>>> https://git.kernel.org/pub/scm/network/wireless/iwd.git/tree/doc/test-runner.txt
>>>> Once set up you can just run test-runner with a few flags and you'll
>>>> boot into a shell:
>>>> ./tools/test-runner -k <kernel-image> --hw --start /bin/bash
>>>> Please reach out if you have questions, thanks for looking into
>>>> this.
>>>
>>> Thanks for these details. I reproduced this issue by following your guide.
>>>
>>> Seems the root cause is that the MSI vector assigned to WCN6855 in
>>> qemu is different with that in host. In my case the MSI vector in qemu
>>> is [Address: fee00000  Data: 0020] while in host it is [Address:
>>> fee00578 Data: 0000]. So in qemu ath11k configures MSI vector
>>> [Address: fee00000 Data: 0020] to WCN6855 hardware/firmware, and
>>> firmware uses that vector to fire interrupts to host/qemu. However
>>> host IOMMU doesn't know that vector because the real vector is
>>> [Address: fee00578  Data: 0000], as a result host blocks that
>>> interrupt and reports an error, see below log:
>>>
>>> [ 1414.206069] DMAR: DRHD: handling fault status reg 2
>>> [ 1414.206081] DMAR: [INTR-REMAP] Request device [02:00.0] fault index
>>> 0x0 [fault reason 0x25] Blocked a compatibility format interrupt
>>> request
>>> [ 1414.210334] DMAR: DRHD: handling fault status reg 2
>>> [ 1414.210342] DMAR: [INTR-REMAP] Request device [02:00.0] fault index
>>> 0x0 [fault reason 0x25] Blocked a compatibility format interrupt
>>> request
>>> [ 1414.212496] DMAR: DRHD: handling fault status reg 2
>>> [ 1414.212503] DMAR: [INTR-REMAP] Request device [02:00.0] fault index
>>> 0x0 [fault reason 0x25] Blocked a compatibility format interrupt
>>> request
>>> [ 1414.214600] DMAR: DRHD: handling fault status reg 2
>>>
>>> While I don't think there is a way for qemu/ath11k to get the real MSI
>>> vector from host, I will try to read the vfio code to check further.
>>> Before that, to unblock you, a possible hack is to hard code the MSI
>>> vector in qemu to the same as in host, on condition that the MSI
>>> vector doesn't change.
>>
>> Baochen, awesome that you were able to debug this further. Now we at
>> least know what's the problem.
> 
> It's an interesting problem, I don't think we've seen another device
> where the driver reads the MSI register in order to program another
> hardware entity to match the MSI address and data configuration.
> 
> When assigning a device, the host and guest use entirely separate
> address spaces for MSI interrupts.  When the guest enables MSI, the
> operation is trapped by the VMM and triggers an ioctl to the host to
> perform an equivalent configuration.  Generally the physical device
> will interrupt within the host where it may be directly attached to KVM
> to signal the interrupt, trigger through the VMM, or where
> virtualization hardware supports it, the interrupt can directly trigger
> the vCPU.   From the VM perspective, the guest address/data pair is used
> to signal the interrupt, which is why it makes sense to virtualize the
> MSI registers.
Hi Alex, could you help elaborate more? why from the VM perspective MSI 
virtualization is necessary?

And, maybe a stupid question, is that possible VM/KVM or vfio only 
virtualize write operation to MSI register but leave read operation 
un-virtualized? I am asking this because in that way ath11k may get a 
chance to run in VM after getting the real vector.

> 
> Off hand I don't have a good solution for this, the hardware is
> essentially imposing a unique requirement for MSI programming that the
> driver needs visibility of the physical MSI address and data.  It's
> conceivable that device specific code could either make the physical
> address/data pair visible to the VM or trap the firmware programming to
> inject the correct physical values.  Is there somewhere other than the
> standard MSI capability in config space that the driver could learn the
> physical values, ie. somewhere that isn't virtualized?  Thanks,
I don't think we have such capability in configuration space.

> 
> Alex
>