Hi,
For a while now I've been using vfio-pci to pass through wireless
hardware to a Qemu VM. This makes testing kernel changes quick and easy
compared to swapping the host kernel for reach iterative change. So far
I've had very few issues doing this, maybe I've just been lucky... I
tried doing this with an ath11k (WCN6855) card and ran into issues with
the driver starting up. I'm wondering if its a configuration issue or
just a lack of support by ath11k? The card works just fine when I use it
on my host machine. Based on the logs it may not even be related to
ath11k as wmi-bmof seems to fail first, but I'm not familiar with
anything at the PCI level so I've got no idea whats going on.
My host machine is running 5.19
Booting into a 6.2 kernel with Qemu
Below is a partial kernel log of relevant messages:
[ 0.318120] ath11k_pci 0000:00:06.0: BAR 0: assigned [mem
0x100000000-0x1001fffff 64bit]
[ 0.319006] ath11k_pci 0000:00:06.0: MSI vectors: 1
[ 0.319075] ath11k_pci 0000:00:06.0: wcn6855 hw2.1
[ 0.475326] mhi mhi0: Requested to power ON
[ 0.475333] mhi mhi0: Power on setup success
[ 1.440375] Driver 'wmi-bmof' was unable to register with bus_type
'wmi' because the bus was not initialized.
[ 1.459667] ath11k_pci 0000:00:06.0: chip_id 0x12 chip_family 0xb
board_id 0xff soc_id 0x400c1211
[ 1.459674] ath11k_pci 0000:00:06.0: fw_version 0x1109996e
fw_build_timestamp 2023-12-19 11:11 fw_build_id
WLAN.HSP.1.1-03125-QCAHSPSWPL_V1_V2_SILICONZ_LITE-3.6510.36
[ 1.528705] ath11k_pci 0000:00:06.0: leaving PCI ASPM disabled to
avoid MHI M2 problems
[ 2.537835] ath11k_pci 0000:00:06.0: failed to receive control
response completion, polling..
[ 3.562151] ath11k_pci 0000:00:06.0: Service connect timeout
[ 3.563511] ath11k_pci 0000:00:06.0: failed to connect to HTT: -110
[ 3.565375] ath11k_pci 0000:00:06.0: failed to start core: -110
[ 3.776067] ath11k_pci 0000:00:06.0: ignore reset dev flags 0x4000
[ 13.802149] ath11k_pci 0000:00:06.0: failed to wait wlan mode request
(mode 4): -110
[ 13.802163] ath11k_pci 0000:00:06.0: qmi failed to send wlan mode
off: -110
+ ath11k list
James Prestwood <[email protected]> writes:
> For a while now I've been using vfio-pci to pass through wireless
> hardware to a Qemu VM. This makes testing kernel changes quick and
> easy compared to swapping the host kernel for reach iterative change.
> So far I've had very few issues doing this, maybe I've just been
> lucky... I tried doing this with an ath11k (WCN6855) card and ran into
> issues with the driver starting up. I'm wondering if its a
> configuration issue or just a lack of support by ath11k? The card
> works just fine when I use it on my host machine. Based on the logs it
> may not even be related to ath11k as wmi-bmof seems to fail first, but
> I'm not familiar with anything at the PCI level so I've got no idea
> whats going on.
There's a bug report about this:
https://bugzilla.kernel.org/show_bug.cgi?id=216055
But I have also no idea what is causing this, I guess we are doing
something wrong with the PCI communication? That reminds me, you could
try this in case that helps:
https://patchwork.kernel.org/project/linux-wireless/patch/[email protected]/
--
https://patchwork.kernel.org/project/linux-wireless/list/
https://wireless.wiki.kernel.org/en/developers/documentation/submittingpatches
Hi Kalle,
On 1/10/24 1:00 AM, Kalle Valo wrote:
> + ath11k list
>
> James Prestwood <[email protected]> writes:
>
>> For a while now I've been using vfio-pci to pass through wireless
>> hardware to a Qemu VM. This makes testing kernel changes quick and
>> easy compared to swapping the host kernel for reach iterative change.
>> So far I've had very few issues doing this, maybe I've just been
>> lucky... I tried doing this with an ath11k (WCN6855) card and ran into
>> issues with the driver starting up. I'm wondering if its a
>> configuration issue or just a lack of support by ath11k? The card
>> works just fine when I use it on my host machine. Based on the logs it
>> may not even be related to ath11k as wmi-bmof seems to fail first, but
>> I'm not familiar with anything at the PCI level so I've got no idea
>> whats going on.
> There's a bug report about this:
>
> https://bugzilla.kernel.org/show_bug.cgi?id=216055
Yes I saw that, but apparently didn't read past the first comment... I'm
not seeing any of the DMAR faults but the ath-releated ones look the
same. Good to know its not just me.
>
> But I have also no idea what is causing this, I guess we are doing
> something wrong with the PCI communication? That reminds me, you could
> try this in case that helps:
>
> https://patchwork.kernel.org/project/linux-wireless/patch/[email protected]/
Heh, I saw this pop up a day after I sent this and was wondering. Is
this something I'd need on the host kernel, guest, or both?
Thanks,
James
James Prestwood <[email protected]> writes:
>> But I have also no idea what is causing this, I guess we are doing
>> something wrong with the PCI communication? That reminds me, you could
>> try this in case that helps:
>>
>> https://patchwork.kernel.org/project/linux-wireless/patch/[email protected]/
>
> Heh, I saw this pop up a day after I sent this and was wondering. Is
> this something I'd need on the host kernel, guest, or both?
On the guest where ath11k is running. I'm not optimistic that this would
solve your issue, I suspect there can be also other bugs, but good to
know if the patch changes anything.
--
https://patchwork.kernel.org/project/linux-wireless/list/
https://wireless.wiki.kernel.org/en/developers/documentation/submittingpatches
Hi Kalle,
On 1/10/24 5:49 AM, Kalle Valo wrote:
> James Prestwood <[email protected]> writes:
>
>>> But I have also no idea what is causing this, I guess we are doing
>>> something wrong with the PCI communication? That reminds me, you could
>>> try this in case that helps:
>>>
>>> https://patchwork.kernel.org/project/linux-wireless/patch/[email protected]/
>> Heh, I saw this pop up a day after I sent this and was wondering. Is
>> this something I'd need on the host kernel, guest, or both?
> On the guest where ath11k is running. I'm not optimistic that this would
> solve your issue, I suspect there can be also other bugs, but good to
> know if the patch changes anything.
Looks the same here, didn't seem to change anything based on the kernel
logs.
Thanks,
James
On 1/10/2024 10:55 PM, James Prestwood wrote:
> Hi Kalle,
>
> On 1/10/24 5:49 AM, Kalle Valo wrote:
>> James Prestwood <[email protected]> writes:
>>
>>>> But I have also no idea what is causing this, I guess we are doing
>>>> something wrong with the PCI communication? That reminds me, you could
>>>> try this in case that helps:
>>>>
>>>> https://patchwork.kernel.org/project/linux-wireless/patch/[email protected]/
>>> Heh, I saw this pop up a day after I sent this and was wondering. Is
>>> this something I'd need on the host kernel, guest, or both?
>> On the guest where ath11k is running. I'm not optimistic that this would
>> solve your issue, I suspect there can be also other bugs, but good to
>> know if the patch changes anything.
>
> Looks the same here, didn't seem to change anything based on the kernel
> logs.
>
> Thanks,
>
> James
>
Could you try this?
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/drivers/net/wireless/ath/ath11k/pci.c?id=39564b475ac5a589e6c22c43a08cbd283c295d2c
Baochen Qiang <[email protected]> writes:
> On 1/10/2024 10:55 PM, James Prestwood wrote:
>> Hi Kalle,
>> On 1/10/24 5:49 AM, Kalle Valo wrote:
>>> James Prestwood <[email protected]> writes:
>>>
>>>>> But I have also no idea what is causing this, I guess we are doing
>>>>> something wrong with the PCI communication? That reminds me, you could
>>>>> try this in case that helps:
>>>>>
>>>>> https://patchwork.kernel.org/project/linux-wireless/patch/[email protected]/
>>>>
>>>> Heh, I saw this pop up a day after I sent this and was wondering. Is
>>>> this something I'd need on the host kernel, guest, or both?
>>>
>>> On the guest where ath11k is running. I'm not optimistic that this would
>>> solve your issue, I suspect there can be also other bugs, but good to
>>> know if the patch changes anything.
>>
>> Looks the same here, didn't seem to change anything based on the
>> kernel logs.
>>
> Could you try this?
>
> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/drivers/net/wireless/ath/ath11k/pci.c?id=39564b475ac5a589e6c22c43a08cbd283c295d2c
This reminds me, I assumed James was testing with ath.git master branch
(which has that commit) but I never checked that. So for testing please
always use the master branch to get the latest and greatest ath11k:
https://git.kernel.org/pub/scm/linux/kernel/git/kvalo/ath.git/
There's a quite long delay from ath.git to official releases.
--
https://patchwork.kernel.org/project/linux-wireless/list/
https://wireless.wiki.kernel.org/en/developers/documentation/submittingpatches
Hi Kalle, Baochen,
On 1/11/24 12:16 AM, Kalle Valo wrote:
> Baochen Qiang <[email protected]> writes:
>
>> On 1/10/2024 10:55 PM, James Prestwood wrote:
>>> Hi Kalle,
>>> On 1/10/24 5:49 AM, Kalle Valo wrote:
>>>> James Prestwood <[email protected]> writes:
>>>>
>>>>>> But I have also no idea what is causing this, I guess we are doing
>>>>>> something wrong with the PCI communication? That reminds me, you could
>>>>>> try this in case that helps:
>>>>>>
>>>>>> https://patchwork.kernel.org/project/linux-wireless/patch/[email protected]/
>>>>> Heh, I saw this pop up a day after I sent this and was wondering. Is
>>>>> this something I'd need on the host kernel, guest, or both?
>>>> On the guest where ath11k is running. I'm not optimistic that this would
>>>> solve your issue, I suspect there can be also other bugs, but good to
>>>> know if the patch changes anything.
>>> Looks the same here, didn't seem to change anything based on the
>>> kernel logs.
>>>
>> Could you try this?
>>
>> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/drivers/net/wireless/ath/ath11k/pci.c?id=39564b475ac5a589e6c22c43a08cbd283c295d2c
> This reminds me, I assumed James was testing with ath.git master branch
> (which has that commit) but I never checked that. So for testing please
> always use the master branch to get the latest and greatest ath11k:
>
> https://git.kernel.org/pub/scm/linux/kernel/git/kvalo/ath.git/
>
> There's a quite long delay from ath.git to official releases.
Good to know, and I was not in fact using that branch. Rebuilt from
ath.git/master but still roughly the same behavior. There does appear to
be more output now though, specifically a firmware crash:
[ 2.281721] ath11k_pci 0000:00:06.0: failed to receive control
response completion, polling..
[ 2.282101] ip (65) used greatest stack depth: 12464 bytes left
[ 3.306039] ath11k_pci 0000:00:06.0: Service connect timeout
[ 3.307588] ath11k_pci 0000:00:06.0: failed to connect to HTT: -110
[ 3.309286] ath11k_pci 0000:00:06.0: failed to start core: -110
[ 3.519637] ath11k_pci 0000:00:06.0: firmware crashed: MHI_CB_EE_RDDM
[ 3.519678] ath11k_pci 0000:00:06.0: ignore reset dev flags 0x4000
[ 3.627087] ath11k_pci 0000:00:06.0: firmware crashed: MHI_CB_EE_RDDM
[ 3.627129] ath11k_pci 0000:00:06.0: ignore reset dev flags 0x4000
[ 13.802105] ath11k_pci 0000:00:06.0: failed to wait wlan mode request
(mode 4): -110
[ 13.802175] ath11k_pci 0000:00:06.0: qmi failed to send wlan mode
off: -110
Thanks,
James
James Prestwood <[email protected]> writes:
> Hi Kalle, Baochen,
>
> On 1/11/24 12:16 AM, Kalle Valo wrote:
>> Baochen Qiang <[email protected]> writes:
>>
>>> On 1/10/2024 10:55 PM, James Prestwood wrote:
>>>> Hi Kalle,
>>>> On 1/10/24 5:49 AM, Kalle Valo wrote:
>>>>> James Prestwood <[email protected]> writes:
>>>>>
>>>>>>> But I have also no idea what is causing this, I guess we are doing
>>>>>>> something wrong with the PCI communication? That reminds me, you could
>>>>>>> try this in case that helps:
>>>>>>>
>>>>>>> https://patchwork.kernel.org/project/linux-wireless/patch/[email protected]/
>>>>>> Heh, I saw this pop up a day after I sent this and was wondering. Is
>>>>>> this something I'd need on the host kernel, guest, or both?
>>>>> On the guest where ath11k is running. I'm not optimistic that this would
>>>>> solve your issue, I suspect there can be also other bugs, but good to
>>>>> know if the patch changes anything.
>>>> Looks the same here, didn't seem to change anything based on the
>>>> kernel logs.
>>>>
>>> Could you try this?
>>>
>>> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/drivers/net/wireless/ath/ath11k/pci.c?id=39564b475ac5a589e6c22c43a08cbd283c295d2c
>> This reminds me, I assumed James was testing with ath.git master branch
>> (which has that commit) but I never checked that. So for testing please
>> always use the master branch to get the latest and greatest ath11k:
>>
>> https://git.kernel.org/pub/scm/linux/kernel/git/kvalo/ath.git/
>>
>> There's a quite long delay from ath.git to official releases.
>
> Good to know, and I was not in fact using that branch. Rebuilt from
> ath.git/master but still roughly the same behavior. There does appear
> to be more output now though, specifically a firmware crash:
>
> [ 2.281721] ath11k_pci 0000:00:06.0: failed to receive control
> response completion, polling..
> [ 2.282101] ip (65) used greatest stack depth: 12464 bytes left
> [ 3.306039] ath11k_pci 0000:00:06.0: Service connect timeout
> [ 3.307588] ath11k_pci 0000:00:06.0: failed to connect to HTT: -110
> [ 3.309286] ath11k_pci 0000:00:06.0: failed to start core: -110
> [ 3.519637] ath11k_pci 0000:00:06.0: firmware crashed: MHI_CB_EE_RDDM
> [ 3.519678] ath11k_pci 0000:00:06.0: ignore reset dev flags 0x4000
> [ 3.627087] ath11k_pci 0000:00:06.0: firmware crashed: MHI_CB_EE_RDDM
> [ 3.627129] ath11k_pci 0000:00:06.0: ignore reset dev flags 0x4000
> [ 13.802105] ath11k_pci 0000:00:06.0: failed to wait wlan mode
> request (mode 4): -110
> [ 13.802175] ath11k_pci 0000:00:06.0: qmi failed to send wlan mode
> off: -110
Ok, that's progress now. Can you try next try the iommu patch[1] we
talked about earlier? It's already in master-pending branch (along with
other pending patches) so you can use that branch if you want.
[1] https://patchwork.kernel.org/project/linux-wireless/patch/[email protected]/
--
https://patchwork.kernel.org/project/linux-wireless/list/
https://wireless.wiki.kernel.org/en/developers/documentation/submittingpatches
On 1/11/24 5:11 AM, Kalle Valo wrote:
> James Prestwood <[email protected]> writes:
>
>> Hi Kalle, Baochen,
>>
>> On 1/11/24 12:16 AM, Kalle Valo wrote:
>>> Baochen Qiang <[email protected]> writes:
>>>
>>>> On 1/10/2024 10:55 PM, James Prestwood wrote:
>>>>> Hi Kalle,
>>>>> On 1/10/24 5:49 AM, Kalle Valo wrote:
>>>>>> James Prestwood <[email protected]> writes:
>>>>>>
>>>>>>>> But I have also no idea what is causing this, I guess we are doing
>>>>>>>> something wrong with the PCI communication? That reminds me, you could
>>>>>>>> try this in case that helps:
>>>>>>>>
>>>>>>>> https://patchwork.kernel.org/project/linux-wireless/patch/[email protected]/
>>>>>>> Heh, I saw this pop up a day after I sent this and was wondering. Is
>>>>>>> this something I'd need on the host kernel, guest, or both?
>>>>>> On the guest where ath11k is running. I'm not optimistic that this would
>>>>>> solve your issue, I suspect there can be also other bugs, but good to
>>>>>> know if the patch changes anything.
>>>>> Looks the same here, didn't seem to change anything based on the
>>>>> kernel logs.
>>>>>
>>>> Could you try this?
>>>>
>>>> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/drivers/net/wireless/ath/ath11k/pci.c?id=39564b475ac5a589e6c22c43a08cbd283c295d2c
>>> This reminds me, I assumed James was testing with ath.git master branch
>>> (which has that commit) but I never checked that. So for testing please
>>> always use the master branch to get the latest and greatest ath11k:
>>>
>>> https://git.kernel.org/pub/scm/linux/kernel/git/kvalo/ath.git/
>>>
>>> There's a quite long delay from ath.git to official releases.
>> Good to know, and I was not in fact using that branch. Rebuilt from
>> ath.git/master but still roughly the same behavior. There does appear
>> to be more output now though, specifically a firmware crash:
>>
>> [ 2.281721] ath11k_pci 0000:00:06.0: failed to receive control
>> response completion, polling..
>> [ 2.282101] ip (65) used greatest stack depth: 12464 bytes left
>> [ 3.306039] ath11k_pci 0000:00:06.0: Service connect timeout
>> [ 3.307588] ath11k_pci 0000:00:06.0: failed to connect to HTT: -110
>> [ 3.309286] ath11k_pci 0000:00:06.0: failed to start core: -110
>> [ 3.519637] ath11k_pci 0000:00:06.0: firmware crashed: MHI_CB_EE_RDDM
>> [ 3.519678] ath11k_pci 0000:00:06.0: ignore reset dev flags 0x4000
>> [ 3.627087] ath11k_pci 0000:00:06.0: firmware crashed: MHI_CB_EE_RDDM
>> [ 3.627129] ath11k_pci 0000:00:06.0: ignore reset dev flags 0x4000
>> [ 13.802105] ath11k_pci 0000:00:06.0: failed to wait wlan mode
>> request (mode 4): -110
>> [ 13.802175] ath11k_pci 0000:00:06.0: qmi failed to send wlan mode
>> off: -110
> Ok, that's progress now. Can you try next try the iommu patch[1] we
> talked about earlier? It's already in master-pending branch (along with
> other pending patches) so you can use that branch if you want.
>
> [1] https://patchwork.kernel.org/project/linux-wireless/patch/[email protected]/
Same result unfortunately, tried both with just [1] applied to ath.git
and at HEAD of master-pending.
Thanks,
James
On 1/11/2024 9:38 PM, James Prestwood wrote:
>
> On 1/11/24 5:11 AM, Kalle Valo wrote:
>> James Prestwood <[email protected]> writes:
>>
>>> Hi Kalle, Baochen,
>>>
>>> On 1/11/24 12:16 AM, Kalle Valo wrote:
>>>> Baochen Qiang <[email protected]> writes:
>>>>
>>>>> On 1/10/2024 10:55 PM, James Prestwood wrote:
>>>>>> Hi Kalle,
>>>>>> On 1/10/24 5:49 AM, Kalle Valo wrote:
>>>>>>> James Prestwood <[email protected]> writes:
>>>>>>>
>>>>>>>>> But I have also no idea what is causing this, I guess we are doing
>>>>>>>>> something wrong with the PCI communication? That reminds me,
>>>>>>>>> you could
>>>>>>>>> try this in case that helps:
>>>>>>>>>
>>>>>>>>> https://patchwork.kernel.org/project/linux-wireless/patch/[email protected]/
>>>>>>>> Heh, I saw this pop up a day after I sent this and was
>>>>>>>> wondering. Is
>>>>>>>> this something I'd need on the host kernel, guest, or both?
>>>>>>> On the guest where ath11k is running. I'm not optimistic that
>>>>>>> this would
>>>>>>> solve your issue, I suspect there can be also other bugs, but
>>>>>>> good to
>>>>>>> know if the patch changes anything.
>>>>>> Looks the same here, didn't seem to change anything based on the
>>>>>> kernel logs.
>>>>>>
>>>>> Could you try this?
>>>>>
>>>>> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/drivers/net/wireless/ath/ath11k/pci.c?id=39564b475ac5a589e6c22c43a08cbd283c295d2c
>>>> This reminds me, I assumed James was testing with ath.git master branch
>>>> (which has that commit) but I never checked that. So for testing please
>>>> always use the master branch to get the latest and greatest ath11k:
>>>>
>>>> https://git.kernel.org/pub/scm/linux/kernel/git/kvalo/ath.git/
>>>>
>>>> There's a quite long delay from ath.git to official releases.
>>> Good to know, and I was not in fact using that branch. Rebuilt from
>>> ath.git/master but still roughly the same behavior. There does appear
>>> to be more output now though, specifically a firmware crash:
>>>
>>> [ 2.281721] ath11k_pci 0000:00:06.0: failed to receive control
>>> response completion, polling..
>>> [ 2.282101] ip (65) used greatest stack depth: 12464 bytes left
>>> [ 3.306039] ath11k_pci 0000:00:06.0: Service connect timeout
>>> [ 3.307588] ath11k_pci 0000:00:06.0: failed to connect to HTT: -110
>>> [ 3.309286] ath11k_pci 0000:00:06.0: failed to start core: -110
>>> [ 3.519637] ath11k_pci 0000:00:06.0: firmware crashed: MHI_CB_EE_RDDM
>>> [ 3.519678] ath11k_pci 0000:00:06.0: ignore reset dev flags 0x4000
>>> [ 3.627087] ath11k_pci 0000:00:06.0: firmware crashed: MHI_CB_EE_RDDM
>>> [ 3.627129] ath11k_pci 0000:00:06.0: ignore reset dev flags 0x4000
>>> [ 13.802105] ath11k_pci 0000:00:06.0: failed to wait wlan mode
>>> request (mode 4): -110
>>> [ 13.802175] ath11k_pci 0000:00:06.0: qmi failed to send wlan mode
>>> off: -110
>> Ok, that's progress now. Can you try next try the iommu patch[1] we
>> talked about earlier? It's already in master-pending branch (along with
>> other pending patches) so you can use that branch if you want.
>>
>> [1]
>> https://patchwork.kernel.org/project/linux-wireless/patch/[email protected]/
>
> Same result unfortunately, tried both with just [1] applied to ath.git
> and at HEAD of master-pending.
>
> Thanks,
>
> James
Strange that still fails. Are you now seeing this error in your host or
your Qemu? or both?
Could you share your test steps? And if you can share please be as
detailed as possible since I'm not familiar with passing WLAN hardware
to a VM using vfio-pci.
>
>
Hi,
On 1/11/24 6:04 PM, Baochen Qiang wrote:
>
>
> On 1/11/2024 9:38 PM, James Prestwood wrote:
>>
>> On 1/11/24 5:11 AM, Kalle Valo wrote:
>>> James Prestwood <[email protected]> writes:
>>>
>>>> Hi Kalle, Baochen,
>>>>
>>>> On 1/11/24 12:16 AM, Kalle Valo wrote:
>>>>> Baochen Qiang <[email protected]> writes:
>>>>>
>>>>>> On 1/10/2024 10:55 PM, James Prestwood wrote:
>>>>>>> Hi Kalle,
>>>>>>> On 1/10/24 5:49 AM, Kalle Valo wrote:
>>>>>>>> James Prestwood <[email protected]> writes:
>>>>>>>>
>>>>>>>>>> But I have also no idea what is causing this, I guess we are
>>>>>>>>>> doing
>>>>>>>>>> something wrong with the PCI communication? That reminds me,
>>>>>>>>>> you could
>>>>>>>>>> try this in case that helps:
>>>>>>>>>>
>>>>>>>>>> https://patchwork.kernel.org/project/linux-wireless/patch/[email protected]/
>>>>>>>>>>
>>>>>>>>> Heh, I saw this pop up a day after I sent this and was
>>>>>>>>> wondering. Is
>>>>>>>>> this something I'd need on the host kernel, guest, or both?
>>>>>>>> On the guest where ath11k is running. I'm not optimistic that
>>>>>>>> this would
>>>>>>>> solve your issue, I suspect there can be also other bugs, but
>>>>>>>> good to
>>>>>>>> know if the patch changes anything.
>>>>>>> Looks the same here, didn't seem to change anything based on the
>>>>>>> kernel logs.
>>>>>>>
>>>>>> Could you try this?
>>>>>>
>>>>>> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/drivers/net/wireless/ath/ath11k/pci.c?id=39564b475ac5a589e6c22c43a08cbd283c295d2c
>>>>>>
>>>>> This reminds me, I assumed James was testing with ath.git master
>>>>> branch
>>>>> (which has that commit) but I never checked that. So for testing
>>>>> please
>>>>> always use the master branch to get the latest and greatest ath11k:
>>>>>
>>>>> https://git.kernel.org/pub/scm/linux/kernel/git/kvalo/ath.git/
>>>>>
>>>>> There's a quite long delay from ath.git to official releases.
>>>> Good to know, and I was not in fact using that branch. Rebuilt from
>>>> ath.git/master but still roughly the same behavior. There does appear
>>>> to be more output now though, specifically a firmware crash:
>>>>
>>>> [ 2.281721] ath11k_pci 0000:00:06.0: failed to receive control
>>>> response completion, polling..
>>>> [ 2.282101] ip (65) used greatest stack depth: 12464 bytes left
>>>> [ 3.306039] ath11k_pci 0000:00:06.0: Service connect timeout
>>>> [ 3.307588] ath11k_pci 0000:00:06.0: failed to connect to HTT: -110
>>>> [ 3.309286] ath11k_pci 0000:00:06.0: failed to start core: -110
>>>> [ 3.519637] ath11k_pci 0000:00:06.0: firmware crashed:
>>>> MHI_CB_EE_RDDM
>>>> [ 3.519678] ath11k_pci 0000:00:06.0: ignore reset dev flags 0x4000
>>>> [ 3.627087] ath11k_pci 0000:00:06.0: firmware crashed:
>>>> MHI_CB_EE_RDDM
>>>> [ 3.627129] ath11k_pci 0000:00:06.0: ignore reset dev flags 0x4000
>>>> [ 13.802105] ath11k_pci 0000:00:06.0: failed to wait wlan mode
>>>> request (mode 4): -110
>>>> [ 13.802175] ath11k_pci 0000:00:06.0: qmi failed to send wlan mode
>>>> off: -110
>>> Ok, that's progress now. Can you try next try the iommu patch[1] we
>>> talked about earlier? It's already in master-pending branch (along with
>>> other pending patches) so you can use that branch if you want.
>>>
>>> [1]
>>> https://patchwork.kernel.org/project/linux-wireless/patch/[email protected]/
>>
>> Same result unfortunately, tried both with just [1] applied to
>> ath.git and at HEAD of master-pending.
>>
>> Thanks,
>>
>> James
> Strange that still fails. Are you now seeing this error in your host
> or your Qemu? or both?
> Could you share your test steps? And if you can share please be as
> detailed as possible since I'm not familiar with passing WLAN hardware
> to a VM using vfio-pci.
Just in Qemu, the hardware works fine on my host machine.
I basically follow this guide to set it up, its written in the context
of GPUs/libvirt but the host setup is exactly the same. By no means do
you need to read it all, once you set the vfio-pci.ids and see your
unclaimed adapter you can stop:
https://wiki.archlinux.org/title/PCI_passthrough_via_OVMF
In short you should be able to set the following host kernel options and
reboot (assuming your motherboard/hardware is compatible):
intel_iommu=on iommu=pt vfio-pci.ids=17cb:1103
Obviously change the device/vendor IDs to whatever ath11k hw you have.
Once the host is rebooted you should see your wlan adapter as UNCLAIMED,
showing the driver in use as vfio-pci. If not, its likely your
motherboard just isn't compatible, the device has to be in its own IOMMU
group (you could try switching PCI ports if this is the case).
I then build a "kvm_guest.config" kernel with the driver/firmware for
ath11k and boot into that with the following Qemu options:
-enable-kvm -device -vfio-pci,host=<PCI address>
If it seems easier you could also utilize IWD's test-runner which
handles launching the Qemu kernel automatically, detecting any
vfio-devices and passes them through and mounts some useful host folders
into the VM. Its actually a very good general purpose tool for kernel
testing, not just for IWD:
https://git.kernel.org/pub/scm/network/wireless/iwd.git/tree/doc/test-runner.txt
Once set up you can just run test-runner with a few flags and you'll
boot into a shell:
./tools/test-runner -k <kernel-image> --hw --start /bin/bash
Please reach out if you have questions, thanks for looking into this.
On 1/12/2024 8:47 PM, James Prestwood wrote:
> Hi,
>
> On 1/11/24 6:04 PM, Baochen Qiang wrote:
>>
>>
>> On 1/11/2024 9:38 PM, James Prestwood wrote:
>>>
>>> On 1/11/24 5:11 AM, Kalle Valo wrote:
>>>> James Prestwood <[email protected]> writes:
>>>>
>>>>> Hi Kalle, Baochen,
>>>>>
>>>>> On 1/11/24 12:16 AM, Kalle Valo wrote:
>>>>>> Baochen Qiang <[email protected]> writes:
>>>>>>
>>>>>>> On 1/10/2024 10:55 PM, James Prestwood wrote:
>>>>>>>> Hi Kalle,
>>>>>>>> On 1/10/24 5:49 AM, Kalle Valo wrote:
>>>>>>>>> James Prestwood <[email protected]> writes:
>>>>>>>>>
>>>>>>>>>>> But I have also no idea what is causing this, I guess we are
>>>>>>>>>>> doing
>>>>>>>>>>> something wrong with the PCI communication? That reminds me,
>>>>>>>>>>> you could
>>>>>>>>>>> try this in case that helps:
>>>>>>>>>>>
>>>>>>>>>>> https://patchwork.kernel.org/project/linux-wireless/patch/[email protected]/
>>>>>>>>>> Heh, I saw this pop up a day after I sent this and was
>>>>>>>>>> wondering. Is
>>>>>>>>>> this something I'd need on the host kernel, guest, or both?
>>>>>>>>> On the guest where ath11k is running. I'm not optimistic that
>>>>>>>>> this would
>>>>>>>>> solve your issue, I suspect there can be also other bugs, but
>>>>>>>>> good to
>>>>>>>>> know if the patch changes anything.
>>>>>>>> Looks the same here, didn't seem to change anything based on the
>>>>>>>> kernel logs.
>>>>>>>>
>>>>>>> Could you try this?
>>>>>>>
>>>>>>> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/drivers/net/wireless/ath/ath11k/pci.c?id=39564b475ac5a589e6c22c43a08cbd283c295d2c
>>>>>> This reminds me, I assumed James was testing with ath.git master
>>>>>> branch
>>>>>> (which has that commit) but I never checked that. So for testing
>>>>>> please
>>>>>> always use the master branch to get the latest and greatest ath11k:
>>>>>>
>>>>>> https://git.kernel.org/pub/scm/linux/kernel/git/kvalo/ath.git/
>>>>>>
>>>>>> There's a quite long delay from ath.git to official releases.
>>>>> Good to know, and I was not in fact using that branch. Rebuilt from
>>>>> ath.git/master but still roughly the same behavior. There does appear
>>>>> to be more output now though, specifically a firmware crash:
>>>>>
>>>>> [ 2.281721] ath11k_pci 0000:00:06.0: failed to receive control
>>>>> response completion, polling..
>>>>> [ 2.282101] ip (65) used greatest stack depth: 12464 bytes left
>>>>> [ 3.306039] ath11k_pci 0000:00:06.0: Service connect timeout
>>>>> [ 3.307588] ath11k_pci 0000:00:06.0: failed to connect to HTT: -110
>>>>> [ 3.309286] ath11k_pci 0000:00:06.0: failed to start core: -110
>>>>> [ 3.519637] ath11k_pci 0000:00:06.0: firmware crashed:
>>>>> MHI_CB_EE_RDDM
>>>>> [ 3.519678] ath11k_pci 0000:00:06.0: ignore reset dev flags 0x4000
>>>>> [ 3.627087] ath11k_pci 0000:00:06.0: firmware crashed:
>>>>> MHI_CB_EE_RDDM
>>>>> [ 3.627129] ath11k_pci 0000:00:06.0: ignore reset dev flags 0x4000
>>>>> [ 13.802105] ath11k_pci 0000:00:06.0: failed to wait wlan mode
>>>>> request (mode 4): -110
>>>>> [ 13.802175] ath11k_pci 0000:00:06.0: qmi failed to send wlan mode
>>>>> off: -110
>>>> Ok, that's progress now. Can you try next try the iommu patch[1] we
>>>> talked about earlier? It's already in master-pending branch (along with
>>>> other pending patches) so you can use that branch if you want.
>>>>
>>>> [1]
>>>> https://patchwork.kernel.org/project/linux-wireless/patch/[email protected]/
>>>
>>> Same result unfortunately, tried both with just [1] applied to
>>> ath.git and at HEAD of master-pending.
>>>
>>> Thanks,
>>>
>>> James
>> Strange that still fails. Are you now seeing this error in your host
>> or your Qemu? or both?
>> Could you share your test steps? And if you can share please be as
>> detailed as possible since I'm not familiar with passing WLAN hardware
>> to a VM using vfio-pci.
>
> Just in Qemu, the hardware works fine on my host machine.
>
> I basically follow this guide to set it up, its written in the context
> of GPUs/libvirt but the host setup is exactly the same. By no means do
> you need to read it all, once you set the vfio-pci.ids and see your
> unclaimed adapter you can stop:
>
> https://wiki.archlinux.org/title/PCI_passthrough_via_OVMF
>
> In short you should be able to set the following host kernel options and
> reboot (assuming your motherboard/hardware is compatible):
>
> intel_iommu=on iommu=pt vfio-pci.ids=17cb:1103
>
> Obviously change the device/vendor IDs to whatever ath11k hw you have.
> Once the host is rebooted you should see your wlan adapter as UNCLAIMED,
> showing the driver in use as vfio-pci. If not, its likely your
> motherboard just isn't compatible, the device has to be in its own IOMMU
> group (you could try switching PCI ports if this is the case).
>
> I then build a "kvm_guest.config" kernel with the driver/firmware for
> ath11k and boot into that with the following Qemu options:
>
> -enable-kvm -device -vfio-pci,host=<PCI address>
>
> If it seems easier you could also utilize IWD's test-runner which
> handles launching the Qemu kernel automatically, detecting any
> vfio-devices and passes them through and mounts some useful host folders
> into the VM. Its actually a very good general purpose tool for kernel
> testing, not just for IWD:
>
> https://git.kernel.org/pub/scm/network/wireless/iwd.git/tree/doc/test-runner.txt
>
> Once set up you can just run test-runner with a few flags and you'll
> boot into a shell:
>
> ./tools/test-runner -k <kernel-image> --hw --start /bin/bash
>
> Please reach out if you have questions, thanks for looking into this.
>
Thanks for these details. I reproduced this issue by following your guide.
Seems the root cause is that the MSI vector assigned to WCN6855 in qemu
is different with that in host. In my case the MSI vector in qemu is
[Address: fee00000 Data: 0020] while in host it is [Address: fee00578
Data: 0000]. So in qemu ath11k configures MSI vector [Address: fee00000
Data: 0020] to WCN6855 hardware/firmware, and firmware uses that vector
to fire interrupts to host/qemu. However host IOMMU doesn't know that
vector because the real vector is [Address: fee00578 Data: 0000], as a
result host blocks that interrupt and reports an error, see below log:
[ 1414.206069] DMAR: DRHD: handling fault status reg 2
[ 1414.206081] DMAR: [INTR-REMAP] Request device [02:00.0] fault index
0x0 [fault reason 0x25] Blocked a compatibility format interrupt request
[ 1414.210334] DMAR: DRHD: handling fault status reg 2
[ 1414.210342] DMAR: [INTR-REMAP] Request device [02:00.0] fault index
0x0 [fault reason 0x25] Blocked a compatibility format interrupt request
[ 1414.212496] DMAR: DRHD: handling fault status reg 2
[ 1414.212503] DMAR: [INTR-REMAP] Request device [02:00.0] fault index
0x0 [fault reason 0x25] Blocked a compatibility format interrupt request
[ 1414.214600] DMAR: DRHD: handling fault status reg 2
While I don't think there is a way for qemu/ath11k to get the real MSI
vector from host, I will try to read the vfio code to check further.
Before that, to unblock you, a possible hack is to hard code the MSI
vector in qemu to the same as in host, on condition that the MSI vector
doesn't change. In my case, the change looks like
diff --git a/drivers/net/wireless/ath/ath11k/pci.c
b/drivers/net/wireless/ath/ath11k/pci.c
index 09e65c5e55c4..89a9bbe9e4d2 100644
--- a/drivers/net/wireless/ath/ath11k/pci.c
+++ b/drivers/net/wireless/ath/ath11k/pci.c
@@ -459,7 +459,12 @@ static int ath11k_pci_alloc_msi(struct ath11k_pci
*ab_pci)
ab->pci.msi.addr_hi = 0;
}
- ath11k_dbg(ab, ATH11K_DBG_PCI, "msi base data is %d\n",
ab->pci.msi.ep_base_data);
+ ab->pci.msi.addr_hi = 0;
+ ab->pci.msi.addr_lo = 0xfee00578;
+ ath11k_dbg(ab, ATH11K_DBG_PCI, "msi addr hi 0x%x lo 0x%x base
data is %d\n",
+ ab->pci.msi.addr_hi,
+ ab->pci.msi.addr_lo,
+ ab->pci.msi.ep_base_data);
return 0;
@@ -487,6 +492,7 @@ static int ath11k_pci_config_msi_data(struct
ath11k_pci *ab_pci)
}
ab_pci->ab->pci.msi.ep_base_data = msi_desc->msg.data;
+ ab_pci->ab->pci.msi.ep_base_data = 0;
ath11k_dbg(ab_pci->ab, ATH11K_DBG_PCI, "after request_irq
msi_ep_base_data %d\n",
ab_pci->ab->pci.msi.ep_base_data);
This hack works on my setup.
Baochen Qiang <[email protected]> writes:
>>> Strange that still fails. Are you now seeing this error in your
>>> host or your Qemu? or both?
>>> Could you share your test steps? And if you can share please be as
>>> detailed as possible since I'm not familiar with passing WLAN
>>> hardware to a VM using vfio-pci.
>>
>> Just in Qemu, the hardware works fine on my host machine.
>> I basically follow this guide to set it up, its written in the
>> context of GPUs/libvirt but the host setup is exactly the same. By
>> no means do you need to read it all, once you set the vfio-pci.ids
>> and see your unclaimed adapter you can stop:
>> https://wiki.archlinux.org/title/PCI_passthrough_via_OVMF
>> In short you should be able to set the following host kernel options
>> and reboot (assuming your motherboard/hardware is compatible):
>> intel_iommu=on iommu=pt vfio-pci.ids=17cb:1103
>> Obviously change the device/vendor IDs to whatever ath11k hw you
>> have. Once the host is rebooted you should see your wlan adapter as
>> UNCLAIMED, showing the driver in use as vfio-pci. If not, its likely
>> your motherboard just isn't compatible, the device has to be in its
>> own IOMMU group (you could try switching PCI ports if this is the
>> case).
>> I then build a "kvm_guest.config" kernel with the driver/firmware
>> for ath11k and boot into that with the following Qemu options:
>> -enable-kvm -device -vfio-pci,host=<PCI address>
>> If it seems easier you could also utilize IWD's test-runner which
>> handles launching the Qemu kernel automatically, detecting any
>> vfio-devices and passes them through and mounts some useful host
>> folders into the VM. Its actually a very good general purpose tool
>> for kernel testing, not just for IWD:
>> https://git.kernel.org/pub/scm/network/wireless/iwd.git/tree/doc/test-runner.txt
>> Once set up you can just run test-runner with a few flags and you'll
>> boot into a shell:
>> ./tools/test-runner -k <kernel-image> --hw --start /bin/bash
>> Please reach out if you have questions, thanks for looking into
>> this.
>
> Thanks for these details. I reproduced this issue by following your guide.
>
> Seems the root cause is that the MSI vector assigned to WCN6855 in
> qemu is different with that in host. In my case the MSI vector in qemu
> is [Address: fee00000 Data: 0020] while in host it is [Address:
> fee00578 Data: 0000]. So in qemu ath11k configures MSI vector
> [Address: fee00000 Data: 0020] to WCN6855 hardware/firmware, and
> firmware uses that vector to fire interrupts to host/qemu. However
> host IOMMU doesn't know that vector because the real vector is
> [Address: fee00578 Data: 0000], as a result host blocks that
> interrupt and reports an error, see below log:
>
> [ 1414.206069] DMAR: DRHD: handling fault status reg 2
> [ 1414.206081] DMAR: [INTR-REMAP] Request device [02:00.0] fault index
> 0x0 [fault reason 0x25] Blocked a compatibility format interrupt
> request
> [ 1414.210334] DMAR: DRHD: handling fault status reg 2
> [ 1414.210342] DMAR: [INTR-REMAP] Request device [02:00.0] fault index
> 0x0 [fault reason 0x25] Blocked a compatibility format interrupt
> request
> [ 1414.212496] DMAR: DRHD: handling fault status reg 2
> [ 1414.212503] DMAR: [INTR-REMAP] Request device [02:00.0] fault index
> 0x0 [fault reason 0x25] Blocked a compatibility format interrupt
> request
> [ 1414.214600] DMAR: DRHD: handling fault status reg 2
>
> While I don't think there is a way for qemu/ath11k to get the real MSI
> vector from host, I will try to read the vfio code to check further.
> Before that, to unblock you, a possible hack is to hard code the MSI
> vector in qemu to the same as in host, on condition that the MSI
> vector doesn't change.
Baochen, awesome that you were able to debug this further. Now we at
least know what's the problem.
I'll add David Woodhouse and the iommu list in hopes that they could
give us ideas how to solve this. Full thread here:
https://lore.kernel.org/all/[email protected]/
--
https://patchwork.kernel.org/project/linux-wireless/list/
https://wireless.wiki.kernel.org/en/developers/documentation/submittingpatches
On Sun, 14 Jan 2024 16:36:02 +0200
Kalle Valo <[email protected]> wrote:
> Baochen Qiang <[email protected]> writes:
>
> >>> Strange that still fails. Are you now seeing this error in your
> >>> host or your Qemu? or both?
> >>> Could you share your test steps? And if you can share please be as
> >>> detailed as possible since I'm not familiar with passing WLAN
> >>> hardware to a VM using vfio-pci.
> >>
> >> Just in Qemu, the hardware works fine on my host machine.
> >> I basically follow this guide to set it up, its written in the
> >> context of GPUs/libvirt but the host setup is exactly the same. By
> >> no means do you need to read it all, once you set the vfio-pci.ids
> >> and see your unclaimed adapter you can stop:
> >> https://wiki.archlinux.org/title/PCI_passthrough_via_OVMF
> >> In short you should be able to set the following host kernel options
> >> and reboot (assuming your motherboard/hardware is compatible):
> >> intel_iommu=on iommu=pt vfio-pci.ids=17cb:1103
> >> Obviously change the device/vendor IDs to whatever ath11k hw you
> >> have. Once the host is rebooted you should see your wlan adapter as
> >> UNCLAIMED, showing the driver in use as vfio-pci. If not, its likely
> >> your motherboard just isn't compatible, the device has to be in its
> >> own IOMMU group (you could try switching PCI ports if this is the
> >> case).
> >> I then build a "kvm_guest.config" kernel with the driver/firmware
> >> for ath11k and boot into that with the following Qemu options:
> >> -enable-kvm -device -vfio-pci,host=<PCI address>
> >> If it seems easier you could also utilize IWD's test-runner which
> >> handles launching the Qemu kernel automatically, detecting any
> >> vfio-devices and passes them through and mounts some useful host
> >> folders into the VM. Its actually a very good general purpose tool
> >> for kernel testing, not just for IWD:
> >> https://git.kernel.org/pub/scm/network/wireless/iwd.git/tree/doc/test-runner.txt
> >> Once set up you can just run test-runner with a few flags and you'll
> >> boot into a shell:
> >> ./tools/test-runner -k <kernel-image> --hw --start /bin/bash
> >> Please reach out if you have questions, thanks for looking into
> >> this.
> >
> > Thanks for these details. I reproduced this issue by following your guide.
> >
> > Seems the root cause is that the MSI vector assigned to WCN6855 in
> > qemu is different with that in host. In my case the MSI vector in qemu
> > is [Address: fee00000 Data: 0020] while in host it is [Address:
> > fee00578 Data: 0000]. So in qemu ath11k configures MSI vector
> > [Address: fee00000 Data: 0020] to WCN6855 hardware/firmware, and
> > firmware uses that vector to fire interrupts to host/qemu. However
> > host IOMMU doesn't know that vector because the real vector is
> > [Address: fee00578 Data: 0000], as a result host blocks that
> > interrupt and reports an error, see below log:
> >
> > [ 1414.206069] DMAR: DRHD: handling fault status reg 2
> > [ 1414.206081] DMAR: [INTR-REMAP] Request device [02:00.0] fault index
> > 0x0 [fault reason 0x25] Blocked a compatibility format interrupt
> > request
> > [ 1414.210334] DMAR: DRHD: handling fault status reg 2
> > [ 1414.210342] DMAR: [INTR-REMAP] Request device [02:00.0] fault index
> > 0x0 [fault reason 0x25] Blocked a compatibility format interrupt
> > request
> > [ 1414.212496] DMAR: DRHD: handling fault status reg 2
> > [ 1414.212503] DMAR: [INTR-REMAP] Request device [02:00.0] fault index
> > 0x0 [fault reason 0x25] Blocked a compatibility format interrupt
> > request
> > [ 1414.214600] DMAR: DRHD: handling fault status reg 2
> >
> > While I don't think there is a way for qemu/ath11k to get the real MSI
> > vector from host, I will try to read the vfio code to check further.
> > Before that, to unblock you, a possible hack is to hard code the MSI
> > vector in qemu to the same as in host, on condition that the MSI
> > vector doesn't change.
>
> Baochen, awesome that you were able to debug this further. Now we at
> least know what's the problem.
It's an interesting problem, I don't think we've seen another device
where the driver reads the MSI register in order to program another
hardware entity to match the MSI address and data configuration.
When assigning a device, the host and guest use entirely separate
address spaces for MSI interrupts. When the guest enables MSI, the
operation is trapped by the VMM and triggers an ioctl to the host to
perform an equivalent configuration. Generally the physical device
will interrupt within the host where it may be directly attached to KVM
to signal the interrupt, trigger through the VMM, or where
virtualization hardware supports it, the interrupt can directly trigger
the vCPU. From the VM perspective, the guest address/data pair is used
to signal the interrupt, which is why it makes sense to virtualize the
MSI registers.
Off hand I don't have a good solution for this, the hardware is
essentially imposing a unique requirement for MSI programming that the
driver needs visibility of the physical MSI address and data. It's
conceivable that device specific code could either make the physical
address/data pair visible to the VM or trap the firmware programming to
inject the correct physical values. Is there somewhere other than the
standard MSI capability in config space that the driver could learn the
physical values, ie. somewhere that isn't virtualized? Thanks,
Alex
On 1/16/2024 1:46 AM, Alex Williamson wrote:
> On Sun, 14 Jan 2024 16:36:02 +0200
> Kalle Valo <[email protected]> wrote:
>
>> Baochen Qiang <[email protected]> writes:
>>
>>>>> Strange that still fails. Are you now seeing this error in your
>>>>> host or your Qemu? or both?
>>>>> Could you share your test steps? And if you can share please be as
>>>>> detailed as possible since I'm not familiar with passing WLAN
>>>>> hardware to a VM using vfio-pci.
>>>>
>>>> Just in Qemu, the hardware works fine on my host machine.
>>>> I basically follow this guide to set it up, its written in the
>>>> context of GPUs/libvirt but the host setup is exactly the same. By
>>>> no means do you need to read it all, once you set the vfio-pci.ids
>>>> and see your unclaimed adapter you can stop:
>>>> https://wiki.archlinux.org/title/PCI_passthrough_via_OVMF
>>>> In short you should be able to set the following host kernel options
>>>> and reboot (assuming your motherboard/hardware is compatible):
>>>> intel_iommu=on iommu=pt vfio-pci.ids=17cb:1103
>>>> Obviously change the device/vendor IDs to whatever ath11k hw you
>>>> have. Once the host is rebooted you should see your wlan adapter as
>>>> UNCLAIMED, showing the driver in use as vfio-pci. If not, its likely
>>>> your motherboard just isn't compatible, the device has to be in its
>>>> own IOMMU group (you could try switching PCI ports if this is the
>>>> case).
>>>> I then build a "kvm_guest.config" kernel with the driver/firmware
>>>> for ath11k and boot into that with the following Qemu options:
>>>> -enable-kvm -device -vfio-pci,host=<PCI address>
>>>> If it seems easier you could also utilize IWD's test-runner which
>>>> handles launching the Qemu kernel automatically, detecting any
>>>> vfio-devices and passes them through and mounts some useful host
>>>> folders into the VM. Its actually a very good general purpose tool
>>>> for kernel testing, not just for IWD:
>>>> https://git.kernel.org/pub/scm/network/wireless/iwd.git/tree/doc/test-runner.txt
>>>> Once set up you can just run test-runner with a few flags and you'll
>>>> boot into a shell:
>>>> ./tools/test-runner -k <kernel-image> --hw --start /bin/bash
>>>> Please reach out if you have questions, thanks for looking into
>>>> this.
>>>
>>> Thanks for these details. I reproduced this issue by following your guide.
>>>
>>> Seems the root cause is that the MSI vector assigned to WCN6855 in
>>> qemu is different with that in host. In my case the MSI vector in qemu
>>> is [Address: fee00000 Data: 0020] while in host it is [Address:
>>> fee00578 Data: 0000]. So in qemu ath11k configures MSI vector
>>> [Address: fee00000 Data: 0020] to WCN6855 hardware/firmware, and
>>> firmware uses that vector to fire interrupts to host/qemu. However
>>> host IOMMU doesn't know that vector because the real vector is
>>> [Address: fee00578 Data: 0000], as a result host blocks that
>>> interrupt and reports an error, see below log:
>>>
>>> [ 1414.206069] DMAR: DRHD: handling fault status reg 2
>>> [ 1414.206081] DMAR: [INTR-REMAP] Request device [02:00.0] fault index
>>> 0x0 [fault reason 0x25] Blocked a compatibility format interrupt
>>> request
>>> [ 1414.210334] DMAR: DRHD: handling fault status reg 2
>>> [ 1414.210342] DMAR: [INTR-REMAP] Request device [02:00.0] fault index
>>> 0x0 [fault reason 0x25] Blocked a compatibility format interrupt
>>> request
>>> [ 1414.212496] DMAR: DRHD: handling fault status reg 2
>>> [ 1414.212503] DMAR: [INTR-REMAP] Request device [02:00.0] fault index
>>> 0x0 [fault reason 0x25] Blocked a compatibility format interrupt
>>> request
>>> [ 1414.214600] DMAR: DRHD: handling fault status reg 2
>>>
>>> While I don't think there is a way for qemu/ath11k to get the real MSI
>>> vector from host, I will try to read the vfio code to check further.
>>> Before that, to unblock you, a possible hack is to hard code the MSI
>>> vector in qemu to the same as in host, on condition that the MSI
>>> vector doesn't change.
>>
>> Baochen, awesome that you were able to debug this further. Now we at
>> least know what's the problem.
>
> It's an interesting problem, I don't think we've seen another device
> where the driver reads the MSI register in order to program another
> hardware entity to match the MSI address and data configuration.
>
> When assigning a device, the host and guest use entirely separate
> address spaces for MSI interrupts. When the guest enables MSI, the
> operation is trapped by the VMM and triggers an ioctl to the host to
> perform an equivalent configuration. Generally the physical device
> will interrupt within the host where it may be directly attached to KVM
> to signal the interrupt, trigger through the VMM, or where
> virtualization hardware supports it, the interrupt can directly trigger
> the vCPU. From the VM perspective, the guest address/data pair is used
> to signal the interrupt, which is why it makes sense to virtualize the
> MSI registers.
Hi Alex, could you help elaborate more? why from the VM perspective MSI
virtualization is necessary?
And, maybe a stupid question, is that possible VM/KVM or vfio only
virtualize write operation to MSI register but leave read operation
un-virtualized? I am asking this because in that way ath11k may get a
chance to run in VM after getting the real vector.
>
> Off hand I don't have a good solution for this, the hardware is
> essentially imposing a unique requirement for MSI programming that the
> driver needs visibility of the physical MSI address and data. It's
> conceivable that device specific code could either make the physical
> address/data pair visible to the VM or trap the firmware programming to
> inject the correct physical values. Is there somewhere other than the
> standard MSI capability in config space that the driver could learn the
> physical values, ie. somewhere that isn't virtualized? Thanks,
I don't think we have such capability in configuration space.
>
> Alex
>
On Tue, 2024-01-16 at 18:08 +0800, Baochen Qiang wrote:
>
>
> On 1/16/2024 1:46 AM, Alex Williamson wrote:
> > On Sun, 14 Jan 2024 16:36:02 +0200
> > Kalle Valo <[email protected]> wrote:
> >
> > > Baochen Qiang <[email protected]> writes:
> > >
> > > > > > Strange that still fails. Are you now seeing this error in your
> > > > > > host or your Qemu? or both?
> > > > > > Could you share your test steps? And if you can share please be as
> > > > > > detailed as possible since I'm not familiar with passing WLAN
> > > > > > hardware to a VM using vfio-pci.
> > > > >
> > > > > Just in Qemu, the hardware works fine on my host machine.
> > > > > I basically follow this guide to set it up, its written in the
> > > > > context of GPUs/libvirt but the host setup is exactly the same. By
> > > > > no means do you need to read it all, once you set the vfio-pci.ids
> > > > > and see your unclaimed adapter you can stop:
> > > > > https://wiki.archlinux.org/title/PCI_passthrough_via_OVMF
> > > > > In short you should be able to set the following host kernel options
> > > > > and reboot (assuming your motherboard/hardware is compatible):
> > > > > intel_iommu=on iommu=pt vfio-pci.ids=17cb:1103
> > > > > Obviously change the device/vendor IDs to whatever ath11k hw you
> > > > > have. Once the host is rebooted you should see your wlan adapter as
> > > > > UNCLAIMED, showing the driver in use as vfio-pci. If not, its likely
> > > > > your motherboard just isn't compatible, the device has to be in its
> > > > > own IOMMU group (you could try switching PCI ports if this is the
> > > > > case).
> > > > > I then build a "kvm_guest.config" kernel with the driver/firmware
> > > > > for ath11k and boot into that with the following Qemu options:
> > > > > -enable-kvm -device -vfio-pci,host=<PCI address>
> > > > > If it seems easier you could also utilize IWD's test-runner which
> > > > > handles launching the Qemu kernel automatically, detecting any
> > > > > vfio-devices and passes them through and mounts some useful host
> > > > > folders into the VM. Its actually a very good general purpose tool
> > > > > for kernel testing, not just for IWD:
> > > > > https://git.kernel.org/pub/scm/network/wireless/iwd.git/tree/doc/test-runner.txt
> > > > > Once set up you can just run test-runner with a few flags and you'll
> > > > > boot into a shell:
> > > > > ./tools/test-runner -k <kernel-image> --hw --start /bin/bash
> > > > > Please reach out if you have questions, thanks for looking into
> > > > > this.
> > > >
> > > > Thanks for these details. I reproduced this issue by following your guide.
> > > >
> > > > Seems the root cause is that the MSI vector assigned to WCN6855 in
> > > > qemu is different with that in host. In my case the MSI vector in qemu
> > > > is [Address: fee00000 Data: 0020] while in host it is [Address:
> > > > fee00578 Data: 0000]. So in qemu ath11k configures MSI vector
> > > > [Address: fee00000 Data: 0020] to WCN6855 hardware/firmware, and
> > > > firmware uses that vector to fire interrupts to host/qemu. However
> > > > host IOMMU doesn't know that vector because the real vector is
> > > > [Address: fee00578 Data: 0000], as a result host blocks that
> > > > interrupt and reports an error, see below log:
> > > >
> > > > [ 1414.206069] DMAR: DRHD: handling fault status reg 2
> > > > [ 1414.206081] DMAR: [INTR-REMAP] Request device [02:00.0] fault index
> > > > 0x0 [fault reason 0x25] Blocked a compatibility format interrupt
> > > > request
> > > > [ 1414.210334] DMAR: DRHD: handling fault status reg 2
> > > > [ 1414.210342] DMAR: [INTR-REMAP] Request device [02:00.0] fault index
> > > > 0x0 [fault reason 0x25] Blocked a compatibility format interrupt
> > > > request
> > > > [ 1414.212496] DMAR: DRHD: handling fault status reg 2
> > > > [ 1414.212503] DMAR: [INTR-REMAP] Request device [02:00.0] fault index
> > > > 0x0 [fault reason 0x25] Blocked a compatibility format interrupt
> > > > request
> > > > [ 1414.214600] DMAR: DRHD: handling fault status reg 2
> > > >
> > > > While I don't think there is a way for qemu/ath11k to get the real MSI
> > > > vector from host, I will try to read the vfio code to check further.
> > > > Before that, to unblock you, a possible hack is to hard code the MSI
> > > > vector in qemu to the same as in host, on condition that the MSI
> > > > vector doesn't change.
> > >
> > > Baochen, awesome that you were able to debug this further. Now we at
> > > least know what's the problem.
> >
> > It's an interesting problem, I don't think we've seen another device
> > where the driver reads the MSI register in order to program another
> > hardware entity to match the MSI address and data configuration.
> >
> > When assigning a device, the host and guest use entirely separate
> > address spaces for MSI interrupts. When the guest enables MSI, the
> > operation is trapped by the VMM and triggers an ioctl to the host to
> > perform an equivalent configuration. Generally the physical device
> > will interrupt within the host where it may be directly attached to KVM
> > to signal the interrupt, trigger through the VMM, or where
> > virtualization hardware supports it, the interrupt can directly trigger
> > the vCPU. From the VM perspective, the guest address/data pair is used
> > to signal the interrupt, which is why it makes sense to virtualize the
> > MSI registers.
>
> Hi Alex, could you help elaborate more? why from the VM perspective MSI
> virtualization is necessary?
An MSI is just a write to physical memory space. You can even use it
like that; configure the device to just write 4 bytes to some address
in a struct in memory to show that it needs attention, and you then
poll that memory.
But mostly we don't (ab)use it like that, of course. We tell the device
to write to a special range of the physical address space where the
interrupt controller lives — the range from 0xfee00000 to 0xfeefffff.
The low 20 bits of the address, and the 32 bits of data written to that
address, tell the interrupt controller which CPU to interrupt, and
which vector to raise on the CPU (as well as some other details and
weird interrupt modes which are theoretically encodable).
So in your example, the guest writes [Address: fee00000 Data: 0020]
which means it wants vector 0x20 on CPU#0 (well, the CPU with APICID
0). But that's what the *guest* wants. If we just blindly programmed
that into the hardware, the hardware would deliver vector 0x20 to the
host's CPU0... which would be very confused by it.
The host has a driver for that device, probably the VFIO driver. The
host registers its own interrupt handlers for the real hardware,
decides which *host* CPU (and vector) should be notified when something
happens. And when that happens, the VFIO driver will raise an event on
an eventfd, which will notify QEMU to inject the appropriate interrupt
into the guest.
So... when the guest enables the MSI, that's trapped by QEMU which
remembers which *guest* CPU/vector the interrupt should go to. QEMU
tells VFIO to enable the corresponding interrupt, and what gets
programmed into the actual hardware is up to the *host* operating
system; nothing to do with the guest's information at all.
Then when the actual hardware raises the interrupt, the VFIO interrupt
handler runs in the guest, signals an event on the eventfd, and QEMU
receives that and injects the event into the appropriate guest vCPU.
(In practice QEMU doesn't do it these days; there's actually a shortcut
which improves latency by allowing the kernel to deliver the event to
the guest directly, connecting the eventfd directly to the KVM irq
routing table.)
Interrupt remapping is probably not important here, but I'll explain it
briefly anyway. With interrupt remapping, the IOMMU handles the
'memory' write from the device, just as it handles all other memory
transactions. One of the reasons for interrupt remapping is that the
original definitions of the bits in the MSI (the low 20 bits of the
address and the 32 bits of what's written) only had 8 bits for the
target CPU APICID. And we have bigger systems than that now.
So by using one of the spare bits in the MSI message, we can indicate
that this isn't just a directly-encoded cpu/vector in "Compatibility
Format", but is a "Remappable Format" interrupt. Instead of the
cpu/vector it just contains an index in to the IOMMU's Interrupt
Redirection Table. Which *does* have a full 32-bits for the target APIC
ID. That's why x2apic support (which gives us support for >254 CPUs)
depends on interrupt remapping.
The other thing that the IOMMU can do in modern systems is *posted*
interrupts. Where the entry in the IOMMU's IRT doesn't just specify the
host's CPU/vector, but actually specifies a *vCPU* to deliver the
interrupt to.
All of which is mostly irrelevant as it's just another bypass
optimisation to improve latency. The key here is that what the guest
writes to its emulated MSI table and what the host writes to the real
hardware are not at all related.
If we had had this posted interrupt support from the beginning, perhaps
we could have have a much simpler model — we just let the guest write
its intended (v)CPU#/vector *directly* to the MSI table in the device,
and let the IOMMU fix it up by having a table pointing to the
appropriate set of vCPUs. But that isn't how it happened. The model we
have is that the VMM has to *emulate* the config space and handle the
interrupts as described above.
This means that whenever a device has a non-standard way of configuring
MSIs, the VMM has to understand and intercept that. I believe we've
even seen some Atheros devices with the MSI target in some weird MMIO
registers instead of the standard location, so we've had to hack QEMU
to handle those too?
> And, maybe a stupid question, is that possible VM/KVM or vfio only
> virtualize write operation to MSI register but leave read operation
> un-virtualized? I am asking this because in that way ath11k may get a
> chance to run in VM after getting the real vector.
That might confuse a number of operating systems. Especially if they
mask/unmask by reading the register, flipping the mask bit and writing
back again.
How exactly is the content of this register then given back to the
firmware? Is that communication snoopable by the VMM?
> >
> > Off hand I don't have a good solution for this, the hardware is
> > essentially imposing a unique requirement for MSI programming that the
> > driver needs visibility of the physical MSI address and data.
> >
Strictly, the driver doesn't need visibility to the actual values used
by the hardware. Another way of it looking at it would be to say that
the driver programs the MSI through this non-standard method, it just
needs the VMM to trap and handle that, just as the VMM does for the
standard MSI table.
Which is what I thought we'd already seen on some Atheros devices.
> > It's
> > conceivable that device specific code could either make the physical
> > address/data pair visible to the VM or trap the firmware programming to
> > inject the correct physical values. Is there somewhere other than the
> > standard MSI capability in config space that the driver could learn the
> > physical values, ie. somewhere that isn't virtualized? Thanks,
>
> I don't think we have such capability in configuration space.
Configuration space is a complete fiction though; it's all emulated. We
can do anything we like. Or we can have a PV hypercall which will
report it. I don't know that we'd *want* to, but all things are
possible.
Hi Baochen,
On 1/14/24 4:37 AM, Baochen Qiang wrote:
>
>
> On 1/12/2024 8:47 PM, James Prestwood wrote:
>> Hi,
>>
>> On 1/11/24 6:04 PM, Baochen Qiang wrote:
>>>
>>>
>>> On 1/11/2024 9:38 PM, James Prestwood wrote:
>>>>
>>>> On 1/11/24 5:11 AM, Kalle Valo wrote:
>>>>> James Prestwood <[email protected]> writes:
>>>>>
>>>>>> Hi Kalle, Baochen,
>>>>>>
>>>>>> On 1/11/24 12:16 AM, Kalle Valo wrote:
>>>>>>> Baochen Qiang <[email protected]> writes:
>>>>>>>
>>>>>>>> On 1/10/2024 10:55 PM, James Prestwood wrote:
>>>>>>>>> Hi Kalle,
>>>>>>>>> On 1/10/24 5:49 AM, Kalle Valo wrote:
>>>>>>>>>> James Prestwood <[email protected]> writes:
>>>>>>>>>>
>>>>>>>>>>>> But I have also no idea what is causing this, I guess we
>>>>>>>>>>>> are doing
>>>>>>>>>>>> something wrong with the PCI communication? That reminds
>>>>>>>>>>>> me, you could
>>>>>>>>>>>> try this in case that helps:
>>>>>>>>>>>>
>>>>>>>>>>>> https://patchwork.kernel.org/project/linux-wireless/patch/[email protected]/
>>>>>>>>>>>>
>>>>>>>>>>> Heh, I saw this pop up a day after I sent this and was
>>>>>>>>>>> wondering. Is
>>>>>>>>>>> this something I'd need on the host kernel, guest, or both?
>>>>>>>>>> On the guest where ath11k is running. I'm not optimistic that
>>>>>>>>>> this would
>>>>>>>>>> solve your issue, I suspect there can be also other bugs, but
>>>>>>>>>> good to
>>>>>>>>>> know if the patch changes anything.
>>>>>>>>> Looks the same here, didn't seem to change anything based on the
>>>>>>>>> kernel logs.
>>>>>>>>>
>>>>>>>> Could you try this?
>>>>>>>>
>>>>>>>> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/drivers/net/wireless/ath/ath11k/pci.c?id=39564b475ac5a589e6c22c43a08cbd283c295d2c
>>>>>>>>
>>>>>>> This reminds me, I assumed James was testing with ath.git master
>>>>>>> branch
>>>>>>> (which has that commit) but I never checked that. So for testing
>>>>>>> please
>>>>>>> always use the master branch to get the latest and greatest ath11k:
>>>>>>>
>>>>>>> https://git.kernel.org/pub/scm/linux/kernel/git/kvalo/ath.git/
>>>>>>>
>>>>>>> There's a quite long delay from ath.git to official releases.
>>>>>> Good to know, and I was not in fact using that branch. Rebuilt from
>>>>>> ath.git/master but still roughly the same behavior. There does
>>>>>> appear
>>>>>> to be more output now though, specifically a firmware crash:
>>>>>>
>>>>>> [ 2.281721] ath11k_pci 0000:00:06.0: failed to receive control
>>>>>> response completion, polling..
>>>>>> [ 2.282101] ip (65) used greatest stack depth: 12464 bytes left
>>>>>> [ 3.306039] ath11k_pci 0000:00:06.0: Service connect timeout
>>>>>> [ 3.307588] ath11k_pci 0000:00:06.0: failed to connect to HTT:
>>>>>> -110
>>>>>> [ 3.309286] ath11k_pci 0000:00:06.0: failed to start core: -110
>>>>>> [ 3.519637] ath11k_pci 0000:00:06.0: firmware crashed:
>>>>>> MHI_CB_EE_RDDM
>>>>>> [ 3.519678] ath11k_pci 0000:00:06.0: ignore reset dev flags
>>>>>> 0x4000
>>>>>> [ 3.627087] ath11k_pci 0000:00:06.0: firmware crashed:
>>>>>> MHI_CB_EE_RDDM
>>>>>> [ 3.627129] ath11k_pci 0000:00:06.0: ignore reset dev flags
>>>>>> 0x4000
>>>>>> [ 13.802105] ath11k_pci 0000:00:06.0: failed to wait wlan mode
>>>>>> request (mode 4): -110
>>>>>> [ 13.802175] ath11k_pci 0000:00:06.0: qmi failed to send wlan mode
>>>>>> off: -110
>>>>> Ok, that's progress now. Can you try next try the iommu patch[1] we
>>>>> talked about earlier? It's already in master-pending branch (along
>>>>> with
>>>>> other pending patches) so you can use that branch if you want.
>>>>>
>>>>> [1]
>>>>> https://patchwork.kernel.org/project/linux-wireless/patch/[email protected]/
>>>>
>>>> Same result unfortunately, tried both with just [1] applied to
>>>> ath.git and at HEAD of master-pending.
>>>>
>>>> Thanks,
>>>>
>>>> James
>>> Strange that still fails. Are you now seeing this error in your host
>>> or your Qemu? or both?
>>> Could you share your test steps? And if you can share please be as
>>> detailed as possible since I'm not familiar with passing WLAN
>>> hardware to a VM using vfio-pci.
>>
>> Just in Qemu, the hardware works fine on my host machine.
>>
>> I basically follow this guide to set it up, its written in the
>> context of GPUs/libvirt but the host setup is exactly the same. By no
>> means do you need to read it all, once you set the vfio-pci.ids and
>> see your unclaimed adapter you can stop:
>>
>> https://wiki.archlinux.org/title/PCI_passthrough_via_OVMF
>>
>> In short you should be able to set the following host kernel options
>> and reboot (assuming your motherboard/hardware is compatible):
>>
>> intel_iommu=on iommu=pt vfio-pci.ids=17cb:1103
>>
>> Obviously change the device/vendor IDs to whatever ath11k hw you
>> have. Once the host is rebooted you should see your wlan adapter as
>> UNCLAIMED, showing the driver in use as vfio-pci. If not, its likely
>> your motherboard just isn't compatible, the device has to be in its
>> own IOMMU group (you could try switching PCI ports if this is the case).
>>
>> I then build a "kvm_guest.config" kernel with the driver/firmware for
>> ath11k and boot into that with the following Qemu options:
>>
>> -enable-kvm -device -vfio-pci,host=<PCI address>
>>
>> If it seems easier you could also utilize IWD's test-runner which
>> handles launching the Qemu kernel automatically, detecting any
>> vfio-devices and passes them through and mounts some useful host
>> folders into the VM. Its actually a very good general purpose tool
>> for kernel testing, not just for IWD:
>>
>> https://git.kernel.org/pub/scm/network/wireless/iwd.git/tree/doc/test-runner.txt
>>
>>
>> Once set up you can just run test-runner with a few flags and you'll
>> boot into a shell:
>>
>> ./tools/test-runner -k <kernel-image> --hw --start /bin/bash
>>
>> Please reach out if you have questions, thanks for looking into this.
>>
> Thanks for these details. I reproduced this issue by following your
> guide.
>
> Seems the root cause is that the MSI vector assigned to WCN6855 in
> qemu is different with that in host. In my case the MSI vector in qemu
> is [Address: fee00000 Data: 0020] while in host it is [Address:
> fee00578 Data: 0000]. So in qemu ath11k configures MSI vector
> [Address: fee00000 Data: 0020] to WCN6855 hardware/firmware, and
> firmware uses that vector to fire interrupts to host/qemu. However
> host IOMMU doesn't know that vector because the real vector is
> [Address: fee00578 Data: 0000], as a result host blocks that
> interrupt and reports an error, see below log:
>
> [ 1414.206069] DMAR: DRHD: handling fault status reg 2
> [ 1414.206081] DMAR: [INTR-REMAP] Request device [02:00.0] fault index
> 0x0 [fault reason 0x25] Blocked a compatibility format interrupt request
> [ 1414.210334] DMAR: DRHD: handling fault status reg 2
> [ 1414.210342] DMAR: [INTR-REMAP] Request device [02:00.0] fault index
> 0x0 [fault reason 0x25] Blocked a compatibility format interrupt request
> [ 1414.212496] DMAR: DRHD: handling fault status reg 2
> [ 1414.212503] DMAR: [INTR-REMAP] Request device [02:00.0] fault index
> 0x0 [fault reason 0x25] Blocked a compatibility format interrupt request
> [ 1414.214600] DMAR: DRHD: handling fault status reg 2
>
> While I don't think there is a way for qemu/ath11k to get the real MSI
> vector from host, I will try to read the vfio code to check further.
> Before that, to unblock you, a possible hack is to hard code the MSI
> vector in qemu to the same as in host, on condition that the MSI
> vector doesn't change. In my case, the change looks like
>
> diff --git a/drivers/net/wireless/ath/ath11k/pci.c
> b/drivers/net/wireless/ath/ath11k/pci.c
> index 09e65c5e55c4..89a9bbe9e4d2 100644
> --- a/drivers/net/wireless/ath/ath11k/pci.c
> +++ b/drivers/net/wireless/ath/ath11k/pci.c
> @@ -459,7 +459,12 @@ static int ath11k_pci_alloc_msi(struct ath11k_pci
> *ab_pci)
> ab->pci.msi.addr_hi = 0;
> }
>
> - ath11k_dbg(ab, ATH11K_DBG_PCI, "msi base data is %d\n",
> ab->pci.msi.ep_base_data);
> + ab->pci.msi.addr_hi = 0;
> + ab->pci.msi.addr_lo = 0xfee00578;
> + ath11k_dbg(ab, ATH11K_DBG_PCI, "msi addr hi 0x%x lo 0x%x base
> data is %d\n",
> + ab->pci.msi.addr_hi,
> + ab->pci.msi.addr_lo,
> + ab->pci.msi.ep_base_data);
>
> return 0;
>
> @@ -487,6 +492,7 @@ static int ath11k_pci_config_msi_data(struct
> ath11k_pci *ab_pci)
> }
>
> ab_pci->ab->pci.msi.ep_base_data = msi_desc->msg.data;
> + ab_pci->ab->pci.msi.ep_base_data = 0;
>
> ath11k_dbg(ab_pci->ab, ATH11K_DBG_PCI, "after request_irq
> msi_ep_base_data %d\n",
> ab_pci->ab->pci.msi.ep_base_data);
>
>
> This hack works on my setup.
Progress! Thank you. This didn't work for me but its likely because my
host MSI vector is not fee00578. Where did you come up with this value?
I don't see anything in the dmesg logs, or in lspci etc.
Thanks,
James
On Tue, Jan 16, 2024 at 11:41:19AM +0100, David Woodhouse wrote:
> If we had had this posted interrupt support from the beginning, perhaps
> we could have have a much simpler model — we just let the guest write
> its intended (v)CPU#/vector *directly* to the MSI table in the device,
> and let the IOMMU fix it up by having a table pointing to the
> appropriate set of vCPUs. But that isn't how it happened. The model we
> have is that the VMM has to *emulate* the config space and handle the
> interrupts as described above.
I do have a strong desire to rework things to be more like this, just
not time yet :)
We have enough real problems related to the fake interrupt data in
the guest.
This ath11k thing sounds more like IMS really - it makes zero sense
that a device would be designed where the MSI vector has to be copied
to another location - most likely the other location is another
interrupt source that can be programmed independently, with its own
irqchip, etc? Linux supports this now. Thomas and Intel did it to
support SIOV IMS.
Are you sure you have implemented your Linux driver correctly? :)
Of course IMS doesn't work in VMs, but that is a big motivation to fix
the irq organizing. At least you'd know why the device is broken :)
Jason
On Tue, 16 Jan 2024 11:41:19 +0100
David Woodhouse <[email protected]> wrote:
> On Tue, 2024-01-16 at 18:08 +0800, Baochen Qiang wrote:
> >
> >
> > On 1/16/2024 1:46 AM, Alex Williamson wrote:
> > > On Sun, 14 Jan 2024 16:36:02 +0200
> > > Kalle Valo <[email protected]> wrote:
> > >
> > > > Baochen Qiang <[email protected]> writes:
> > > >
> > > > > > > Strange that still fails. Are you now seeing this error in your
> > > > > > > host or your Qemu? or both?
> > > > > > > Could you share your test steps? And if you can share please be as
> > > > > > > detailed as possible since I'm not familiar with passing WLAN
> > > > > > > hardware to a VM using vfio-pci.
> > > > > >
> > > > > > Just in Qemu, the hardware works fine on my host machine.
> > > > > > I basically follow this guide to set it up, its written in the
> > > > > > context of GPUs/libvirt but the host setup is exactly the same. By
> > > > > > no means do you need to read it all, once you set the vfio-pci.ids
> > > > > > and see your unclaimed adapter you can stop:
> > > > > > https://wiki.archlinux.org/title/PCI_passthrough_via_OVMF
> > > > > > In short you should be able to set the following host kernel options
> > > > > > and reboot (assuming your motherboard/hardware is compatible):
> > > > > > intel_iommu=on iommu=pt vfio-pci.ids=17cb:1103
> > > > > > Obviously change the device/vendor IDs to whatever ath11k hw you
> > > > > > have. Once the host is rebooted you should see your wlan adapter as
> > > > > > UNCLAIMED, showing the driver in use as vfio-pci. If not, its likely
> > > > > > your motherboard just isn't compatible, the device has to be in its
> > > > > > own IOMMU group (you could try switching PCI ports if this is the
> > > > > > case).
> > > > > > I then build a "kvm_guest.config" kernel with the driver/firmware
> > > > > > for ath11k and boot into that with the following Qemu options:
> > > > > > -enable-kvm -device -vfio-pci,host=<PCI address>
> > > > > > If it seems easier you could also utilize IWD's test-runner which
> > > > > > handles launching the Qemu kernel automatically, detecting any
> > > > > > vfio-devices and passes them through and mounts some useful host
> > > > > > folders into the VM. Its actually a very good general purpose tool
> > > > > > for kernel testing, not just for IWD:
> > > > > > https://git.kernel.org/pub/scm/network/wireless/iwd.git/tree/doc/test-runner.txt
> > > > > > Once set up you can just run test-runner with a few flags and you'll
> > > > > > boot into a shell:
> > > > > > ./tools/test-runner -k <kernel-image> --hw --start /bin/bash
> > > > > > Please reach out if you have questions, thanks for looking into
> > > > > > this.
> > > > >
> > > > > Thanks for these details. I reproduced this issue by following your guide.
> > > > >
> > > > > Seems the root cause is that the MSI vector assigned to WCN6855 in
> > > > > qemu is different with that in host. In my case the MSI vector in qemu
> > > > > is [Address: fee00000 Data: 0020] while in host it is [Address:
> > > > > fee00578 Data: 0000]. So in qemu ath11k configures MSI vector
> > > > > [Address: fee00000 Data: 0020] to WCN6855 hardware/firmware, and
> > > > > firmware uses that vector to fire interrupts to host/qemu. However
> > > > > host IOMMU doesn't know that vector because the real vector is
> > > > > [Address: fee00578 Data: 0000], as a result host blocks that
> > > > > interrupt and reports an error, see below log:
> > > > >
> > > > > [ 1414.206069] DMAR: DRHD: handling fault status reg 2
> > > > > [ 1414.206081] DMAR: [INTR-REMAP] Request device [02:00.0] fault index
> > > > > 0x0 [fault reason 0x25] Blocked a compatibility format interrupt
> > > > > request
> > > > > [ 1414.210334] DMAR: DRHD: handling fault status reg 2
> > > > > [ 1414.210342] DMAR: [INTR-REMAP] Request device [02:00.0] fault index
> > > > > 0x0 [fault reason 0x25] Blocked a compatibility format interrupt
> > > > > request
> > > > > [ 1414.212496] DMAR: DRHD: handling fault status reg 2
> > > > > [ 1414.212503] DMAR: [INTR-REMAP] Request device [02:00.0] fault index
> > > > > 0x0 [fault reason 0x25] Blocked a compatibility format interrupt
> > > > > request
> > > > > [ 1414.214600] DMAR: DRHD: handling fault status reg 2
> > > > >
> > > > > While I don't think there is a way for qemu/ath11k to get the real MSI
> > > > > vector from host, I will try to read the vfio code to check further.
> > > > > Before that, to unblock you, a possible hack is to hard code the MSI
> > > > > vector in qemu to the same as in host, on condition that the MSI
> > > > > vector doesn't change.
> > > >
> > > > Baochen, awesome that you were able to debug this further. Now we at
> > > > least know what's the problem.
> > >
> > > It's an interesting problem, I don't think we've seen another device
> > > where the driver reads the MSI register in order to program another
> > > hardware entity to match the MSI address and data configuration.
> > >
> > > When assigning a device, the host and guest use entirely separate
> > > address spaces for MSI interrupts. When the guest enables MSI, the
> > > operation is trapped by the VMM and triggers an ioctl to the host to
> > > perform an equivalent configuration. Generally the physical device
> > > will interrupt within the host where it may be directly attached to KVM
> > > to signal the interrupt, trigger through the VMM, or where
> > > virtualization hardware supports it, the interrupt can directly trigger
> > > the vCPU. From the VM perspective, the guest address/data pair is used
> > > to signal the interrupt, which is why it makes sense to virtualize the
> > > MSI registers.
> >
> > Hi Alex, could you help elaborate more? why from the VM perspective MSI
> > virtualization is necessary?
>
> An MSI is just a write to physical memory space. You can even use it
> like that; configure the device to just write 4 bytes to some address
> in a struct in memory to show that it needs attention, and you then
> poll that memory.
>
> But mostly we don't (ab)use it like that, of course. We tell the device
> to write to a special range of the physical address space where the
> interrupt controller lives — the range from 0xfee00000 to 0xfeefffff.
> The low 20 bits of the address, and the 32 bits of data written to that
> address, tell the interrupt controller which CPU to interrupt, and
> which vector to raise on the CPU (as well as some other details and
> weird interrupt modes which are theoretically encodable).
>
> So in your example, the guest writes [Address: fee00000 Data: 0020]
> which means it wants vector 0x20 on CPU#0 (well, the CPU with APICID
> 0). But that's what the *guest* wants. If we just blindly programmed
> that into the hardware, the hardware would deliver vector 0x20 to the
> host's CPU0... which would be very confused by it.
>
> The host has a driver for that device, probably the VFIO driver. The
> host registers its own interrupt handlers for the real hardware,
> decides which *host* CPU (and vector) should be notified when something
> happens. And when that happens, the VFIO driver will raise an event on
> an eventfd, which will notify QEMU to inject the appropriate interrupt
> into the guest.
>
> So... when the guest enables the MSI, that's trapped by QEMU which
> remembers which *guest* CPU/vector the interrupt should go to. QEMU
> tells VFIO to enable the corresponding interrupt, and what gets
> programmed into the actual hardware is up to the *host* operating
> system; nothing to do with the guest's information at all.
>
> Then when the actual hardware raises the interrupt, the VFIO interrupt
> handler runs in the guest, signals an event on the eventfd, and QEMU
s/guest/host/
> receives that and injects the event into the appropriate guest vCPU.
>
> (In practice QEMU doesn't do it these days; there's actually a shortcut
> which improves latency by allowing the kernel to deliver the event to
> the guest directly, connecting the eventfd directly to the KVM irq
> routing table.)
>
>
> Interrupt remapping is probably not important here, but I'll explain it
> briefly anyway. With interrupt remapping, the IOMMU handles the
> 'memory' write from the device, just as it handles all other memory
> transactions. One of the reasons for interrupt remapping is that the
> original definitions of the bits in the MSI (the low 20 bits of the
> address and the 32 bits of what's written) only had 8 bits for the
> target CPU APICID. And we have bigger systems than that now.
>
> So by using one of the spare bits in the MSI message, we can indicate
> that this isn't just a directly-encoded cpu/vector in "Compatibility
> Format", but is a "Remappable Format" interrupt. Instead of the
> cpu/vector it just contains an index in to the IOMMU's Interrupt
> Redirection Table. Which *does* have a full 32-bits for the target APIC
> ID. That's why x2apic support (which gives us support for >254 CPUs)
> depends on interrupt remapping.
>
> The other thing that the IOMMU can do in modern systems is *posted*
> interrupts. Where the entry in the IOMMU's IRT doesn't just specify the
> host's CPU/vector, but actually specifies a *vCPU* to deliver the
> interrupt to.
>
> All of which is mostly irrelevant as it's just another bypass
> optimisation to improve latency. The key here is that what the guest
> writes to its emulated MSI table and what the host writes to the real
> hardware are not at all related.
>
> If we had had this posted interrupt support from the beginning, perhaps
> we could have have a much simpler model — we just let the guest write
> its intended (v)CPU#/vector *directly* to the MSI table in the device,
> and let the IOMMU fix it up by having a table pointing to the
> appropriate set of vCPUs. But that isn't how it happened. The model we
> have is that the VMM has to *emulate* the config space and handle the
> interrupts as described above.
>
> This means that whenever a device has a non-standard way of configuring
> MSIs, the VMM has to understand and intercept that. I believe we've
> even seen some Atheros devices with the MSI target in some weird MMIO
> registers instead of the standard location, so we've had to hack QEMU
> to handle those too?
>
> > And, maybe a stupid question, is that possible VM/KVM or vfio only
> > virtualize write operation to MSI register but leave read operation
> > un-virtualized? I am asking this because in that way ath11k may get a
> > chance to run in VM after getting the real vector.
>
> That might confuse a number of operating systems. Especially if they
> mask/unmask by reading the register, flipping the mask bit and writing
> back again.
>
> How exactly is the content of this register then given back to the
> firmware? Is that communication snoopable by the VMM?
>
>
> > >
> > > Off hand I don't have a good solution for this, the hardware is
> > > essentially imposing a unique requirement for MSI programming that the
> > > driver needs visibility of the physical MSI address and data.
> > >
>
> Strictly, the driver doesn't need visibility to the actual values used
> by the hardware. Another way of it looking at it would be to say that
> the driver programs the MSI through this non-standard method, it just
> needs the VMM to trap and handle that, just as the VMM does for the
> standard MSI table.
>
> Which is what I thought we'd already seen on some Atheros devices.
>
> > > It's
> > > conceivable that device specific code could either make the physical
> > > address/data pair visible to the VM or trap the firmware programming to
> > > inject the correct physical values. Is there somewhere other than the
> > > standard MSI capability in config space that the driver could learn the
> > > physical values, ie. somewhere that isn't virtualized? Thanks,
> >
> > I don't think we have such capability in configuration space.
>
> Configuration space is a complete fiction though; it's all emulated. We
> can do anything we like. Or we can have a PV hypercall which will
> report it. I don't know that we'd *want* to, but all things are
> possible.
RTL8169 has a back door to the MSI-X vector table, maybe that's the one
you're thinking of. Alternate methods for the driver to access config
space is common on GPUs, presumably because they require extensive
vBIOS support and IO port and MMIO windows through which pre-boot code
can interact with config space is faster and easier than standard
config accesses. Much of the work of assigning a GPU to a VM is to
wrap those alternate methods in virtualization to keep the driver
working within the guest address space.
The fictitious config space was my thought too, an ath11k vfio-pci
variant driver could insert a vendor defined capability into config
space to expose the physical MSI address/data. The driver would know
by the presence of the capability that it's running in a VM and to
prefer that mechanism to retrieve MSI address and data.
Alternatively as also suggested here, if programming of the firmware
with the MSI address/data is something that a hypervisor could trap,
then we might be able to make it transparent to the guest. For example
if it were programmed via MMIO, the guest address/data values could be
auto-magically replaced with physical values. Since QEMU doesn't know
the physical values, this would also likely be through a device
specific extension to vfio-pci through a variant driver, or maybe some
combination of variant driver and QEMU if we need to make trapping
conditional in order to avoid a performance penalty.
This is essentially device specific interrupt programming, which either
needs to be virtualized (performed by the VMM) or paravirtualized
(performed in cooperation with the guest). This is also something to
keep in mind relative to the initial source of this issue, ie. testing
device drivers and hardware under device assignment. There can be
subtle differences. Thanks,
Alex
On 1/16/2024 10:28 AM, Alex Williamson wrote:
> On Tue, 16 Jan 2024 11:41:19 +0100
> David Woodhouse <[email protected]> wrote:
>
>> On Tue, 2024-01-16 at 18:08 +0800, Baochen Qiang wrote:
>>>
>>>
>>> On 1/16/2024 1:46 AM, Alex Williamson wrote:
>>>> On Sun, 14 Jan 2024 16:36:02 +0200
>>>> Kalle Valo <[email protected]> wrote:
>>>>
>>>>> Baochen Qiang <[email protected]> writes:
>>>>>
>>>>>>>> Strange that still fails. Are you now seeing this error in your
>>>>>>>> host or your Qemu? or both?
>>>>>>>> Could you share your test steps? And if you can share please be as
>>>>>>>> detailed as possible since I'm not familiar with passing WLAN
>>>>>>>> hardware to a VM using vfio-pci.
>>>>>>>
>>>>>>> Just in Qemu, the hardware works fine on my host machine.
>>>>>>> I basically follow this guide to set it up, its written in the
>>>>>>> context of GPUs/libvirt but the host setup is exactly the same. By
>>>>>>> no means do you need to read it all, once you set the vfio-pci.ids
>>>>>>> and see your unclaimed adapter you can stop:
>>>>>>> https://wiki.archlinux.org/title/PCI_passthrough_via_OVMF
>>>>>>> In short you should be able to set the following host kernel options
>>>>>>> and reboot (assuming your motherboard/hardware is compatible):
>>>>>>> intel_iommu=on iommu=pt vfio-pci.ids=17cb:1103
>>>>>>> Obviously change the device/vendor IDs to whatever ath11k hw you
>>>>>>> have. Once the host is rebooted you should see your wlan adapter as
>>>>>>> UNCLAIMED, showing the driver in use as vfio-pci. If not, its likely
>>>>>>> your motherboard just isn't compatible, the device has to be in its
>>>>>>> own IOMMU group (you could try switching PCI ports if this is the
>>>>>>> case).
>>>>>>> I then build a "kvm_guest.config" kernel with the driver/firmware
>>>>>>> for ath11k and boot into that with the following Qemu options:
>>>>>>> -enable-kvm -device -vfio-pci,host=<PCI address>
>>>>>>> If it seems easier you could also utilize IWD's test-runner which
>>>>>>> handles launching the Qemu kernel automatically, detecting any
>>>>>>> vfio-devices and passes them through and mounts some useful host
>>>>>>> folders into the VM. Its actually a very good general purpose tool
>>>>>>> for kernel testing, not just for IWD:
>>>>>>> https://git.kernel.org/pub/scm/network/wireless/iwd.git/tree/doc/test-runner.txt
>>>>>>> Once set up you can just run test-runner with a few flags and you'll
>>>>>>> boot into a shell:
>>>>>>> ./tools/test-runner -k <kernel-image> --hw --start /bin/bash
>>>>>>> Please reach out if you have questions, thanks for looking into
>>>>>>> this.
>>>>>>
>>>>>> Thanks for these details. I reproduced this issue by following your guide.
>>>>>>
>>>>>> Seems the root cause is that the MSI vector assigned to WCN6855 in
>>>>>> qemu is different with that in host. In my case the MSI vector in qemu
>>>>>> is [Address: fee00000 Data: 0020] while in host it is [Address:
>>>>>> fee00578 Data: 0000]. So in qemu ath11k configures MSI vector
>>>>>> [Address: fee00000 Data: 0020] to WCN6855 hardware/firmware, and
>>>>>> firmware uses that vector to fire interrupts to host/qemu. However
>>>>>> host IOMMU doesn't know that vector because the real vector is
>>>>>> [Address: fee00578 Data: 0000], as a result host blocks that
>>>>>> interrupt and reports an error, see below log:
>>>>>>
>>>>>> [ 1414.206069] DMAR: DRHD: handling fault status reg 2
>>>>>> [ 1414.206081] DMAR: [INTR-REMAP] Request device [02:00.0] fault index
>>>>>> 0x0 [fault reason 0x25] Blocked a compatibility format interrupt
>>>>>> request
>>>>>> [ 1414.210334] DMAR: DRHD: handling fault status reg 2
>>>>>> [ 1414.210342] DMAR: [INTR-REMAP] Request device [02:00.0] fault index
>>>>>> 0x0 [fault reason 0x25] Blocked a compatibility format interrupt
>>>>>> request
>>>>>> [ 1414.212496] DMAR: DRHD: handling fault status reg 2
>>>>>> [ 1414.212503] DMAR: [INTR-REMAP] Request device [02:00.0] fault index
>>>>>> 0x0 [fault reason 0x25] Blocked a compatibility format interrupt
>>>>>> request
>>>>>> [ 1414.214600] DMAR: DRHD: handling fault status reg 2
>>>>>>
>>>>>> While I don't think there is a way for qemu/ath11k to get the real MSI
>>>>>> vector from host, I will try to read the vfio code to check further.
>>>>>> Before that, to unblock you, a possible hack is to hard code the MSI
>>>>>> vector in qemu to the same as in host, on condition that the MSI
>>>>>> vector doesn't change.
>>>>>
>>>>> Baochen, awesome that you were able to debug this further. Now we at
>>>>> least know what's the problem.
>>>>
>>>> It's an interesting problem, I don't think we've seen another device
>>>> where the driver reads the MSI register in order to program another
>>>> hardware entity to match the MSI address and data configuration.
>>>>
>>>> When assigning a device, the host and guest use entirely separate
>>>> address spaces for MSI interrupts. When the guest enables MSI, the
>>>> operation is trapped by the VMM and triggers an ioctl to the host to
>>>> perform an equivalent configuration. Generally the physical device
>>>> will interrupt within the host where it may be directly attached to KVM
>>>> to signal the interrupt, trigger through the VMM, or where
>>>> virtualization hardware supports it, the interrupt can directly trigger
>>>> the vCPU. From the VM perspective, the guest address/data pair is used
>>>> to signal the interrupt, which is why it makes sense to virtualize the
>>>> MSI registers.
>>>
>>> Hi Alex, could you help elaborate more? why from the VM perspective MSI
>>> virtualization is necessary?
>>
>> An MSI is just a write to physical memory space. You can even use it
>> like that; configure the device to just write 4 bytes to some address
>> in a struct in memory to show that it needs attention, and you then
>> poll that memory.
>>
>> But mostly we don't (ab)use it like that, of course. We tell the device
>> to write to a special range of the physical address space where the
>> interrupt controller lives — the range from 0xfee00000 to 0xfeefffff.
>> The low 20 bits of the address, and the 32 bits of data written to that
>> address, tell the interrupt controller which CPU to interrupt, and
>> which vector to raise on the CPU (as well as some other details and
>> weird interrupt modes which are theoretically encodable).
>>
>> So in your example, the guest writes [Address: fee00000 Data: 0020]
>> which means it wants vector 0x20 on CPU#0 (well, the CPU with APICID
>> 0). But that's what the *guest* wants. If we just blindly programmed
>> that into the hardware, the hardware would deliver vector 0x20 to the
>> host's CPU0... which would be very confused by it.
>>
>> The host has a driver for that device, probably the VFIO driver. The
>> host registers its own interrupt handlers for the real hardware,
>> decides which *host* CPU (and vector) should be notified when something
>> happens. And when that happens, the VFIO driver will raise an event on
>> an eventfd, which will notify QEMU to inject the appropriate interrupt
>> into the guest.
>>
>> So... when the guest enables the MSI, that's trapped by QEMU which
>> remembers which *guest* CPU/vector the interrupt should go to. QEMU
>> tells VFIO to enable the corresponding interrupt, and what gets
>> programmed into the actual hardware is up to the *host* operating
>> system; nothing to do with the guest's information at all.
>>
>> Then when the actual hardware raises the interrupt, the VFIO interrupt
>> handler runs in the guest, signals an event on the eventfd, and QEMU
>
> s/guest/host/
>
>> receives that and injects the event into the appropriate guest vCPU.
>>
>> (In practice QEMU doesn't do it these days; there's actually a shortcut
>> which improves latency by allowing the kernel to deliver the event to
>> the guest directly, connecting the eventfd directly to the KVM irq
>> routing table.)
>>
>>
>> Interrupt remapping is probably not important here, but I'll explain it
>> briefly anyway. With interrupt remapping, the IOMMU handles the
>> 'memory' write from the device, just as it handles all other memory
>> transactions. One of the reasons for interrupt remapping is that the
>> original definitions of the bits in the MSI (the low 20 bits of the
>> address and the 32 bits of what's written) only had 8 bits for the
>> target CPU APICID. And we have bigger systems than that now.
>>
>> So by using one of the spare bits in the MSI message, we can indicate
>> that this isn't just a directly-encoded cpu/vector in "Compatibility
>> Format", but is a "Remappable Format" interrupt. Instead of the
>> cpu/vector it just contains an index in to the IOMMU's Interrupt
>> Redirection Table. Which *does* have a full 32-bits for the target APIC
>> ID. That's why x2apic support (which gives us support for >254 CPUs)
>> depends on interrupt remapping.
>>
>> The other thing that the IOMMU can do in modern systems is *posted*
>> interrupts. Where the entry in the IOMMU's IRT doesn't just specify the
>> host's CPU/vector, but actually specifies a *vCPU* to deliver the
>> interrupt to.
>>
>> All of which is mostly irrelevant as it's just another bypass
>> optimisation to improve latency. The key here is that what the guest
>> writes to its emulated MSI table and what the host writes to the real
>> hardware are not at all related.
>>
>> If we had had this posted interrupt support from the beginning, perhaps
>> we could have have a much simpler model — we just let the guest write
>> its intended (v)CPU#/vector *directly* to the MSI table in the device,
>> and let the IOMMU fix it up by having a table pointing to the
>> appropriate set of vCPUs. But that isn't how it happened. The model we
>> have is that the VMM has to *emulate* the config space and handle the
>> interrupts as described above.
>>
>> This means that whenever a device has a non-standard way of configuring
>> MSIs, the VMM has to understand and intercept that. I believe we've
>> even seen some Atheros devices with the MSI target in some weird MMIO
>> registers instead of the standard location, so we've had to hack QEMU
>> to handle those too?
>>
>>> And, maybe a stupid question, is that possible VM/KVM or vfio only
>>> virtualize write operation to MSI register but leave read operation
>>> un-virtualized? I am asking this because in that way ath11k may get a
>>> chance to run in VM after getting the real vector.
>>
>> That might confuse a number of operating systems. Especially if they
>> mask/unmask by reading the register, flipping the mask bit and writing
>> back again.
>>
>> How exactly is the content of this register then given back to the
>> firmware? Is that communication snoopable by the VMM?
>>
>>
>>>>
>>>> Off hand I don't have a good solution for this, the hardware is
>>>> essentially imposing a unique requirement for MSI programming that the
>>>> driver needs visibility of the physical MSI address and data.
>>>>
>>
>> Strictly, the driver doesn't need visibility to the actual values used
>> by the hardware. Another way of it looking at it would be to say that
>> the driver programs the MSI through this non-standard method, it just
>> needs the VMM to trap and handle that, just as the VMM does for the
>> standard MSI table.
>>
>> Which is what I thought we'd already seen on some Atheros devices.
>>
>>>> It's
>>>> conceivable that device specific code could either make the physical
>>>> address/data pair visible to the VM or trap the firmware programming to
>>>> inject the correct physical values. Is there somewhere other than the
>>>> standard MSI capability in config space that the driver could learn the
>>>> physical values, ie. somewhere that isn't virtualized? Thanks,
>>>
>>> I don't think we have such capability in configuration space.
>>
>> Configuration space is a complete fiction though; it's all emulated. We
>> can do anything we like. Or we can have a PV hypercall which will
>> report it. I don't know that we'd *want* to, but all things are
>> possible.
>
> RTL8169 has a back door to the MSI-X vector table, maybe that's the one
> you're thinking of. Alternate methods for the driver to access config
> space is common on GPUs, presumably because they require extensive
> vBIOS support and IO port and MMIO windows through which pre-boot code
> can interact with config space is faster and easier than standard
> config accesses. Much of the work of assigning a GPU to a VM is to
> wrap those alternate methods in virtualization to keep the driver
> working within the guest address space.
>
> The fictitious config space was my thought too, an ath11k vfio-pci
> variant driver could insert a vendor defined capability into config
> space to expose the physical MSI address/data. The driver would know
> by the presence of the capability that it's running in a VM and to
> prefer that mechanism to retrieve MSI address and data.
>
> Alternatively as also suggested here, if programming of the firmware
> with the MSI address/data is something that a hypervisor could trap,
> then we might be able to make it transparent to the guest. For example
> if it were programmed via MMIO, the guest address/data values could be
> auto-magically replaced with physical values. Since QEMU doesn't know
> the physical values, this would also likely be through a device
> specific extension to vfio-pci through a variant driver, or maybe some
> combination of variant driver and QEMU if we need to make trapping
> conditional in order to avoid a performance penalty.
>
> This is essentially device specific interrupt programming, which either
> needs to be virtualized (performed by the VMM) or paravirtualized
> (performed in cooperation with the guest). This is also something to
> keep in mind relative to the initial source of this issue, ie. testing
> device drivers and hardware under device assignment. There can be
> subtle differences. Thanks,
>
> Alex
>
>
+ [email protected] for added visibility and advice
Full thread:
<https://lore.kernel.org/all/[email protected]/>
On 1/16/2024 9:05 PM, James Prestwood wrote:
> Hi Baochen,
>
> On 1/14/24 4:37 AM, Baochen Qiang wrote:
>>
>>
>> On 1/12/2024 8:47 PM, James Prestwood wrote:
>>> Hi,
>>>
>>> On 1/11/24 6:04 PM, Baochen Qiang wrote:
>>>>
>>>>
>>>> On 1/11/2024 9:38 PM, James Prestwood wrote:
>>>>>
>>>>> On 1/11/24 5:11 AM, Kalle Valo wrote:
>>>>>> James Prestwood <[email protected]> writes:
>>>>>>
>>>>>>> Hi Kalle, Baochen,
>>>>>>>
>>>>>>> On 1/11/24 12:16 AM, Kalle Valo wrote:
>>>>>>>> Baochen Qiang <[email protected]> writes:
>>>>>>>>
>>>>>>>>> On 1/10/2024 10:55 PM, James Prestwood wrote:
>>>>>>>>>> Hi Kalle,
>>>>>>>>>> On 1/10/24 5:49 AM, Kalle Valo wrote:
>>>>>>>>>>> James Prestwood <[email protected]> writes:
>>>>>>>>>>>
>>>>>>>>>>>>> But I have also no idea what is causing this, I guess we
>>>>>>>>>>>>> are doing
>>>>>>>>>>>>> something wrong with the PCI communication? That reminds
>>>>>>>>>>>>> me, you could
>>>>>>>>>>>>> try this in case that helps:
>>>>>>>>>>>>>
>>>>>>>>>>>>> https://patchwork.kernel.org/project/linux-wireless/patch/[email protected]/
>>>>>>>>>>>> Heh, I saw this pop up a day after I sent this and was
>>>>>>>>>>>> wondering. Is
>>>>>>>>>>>> this something I'd need on the host kernel, guest, or both?
>>>>>>>>>>> On the guest where ath11k is running. I'm not optimistic that
>>>>>>>>>>> this would
>>>>>>>>>>> solve your issue, I suspect there can be also other bugs, but
>>>>>>>>>>> good to
>>>>>>>>>>> know if the patch changes anything.
>>>>>>>>>> Looks the same here, didn't seem to change anything based on the
>>>>>>>>>> kernel logs.
>>>>>>>>>>
>>>>>>>>> Could you try this?
>>>>>>>>>
>>>>>>>>> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/drivers/net/wireless/ath/ath11k/pci.c?id=39564b475ac5a589e6c22c43a08cbd283c295d2c
>>>>>>>> This reminds me, I assumed James was testing with ath.git master
>>>>>>>> branch
>>>>>>>> (which has that commit) but I never checked that. So for testing
>>>>>>>> please
>>>>>>>> always use the master branch to get the latest and greatest ath11k:
>>>>>>>>
>>>>>>>> https://git.kernel.org/pub/scm/linux/kernel/git/kvalo/ath.git/
>>>>>>>>
>>>>>>>> There's a quite long delay from ath.git to official releases.
>>>>>>> Good to know, and I was not in fact using that branch. Rebuilt from
>>>>>>> ath.git/master but still roughly the same behavior. There does
>>>>>>> appear
>>>>>>> to be more output now though, specifically a firmware crash:
>>>>>>>
>>>>>>> [ 2.281721] ath11k_pci 0000:00:06.0: failed to receive control
>>>>>>> response completion, polling..
>>>>>>> [ 2.282101] ip (65) used greatest stack depth: 12464 bytes left
>>>>>>> [ 3.306039] ath11k_pci 0000:00:06.0: Service connect timeout
>>>>>>> [ 3.307588] ath11k_pci 0000:00:06.0: failed to connect to HTT:
>>>>>>> -110
>>>>>>> [ 3.309286] ath11k_pci 0000:00:06.0: failed to start core: -110
>>>>>>> [ 3.519637] ath11k_pci 0000:00:06.0: firmware crashed:
>>>>>>> MHI_CB_EE_RDDM
>>>>>>> [ 3.519678] ath11k_pci 0000:00:06.0: ignore reset dev flags
>>>>>>> 0x4000
>>>>>>> [ 3.627087] ath11k_pci 0000:00:06.0: firmware crashed:
>>>>>>> MHI_CB_EE_RDDM
>>>>>>> [ 3.627129] ath11k_pci 0000:00:06.0: ignore reset dev flags
>>>>>>> 0x4000
>>>>>>> [ 13.802105] ath11k_pci 0000:00:06.0: failed to wait wlan mode
>>>>>>> request (mode 4): -110
>>>>>>> [ 13.802175] ath11k_pci 0000:00:06.0: qmi failed to send wlan mode
>>>>>>> off: -110
>>>>>> Ok, that's progress now. Can you try next try the iommu patch[1] we
>>>>>> talked about earlier? It's already in master-pending branch (along
>>>>>> with
>>>>>> other pending patches) so you can use that branch if you want.
>>>>>>
>>>>>> [1]
>>>>>> https://patchwork.kernel.org/project/linux-wireless/patch/[email protected]/
>>>>>
>>>>> Same result unfortunately, tried both with just [1] applied to
>>>>> ath.git and at HEAD of master-pending.
>>>>>
>>>>> Thanks,
>>>>>
>>>>> James
>>>> Strange that still fails. Are you now seeing this error in your host
>>>> or your Qemu? or both?
>>>> Could you share your test steps? And if you can share please be as
>>>> detailed as possible since I'm not familiar with passing WLAN
>>>> hardware to a VM using vfio-pci.
>>>
>>> Just in Qemu, the hardware works fine on my host machine.
>>>
>>> I basically follow this guide to set it up, its written in the
>>> context of GPUs/libvirt but the host setup is exactly the same. By no
>>> means do you need to read it all, once you set the vfio-pci.ids and
>>> see your unclaimed adapter you can stop:
>>>
>>> https://wiki.archlinux.org/title/PCI_passthrough_via_OVMF
>>>
>>> In short you should be able to set the following host kernel options
>>> and reboot (assuming your motherboard/hardware is compatible):
>>>
>>> intel_iommu=on iommu=pt vfio-pci.ids=17cb:1103
>>>
>>> Obviously change the device/vendor IDs to whatever ath11k hw you
>>> have. Once the host is rebooted you should see your wlan adapter as
>>> UNCLAIMED, showing the driver in use as vfio-pci. If not, its likely
>>> your motherboard just isn't compatible, the device has to be in its
>>> own IOMMU group (you could try switching PCI ports if this is the case).
>>>
>>> I then build a "kvm_guest.config" kernel with the driver/firmware for
>>> ath11k and boot into that with the following Qemu options:
>>>
>>> -enable-kvm -device -vfio-pci,host=<PCI address>
>>>
>>> If it seems easier you could also utilize IWD's test-runner which
>>> handles launching the Qemu kernel automatically, detecting any
>>> vfio-devices and passes them through and mounts some useful host
>>> folders into the VM. Its actually a very good general purpose tool
>>> for kernel testing, not just for IWD:
>>>
>>> https://git.kernel.org/pub/scm/network/wireless/iwd.git/tree/doc/test-runner.txt
>>>
>>> Once set up you can just run test-runner with a few flags and you'll
>>> boot into a shell:
>>>
>>> ./tools/test-runner -k <kernel-image> --hw --start /bin/bash
>>>
>>> Please reach out if you have questions, thanks for looking into this.
>>>
>> Thanks for these details. I reproduced this issue by following your
>> guide.
>>
>> Seems the root cause is that the MSI vector assigned to WCN6855 in
>> qemu is different with that in host. In my case the MSI vector in qemu
>> is [Address: fee00000 Data: 0020] while in host it is [Address:
>> fee00578 Data: 0000]. So in qemu ath11k configures MSI vector
>> [Address: fee00000 Data: 0020] to WCN6855 hardware/firmware, and
>> firmware uses that vector to fire interrupts to host/qemu. However
>> host IOMMU doesn't know that vector because the real vector is
>> [Address: fee00578 Data: 0000], as a result host blocks that
>> interrupt and reports an error, see below log:
>>
>> [ 1414.206069] DMAR: DRHD: handling fault status reg 2
>> [ 1414.206081] DMAR: [INTR-REMAP] Request device [02:00.0] fault index
>> 0x0 [fault reason 0x25] Blocked a compatibility format interrupt request
>> [ 1414.210334] DMAR: DRHD: handling fault status reg 2
>> [ 1414.210342] DMAR: [INTR-REMAP] Request device [02:00.0] fault index
>> 0x0 [fault reason 0x25] Blocked a compatibility format interrupt request
>> [ 1414.212496] DMAR: DRHD: handling fault status reg 2
>> [ 1414.212503] DMAR: [INTR-REMAP] Request device [02:00.0] fault index
>> 0x0 [fault reason 0x25] Blocked a compatibility format interrupt request
>> [ 1414.214600] DMAR: DRHD: handling fault status reg 2
>>
>> While I don't think there is a way for qemu/ath11k to get the real MSI
>> vector from host, I will try to read the vfio code to check further.
>> Before that, to unblock you, a possible hack is to hard code the MSI
>> vector in qemu to the same as in host, on condition that the MSI
>> vector doesn't change. In my case, the change looks like
>>
>> diff --git a/drivers/net/wireless/ath/ath11k/pci.c
>> b/drivers/net/wireless/ath/ath11k/pci.c
>> index 09e65c5e55c4..89a9bbe9e4d2 100644
>> --- a/drivers/net/wireless/ath/ath11k/pci.c
>> +++ b/drivers/net/wireless/ath/ath11k/pci.c
>> @@ -459,7 +459,12 @@ static int ath11k_pci_alloc_msi(struct ath11k_pci
>> *ab_pci)
>> ab->pci.msi.addr_hi = 0;
>> }
>>
>> - ath11k_dbg(ab, ATH11K_DBG_PCI, "msi base data is %d\n",
>> ab->pci.msi.ep_base_data);
>> + ab->pci.msi.addr_hi = 0;
>> + ab->pci.msi.addr_lo = 0xfee00578;
>> + ath11k_dbg(ab, ATH11K_DBG_PCI, "msi addr hi 0x%x lo 0x%x base
>> data is %d\n",
>> + ab->pci.msi.addr_hi,
>> + ab->pci.msi.addr_lo,
>> + ab->pci.msi.ep_base_data);
>>
>> return 0;
>>
>> @@ -487,6 +492,7 @@ static int ath11k_pci_config_msi_data(struct
>> ath11k_pci *ab_pci)
>> }
>>
>> ab_pci->ab->pci.msi.ep_base_data = msi_desc->msg.data;
>> + ab_pci->ab->pci.msi.ep_base_data = 0;
>>
>> ath11k_dbg(ab_pci->ab, ATH11K_DBG_PCI, "after request_irq
>> msi_ep_base_data %d\n",
>> ab_pci->ab->pci.msi.ep_base_data);
>>
>>
>> This hack works on my setup.
>
> Progress! Thank you. This didn't work for me but its likely because my
> host MSI vector is not fee00578. Where did you come up with this value?
It could, and most likely, be different from machine to machine.
> I don't see anything in the dmesg logs, or in lspci etc.
>
fee00578 is the physical MSI vector so I got it using lspci in host, see
...
Capabilities: [50] MSI: Enable+ Count=1/32 Maskable+ 64bit-
Address: fee00578 Data: 0000
Masking: fffffffe Pending: 00000000
...
> Thanks,
>
> James
>
On 1/16/2024 6:41 PM, David Woodhouse wrote:
> On Tue, 2024-01-16 at 18:08 +0800, Baochen Qiang wrote:
>>
>>
>> On 1/16/2024 1:46 AM, Alex Williamson wrote:
>>> On Sun, 14 Jan 2024 16:36:02 +0200
>>> Kalle Valo <[email protected]> wrote:
>>>
>>>> Baochen Qiang <[email protected]> writes:
>>>>
>>>>>>> Strange that still fails. Are you now seeing this error in your
>>>>>>> host or your Qemu? or both?
>>>>>>> Could you share your test steps? And if you can share please be as
>>>>>>> detailed as possible since I'm not familiar with passing WLAN
>>>>>>> hardware to a VM using vfio-pci.
>>>>>>
>>>>>> Just in Qemu, the hardware works fine on my host machine.
>>>>>> I basically follow this guide to set it up, its written in the
>>>>>> context of GPUs/libvirt but the host setup is exactly the same. By
>>>>>> no means do you need to read it all, once you set the vfio-pci.ids
>>>>>> and see your unclaimed adapter you can stop:
>>>>>> https://wiki.archlinux.org/title/PCI_passthrough_via_OVMF
>>>>>> In short you should be able to set the following host kernel options
>>>>>> and reboot (assuming your motherboard/hardware is compatible):
>>>>>> intel_iommu=on iommu=pt vfio-pci.ids=17cb:1103
>>>>>> Obviously change the device/vendor IDs to whatever ath11k hw you
>>>>>> have. Once the host is rebooted you should see your wlan adapter as
>>>>>> UNCLAIMED, showing the driver in use as vfio-pci. If not, its likely
>>>>>> your motherboard just isn't compatible, the device has to be in its
>>>>>> own IOMMU group (you could try switching PCI ports if this is the
>>>>>> case).
>>>>>> I then build a "kvm_guest.config" kernel with the driver/firmware
>>>>>> for ath11k and boot into that with the following Qemu options:
>>>>>> -enable-kvm -device -vfio-pci,host=<PCI address>
>>>>>> If it seems easier you could also utilize IWD's test-runner which
>>>>>> handles launching the Qemu kernel automatically, detecting any
>>>>>> vfio-devices and passes them through and mounts some useful host
>>>>>> folders into the VM. Its actually a very good general purpose tool
>>>>>> for kernel testing, not just for IWD:
>>>>>> https://git.kernel.org/pub/scm/network/wireless/iwd.git/tree/doc/test-runner.txt
>>>>>> Once set up you can just run test-runner with a few flags and you'll
>>>>>> boot into a shell:
>>>>>> ./tools/test-runner -k <kernel-image> --hw --start /bin/bash
>>>>>> Please reach out if you have questions, thanks for looking into
>>>>>> this.
>>>>>
>>>>> Thanks for these details. I reproduced this issue by following your guide.
>>>>>
>>>>> Seems the root cause is that the MSI vector assigned to WCN6855 in
>>>>> qemu is different with that in host. In my case the MSI vector in qemu
>>>>> is [Address: fee00000 Data: 0020] while in host it is [Address:
>>>>> fee00578 Data: 0000]. So in qemu ath11k configures MSI vector
>>>>> [Address: fee00000 Data: 0020] to WCN6855 hardware/firmware, and
>>>>> firmware uses that vector to fire interrupts to host/qemu. However
>>>>> host IOMMU doesn't know that vector because the real vector is
>>>>> [Address: fee00578 Data: 0000], as a result host blocks that
>>>>> interrupt and reports an error, see below log:
>>>>>
>>>>> [ 1414.206069] DMAR: DRHD: handling fault status reg 2
>>>>> [ 1414.206081] DMAR: [INTR-REMAP] Request device [02:00.0] fault index
>>>>> 0x0 [fault reason 0x25] Blocked a compatibility format interrupt
>>>>> request
>>>>> [ 1414.210334] DMAR: DRHD: handling fault status reg 2
>>>>> [ 1414.210342] DMAR: [INTR-REMAP] Request device [02:00.0] fault index
>>>>> 0x0 [fault reason 0x25] Blocked a compatibility format interrupt
>>>>> request
>>>>> [ 1414.212496] DMAR: DRHD: handling fault status reg 2
>>>>> [ 1414.212503] DMAR: [INTR-REMAP] Request device [02:00.0] fault index
>>>>> 0x0 [fault reason 0x25] Blocked a compatibility format interrupt
>>>>> request
>>>>> [ 1414.214600] DMAR: DRHD: handling fault status reg 2
>>>>>
>>>>> While I don't think there is a way for qemu/ath11k to get the real MSI
>>>>> vector from host, I will try to read the vfio code to check further.
>>>>> Before that, to unblock you, a possible hack is to hard code the MSI
>>>>> vector in qemu to the same as in host, on condition that the MSI
>>>>> vector doesn't change.
>>>>
>>>> Baochen, awesome that you were able to debug this further. Now we at
>>>> least know what's the problem.
>>>
>>> It's an interesting problem, I don't think we've seen another device
>>> where the driver reads the MSI register in order to program another
>>> hardware entity to match the MSI address and data configuration.
>>>
>>> When assigning a device, the host and guest use entirely separate
>>> address spaces for MSI interrupts. When the guest enables MSI, the
>>> operation is trapped by the VMM and triggers an ioctl to the host to
>>> perform an equivalent configuration. Generally the physical device
>>> will interrupt within the host where it may be directly attached to KVM
>>> to signal the interrupt, trigger through the VMM, or where
>>> virtualization hardware supports it, the interrupt can directly trigger
>>> the vCPU. From the VM perspective, the guest address/data pair is used
>>> to signal the interrupt, which is why it makes sense to virtualize the
>>> MSI registers.
>>
>> Hi Alex, could you help elaborate more? why from the VM perspective MSI
>> virtualization is necessary?
>
> An MSI is just a write to physical memory space. You can even use it
> like that; configure the device to just write 4 bytes to some address
> in a struct in memory to show that it needs attention, and you then
> poll that memory.
>
> But mostly we don't (ab)use it like that, of course. We tell the device
> to write to a special range of the physical address space where the
> interrupt controller lives — the range from 0xfee00000 to 0xfeefffff.
> The low 20 bits of the address, and the 32 bits of data written to that
> address, tell the interrupt controller which CPU to interrupt, and
> which vector to raise on the CPU (as well as some other details and
> weird interrupt modes which are theoretically encodable).
>
> So in your example, the guest writes [Address: fee00000 Data: 0020]
> which means it wants vector 0x20 on CPU#0 (well, the CPU with APICID
> 0). But that's what the *guest* wants. If we just blindly programmed
> that into the hardware, the hardware would deliver vector 0x20 to the
> host's CPU0... which would be very confused by it.
>
> The host has a driver for that device, probably the VFIO driver. The
> host registers its own interrupt handlers for the real hardware,
> decides which *host* CPU (and vector) should be notified when something
> happens. And when that happens, the VFIO driver will raise an event on
> an eventfd, which will notify QEMU to inject the appropriate interrupt
> into the guest.
>
> So... when the guest enables the MSI, that's trapped by QEMU which
> remembers which *guest* CPU/vector the interrupt should go to. QEMU
> tells VFIO to enable the corresponding interrupt, and what gets
> programmed into the actual hardware is up to the *host* operating
> system; nothing to do with the guest's information at all.
>
> Then when the actual hardware raises the interrupt, the VFIO interrupt
> handler runs in the guest, signals an event on the eventfd, and QEMU
> receives that and injects the event into the appropriate guest vCPU.
>
> (In practice QEMU doesn't do it these days; there's actually a shortcut
> which improves latency by allowing the kernel to deliver the event to
> the guest directly, connecting the eventfd directly to the KVM irq
> routing table.)
>
>
> Interrupt remapping is probably not important here, but I'll explain it
> briefly anyway. With interrupt remapping, the IOMMU handles the
> 'memory' write from the device, just as it handles all other memory
> transactions. One of the reasons for interrupt remapping is that the
> original definitions of the bits in the MSI (the low 20 bits of the
> address and the 32 bits of what's written) only had 8 bits for the
> target CPU APICID. And we have bigger systems than that now.
>
> So by using one of the spare bits in the MSI message, we can indicate
> that this isn't just a directly-encoded cpu/vector in "Compatibility
> Format", but is a "Remappable Format" interrupt. Instead of the
> cpu/vector it just contains an index in to the IOMMU's Interrupt
> Redirection Table. Which *does* have a full 32-bits for the target APIC
> ID. That's why x2apic support (which gives us support for >254 CPUs)
> depends on interrupt remapping.
>
> The other thing that the IOMMU can do in modern systems is *posted*
> interrupts. Where the entry in the IOMMU's IRT doesn't just specify the
> host's CPU/vector, but actually specifies a *vCPU* to deliver the
> interrupt to.
>
> All of which is mostly irrelevant as it's just another bypass
> optimisation to improve latency. The key here is that what the guest
> writes to its emulated MSI table and what the host writes to the real
> hardware are not at all related.
>
Thanks. A really detailed and clear explanation.
> If we had had this posted interrupt support from the beginning, perhaps
> we could have have a much simpler model — we just let the guest write
> its intended (v)CPU#/vector *directly* to the MSI table in the device,
> and let the IOMMU fix it up by having a table pointing to the
> appropriate set of vCPUs. But that isn't how it happened. The model we
> have is that the VMM has to *emulate* the config space and handle the
> interrupts as described above.
>
> This means that whenever a device has a non-standard way of configuring
> MSIs, the VMM has to understand and intercept that. I believe we've
> even seen some Atheros devices with the MSI target in some weird MMIO
> registers instead of the standard location, so we've had to hack QEMU
> to handle those too?
>
>> And, maybe a stupid question, is that possible VM/KVM or vfio only
>> virtualize write operation to MSI register but leave read operation
>> un-virtualized? I am asking this because in that way ath11k may get a
>> chance to run in VM after getting the real vector.
>
> That might confuse a number of operating systems. Especially if they
> mask/unmask by reading the register, flipping the mask bit and writing
> back again.
>
> How exactly is the content of this register then given back to the
> firmware? Is that communication snoopable by the VMM?
By programming it to a MMIO register. It is a non-standard register and
also device specific, not sure snoopable or not by the VMM.
>
>
>>>
>>> Off hand I don't have a good solution for this, the hardware is
>>> essentially imposing a unique requirement for MSI programming that the
>>> driver needs visibility of the physical MSI address and data.
>>>
>
> Strictly, the driver doesn't need visibility to the actual values used
> by the hardware. Another way of it looking at it would be to say that
> the driver programs the MSI through this non-standard method, it just
> needs the VMM to trap and handle that, just as the VMM does for the
> standard MSI table.
>
> Which is what I thought we'd already seen on some Atheros devices.
>
>>> It's
>>> conceivable that device specific code could either make the physical
>>> address/data pair visible to the VM or trap the firmware programming to
>>> inject the correct physical values. Is there somewhere other than the
>>> standard MSI capability in config space that the driver could learn the
>>> physical values, ie. somewhere that isn't virtualized? Thanks,
>>
>> I don't think we have such capability in configuration space.
>
> Configuration space is a complete fiction though; it's all emulated. We
> can do anything we like. Or we can have a PV hypercall which will
> report it. I don't know that we'd *want* to, but all things are
> possible.
OK, I get the point now.
>
Hi Baochen,
<snip>
>>>
>>> While I don't think there is a way for qemu/ath11k to get the real
>>> MSI vector from host, I will try to read the vfio code to check
>>> further. Before that, to unblock you, a possible hack is to hard
>>> code the MSI vector in qemu to the same as in host, on condition
>>> that the MSI vector doesn't change. In my case, the change looks like
>>>
>>> diff --git a/drivers/net/wireless/ath/ath11k/pci.c
>>> b/drivers/net/wireless/ath/ath11k/pci.c
>>> index 09e65c5e55c4..89a9bbe9e4d2 100644
>>> --- a/drivers/net/wireless/ath/ath11k/pci.c
>>> +++ b/drivers/net/wireless/ath/ath11k/pci.c
>>> @@ -459,7 +459,12 @@ static int ath11k_pci_alloc_msi(struct
>>> ath11k_pci *ab_pci)
>>> ab->pci.msi.addr_hi = 0;
>>> }
>>>
>>> - ath11k_dbg(ab, ATH11K_DBG_PCI, "msi base data is %d\n",
>>> ab->pci.msi.ep_base_data);
>>> + ab->pci.msi.addr_hi = 0;
>>> + ab->pci.msi.addr_lo = 0xfee00578;
>>> + ath11k_dbg(ab, ATH11K_DBG_PCI, "msi addr hi 0x%x lo 0x%x
>>> base data is %d\n",
>>> + ab->pci.msi.addr_hi,
>>> + ab->pci.msi.addr_lo,
>>> + ab->pci.msi.ep_base_data);
>>>
>>> return 0;
>>>
>>> @@ -487,6 +492,7 @@ static int ath11k_pci_config_msi_data(struct
>>> ath11k_pci *ab_pci)
>>> }
>>>
>>> ab_pci->ab->pci.msi.ep_base_data = msi_desc->msg.data;
>>> + ab_pci->ab->pci.msi.ep_base_data = 0;
>>>
>>> ath11k_dbg(ab_pci->ab, ATH11K_DBG_PCI, "after request_irq
>>> msi_ep_base_data %d\n",
>>> ab_pci->ab->pci.msi.ep_base_data);
>>>
>>>
>>> This hack works on my setup.
>>
>> Progress! Thank you. This didn't work for me but its likely because
>> my host MSI vector is not fee00578. Where did you come up with this
>> value?
> It could, and most likely, be different from machine to machine.
>
>> I don't see anything in the dmesg logs, or in lspci etc.
>>
> fee00578 is the physical MSI vector so I got it using lspci in host, see
> ...
> Capabilities: [50] MSI: Enable+ Count=1/32 Maskable+ 64bit-
> Address: fee00578 Data: 0000
> Masking: fffffffe Pending: 00000000
> ...
Mine looks like this:
...
Capabilities: [50] MSI: Enable- Count=1/32 Maskable+ 64bit-
Address: 00000000 Data: 0000
Masking: 00000000 Pending: 00000000
...
I've adjusted the patch:
diff --git a/drivers/net/wireless/ath/ath11k/pci.c
b/drivers/net/wireless/ath/ath11k/pci.c
index 09e65c5e55..1cc7115582 100644
--- a/drivers/net/wireless/ath/ath11k/pci.c
+++ b/drivers/net/wireless/ath/ath11k/pci.c
@@ -459,7 +459,12 @@ static int ath11k_pci_alloc_msi(struct ath11k_pci
*ab_pci)
ab->pci.msi.addr_hi = 0;
}
- ath11k_dbg(ab, ATH11K_DBG_PCI, "msi base data is %d\n",
ab->pci.msi.ep_base_data);
+ ab->pci.msi.addr_hi = 0;
+ ab->pci.msi.addr_lo = 0;
+ ath11k_dbg(ab, ATH11K_DBG_PCI, "msi addr hi 0x%x lo 0x%x base
data is %d\n",
+ ab->pci.msi.addr_hi,
+ ab->pci.msi.addr_lo,
+ ab->pci.msi.ep_base_data);
return 0;
@@ -487,6 +492,7 @@ static int ath11k_pci_config_msi_data(struct
ath11k_pci *ab_pci)
}
ab_pci->ab->pci.msi.ep_base_data = msi_desc->msg.data;
+ ab_pci->ab->pci.msi.ep_base_data = 0;
ath11k_dbg(ab_pci->ab, ATH11K_DBG_PCI, "after request_irq
msi_ep_base_data %d\n",
ab_pci->ab->pci.msi.ep_base_data);
But still getting the same errors:
[ 3.563057] ath11k_pci 0000:00:06.0: Service connect timeout
[ 3.565044] ath11k_pci 0000:00:06.0: failed to connect to HTT: -110
[ 3.567031] ath11k_pci 0000:00:06.0: failed to start core: -110
[ 3.777514] ath11k_pci 0000:00:06.0: firmware crashed: MHI_CB_EE_RDDM
[ 3.777555] ath11k_pci 0000:00:06.0: ignore reset dev flags 0x4000
[ 3.885137] ath11k_pci 0000:00:06.0: firmware crashed: MHI_CB_EE_RDDM
[ 3.885178] ath11k_pci 0000:00:06.0: ignore reset dev flags 0x4000
I know this isn't a proper fix, so if its something that needs more
thought than just hard-coded values I understand.
Thanks,
James
>
>> Thanks,
>>
>> James
>>
James Prestwood <[email protected]> writes:
>>> I don't see anything in the dmesg logs, or in lspci etc.
>>>
>> fee00578 is the physical MSI vector so I got it using lspci in host, see
>> ...
>> Capabilities: [50] MSI: Enable+ Count=1/32 Maskable+ 64bit-
>> Address: fee00578 Data: 0000
>> Masking: fffffffe Pending: 00000000
>> ...
>
> Mine looks like this:
>
> ...
>
> Capabilities: [50] MSI: Enable- Count=1/32 Maskable+ 64bit-
> Address: 00000000 Data: 0000
> Masking: 00000000 Pending: 00000000
And that lspci output is from the host (not the guest)?
--
https://patchwork.kernel.org/project/linux-wireless/list/
https://wireless.wiki.kernel.org/en/developers/documentation/submittingpatches
Hi Kalle,
On 1/17/24 5:43 AM, Kalle Valo wrote:
> James Prestwood <[email protected]> writes:
>
>>>> I don't see anything in the dmesg logs, or in lspci etc.
>>>>
>>> fee00578 is the physical MSI vector so I got it using lspci in host, see
>>> ...
>>> Capabilities: [50] MSI: Enable+ Count=1/32 Maskable+ 64bit-
>>> Address: fee00578 Data: 0000
>>> Masking: fffffffe Pending: 00000000
>>> ...
>> Mine looks like this:
>>
>> ...
>>
>> Capabilities: [50] MSI: Enable- Count=1/32 Maskable+ 64bit-
>> Address: 00000000 Data: 0000
>> Masking: 00000000 Pending: 00000000
> And that lspci output is from the host (not the guest)?
Yes, that is on the host. On the guest I have:
Capabilities: [50] MSI: Enable+ Count=1/32 Maskable+ 64bit-
Address: fee01004 Data: 0028
Masking: fffffffe Pending: 00000000
And I just tried those values, i.e:
- ath11k_dbg(ab, ATH11K_DBG_PCI, "msi base data is %d\n",
ab->pci.msi.ep_base_data);
+ ab->pci.msi.addr_hi = 0;
+ ab->pci.msi.addr_lo = 0xfee01004;
+ ath11k_dbg(ab, ATH11K_DBG_PCI, "msi addr hi 0x%x lo 0x%x base
data is %d\n",
+ ab->pci.msi.addr_hi,
+ ab->pci.msi.addr_lo,
+ ab->pci.msi.ep_base_data);
Though wasn't sure for "Data"/ep_base_data since the kernel uses a u32
but lspci appears to show a 16 bit value. I tried:
+ ab_pci->ab->pci.msi.ep_base_data = 0;
+ ab_pci->ab->pci.msi.ep_base_data = 0x0028;
+ ab_pci->ab->pci.msi.ep_base_data = 0x0028000;
(maybe thats entirely the wrong change)
>
On 1/17/2024 9:20 PM, James Prestwood wrote:
> Hi Baochen,
>
> <snip>
>
>>>>
>>>> While I don't think there is a way for qemu/ath11k to get the real
>>>> MSI vector from host, I will try to read the vfio code to check
>>>> further. Before that, to unblock you, a possible hack is to hard
>>>> code the MSI vector in qemu to the same as in host, on condition
>>>> that the MSI vector doesn't change. In my case, the change looks like
>>>>
>>>> diff --git a/drivers/net/wireless/ath/ath11k/pci.c
>>>> b/drivers/net/wireless/ath/ath11k/pci.c
>>>> index 09e65c5e55c4..89a9bbe9e4d2 100644
>>>> --- a/drivers/net/wireless/ath/ath11k/pci.c
>>>> +++ b/drivers/net/wireless/ath/ath11k/pci.c
>>>> @@ -459,7 +459,12 @@ static int ath11k_pci_alloc_msi(struct
>>>> ath11k_pci *ab_pci)
>>>> ab->pci.msi.addr_hi = 0;
>>>> }
>>>>
>>>> - ath11k_dbg(ab, ATH11K_DBG_PCI, "msi base data is %d\n",
>>>> ab->pci.msi.ep_base_data);
>>>> + ab->pci.msi.addr_hi = 0;
>>>> + ab->pci.msi.addr_lo = 0xfee00578;
>>>> + ath11k_dbg(ab, ATH11K_DBG_PCI, "msi addr hi 0x%x lo 0x%x
>>>> base data is %d\n",
>>>> + ab->pci.msi.addr_hi,
>>>> + ab->pci.msi.addr_lo,
>>>> + ab->pci.msi.ep_base_data);
>>>>
>>>> return 0;
>>>>
>>>> @@ -487,6 +492,7 @@ static int ath11k_pci_config_msi_data(struct
>>>> ath11k_pci *ab_pci)
>>>> }
>>>>
>>>> ab_pci->ab->pci.msi.ep_base_data = msi_desc->msg.data;
>>>> + ab_pci->ab->pci.msi.ep_base_data = 0;
>>>>
>>>> ath11k_dbg(ab_pci->ab, ATH11K_DBG_PCI, "after request_irq
>>>> msi_ep_base_data %d\n",
>>>> ab_pci->ab->pci.msi.ep_base_data);
>>>>
>>>>
>>>> This hack works on my setup.
>>>
>>> Progress! Thank you. This didn't work for me but its likely because
>>> my host MSI vector is not fee00578. Where did you come up with this
>>> value?
>> It could, and most likely, be different from machine to machine.
>>
>>> I don't see anything in the dmesg logs, or in lspci etc.
>>>
>> fee00578 is the physical MSI vector so I got it using lspci in host, see
>> ...
>> Capabilities: [50] MSI: Enable+ Count=1/32 Maskable+ 64bit-
>> Address: fee00578 Data: 0000
>> Masking: fffffffe Pending: 00000000
>> ...
>
> Mine looks like this:
>
> ...
>
> Capabilities: [50] MSI: Enable- Count=1/32 Maskable+ 64bit-
> Address: 00000000 Data: 0000
> Masking: 00000000 Pending: 00000000
I guess you got this output when qemu/ath11k is not running, either qemu
not started or ath11k module not loaded, right? Since MSI is not enabled
in this output.
Sorry I should have made it clear:
step1: start qemu and load ath11k without changing anything, then you
get those errors;
step2: keep qemu running and ath11k there, don't unload it. Then check
lspci output in host to get the MSI vector.
step3: Back in qemu, hard code MSI vector to the one you get in step2,
then rebuild and reload ath11k.
> ...
>
> I've adjusted the patch:
>
> diff --git a/drivers/net/wireless/ath/ath11k/pci.c
> b/drivers/net/wireless/ath/ath11k/pci.c
> index 09e65c5e55..1cc7115582 100644
> --- a/drivers/net/wireless/ath/ath11k/pci.c
> +++ b/drivers/net/wireless/ath/ath11k/pci.c
> @@ -459,7 +459,12 @@ static int ath11k_pci_alloc_msi(struct ath11k_pci
> *ab_pci)
> ab->pci.msi.addr_hi = 0;
> }
>
> - ath11k_dbg(ab, ATH11K_DBG_PCI, "msi base data is %d\n",
> ab->pci.msi.ep_base_data);
> + ab->pci.msi.addr_hi = 0;
> + ab->pci.msi.addr_lo = 0;
> + ath11k_dbg(ab, ATH11K_DBG_PCI, "msi addr hi 0x%x lo 0x%x base
> data is %d\n",
> + ab->pci.msi.addr_hi,
> + ab->pci.msi.addr_lo,
> + ab->pci.msi.ep_base_data);
>
> return 0;
>
> @@ -487,6 +492,7 @@ static int ath11k_pci_config_msi_data(struct
> ath11k_pci *ab_pci)
> }
>
> ab_pci->ab->pci.msi.ep_base_data = msi_desc->msg.data;
> + ab_pci->ab->pci.msi.ep_base_data = 0;
>
> ath11k_dbg(ab_pci->ab, ATH11K_DBG_PCI, "after request_irq
> msi_ep_base_data %d\n",
> ab_pci->ab->pci.msi.ep_base_data);
>
> But still getting the same errors:
>
> [ 3.563057] ath11k_pci 0000:00:06.0: Service connect timeout
> [ 3.565044] ath11k_pci 0000:00:06.0: failed to connect to HTT: -110
> [ 3.567031] ath11k_pci 0000:00:06.0: failed to start core: -110
> [ 3.777514] ath11k_pci 0000:00:06.0: firmware crashed: MHI_CB_EE_RDDM
> [ 3.777555] ath11k_pci 0000:00:06.0: ignore reset dev flags 0x4000
> [ 3.885137] ath11k_pci 0000:00:06.0: firmware crashed: MHI_CB_EE_RDDM
> [ 3.885178] ath11k_pci 0000:00:06.0: ignore reset dev flags 0x4000
>
> I know this isn't a proper fix, so if its something that needs more
> thought than just hard-coded values I understand.
>
> Thanks,
>
> James
>
>>
>>> Thanks,
>>>
>>> James
>>>
James Prestwood <[email protected]> writes:
>> Sorry I should have made it clear:
>> step1: start qemu and load ath11k without changing anything, then
>> you get those errors;
>> step2: keep qemu running and ath11k there, don't unload it. Then
>> check lspci output in host to get the MSI vector.
>> step3: Back in qemu, hard code MSI vector to the one you get in
>> step2, then rebuild and reload ath11k.
>
> Thank you for the clarification, this worked like a charm!
Very nice, huge thanks to Baochen for finding the workaround! James, let
us know how it works for you in the long run. This would definitely help
automating ath11k testing so I'm very interested about this.
--
https://patchwork.kernel.org/project/linux-wireless/list/
https://wireless.wiki.kernel.org/en/developers/documentation/submittingpatches
Hi Baochen,
I guess you got this output when qemu/ath11k is not running, either qemu
not started or ath11k module not loaded, right? Since MSI is not enabled
in this output.
>
> Sorry I should have made it clear:
> step1: start qemu and load ath11k without changing anything, then you
> get those errors;
> step2: keep qemu running and ath11k there, don't unload it. Then check
> lspci output in host to get the MSI vector.
> step3: Back in qemu, hard code MSI vector to the one you get in step2,
> then rebuild and reload ath11k.
>
Thank you for the clarification, this worked like a charm!
Thanks,
James
Hi Kalle,
On 1/19/24 9:57 AM, Kalle Valo wrote:
> James Prestwood <[email protected]> writes:
>
>>> Sorry I should have made it clear:
>>> step1: start qemu and load ath11k without changing anything, then
>>> you get those errors;
>>> step2: keep qemu running and ath11k there, don't unload it. Then
>>> check lspci output in host to get the MSI vector.
>>> step3: Back in qemu, hard code MSI vector to the one you get in
>>> step2, then rebuild and reload ath11k.
>> Thank you for the clarification, this worked like a charm!
> Very nice, huge thanks to Baochen for finding the workaround! James, let
> us know how it works for you in the long run. This would definitely help
> automating ath11k testing so I'm very interested about this.
Yes thank you Baochen!
You mean using PCI pass-through? Yes, its immensely useful for testing
kernel changes. No swapping host kernels/reboots and you can build a
very minimal kernel, mine takes ~10 min to build initially then
iterative changes take ~10 seconds.
We have also integrated this into IWD's autotest framework so you can,
in theory, run tests you normally run using mac80211_hwsim but using
real hardware (assuming you aren't needing to modify frames). I did at
one point have a system set up with 4-5 wifi cards and could run python
tests against them, bring up an AP, connect, etc. It was pretty neat.
Thanks,
James
Hi Kalle/Baochen,
On 1/19/24 9:57 AM, Kalle Valo wrote:
> James Prestwood <[email protected]> writes:
>
>>> Sorry I should have made it clear:
>>> step1: start qemu and load ath11k without changing anything, then
>>> you get those errors;
>>> step2: keep qemu running and ath11k there, don't unload it. Then
>>> check lspci output in host to get the MSI vector.
>>> step3: Back in qemu, hard code MSI vector to the one you get in
>>> step2, then rebuild and reload ath11k.
>> Thank you for the clarification, this worked like a charm!
> Very nice, huge thanks to Baochen for finding the workaround! James, let
> us know how it works for you in the long run. This would definitely help
> automating ath11k testing so I'm very interested about this.
It appears this hack isn't a reliable solution long term. It did survive
several reboots but then I was back to the same behavior. Looked at the
MSI vector and it had changed:
It went from fee00558 to feee00538
But at least I can just modify this and get it working again.
Thanks,
James
>
On 1/27/2024 2:20 AM, James Prestwood wrote:
> Hi Kalle/Baochen,
>
> On 1/19/24 9:57 AM, Kalle Valo wrote:
>> James Prestwood <[email protected]> writes:
>>
>>>> Sorry I should have made it clear:
>>>> step1: start qemu and load ath11k without changing anything, then
>>>> you get those errors;
>>>> step2: keep qemu running and ath11k there, don't unload it. Then
>>>> check lspci output in host to get the MSI vector.
>>>> step3: Back in qemu, hard code MSI vector to the one you get in
>>>> step2, then rebuild and reload ath11k.
>>> Thank you for the clarification, this worked like a charm!
>> Very nice, huge thanks to Baochen for finding the workaround! James, let
>> us know how it works for you in the long run. This would definitely help
>> automating ath11k testing so I'm very interested about this.
>
> It appears this hack isn't a reliable solution long term. It did survive
> several reboots but then I was back to the same behavior. Looked at the
> MSI vector and it had changed:
>
> It went from fee00558 to feee00538
>
Yes, the MSI vector allocation is up to the kernel, and several factors
are considered when doing this. Among them, the most important are the
IRQ load balance and available vectors on a specific CPU. As a result,
the vector changes, not only when you request it (as you already see),
but also in run time.
> But at least I can just modify this and get it working again.
>
> Thanks,
>
> James
>
>>
Given the "ewww" patch to make a module parameter workaround for this:
https://lore.kernel.org/linux-wireless/[email protected]/T/#u
On Tue, 2024-01-16 at 11:41 +0100, David Woodhouse wrote:
>
> How exactly is the content of this register then given back to the
> firmware? Is that communication snoopable by the VMM?
If I'm reading the code correctly, it's just a write to a register in a
memory mapped region, possibly in multiple locations (different queues
or something). Possibly indirect (see __ath11k_pcic_write32), but
someone would have to trace it or know the HW better to understand which
locations it's written to.
But yeah seems totally feasible to just translate that back in the VMM.
> > > Off hand I don't have a good solution for this, the hardware is
> > > essentially imposing a unique requirement for MSI programming that the
> > > driver needs visibility of the physical MSI address and data.
> > >
>
> Strictly, the driver doesn't need visibility to the actual values used
> by the hardware. Another way of it looking at it would be to say that
> the driver programs the MSI through this non-standard method, it just
> needs the VMM to trap and handle that, just as the VMM does for the
> standard MSI table.
Indeed. Much better than having a module parameter.
> Which is what I thought we'd already seen on some Atheros devices.
It probably also affects ath12k, seems similar.
johannes