Received-SPF: pass (google.com: domain of linux-wireless+bounces-2093-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) client-ip=2604:1380:45e3:2400::1;
Message-ID: <cdec259a-b779-4856-8585-f285179e3671@gmail.com>
Date: Wed, 17 Jan 2024 05:20:24 -0800
Precedence: bulk
MIME-Version: 1.0
User-Agent: Mozilla Thunderbird
Subject: Re: ath11k and vfio-pci support
Content-Language: en-US
To: Baochen Qiang <quic_bqiang@quicinc.com>, Kalle Valo <kvalo@kernel.org>
Cc: linux-wireless@vger.kernel.org, ath11k@lists.infradead.org
References: <adcb785e-4dc7-4c4a-b341-d53b72e13467@gmail.com>
 <8734v5zhol.fsf@kernel.org> <87fa5220-6fd9-433d-879b-c55ac67a0748@gmail.com>
 <87r0ipcn7j.fsf@kernel.org> <356e0b05-f396-4ad7-9b29-c492b54af834@gmail.com>
 <26119c3f-9012-47bb-948e-7e976d4773a7@quicinc.com>
 <87mstccmk6.fsf@kernel.org> <df9fd970-5af3-468c-b1f1-18f91215cf44@gmail.com>
 <8734v4auc4.fsf@kernel.org> <e8878979-1f3f-4635-a716-9ac381c617d9@gmail.com>
 <285b84d0-229c-4c83-a7d6-4c3c23139597@quicinc.com>
 <4607fb37-8227-49a3-9e8c-10c9b117ec7b@gmail.com>
 <3d22a730-aee5-4f2a-9ddc-b4b5bd4d62fe@quicinc.com>
 <ee0280fd-032c-4f45-a3f9-50d96d8bed6d@gmail.com>
 <16cfd010-b62d-4385-92d1-002820a8db38@quicinc.com>
From: James Prestwood <prestwoj@gmail.com>
In-Reply-To: <16cfd010-b62d-4385-92d1-002820a8db38@quicinc.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit

Hi Baochen,

<snip>

>>>
>>> While I don't think there is a way for qemu/ath11k to get the real 
>>> MSI vector from host, I will try to read the vfio code to check 
>>> further. Before that, to unblock you, a possible hack is to hard 
>>> code the MSI vector in qemu to the same as in host, on condition 
>>> that the MSI vector doesn't change. In my case, the change looks like
>>>
>>> diff --git a/drivers/net/wireless/ath/ath11k/pci.c 
>>> b/drivers/net/wireless/ath/ath11k/pci.c
>>> index 09e65c5e55c4..89a9bbe9e4d2 100644
>>> --- a/drivers/net/wireless/ath/ath11k/pci.c
>>> +++ b/drivers/net/wireless/ath/ath11k/pci.c
>>> @@ -459,7 +459,12 @@ static int ath11k_pci_alloc_msi(struct 
>>> ath11k_pci *ab_pci)
>>>                 ab->pci.msi.addr_hi = 0;
>>>         }
>>>
>>> -       ath11k_dbg(ab, ATH11K_DBG_PCI, "msi base data is %d\n", 
>>> ab->pci.msi.ep_base_data);
>>> +       ab->pci.msi.addr_hi = 0;
>>> +       ab->pci.msi.addr_lo = 0xfee00578;
>>> +       ath11k_dbg(ab, ATH11K_DBG_PCI, "msi addr hi 0x%x lo 0x%x 
>>> base data is %d\n",
>>> +                  ab->pci.msi.addr_hi,
>>> +                  ab->pci.msi.addr_lo,
>>> +                  ab->pci.msi.ep_base_data);
>>>
>>>         return 0;
>>>
>>> @@ -487,6 +492,7 @@ static int ath11k_pci_config_msi_data(struct 
>>> ath11k_pci *ab_pci)
>>>         }
>>>
>>>         ab_pci->ab->pci.msi.ep_base_data = msi_desc->msg.data;
>>> +       ab_pci->ab->pci.msi.ep_base_data = 0;
>>>
>>>         ath11k_dbg(ab_pci->ab, ATH11K_DBG_PCI, "after request_irq 
>>> msi_ep_base_data %d\n",
>>>                    ab_pci->ab->pci.msi.ep_base_data);
>>>
>>>
>>> This hack works on my setup.
>>
>> Progress! Thank you. This didn't work for me but its likely because 
>> my host MSI vector is not fee00578. Where did you come up with this 
>> value? 
> It could, and most likely, be different from machine to machine.
>
>> I don't see anything in the dmesg logs, or in lspci etc.
>>
> fee00578 is the physical MSI vector so I got it using lspci in host, see
> ...
>         Capabilities: [50] MSI: Enable+ Count=1/32 Maskable+ 64bit-
>                 Address: fee00578  Data: 0000
>                 Masking: fffffffe  Pending: 00000000
> ...

Mine looks like this:

...

     Capabilities: [50] MSI: Enable- Count=1/32 Maskable+ 64bit-
         Address: 00000000  Data: 0000
         Masking: 00000000  Pending: 00000000
...

I've adjusted the patch:

diff --git a/drivers/net/wireless/ath/ath11k/pci.c 
b/drivers/net/wireless/ath/ath11k/pci.c
index 09e65c5e55..1cc7115582 100644
--- a/drivers/net/wireless/ath/ath11k/pci.c
+++ b/drivers/net/wireless/ath/ath11k/pci.c
@@ -459,7 +459,12 @@ static int ath11k_pci_alloc_msi(struct ath11k_pci 
*ab_pci)
                 ab->pci.msi.addr_hi = 0;
         }

-       ath11k_dbg(ab, ATH11K_DBG_PCI, "msi base data is %d\n", 
ab->pci.msi.ep_base_data);
+        ab->pci.msi.addr_hi = 0;
+        ab->pci.msi.addr_lo = 0;
+        ath11k_dbg(ab, ATH11K_DBG_PCI, "msi addr hi 0x%x lo 0x%x base 
data is %d\n",
+                   ab->pci.msi.addr_hi,
+                   ab->pci.msi.addr_lo,
+                   ab->pci.msi.ep_base_data);

         return 0;

@@ -487,6 +492,7 @@ static int ath11k_pci_config_msi_data(struct 
ath11k_pci *ab_pci)
         }

         ab_pci->ab->pci.msi.ep_base_data = msi_desc->msg.data;
+       ab_pci->ab->pci.msi.ep_base_data = 0;

         ath11k_dbg(ab_pci->ab, ATH11K_DBG_PCI, "after request_irq 
msi_ep_base_data %d\n",
                    ab_pci->ab->pci.msi.ep_base_data);

But still getting the same errors:

[    3.563057] ath11k_pci 0000:00:06.0: Service connect timeout
[    3.565044] ath11k_pci 0000:00:06.0: failed to connect to HTT: -110
[    3.567031] ath11k_pci 0000:00:06.0: failed to start core: -110
[    3.777514] ath11k_pci 0000:00:06.0: firmware crashed: MHI_CB_EE_RDDM
[    3.777555] ath11k_pci 0000:00:06.0: ignore reset dev flags 0x4000
[    3.885137] ath11k_pci 0000:00:06.0: firmware crashed: MHI_CB_EE_RDDM
[    3.885178] ath11k_pci 0000:00:06.0: ignore reset dev flags 0x4000

I know this isn't a proper fix, so if its something that needs more 
thought than just hard-coded values I understand.

Thanks,

James

>
>> Thanks,
>>
>> James
>>