Received-SPF: pass (google.com: domain of linux-wireless+bounces-2135-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:40f1:3f00::1 as permitted sender) client-ip=2604:1380:40f1:3f00::1;
Message-ID: <21e925bc-ee3e-4421-949b-b3839d2abdef@quicinc.com>
Date: Thu, 18 Jan 2024 10:09:49 +0800
Precedence: bulk
MIME-Version: 1.0
User-Agent: Mozilla Thunderbird
Subject: Re: ath11k and vfio-pci support
Content-Language: en-US
To: James Prestwood <prestwoj@gmail.com>, Kalle Valo <kvalo@kernel.org>
CC: <linux-wireless@vger.kernel.org>, <ath11k@lists.infradead.org>
References: <adcb785e-4dc7-4c4a-b341-d53b72e13467@gmail.com>
 <8734v5zhol.fsf@kernel.org> <87fa5220-6fd9-433d-879b-c55ac67a0748@gmail.com>
 <87r0ipcn7j.fsf@kernel.org> <356e0b05-f396-4ad7-9b29-c492b54af834@gmail.com>
 <26119c3f-9012-47bb-948e-7e976d4773a7@quicinc.com>
 <87mstccmk6.fsf@kernel.org> <df9fd970-5af3-468c-b1f1-18f91215cf44@gmail.com>
 <8734v4auc4.fsf@kernel.org> <e8878979-1f3f-4635-a716-9ac381c617d9@gmail.com>
 <285b84d0-229c-4c83-a7d6-4c3c23139597@quicinc.com>
 <4607fb37-8227-49a3-9e8c-10c9b117ec7b@gmail.com>
 <3d22a730-aee5-4f2a-9ddc-b4b5bd4d62fe@quicinc.com>
 <ee0280fd-032c-4f45-a3f9-50d96d8bed6d@gmail.com>
 <16cfd010-b62d-4385-92d1-002820a8db38@quicinc.com>
 <cdec259a-b779-4856-8585-f285179e3671@gmail.com>
From: Baochen Qiang <quic_bqiang@quicinc.com>
In-Reply-To: <cdec259a-b779-4856-8585-f285179e3671@gmail.com>
Content-Type: text/plain; charset="UTF-8"; format=flowed
Content-Transfer-Encoding: 8bit


On 1/17/2024 9:20 PM, James Prestwood wrote:
> Hi Baochen,
> 
> <snip>
> 
>>>>
>>>> While I don't think there is a way for qemu/ath11k to get the real 
>>>> MSI vector from host, I will try to read the vfio code to check 
>>>> further. Before that, to unblock you, a possible hack is to hard 
>>>> code the MSI vector in qemu to the same as in host, on condition 
>>>> that the MSI vector doesn't change. In my case, the change looks like
>>>>
>>>> diff --git a/drivers/net/wireless/ath/ath11k/pci.c 
>>>> b/drivers/net/wireless/ath/ath11k/pci.c
>>>> index 09e65c5e55c4..89a9bbe9e4d2 100644
>>>> --- a/drivers/net/wireless/ath/ath11k/pci.c
>>>> +++ b/drivers/net/wireless/ath/ath11k/pci.c
>>>> @@ -459,7 +459,12 @@ static int ath11k_pci_alloc_msi(struct 
>>>> ath11k_pci *ab_pci)
>>>>                 ab->pci.msi.addr_hi = 0;
>>>>         }
>>>>
>>>> -       ath11k_dbg(ab, ATH11K_DBG_PCI, "msi base data is %d\n", 
>>>> ab->pci.msi.ep_base_data);
>>>> +       ab->pci.msi.addr_hi = 0;
>>>> +       ab->pci.msi.addr_lo = 0xfee00578;
>>>> +       ath11k_dbg(ab, ATH11K_DBG_PCI, "msi addr hi 0x%x lo 0x%x 
>>>> base data is %d\n",
>>>> +                  ab->pci.msi.addr_hi,
>>>> +                  ab->pci.msi.addr_lo,
>>>> +                  ab->pci.msi.ep_base_data);
>>>>
>>>>         return 0;
>>>>
>>>> @@ -487,6 +492,7 @@ static int ath11k_pci_config_msi_data(struct 
>>>> ath11k_pci *ab_pci)
>>>>         }
>>>>
>>>>         ab_pci->ab->pci.msi.ep_base_data = msi_desc->msg.data;
>>>> +       ab_pci->ab->pci.msi.ep_base_data = 0;
>>>>
>>>>         ath11k_dbg(ab_pci->ab, ATH11K_DBG_PCI, "after request_irq 
>>>> msi_ep_base_data %d\n",
>>>>                    ab_pci->ab->pci.msi.ep_base_data);
>>>>
>>>>
>>>> This hack works on my setup.
>>>
>>> Progress! Thank you. This didn't work for me but its likely because 
>>> my host MSI vector is not fee00578. Where did you come up with this 
>>> value? 
>> It could, and most likely, be different from machine to machine.
>>
>>> I don't see anything in the dmesg logs, or in lspci etc.
>>>
>> fee00578 is the physical MSI vector so I got it using lspci in host, see
>> ...
>>         Capabilities: [50] MSI: Enable+ Count=1/32 Maskable+ 64bit-
>>                 Address: fee00578  Data: 0000
>>                 Masking: fffffffe  Pending: 00000000
>> ...
> 
> Mine looks like this:
> 
> ...
> 
>      Capabilities: [50] MSI: Enable- Count=1/32 Maskable+ 64bit-
>          Address: 00000000  Data: 0000
>          Masking: 00000000  Pending: 00000000
I guess you got this output when qemu/ath11k is not running, either qemu 
not started or ath11k module not loaded, right? Since MSI is not enabled 
in this output.

Sorry I should have made it clear:
step1: start qemu and load ath11k without changing anything, then you 
get those errors;
step2: keep qemu running and ath11k there, don't unload it. Then check 
lspci output in host to get the MSI vector.
step3: Back in qemu, hard code MSI vector to the one you get in step2, 
then rebuild and reload ath11k.

> ...
> 
> I've adjusted the patch:
> 
> diff --git a/drivers/net/wireless/ath/ath11k/pci.c 
> b/drivers/net/wireless/ath/ath11k/pci.c
> index 09e65c5e55..1cc7115582 100644
> --- a/drivers/net/wireless/ath/ath11k/pci.c
> +++ b/drivers/net/wireless/ath/ath11k/pci.c
> @@ -459,7 +459,12 @@ static int ath11k_pci_alloc_msi(struct ath11k_pci 
> *ab_pci)
>                  ab->pci.msi.addr_hi = 0;
>          }
> 
> -       ath11k_dbg(ab, ATH11K_DBG_PCI, "msi base data is %d\n", 
> ab->pci.msi.ep_base_data);
> +        ab->pci.msi.addr_hi = 0;
> +        ab->pci.msi.addr_lo = 0;
> +        ath11k_dbg(ab, ATH11K_DBG_PCI, "msi addr hi 0x%x lo 0x%x base 
> data is %d\n",
> +                   ab->pci.msi.addr_hi,
> +                   ab->pci.msi.addr_lo,
> +                   ab->pci.msi.ep_base_data);
> 
>          return 0;
> 
> @@ -487,6 +492,7 @@ static int ath11k_pci_config_msi_data(struct 
> ath11k_pci *ab_pci)
>          }
> 
>          ab_pci->ab->pci.msi.ep_base_data = msi_desc->msg.data;
> +       ab_pci->ab->pci.msi.ep_base_data = 0;
> 
>          ath11k_dbg(ab_pci->ab, ATH11K_DBG_PCI, "after request_irq 
> msi_ep_base_data %d\n",
>                     ab_pci->ab->pci.msi.ep_base_data);
> 
> But still getting the same errors:
> 
> [    3.563057] ath11k_pci 0000:00:06.0: Service connect timeout
> [    3.565044] ath11k_pci 0000:00:06.0: failed to connect to HTT: -110
> [    3.567031] ath11k_pci 0000:00:06.0: failed to start core: -110
> [    3.777514] ath11k_pci 0000:00:06.0: firmware crashed: MHI_CB_EE_RDDM
> [    3.777555] ath11k_pci 0000:00:06.0: ignore reset dev flags 0x4000
> [    3.885137] ath11k_pci 0000:00:06.0: firmware crashed: MHI_CB_EE_RDDM
> [    3.885178] ath11k_pci 0000:00:06.0: ignore reset dev flags 0x4000
> 
> I know this isn't a proper fix, so if its something that needs more 
> thought than just hard-coded values I understand.
> 
> Thanks,
> 
> James
> 
>>
>>> Thanks,
>>>
>>> James
>>>