Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <ca71f97e-ce0b-945e-4e86-49f485fa7d5b@intel.com>
Date:   Fri, 5 May 2023 10:21:22 -0700
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:102.0) Gecko/20100101
 Firefox/102.0 Thunderbird/102.10.1
Subject: Re: [PATCH V4 10/11] vfio/pci: Support dynamic MSI-X
Content-Language: en-US
To:     "Tian, Kevin" <kevin.tian@intel.com>,
        "jgg@nvidia.com" <jgg@nvidia.com>,
        "yishaih@nvidia.com" <yishaih@nvidia.com>,
        "shameerali.kolothum.thodi@huawei.com" 
        <shameerali.kolothum.thodi@huawei.com>,
        "alex.williamson@redhat.com" <alex.williamson@redhat.com>
CC:     "tglx@linutronix.de" <tglx@linutronix.de>,
        "darwi@linutronix.de" <darwi@linutronix.de>,
        "kvm@vger.kernel.org" <kvm@vger.kernel.org>,
        "Jiang, Dave" <dave.jiang@intel.com>,
        "Liu, Jing2" <jing2.liu@intel.com>,
        "Raj, Ashok" <ashok.raj@intel.com>,
        "Yu, Fenghua" <fenghua.yu@intel.com>,
        "tom.zanussi@linux.intel.com" <tom.zanussi@linux.intel.com>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>
References: <cover.1682615447.git.reinette.chatre@intel.com>
 <c4c582970fbeaf4b6000845c400aa4c6b7bb2f13.1682615447.git.reinette.chatre@intel.com>
 <BN9PR11MB5276B67702AACB0B5BF1EC0A8C6B9@BN9PR11MB5276.namprd11.prod.outlook.com>
 <296ec21f-fc7d-eaf2-484c-27ae8815c5a8@intel.com>
 <BN9PR11MB5276ED7B47909222093E92438C729@BN9PR11MB5276.namprd11.prod.outlook.com>
From:   Reinette Chatre <reinette.chatre@intel.com>
In-Reply-To: <BN9PR11MB5276ED7B47909222093E92438C729@BN9PR11MB5276.namprd11.prod.outlook.com>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?ZGl2dnhyallCQ0lpcTVSVmRNQ3VBWEZ6THNIUXkxbGRUNEZ6MEVaSVVEa2xF?=
 =?utf-8?B?cExGUGs2OUJnVWxwNStLUnJMY2M5SE4rYzB6amswcHJYUnF3bG9NQXdWMHpV?=
 =?utf-8?B?OUQ1L0diZHNubXc1eXpTRjBVVUpQcnNON2V5djVEd3NZRjg1TVNaV1I5SXpG?=
 =?utf-8?B?WGhIUlhkOE1UUW9CWGF0YTJ6NjVTNWpvK0VwK1czYUxtTVY0b1dpTGRXZGVR?=
 =?utf-8?B?by9iWUFIc0g5eTVFOEhxay9VZndOaXRmN1MrM0QvK1pkbkZPWkE0UmlTOG0y?=
 =?utf-8?B?M3Bla3VCNlVKdm0vMFdvaVhad1owNmkvbHVQaDd3eWRxQVB6ajZxL0dJZ0RG?=
 =?utf-8?B?ZlBEYVhiVmFwUkwwY1pFR0Rjb2JSVnlXZ1AyL2J5U1RvNHdQeXRac1dNQWRX?=
 =?utf-8?B?N2pyaWE1ZktKdEp1dDRTWWZZWlFXc3lub3NtQm5CQUluOE5hUDd4Qk1Ra2Jv?=
 =?utf-8?B?aHpZdEJwMzVVYjNBU0NwaTlpREh0ekpiUFZ5NjB1VUpqM1BHNVU5VnpqZ0NY?=
 =?utf-8?B?dmdVbnJUSFE1d1VzemYrNTExQkRtcnkvTjdOTVNYRUtoalFIZnU5cnZXdGgr?=
 =?utf-8?B?V1hZYWthd215YlNrNUNJNkp1OVViM0xjNkN2bkU2OWJyTG9nOTd2L0ZJcEdt?=
 =?utf-8?B?QXByTkkraGRETGNmOXYyK1lPazBBT1ErbWxPcGVNMS9oTDhWWTNBSTFhU2E5?=
 =?utf-8?B?dkRMR0lJUEdtcm82ckRaQjJmTFJLZ3lFMVdZRFI5ODYzMjFZOGZ1OG5LS3Nx?=
 =?utf-8?B?OUthSExSOG02cmRCejZ4UDhhalcxb283ckpWcDM0NGxMeks3ZE92QXgrU0hp?=
 =?utf-8?B?OUR1ODJhOURwSnE2NGRnM0VPK25GaUdKUjdqa0VWV1ZaOUdHNi9kUitLdGNR?=
 =?utf-8?B?UEJxVTBYMDl4N1QwNWVDOTJpcWgrVXRqYkhaRXRxV0srLzc2aDIyWE5LdVFy?=
 =?utf-8?B?SVFGbGNxbWgrU0JUMHJjNnZGOTJOYnRYVnZKS29ac3FHQ3h2SzhFKy9oejcx?=
 =?utf-8?B?NjdidGd0YTRXTHZwdG1nSTRvajU2cWE3ZWNxR0tuRks3NVF1a3lZYnZtRGpT?=
 =?utf-8?B?RXhoc0JucDBvWisrb0RTQ0pnM3ZuUklsTlFycVFGMTVEUmdjZzRyTnZIckUv?=
 =?utf-8?B?UjdrbVVWRnAyK0RuRlNaRnpwVXJEeFNBU3c0eXhDUWlHeDZXRzFIZ2hVc3NV?=
 =?utf-8?B?Yk8zL2hyWVNpQXU4RlRvQUxOSVFIak1GcXhhN0duS1RoOHB3REw3ajZML1U5?=
 =?utf-8?B?aWE4ck1FVVdINzRjQVBkWm5IdFNpTDdFa2k0T3hkSEVRU1FwVzhmVVBlQitJ?=
 =?utf-8?B?TlBBT2RldGxYeFcwZ2VtOG1YeWZrSTBZK01UNkNGYXdZUnRvMG53VWxkYXBW?=
 =?utf-8?B?NFJ5SFhUZkxJU3k2N21vRzBqdS9BSTRpS2x0UCtpSCtjOWt6QWhYaGRYK0N1?=
 =?utf-8?B?VnJyMklpSFBva1VueUFtaG1JenJhVkhIR3RqR001QmtYSDBpSWZRSTA3ZTJh?=
 =?utf-8?B?NnQrckt5T2RnRkNna2R0NGlzSWpFSXRBQVAvdWdoK3ZnTERUUXM3RUp3bzlv?=
 =?utf-8?B?dTFSVjdtQ09LMUtJZUFrNkNkVzNwRHVmRXA2b0pDM2s1NXBVenBPL0dWaWQ0?=
 =?utf-8?B?ZDJxVHUwQWtjWDZlQ3FXUDFlUWl6a0dxbXUvMDNld1FUUEZTS0Q3YVVlR2hS?=
 =?utf-8?B?dlNHNlI5OVB2Tk9ZcjlveUJING1uNnlXMU4ycDMwTFEyNFh1OE9sM25RR3hn?=
 =?utf-8?B?a1N2NE9FTVVTSExPT29WNDlSZWRTTmxVcndwYWRJalY0QXNnMnh4MmNQTGYv?=
 =?utf-8?B?SmpiQVM0OUZvdmtRK0ZMTXNyYTZoZDlnMDQ3NnBxczUvWWJRc0F1TnVBa1VT?=
 =?utf-8?B?aXpRenFxZEpGSER3a09pV1IzeURSbWtIYkxnUUw2VUdjNHR3T01hUTRvL3BH?=
 =?utf-8?B?ZktzVkVPYW90aFlLSXk0QlZWMzkwOHNoSGIyeGxwQkp3SWFEMGNMakI4eW1v?=
 =?utf-8?B?RDY5NjNzKzBlQzVHSTFpck9jMCtkZVZySkVXRExPMVhpMU11NWYzcU5PSmNL?=
 =?utf-8?B?S0szSlBRTjB4YTdTaE15Y1FsMEszRnFSM2RNVEc3d3Q2UnZjTG9Ta3p5dzBp?=
 =?utf-8?B?U1ZiUXlsbEg0YS9kVjJrbjZ5S2RvbkhQUlEzSTVicVNVeUxmU2pmeUtEL1F2?=
 =?utf-8?B?TUE9PQ==?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 2763bbdc-3622-4b72-4b1f-08db4d8d2705
X-MS-Exchange-CrossTenant-AuthSource: SJ2PR11MB7573.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 05 May 2023 17:21:24.4922
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: XAHL9DldYgCBpOfAUCRar5hAK9o036mvU75+qIyQ+qasZWl08eFcnTAoHZcvtaX3M/tGg2bK7RqWqAt2fol++7NU/yWFGWGQEu8ie6tjtl0=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN0PR11MB6160
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-8.8 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,NICE_REPLY_A,
        RCVD_IN_DNSWL_MED,RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL,SPF_HELO_NONE,
        SPF_NONE,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no
        version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

Hi Kevin,

On 5/5/2023 1:10 AM, Tian, Kevin wrote:
>> From: Chatre, Reinette <reinette.chatre@intel.com>
>> Sent: Saturday, April 29, 2023 2:35 AM
>> On 4/27/2023 11:50 PM, Tian, Kevin wrote:
>>>> From: Chatre, Reinette <reinette.chatre@intel.com>
>>>> Sent: Friday, April 28, 2023 1:36 AM

...

>>>> +/*
>>>> + * Return Linux IRQ number of an MSI or MSI-X device interrupt vector.
>>>> + * If a Linux IRQ number is not available then a new interrupt will be
>>>> + * allocated if dynamic MSI-X is supported.
>>>> + */
>>>> +static int vfio_msi_alloc_irq(struct vfio_pci_core_device *vdev,
>>>> +			      unsigned int vector, bool msix)
>>>> +{
>>>> +	struct pci_dev *pdev = vdev->pdev;
>>>> +	struct msi_map map;
>>>> +	int irq;
>>>> +	u16 cmd;
>>>> +
>>>> +	irq = pci_irq_vector(pdev, vector);
>>>> +	if (irq > 0 || !msix || !vdev->has_dyn_msix)
>>>> +		return irq;
>>>
>>> if (irq >= 0 || ...)
>>>
>>
>> I am not sure about this request because pci_irq_vector() cannot return 0.
>> The Linux interrupt number will be > 0 on success. 0 means "not found"
>> (see msi_get_virq()), which is translated to -EINVAL by pci_irq_vector().
>>
> 
> There is a subtle difference between the description and the code of
> pci_irq_vector().
> 
> /**
>  * pci_irq_vector() - Get Linux IRQ number of a device interrupt vector
>  * @dev: the PCI device to operate on
>  * @nr:  device-relative interrupt vector index (0-based); has different
>  *       meanings, depending on interrupt mode:
>  *
>  *         * MSI-X     the index in the MSI-X vector table
>  *         * MSI       the index of the enabled MSI vectors
>  *         * INTx      must be 0
>  *
>  * Return: the Linux IRQ number, or -EINVAL if @nr is out of range
>  */
> 
> From above '0' is a valid irq number.
> 
> then in following code:
> 
> 	irq = msi_get_virq(&dev->dev, nr);
> 	return irq ? irq : -EINVAL;
> 
> '0' is obviously invalid for msi.
> 
> I didn't realize the msi part when reading the patch. It left me in
> confusion that '0' is unhandled as here we only check ">0" while in
> other places "-EINVAL" is checked.
> 
> Not big matter but it sounds slightly clearer to me to follow the
> description of pci_irq_vector() instead of its internal detail. 

I can add an explicit check for '0' and, as you confirmed, this is
invalid for MSI and thus I think it should be treated as an error.
This is perhaps another candidate for a WARN considering that
pci_irq_vector() returning a '0' for MSI indicates a kernel problem .

I now consider taking guidance from pci_irq_get_affinity(). Note that
pci_irq_get_affinity() contains:

const struct cpumask *pci_irq_get_affinity(struct pci_dev *dev, int nr)
{
	int idx, irq = pci_irq_vector(dev, nr);
	...
	if (WARN_ON_ONCE(irq <= 0))
		return NULL;
	...
}	


Would you be ok with something like below?

diff --git a/drivers/vfio/pci/vfio_pci_intrs.c b/drivers/vfio/pci/vfio_pci_intrs.c
index b549f5c97cb8..a8e96254f953 100644
--- a/drivers/vfio/pci/vfio_pci_intrs.c
+++ b/drivers/vfio/pci/vfio_pci_intrs.c
@@ -393,6 +393,8 @@ static int vfio_msi_alloc_irq(struct vfio_pci_core_device *vdev,
 	u16 cmd;
 
 	irq = pci_irq_vector(pdev, vector);
+	if (WARN_ON_ONCE(irq == 0))
+		return -EINVAL;
 	if (irq > 0 || !msix || !vdev->has_dyn_msix)
 		return irq;

I would prefer that vfio_msi_alloc_irq() returns negative errors. This enables
callers to in turn just return the error code on failure (note that dynamic
allocation can return different error codes), not needing to translate 0 into
an error.

Reinette