Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <f0d22db8-213d-92f7-963a-9d015c0a9d79@intel.com>
Date:   Wed, 19 Apr 2023 11:13:29 -0700
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:102.0) Gecko/20100101
 Firefox/102.0 Thunderbird/102.10.0
Subject: Re: [PATCH V3 09/10] vfio/pci: Support dynamic MSI-X
Content-Language: en-US
To:     Alex Williamson <alex.williamson@redhat.com>
CC:     <jgg@nvidia.com>, <yishaih@nvidia.com>,
        <shameerali.kolothum.thodi@huawei.com>, <kevin.tian@intel.com>,
        <tglx@linutronix.de>, <darwi@linutronix.de>, <kvm@vger.kernel.org>,
        <dave.jiang@intel.com>, <jing2.liu@intel.com>,
        <ashok.raj@intel.com>, <fenghua.yu@intel.com>,
        <tom.zanussi@linux.intel.com>, <linux-kernel@vger.kernel.org>
References: <cover.1681837892.git.reinette.chatre@intel.com>
 <86cda5cf2742feff3b14954284fb509863355050.1681837892.git.reinette.chatre@intel.com>
 <20230418163829.149e8881.alex.williamson@redhat.com>
From:   Reinette Chatre <reinette.chatre@intel.com>
In-Reply-To: <20230418163829.149e8881.alex.williamson@redhat.com>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?SzFlUG5UZ3dqRm9kRXk1Qm9pTWRqRmU2cVZzZGZ3eWl0OUFueFdZZ3ZSTE5L?=
 =?utf-8?B?YmhqRDZoaFdXZWY3cEFEMlJIYnJWamduNHFtd20xanBBQ1E2WU9QbWVseExS?=
 =?utf-8?B?aDdpcTZHMUQ4enpVOWFORzdoUnBtamRGbG1vQThta3k5c3ljbGQyTHlHRDA5?=
 =?utf-8?B?R0NTQlNlUFY4WU41QWs1R0Q2czVRKzN0bHVGaDlrRUFFWUlqYmV2eWUzTEtJ?=
 =?utf-8?B?cFZDWkZXSWthUkE1ZHp6RUlHaTFkL1FXNjk0Qmk5c1NoRW0vN1Qya0R1cjNn?=
 =?utf-8?B?WnRJWSt6cnB6eGVOVmpURE83Nkw1QW51TDFadDB3YUdiQk4yNmhYaGRXRmhN?=
 =?utf-8?B?aTg2VU9SdE03aHc1UEc3blowRzJ1UlBLdlJQWXVmdzRmemRQZ25rWU9GUWZU?=
 =?utf-8?B?K2R1TnQwZWhXQ2xnR0hjUHd2a25IOFd4a1h5elpPQnVRNTJwV1FoUllvMHRa?=
 =?utf-8?B?Mk8xektYQ29sbnlnTGUrVnhGQXVQeGlaZnQreGJmMWw3a0RMUGtOb0l6bnM0?=
 =?utf-8?B?c3pSbisxMCtXYjJFNnVsejViblZsaUpuMmhxMUV3dVdlRzNGTUN6ZUgrMDVy?=
 =?utf-8?B?VHVPNi8wQjhuL2dDVXJKeXZ3NEpWQ3dHcU9sbGhtN1M1cjJwc2ttN2pyT3BY?=
 =?utf-8?B?MlBVV2N3SnBOdkhmTFF0R3Qxc0ZMZitLcjFFaUZxMlFQRVZTcWVNd1N0Zlln?=
 =?utf-8?B?ckVUaW81ZXpiZGRmbTVWbzVhUnBhWm02N2dMbElVUVo1MTRpeEhEb2NmeUx6?=
 =?utf-8?B?K0x3dVQwd09oSzNyRkdmY1ZIUW9YTzV4NW92K0lITHZWYmkrU1hwU25rMmJI?=
 =?utf-8?B?L1Nlc0dCcE1mSlJHYTJzaVVQc2FWMUdoYWREZlVlYmlUUnJrbEkvOElXNlRS?=
 =?utf-8?B?ME44dXVXaTBNcFArVTRyRUNvbkl4N1BiNGVaaTNuVDF4RmJsa3ZvOGhDclNI?=
 =?utf-8?B?TTNCbGtmTzl6Wk5aVmFtbWQ2a2gxWTVRS0tzS0ZPQTk4aE1qRjBQZzZXT1kz?=
 =?utf-8?B?b2lIR2FtQVplV1Rsd2xOSkNoYkhidnNacWdtL2IxSlBrZ2lseGdBdkRsSU13?=
 =?utf-8?B?T0poVDNpQ1ZsblVsZjdsS1g5WDh3Umo5TE94YThHZjYrVExscEVmVURhdW0y?=
 =?utf-8?B?MHBSUVpUZThBb2o0WmFCVmRRN2dJeVZoQld2Q09jdzBoWG9XNTBjSHNLc3VQ?=
 =?utf-8?B?R0RWYmpYM1IzZzRQaHd3OUsyUlU1b3N5aGxMUHJmbGg0OVRwZ0tlb2pTaXd3?=
 =?utf-8?B?REpyOGsyUVl3OHExNFkyZU9aZVQwZ0JQM0xNREVwaHplRHNMZlB0SWhHdmxs?=
 =?utf-8?B?UklpQVV6SjFVNUZsR0g1ekhPNHhyS3NOc3pUVXl6MHcyZUJXZ3RRZi9ENHpy?=
 =?utf-8?B?ZXhIRzZydDZ3RUNMSTJ1Q0J0YzF5cEY0c25PVlNtMmNUYWJDRzY0a0RrWlpV?=
 =?utf-8?B?T3JRRjU1aWtNYSt2elk4Zi9XVE9BS1BsNVJiNk9jQ3VOTUJkTTloSWFFYmw5?=
 =?utf-8?B?NXBKaFZQczI3eS9SKzBSNGhjMlhQcUV4Zjh0MmhGMHlWYnJrTjYwRmtoUkNR?=
 =?utf-8?B?am5BQ21kd0dIYWU5RkVRc0N2SktiaXRjUXJMVEZJbmlsdG9zSHJDMnF6L2RK?=
 =?utf-8?B?UlJXYjhTQWlka3QwQitHTktWaEM5aWFsQ01mS3UyQzFrTGtWZVRNb2s0QWd5?=
 =?utf-8?B?YVhxZkpUaDFNL3ZSdWJXQ28xZ1o2YkJhcGF4bnlyQ1N0R0xYN3lqRkFrOU0x?=
 =?utf-8?B?SndRSWFITjNGOXV0ZFduTmxoWVFuQUF4MWRtenBJU0pia01qQmF5N1IzcDls?=
 =?utf-8?B?RE8zMnZDVmVtdWc1a3VvRDZPb3ZsUkNGV1NFM21GWTlpMWh2YlBBMS96SkZ1?=
 =?utf-8?B?S3lWcDZrNW52WC9nTWVwM1ZFd0J2Q2JSdEgxQkVBWGNTK0VUelJqeGs2ZjFT?=
 =?utf-8?B?WTNnN2FMck5VVDkrYzl3YnpHajd3aVBxQzBMQmVqcURmc2RYY3NNWmhEZThW?=
 =?utf-8?B?dEFScml3UGFacUhPVlBET0ZNUXhFMTEvdkpueG80VmpVdmJPRitTcmUzRm1m?=
 =?utf-8?B?QjEyREtnd3hXam04WXp2Y2ZueHhtZEpqUUZkZHJwMzdYOXQwYlc4K0pqYTI5?=
 =?utf-8?B?Zmt4dFQ2c2JmYnBNVjZGQlVHWHh1M0orZmhuOUdUeTdsTkZjYU16QTJoaGF1?=
 =?utf-8?B?WFE9PQ==?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 0ad3bf23-3d87-4407-bad7-08db4101c8f4
X-MS-Exchange-CrossTenant-AuthSource: CY4PR11MB1862.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 19 Apr 2023 18:13:32.9650
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: JXaDQz3LfA4ZVVZ9NF1vzODtk/zzSuPAV09W6vOQ7UmLy3Jiy7zZGRucy0q011IfwPNtEB1gtDWYozWzue7ohTSMGyNLSaeDpWeKoTtTtDw=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ0PR11MB6671
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-7.0 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,NICE_REPLY_A,
        RCVD_IN_DNSWL_MED,SPF_HELO_PASS,SPF_NONE,T_SCC_BODY_TEXT_LINE,
        URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

Hi Alex,

On 4/18/2023 3:38 PM, Alex Williamson wrote:
> On Tue, 18 Apr 2023 10:29:20 -0700
> Reinette Chatre <reinette.chatre@intel.com> wrote:
> 
>> Recently introduced pci_msix_alloc_irq_at() and pci_msix_free_irq()
>> enables an individual MSI-X interrupt to be allocated and freed after
>> MSI-X enabling.
>>
>> Use dynamic MSI-X (if supported by the device) to allocate an interrupt
>> after MSI-X is enabled. An MSI-X interrupt is dynamically allocated at
>> the time a valid eventfd is assigned. This is different behavior from
>> a range provided during MSI-X enabling where interrupts are allocated
>> for the entire range whether a valid eventfd is provided for each
>> interrupt or not.
>>
>> Do not dynamically free interrupts, leave that to when MSI-X is
>> disabled.
> 
> But we do, sometimes, even if essentially only on the error path.  Is
> that worthwhile?  It seems like we could entirely remove
> vfio_msi_free_irq() and rely only on pci_free_irq_vectors() on MSI/X
> teardown.

Yes, it is only on the error path where dynamic MSI-X interrupts are
removed. I do not know how to determine if it is worthwhile. On the
kernel side failure seems unlikely since it would mean memory cannot
be allocated or request_irq() failed. In these cases it may not be
worthwhile since user space may try again and having the interrupt
already allocated would be helpful. The remaining error seems to be
if user space provided an invalid eventfd. An allocation in response
to wrong user input is a concern to me. Should we consider
buggy/malicious users? I am uncertain here so would defer to your
guidance.

> I'd probably also add a comment in the commit log about the theory
> behind not dynamically freeing irqs, ie. latency, reliability, and
> whatever else we used to justify it.  Thanks,

Sure. How about something like below to replace the final sentence of
the changelog:

"When a guest disables an interrupt, user space (Qemu) does not
disable the interrupt but instead assigns it a different trigger. A
common flow is thus for the VFIO_DEVICE_SET_IRQS ioctl() to be 
used to assign a new eventfd to an already enabled interrupt. Freeing
and re-allocating an interrupt in this scenario would add unnecessary
latency as well as uncertainty since the re-allocation may fail. Do
not dynamically free interrupts when an interrupt is disabled, instead
support a subsequent re-enable to draw from the initial allocation when
possible. Leave freeing of interrupts to when MSI-X is disabled."

Reinette