Received-SPF: pass (google.com: domain of linux-kernel+bounces-55646-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.199.223 as permitted sender) client-ip=147.75.199.223;
Message-ID: <51b8337f-fb7a-49d9-90f5-a357dfe3ac5b@intel.com>
Date: Tue, 6 Feb 2024 13:44:32 -0800
User-Agent: Mozilla Thunderbird
Subject: Re: [PATCH 03/17] vfio/pci: Consistently acquire mutex for interrupt
 management
Content-Language: en-US
To: Alex Williamson <alex.williamson@redhat.com>
CC: <jgg@nvidia.com>, <yishaih@nvidia.com>,
	<shameerali.kolothum.thodi@huawei.com>, <kevin.tian@intel.com>,
	<kvm@vger.kernel.org>, <dave.jiang@intel.com>, <ashok.raj@intel.com>,
	<linux-kernel@vger.kernel.org>, <patches@lists.linux.dev>
References: <cover.1706849424.git.reinette.chatre@intel.com>
 <e7d35d7730f3f83417e757bc264a470f8c2671ed.1706849424.git.reinette.chatre@intel.com>
 <20240205153452.4a9bddfd.alex.williamson@redhat.com>
From: Reinette Chatre <reinette.chatre@intel.com>
In-Reply-To: <20240205153452.4a9bddfd.alex.williamson@redhat.com>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: 7bit
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?U29DS3pqNm8yb3FVSzR4akVpWWV5TWNzNTFWSmRROHI5Vzd4NFZEaWJqekxw?=
 =?utf-8?B?ci9jVGZaWU9LL3MrMmUzcVV0Z0NPVThWdXFQNEdmRnJ3Y3d0YklIUndWd2h0?=
 =?utf-8?B?OWtzSjJCSitVclpseUIySnBaWUJOdHdBU0d4cDYyc1hTR29GV2JSYkZpWXR3?=
 =?utf-8?B?RFpQd3RsQnRHWDk0TjZRaENTMXZtUjFjK2k0bk5QQmZ4cXYrcHp2TEdlRFZo?=
 =?utf-8?B?VXIvR0hjSHBhelBDUFVwV2djYlV6Zmlzb0FSK0pXbjlFeEdQR1NnOTBCcnpm?=
 =?utf-8?B?WHFuejR4OFFuNDBVMGNlYlgvUkpURWZOekUrUExqYVZNZlNjQ3JZRjYxbUhW?=
 =?utf-8?B?RnB3bXBWTDN1WklHR2hIb1hpaWVaY3NQNUhHSWEwVjV0NGNwYy9YUUkxaFg4?=
 =?utf-8?B?S21RN1FFenRGM3ZUY1dXdXNlS2RZL0JzdXB3aGtlSEY0WkdUQjA4ZkhlL0Np?=
 =?utf-8?B?clNia1NrS3N1MlYwRG80d1BSdE14V0FiZldseHp5UGwwM01QVHAvV3I4SGRV?=
 =?utf-8?B?dDdpWDNnRlM1VmFoQUVpUGpIeTJhVXJ4K2dQWHVWd3EwUWVDMU4zSlBHR3Qw?=
 =?utf-8?B?WEMzRXdIb3d3Tm1iQkVKUlFNTGZYajA1MVpPZEd4U2R2TjVuZW0vRjZOR1Nh?=
 =?utf-8?B?MDJRUjczaG5rb2VzMkhaRUZMVlZuSmN1ZVlrWEFWMWFNeHhpRloyaWYxVmVm?=
 =?utf-8?B?ejRiVkdyOE1ub2g3WjhhZHdwUGpFV0QvRXhaQ0ZyY0FDNHR5b0NING00NXJo?=
 =?utf-8?B?TWNMdUd4WGFMNVFFaTJuekkzVVJrQXRRVk1saldaaWtVK3B6S3BFdkxHazM5?=
 =?utf-8?B?VXQ3dEZhOGNzOWo4NGUxdVVwVEw1VFBsbStZb2ZUS2I0OW9xdVU1OVB2ZlZ3?=
 =?utf-8?B?ZkpRQ0pVYjdtU2sxUUVjQVdsU2pNUFdiZFVKOVlSdzAycXRBRXJ1U2liSDU2?=
 =?utf-8?B?M295VVh5SFNVNjlja0x6WlNORUJ3ODJZTklUcElieDJETitRWGtSMVFMeFd1?=
 =?utf-8?B?MlFZc21ncm5ZbURwSVQzLzJIU2FBNzJRV1l4OEE5YXVoVkFtNEFycVNETHRj?=
 =?utf-8?B?VHZmTXk2RWt6ZnNvbnVZam55M1dZL0dvVWlzRlFSclJIem1tUjViK0JmeHBZ?=
 =?utf-8?B?WmU1aEVZVGppTXJOdk82MUJvbWQ0SzlPdG9rQ3Fka2t2MFRwd2Z2QTd1QmN2?=
 =?utf-8?B?dDdDVjNZZGxmSmlhTFR5UU1yeVg4YWF6RXhsR013d2N0NTBQVlY0WDFGaUo3?=
 =?utf-8?B?RmdhODRwNzNpZGpvQWlDamFHaGUwTEREZGZNMHlDYmc2VC9ORU9yTEFMeSt1?=
 =?utf-8?B?UnNHR0IweGRVbWxRRUZtK29QaHJFQVltam45WDFpOEh1b0s2UG4wSkxiRTk2?=
 =?utf-8?B?aldteDVXZDhVVTZqUlVHd1poU2pLY241UENxdW8zc1I1UEZMZld5bkg3WTB3?=
 =?utf-8?B?bE40eHBmdXllamMzYngxNzc2bEprb21vSDVQSm40ZnRGUXhFUUlHRHYzSTFZ?=
 =?utf-8?B?SmUySFloUzRWY1ZxNkpoeDM0aGhsaDJCOTMyaElMV0xtdjNCbVZwSWVVVUxl?=
 =?utf-8?B?eGNIb1pueHNXQitPclZQVXlpNGdGR3lNWEZiNTAzVGZ4VEMwbTRJRmVxMC83?=
 =?utf-8?B?NS9jM01hTCs0SElPNTlzb3VCaVNvaUNna2E2UUtZUDNsNXQxUkZ6cEkzbW9F?=
 =?utf-8?B?Y1hQMVdXSDZMQ0VPbEFIRXhJbC82cmx6VkVnSHFsVVdvaUhpVEUxQXZ5cldz?=
 =?utf-8?B?KzJsbFRubTY0a2dscDZmV1B1WTdTU3k1RmlaNmE5VHdGZTVRR0pXeE9CMUVZ?=
 =?utf-8?B?VG5RNDcrb2IzNW14L2VkY09nY2dRaEZ2eEpDWVVhaDJEL3pESGdnVFZZdXRn?=
 =?utf-8?B?NUZyWGtydGVLV3B1NkluZUJsSmtrOWwrVG8rdkZpNU82alAzZElyVFJseEdS?=
 =?utf-8?B?RStuaWo2Q2Y5ZlJZeDlzdVBUYkc0UmxEMlQ2dUZYSnBtTFBZbEVtR2tnN1dQ?=
 =?utf-8?B?NVg1VlFHSXAyRllLODBPaW1QQUxidGJpUTFtenRtaGJGSXNHcDJxMVZKR0la?=
 =?utf-8?B?bDhDZVkwRmxtR0cwaDZPd1hUTkxWeGpqbDVudTUvOHZCR1hXZGhLZjZQVGRs?=
 =?utf-8?B?Y01jZjc0RlpXWGlBVFFieUNzK0RxMHZtOUdjRDFkQUtnR1hUOXFJVDEwSHAy?=
 =?utf-8?B?MlE9PQ==?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 928330f8-5e6b-4167-7fea-08dc275ccece
X-MS-Exchange-CrossTenant-AuthSource: SJ2PR11MB7573.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 06 Feb 2024 21:44:34.1385
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: pv74I5tfLyNM6OTIEtK7H2AZ/gWKiFMexPh2+RhIZJjmy9tJwdQ+Qy4uWGpSEb53UD9eEGaTgrnFqhfV2dMA456xkD4dw2eADbPhVnA51qA=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: IA1PR11MB7367
X-OriginatorOrg: intel.com

Hi Alex,

On 2/5/2024 2:34 PM, Alex Williamson wrote:
> On Thu,  1 Feb 2024 20:56:57 -0800
> Reinette Chatre <reinette.chatre@intel.com> wrote:
> 
>> vfio_pci_set_irqs_ioctl() is the entrypoint for interrupt management
>> via the VFIO_DEVICE_SET_IRQS ioctl(). The igate mutex is obtained
>> before calling vfio_pci_set_irqs_ioctl() for management of all interrupt
>> types to protect against concurrent changes to the eventfds associated
>> with device request notification and error interrupts.
>>
>> The igate mutex is not acquired consistently. The mutex is always
>> (for all interrupt types) acquired from within vfio_pci_ioctl_set_irqs()
>> before calling vfio_pci_set_irqs_ioctl(), but vfio_pci_set_irqs_ioctl() is
>> called via vfio_pci_core_disable() without the mutex held. The latter
>> is expected to be correct if the code flow can be guaranteed that
>> the provided interrupt type is not a device request notification or error
>> interrupt.
> 
> The latter is correct because it's always a physical interrupt type
> (INTx/MSI/MSIX), vdev->irq_type dictates this, and the interrupt code
> prevents the handler from being called after the interrupt is disabled.

Thank you for confirming. 

> It's intentional that we don't acquire igate here since we only need to
> prevent a race with concurrent user access, which cannot occur in the
> fd release path.  The igate mutex is acquired consistently, where it's
> required. 

Thank you. I do think it will be helpful to document some of this
in the code to help newcomers distinguish the scenarios (more below).

> It would be more forthcoming to describe that potential future emulated
> device interrupts don't make the same guarantees, but if that's true,
> why can't they?

As I understand an emulated interrupt will be triggered by VFIO PCI driver
as a result from, for example, a mmio write from user space. I thus expect
similar locking to existing device request notification and error interrupts.
I would like to focus this series on existing flows though.

>> Move igate mutex acquire and release into vfio_pci_set_irqs_ioctl()
>> to make the locking consistent irrespective of interrupt type.
>> This is one step closer to contain the interrupt management locking
>> internals within the interrupt management code so that the VFIO PCI
>> core can trigger management of the eventfds associated with device
>> request notification and error interrupts without needing to access
>> and manipulate VFIO interrupt management locks and data.
> 
> If all we want to do is move the mutex into vfio_pci_intr.c then we
> could rename to __vfio_pci_set_irqs_ioctl() and create a wrapper around
> it that grabs the mutex.  The disable path could use the lockless
> version and we wouldn't need to clutter the exit path unlocking the
> mutex as done below.  Thanks,

Will do. This creates an opportunity to document the flows involving
the mutex (essentially adding comments that includes your description
above).

Reinette