Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:3 as permitted sender) client-ip=2620:137:e000::3:3;
Message-ID: <4cc6e8ce-7e51-46c9-8587-0d37f0f39dfa@intel.com>
Date:   Mon, 16 Oct 2023 10:48:57 -0700
User-Agent: Mozilla Thunderbird
Subject: Re: [RFC PATCH V2 14/18] vfio/pci: Add core IMS support
To:     "Tian, Kevin" <kevin.tian@intel.com>,
        "jgg@nvidia.com" <jgg@nvidia.com>,
        "yishaih@nvidia.com" <yishaih@nvidia.com>,
        "shameerali.kolothum.thodi@huawei.com" 
        <shameerali.kolothum.thodi@huawei.com>,
        "alex.williamson@redhat.com" <alex.williamson@redhat.com>
CC:     "kvm@vger.kernel.org" <kvm@vger.kernel.org>,
        "Jiang, Dave" <dave.jiang@intel.com>,
        "Liu, Jing2" <jing2.liu@intel.com>,
        "Raj, Ashok" <ashok.raj@intel.com>,
        "Yu, Fenghua" <fenghua.yu@intel.com>,
        "tom.zanussi@linux.intel.com" <tom.zanussi@linux.intel.com>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        "patches@lists.linux.dev" <patches@lists.linux.dev>
References: <cover.1696609476.git.reinette.chatre@intel.com>
 <0f8fb7122814c5c5083799e935d5bd3d27d38aef.1696609476.git.reinette.chatre@intel.com>
 <BN9PR11MB5276891129DCF849D9E6375D8CD2A@BN9PR11MB5276.namprd11.prod.outlook.com>
Content-Language: en-US
From:   Reinette Chatre <reinette.chatre@intel.com>
In-Reply-To: <BN9PR11MB5276891129DCF849D9E6375D8CD2A@BN9PR11MB5276.namprd11.prod.outlook.com>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?WlBQQnNUY3N5czZjWVpiZFZVS2dkT3BMd3piUHNtaVJmTzNUalhhRWlOdEFR?=
 =?utf-8?B?RFhYRDk0UGUyTVNzZHVDbzZhRTU2UXhNMjRqcEl1TUw0elR3NjlEQ3I2eEhI?=
 =?utf-8?B?YWVsK3ZEd0crSi9IOVZKTHFYZ1dUNENPVlBNNi92K1NwNlF5TTZrcUpmakVK?=
 =?utf-8?B?YkFLZnR0UzczaFlvY0laLzRBV1VlM2FCV2c3K0g0eE5NbjJHclJoWDNvamds?=
 =?utf-8?B?ZjYvOEJ1bzZUaUQ4cEtTc2I1RzUzQXFiSXByTWMxSm8xUVMyaDZXblU2dElV?=
 =?utf-8?B?b0JVY28yWUMyQU4wVHNLallMbFV6K2dKZ2p6RFJqOTlaWUpWR1JmSDZaMVRK?=
 =?utf-8?B?WCtSakhUSUVqL1pTbEt5VVBYbk85dEJYMUd0bklPUCtLZkJMOTNORk9IVUVk?=
 =?utf-8?B?aUZSWDF1eXdGOXQ1R0NBNzAyUjhPbVdVcTA0YUZGVmU0OW4wZktCZlhHbFZ4?=
 =?utf-8?B?elk0bUtJNUc1SUFxakRNSjE1T1ZNUnRQaWQ3eUN6b2dTL1Zza1NLOGxWTGJR?=
 =?utf-8?B?NnUzcDZRV2UwUW1PdE01by9ldmo3ZWh6M1hta1lLdWJtS25IR01MUG41TXo2?=
 =?utf-8?B?UnNIUi9LNitKbGdVUzBsRHdkSGFIZEU4c0tUWFdzK2ZYT285SnFnazRwTTZK?=
 =?utf-8?B?aWN1WmQrVzcwMzAxV2xDR1JSalBkeE5XN0YwMUF6YzFodUZ2WUVTYlhFZ2hl?=
 =?utf-8?B?azdaaWkyc2xSUG1CS0E5cGZlQU5rdmRKK3hQSklZWElhVFErdnh6eXZ5aEsv?=
 =?utf-8?B?QVFaZkFybTkvTGxXVVAyWExjaHdxbDFmMmxHb3ZUQkFmQXltMDg1WVRjdEVp?=
 =?utf-8?B?SEdiUFdNTmwya1RSSVQ0UzdYSmprWWFLM25Velk4TnhHQjltcVR5a0h0cEQv?=
 =?utf-8?B?UkQ0QlV6RWFldjIzTUxEVDR4OWRtbWh5MVNib3lxMVJWYllzWEswTjZ0Z2s4?=
 =?utf-8?B?NXlKVEEzOHZFaVphVEp5dkRuekZXQTlWMFMvTEdmWGQrTFJ3NnI3UkxyclBo?=
 =?utf-8?B?eExJai9WSGNlUnRlNWlENTZzRjJ0RWJkeFZwTFRNWXZtUEtEU3JoSlVRbFcw?=
 =?utf-8?B?emJDZ2xxMG1uamwxTVNrZkFEYzRlWVNWZmtxOUZqZC82TkxsT3Y5RUl1VitP?=
 =?utf-8?B?TnEyU3hvbDN3d2N2QTdwL0NYa0h5Y2VBbDZ0Sk5IWm96RjNXaE5ROVVwNHls?=
 =?utf-8?B?UG5BazRiN1d6bXROVVphTFM4aG9RZ2F5UWp0Uk0reTAwOUhWWlNyZ3ZNRkdH?=
 =?utf-8?B?M0JDalpPWWxrb1U3T2tpRE90dW9hcVdycUMzUG5NUEkzZHFSbTRYVGhHNWhY?=
 =?utf-8?B?ZUdDSFBTMGl5ZVlaN2NBZjZXdmFaay83ZEdIZnRnNEJSY21aclpBWmVKK1Jh?=
 =?utf-8?B?YXdyVG5rcHAzQUJsSmc2eU02bVB1a2JaQU1pdWVCdVMzQkR6S2JPVzFHM2FH?=
 =?utf-8?B?MG1TVDcyUkJNeXBValNJQnZMdzl1YVhnVTE5RWlEVmM5YzRpOHlKUFZlN0U1?=
 =?utf-8?B?ZWZXVS9WYUluNXlVQWpLUFlYOWpya0szOVFxdkNOMjRsMVpCV2cwRnU1TVpW?=
 =?utf-8?B?enJxZ3BMejVpelhaakF2VDZQRmR2bDB2TENTa3RxWm0zTitvbXltS3FKNXBv?=
 =?utf-8?B?MHhYRlFXeFdUY2V5ZUpHVC9Qenc4aHd6dHhEWEF1Vkt1YU5VcTJoM3QyR3NG?=
 =?utf-8?B?NnRNaU9oZDJGVUdvK2cvczgva2JBeEdSekNRZSsxRTFMS2pjODBsVW9aQ1E1?=
 =?utf-8?B?WUZCN2xldmRZN2dQdzcwSGhFcC85aUdHdjZrYlpwTFAyTmxxWko1bGI4V2VK?=
 =?utf-8?B?NVN1clhzaUFMNlp3a3N3VGxwRGNvclNKOGlWNFpLRUpGWmpZejNZN25zbTd0?=
 =?utf-8?B?eU9iSG1odkU0YkkzRkRWTEgxSmNwK1g3cVRSZ0RPbEFVVy9oYmJkSUwveTJ6?=
 =?utf-8?B?NDU0S0JYUjZPbkFybnV4b1lSalI5eXpqMVJIRnJjZTZBN1hkcm5xb245R3ps?=
 =?utf-8?B?TkRRa0g0amF5bzdaOW50WFhyOVhDR3NoamV5ODRkOXlnY1NrcmRGRHkwa1I1?=
 =?utf-8?B?aGN5aDlIemoyM3NIZWxNRnJnWHljdGJPOWNDVytJTU5oMnV2Qy94cHZoRFRZ?=
 =?utf-8?B?TUY0dTZ0TE9oMDV1RmdXU2tOa21pR1RRdW9EbmNER0lNTzlQVld2T0ZDVlFM?=
 =?utf-8?B?OXc9PQ==?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 5b93cd70-ca5d-4fad-8405-08dbce702dcf
X-MS-Exchange-CrossTenant-AuthSource: SJ2PR11MB7573.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 16 Oct 2023 17:49:00.8001
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: 3rtyIBPPN5pcjNjrl9i7L81m1JCTMLDyxbfetm3LfAzATDWYwInDEp55WFhvS6xfrqQZ8RaXk/2YatsgoYNXifVPtebwui/iKdC3RnKh9y8=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA1PR11MB5921
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED,
	DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,
	MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=unavailable
	autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lipwig.vger.email
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org
X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (lipwig.vger.email [0.0.0.0]); Mon, 16 Oct 2023 10:49:30 -0700 (PDT)

Hi Kevin,

On 10/13/2023 1:10 AM, Tian, Kevin wrote:
>> From: Chatre, Reinette <reinette.chatre@intel.com>
>> Sent: Saturday, October 7, 2023 12:41 AM
>>
>> A virtual device driver starts by initializing the backend
>> using new vfio_pci_ims_init_intr_ctx(), cleanup using new
>> vfio_pci_ims_release_intr_ctx(). Once initialized the virtual
>> device driver can call vfio_pci_set_irqs_ioctl() to handle the
>> VFIO_DEVICE_SET_IRQS ioctl() after it has validated the parameters
>> to be appropriate for the particular device.
> 
> I wonder whether the code sharing can go deeper from
> vfio_pci_set_irqs_ioctl() all the way down to set_vector_signal()
> with proper abstraction. 

There is a foundational difference in the MSI and IMS interrupt
management that is handled by the separate set_vector_signal()
implementations.

For MSI interrupts the interrupts stay allocated but the individual
interrupt context is always freed and re-allocated.

For IMS the interrupts are always freed and re-allocated (to ensure that
any new cookie is taken into account) while the individual interrupt
context stays allocated (to not lose the cookie value associated
with the individual interrupt).

It may indeed be possible to accommodate this difference with further
abstraction. I will study the code more to explore how this
can be done.

> Then handle emulated interrupt in the
> common code instead of ims specific path. intel gvt also uses
> emulated interrupt, which could be converted to use this library
> too.

Thank you for pointing me to intel gvt. 

> There is some subtle difference between pci/ims backends
> regarding to how set_vector_signal() is coded in this series. But
> it is not intuitive to me whether such a difference is conceptual
> or simply from a coding preference.
> 
> Would you mind doing an exercise whether that is achievable?

I do not mind at all. Will do.

Thank you very much for taking a look and sharing your guidance.

Reinette