Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <1f84c37d-585b-8113-b045-7d6076405e34@intel.com>
Date:   Mon, 8 May 2023 15:52:56 -0700
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:102.0) Gecko/20100101
 Firefox/102.0 Thunderbird/102.10.1
Subject: Re: [PATCH V4 03/11] vfio/pci: Prepare for dynamic interrupt context
 storage
To:     "Tian, Kevin" <kevin.tian@intel.com>,
        "jgg@nvidia.com" <jgg@nvidia.com>,
        "yishaih@nvidia.com" <yishaih@nvidia.com>,
        "shameerali.kolothum.thodi@huawei.com" 
        <shameerali.kolothum.thodi@huawei.com>,
        "alex.williamson@redhat.com" <alex.williamson@redhat.com>
CC:     "tglx@linutronix.de" <tglx@linutronix.de>,
        "darwi@linutronix.de" <darwi@linutronix.de>,
        "kvm@vger.kernel.org" <kvm@vger.kernel.org>,
        "Jiang, Dave" <dave.jiang@intel.com>,
        "Liu, Jing2" <jing2.liu@intel.com>,
        "Raj, Ashok" <ashok.raj@intel.com>,
        "Yu, Fenghua" <fenghua.yu@intel.com>,
        "tom.zanussi@linux.intel.com" <tom.zanussi@linux.intel.com>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>
References: <cover.1682615447.git.reinette.chatre@intel.com>
 <6fcd4019e22931a97d962b6e657e74d6fd1049ba.1682615447.git.reinette.chatre@intel.com>
 <BN9PR11MB5276242E8D9F4C082E6D2CB68C6B9@BN9PR11MB5276.namprd11.prod.outlook.com>
 <07a693e9-b926-6c25-ea55-8465410e214c@intel.com>
 <BN9PR11MB52764DFBE5C3FFA8BEC9E2AA8C729@BN9PR11MB5276.namprd11.prod.outlook.com>
Content-Language: en-US
From:   Reinette Chatre <reinette.chatre@intel.com>
In-Reply-To: <BN9PR11MB52764DFBE5C3FFA8BEC9E2AA8C729@BN9PR11MB5276.namprd11.prod.outlook.com>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?SXRQbloyaGViWFB4TnZkRDZnTjBzUTZqU3ViMnNhUS9jbU9tQm4zaHN4ZWZ0?=
 =?utf-8?B?L20xVkFvOTVTbzhwYy9lMElzZVViR0pQb1JNQVdPR0IvL2tMRVg2Q2NyVTBR?=
 =?utf-8?B?akVkb0xOR1JlS3k3TFZxam84STh4SDczL0pWSlY1STNzMDNkOTdzVU1ybm13?=
 =?utf-8?B?bVltc3RCVjRzbnZIYzFjNFZuZWNRY0hPb3dSdk1OeWhNVHlJQi9yQzgwTjdX?=
 =?utf-8?B?TytQeENNRlE1M2YxK1NoMTNyRXRYMHBOMGF2Y1F4WWI5ZldIWVNLZXVsVDlR?=
 =?utf-8?B?YTByekVPZmtDZlZ5QWdtU3Awc0NsaVhYYTFlTEFDT2xhUEo0VmZFOHFUNFVO?=
 =?utf-8?B?UFV3OGpUY3Y0SSsycEZjRjJ2QXZSVDBmSy9vMEk0WW45VDh2NHNyV0ZNMml4?=
 =?utf-8?B?YU04YnN1YlNzMm1pRDdsRWwvMk94ek90OFdjdjNrT2k3VWNKclJCcGw4SzhV?=
 =?utf-8?B?bEZoZmpPcC9lVkFzNlZDdDN3VERoUlBHWEtmUncyeThNcDR6bDh5OGNQK3Fn?=
 =?utf-8?B?ckJtQ3V0Mmw5Tzl5aDZiRVhpd0ladk8vZXpwaGRRZTdubEdOYUtKU3c1ZUow?=
 =?utf-8?B?ci94T01EVklQUXFnOVZaUk5WWEJBcEo1SkRpanpDZ01EVzBLQWdTemRTVlRw?=
 =?utf-8?B?TW0rQXY2RWFzZHEwTFBGNHJjUm5qZ1QwTUdZb2RMWjJCL05aU3drZURJOWNw?=
 =?utf-8?B?VTBuSG52bzBjTmpHV3EyU3JZSXpXUVVDNFhmeEExUlJXRW5kcDlvQTZQWElj?=
 =?utf-8?B?OVV6K1RnWVdkMWFlVWlxWW12cDFOVmFZQnpMVVdIenVldEdzcTI1UjZzL2Zm?=
 =?utf-8?B?dXdEYkwvMFhySWIydkUyTVg4QnVaaDljQnE3SWt0MUxpOXZrNWVpejJSVE4x?=
 =?utf-8?B?Mkg2MnFjV1pIK09ZMGozT1ZqT1FWaWt0NVFOM0laOXYrcnk0UFZOWXk3UXcv?=
 =?utf-8?B?M3FFNk94MEFOSmY3dVd6TU5uZ2xyZVdtc2M0aVBhUGFHZlM5WTBqWWQ4YUs4?=
 =?utf-8?B?dFNWZlBocDB2TktUbUgvMWNXOTNMSCtjY09VOWh0Ykg4RnRPbk05VlFpdkRB?=
 =?utf-8?B?alJiQkljQ0NzNDl1YWlPV1J2MHJtb01zdnZ1cEF3dGl3TWtGc2g1eUtZRDV6?=
 =?utf-8?B?ZWRBZHpOdXgyY1NRYmhwSWVlTTlMWWRLNW92bFdIOEZ6c0NFbXpDdzgzSmNS?=
 =?utf-8?B?Zi9UNVplVlhwTFdMTWg1NExjUHNUNWhXTDcyZ2tQNDhSZVlneGRzSVRtTEFN?=
 =?utf-8?B?enZaaWtmdWhGaUVlTlNyRWw0UlZlSXl0THR2Q3E2L1U3aTRqYjQyZXB3MkFj?=
 =?utf-8?B?MlM2S2lIQTJUUHg0eHlYeXZPdFBJbVRYbGxqUmxUQmV3K3RVSC9QaDMrVlRR?=
 =?utf-8?B?Z2NLai8yY1V5WmJhSTRRLzRVVXB1NHd0UjdSRm9BL3oxQWw0dDY0S1JFWWVM?=
 =?utf-8?B?VE9VRDRpanFHSU90MmxIYVBoVVIwYWJreUNzeUVRQ3dEN05Td2xGeCt0SVNq?=
 =?utf-8?B?aFVzeHcwQ3dLUnc0OU53KzVSeUUwZmxjenMyS0psYk94cFM0WG9NazB0NHM3?=
 =?utf-8?B?T0tIM3p5RmJFQi9uczVHeTFWVWJNSzRjTXFHWERxa0d3ZzBrK3NGU04rOE9u?=
 =?utf-8?B?TktQVUdtZFhBNEhLOHJxMkJHN04reGhGR2hFTEhjZUcrazVORytlMHNuNDlN?=
 =?utf-8?B?Ri9BRnJhSHRhdkJNZVlOWXVMVURyZnYrWTdaR2Evbmw0akxtazUxTHBpYTFM?=
 =?utf-8?B?MHMxZ21MQ2hOVkJlSDBQaG01d1ViOWVGUXp6V3N4a3JNamRiTURJdkYyTmZE?=
 =?utf-8?B?cnBJL1BxZDVkUjIyWXVRTHNwQStkWnd2M2I0cVpyREVRUUZwYWo5L2ZJSmc4?=
 =?utf-8?B?NWsySUtVZXhGVGwrL3VIcG9UeVJkVFBZTDIzOW40TVgvdU5ja2VWTjRJYVUz?=
 =?utf-8?B?UlU1VDRsV1FSM0JTOXdkSzg0bkZlelBDeXUyQkVvTVZqYk95TEFNbVhlQTEv?=
 =?utf-8?B?K3NWSlpIeTR0M3N6VkhjRzJCVTdtUnJ0TCtVM3JUSjRXbmxMaEF6ejk5R1NS?=
 =?utf-8?B?WHFQNlRGZTdadG85ZFp5MXRDRjhvejBGYmRRRHo5bnNuQXkvcEh4Mm5kUGhE?=
 =?utf-8?B?SjZBbFhOS2xyR0pMVmNJWHV4L1NQbWpJRWtVbEI0Ull6eENqU3lLaGJhUi83?=
 =?utf-8?B?YkE9PQ==?=
X-MS-Exchange-CrossTenant-Network-Message-Id: bbbfcc25-33c9-48f8-ae8a-08db5016f81a
X-MS-Exchange-CrossTenant-AuthSource: SJ2PR11MB7573.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 08 May 2023 22:52:58.7567
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: sjC92vdYZjM4NFAjLuwqidnw5tr72PRpAYPfKiPo70VPmt0+Tmm6XbZXt8h4NZCFjnlW97xwQ0r0diU1SmTDAlQnv8FpgqieEJD5eqMfiQA=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH7PR11MB6556
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-6.2 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,NICE_REPLY_A,
        RCVD_IN_DNSWL_MED,RCVD_IN_MSPIKE_H2,SPF_HELO_NONE,SPF_NONE,
        T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

Hi Kevin,

On 5/5/2023 12:21 AM, Tian, Kevin wrote:
>> From: Chatre, Reinette <reinette.chatre@intel.com>
>> Sent: Saturday, April 29, 2023 2:24 AM
>>
>> Hi Kevin,
>>
>> On 4/27/2023 11:33 PM, Tian, Kevin wrote:
>>>> From: Chatre, Reinette <reinette.chatre@intel.com>
>>>> Sent: Friday, April 28, 2023 1:36 AM
>>>>
>>>> @@ -55,17 +80,28 @@ static void vfio_send_intx_eventfd(void *opaque,
>>>> void *unused)
>>>>  {
>>>>  	struct vfio_pci_core_device *vdev = opaque;
>>>>
>>>> -	if (likely(is_intx(vdev) && !vdev->virq_disabled))
>>>> -		eventfd_signal(vdev->ctx[0].trigger, 1);
>>>> +	if (likely(is_intx(vdev) && !vdev->virq_disabled)) {
>>>> +		struct vfio_pci_irq_ctx *ctx;
>>>> +
>>>> +		ctx = vfio_irq_ctx_get(vdev, 0);
>>>> +		if (!ctx)
>>>> +			return;
>>>
>>> if this error happens it implies a kernel bug since the same check
>>> has been done in vfio_intx_enable(). Then should be a WARN_ON().
>>
>> Sure. Considering that if these are triggered it may result
>> in many instances, so perhaps WARN_ON_ONCE()?
> 
> yes.
> 
>>
>>> ditto for other intx functions which can be called only after intx
>>> is enabled.
>>
>> It seems the instances in this category can be identified as the places
>> where the array contents is currently used without any checks.
>>
>> I am planning on the following changes:
>>
> 
> that looks good to me

Thank you so much for this guidance. After adding these WARNs two of them
were actually encountered and revealed that the interrupt context was retrieved
too early in vfio_pci_intx_mask() and vfio_pci_intx_unmask_handler() in
the scenario where these calls are triggered via config writes while INTx is
disabled. This will be fixed in the next version.

Reinette