Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <64b99d1c-073f-cbc3-6c5a-100fa23bcb13@intel.com>
Date:   Wed, 19 Apr 2023 11:11:48 -0700
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:102.0) Gecko/20100101
 Firefox/102.0 Thunderbird/102.10.0
Subject: Re: [PATCH V3 08/10] vfio/pci: Probe and store ability to support
 dynamic MSI-X
Content-Language: en-US
To:     Alex Williamson <alex.williamson@redhat.com>
CC:     <jgg@nvidia.com>, <yishaih@nvidia.com>,
        <shameerali.kolothum.thodi@huawei.com>, <kevin.tian@intel.com>,
        <tglx@linutronix.de>, <darwi@linutronix.de>, <kvm@vger.kernel.org>,
        <dave.jiang@intel.com>, <jing2.liu@intel.com>,
        <ashok.raj@intel.com>, <fenghua.yu@intel.com>,
        <tom.zanussi@linux.intel.com>, <linux-kernel@vger.kernel.org>
References: <cover.1681837892.git.reinette.chatre@intel.com>
 <0da4830176e9c4a7877aac0611869f341dda831c.1681837892.git.reinette.chatre@intel.com>
 <20230418163803.46a96fdc.alex.williamson@redhat.com>
From:   Reinette Chatre <reinette.chatre@intel.com>
In-Reply-To: <20230418163803.46a96fdc.alex.williamson@redhat.com>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?ZWRiUDJjQVNWRE0yZjNIUkxzSmRlamhTSG9STk9WeGtINGZxMnhJWUxLS0lK?=
 =?utf-8?B?MWl4cjR4SnNTMkZXNGc1cFcxcTRLZStTUkZ3QWlGRHJsem9lQ2JBT1ZNSHhr?=
 =?utf-8?B?YjYwMHlUNVg0UzBxTk9hc3pUenV1aXhOaTFwYkJLTGpleWkzelZYRlhNZ09D?=
 =?utf-8?B?S0RBenB5UnhJVjQzMXFTR0dqSG1xS0hBYUgyMmRZWnJiQWxuY2Z6WlNyT1hU?=
 =?utf-8?B?WDBpaTZoSTJxQ2VKSWpnZkpMTUhqdU9udUxIWjhQS2ozdXozNHFhWm5HZy9U?=
 =?utf-8?B?MGkzR0FTWWFDZnZGWmZXY1ZoV2RNd1haVUVMdkZuZ3pRYXBueDdWaXZCdFAr?=
 =?utf-8?B?YTVyZ1R1SlNFeXZoTHJOK1pGb1R0MjFNUXAzL0hyNUtsTVVwdVZidENoaFJ5?=
 =?utf-8?B?TTZuWkFpaVk5dUhwWTdhYU9DZm5iSU1lN0I1VE5XU2VReHpQbXRwY0dUSDl4?=
 =?utf-8?B?VnExRjhGVlVwUWN5V3BWdGpta2JRMlZqVTdxMkVTKzh1Y2MvN2J4a0g5UEZw?=
 =?utf-8?B?RzVVQ3pFNHBlOFRRTzBBcDhIamhya0s2cTR1Z0lGc2lnNnRiWFdHNWlaZ3Jy?=
 =?utf-8?B?OGNoTVI1bStINkRUMS9uclNYZ0Vub2dFYkhweFF2L3luYzFaQXlRV1czTkY5?=
 =?utf-8?B?OHExN1djUkFhZUpMdnNrNzczRkZiY3V4a2dKNVhkeHVEdlJnenk2QnM5NFN2?=
 =?utf-8?B?RGpKU2lOQVBpMFl3bHpWVVNST3pTRG5rYm1ZSFBYZm9xbXJiTHFULzNSeStQ?=
 =?utf-8?B?c2JtR3I0ZVcrT2JUOFFDeUNXOEVKT3R4NURtR3grd2ZzSHlBTmFwdklCR0s0?=
 =?utf-8?B?SVhwMEpiOTV6YndBV3B6YVpab2laVWt5dzJ3cUZPN21zaFlsVU0wRXZ2cnd2?=
 =?utf-8?B?SlVCZGt6emRGOStsV3pkQTBRcW84ZkFCb25JZUoyaGF6WkVoRXllRXlyOHo1?=
 =?utf-8?B?Qi9aUDFWUy9GSEVCOThacFVhb0xycmxMVjVlU2Y1OUdkOXlvWTQ2VXU1VmI0?=
 =?utf-8?B?dG5sOHd1OUk0ck15bFJsU1pBcWRWOWx3Q0xzb1FLZW9yUGxzVHdoeDJBNFUx?=
 =?utf-8?B?T3ZWNlExazlPdm9lMFliVFE3Q0NWTWdlenpqQWFHQ281a1dud05LbDBwR2lB?=
 =?utf-8?B?c2VwUk5rbEsxNVVIWktBVVVsYTJPREJOdGRpMTBTcXhIV2lISHl2dW1DL3VT?=
 =?utf-8?B?RlkvQWFpaTZCS3hXSGZEV3cvbUJLVnh1VkRTbWVaL1pVTGh5OWR1Tnljd1Iw?=
 =?utf-8?B?TmZTTXJLRkFUTVlmdEpVQjlyNS85K2tpek50VjhpQmVuZVlPUUFuZlFDMTBL?=
 =?utf-8?B?SWpKNFIvc2FDZHVZWTJibDdhL3pxbU9qWjJXdGYvczdGcGQ0c2xqdXpMU0xF?=
 =?utf-8?B?b25hc1RsdlJiME5wNEZ2VUJIZFBPSkdCYnZPMUt6OXprMmNNZ04wZFY3RENu?=
 =?utf-8?B?dll3dzlOLzkxK1l4b3E0SVVPUjBzSG5YMUFGY3NvWlhzWUkxWmtHbHN3aXBW?=
 =?utf-8?B?c08xNHl4UmtENEdKV3hkV2lrcUd2dXQ4Q21JNU9RN2c3US96V2t1bkdGdzdW?=
 =?utf-8?B?U3JoUWV3aEZFN3NFbXhkK21YQzFxRHc3SWdXUThpQkkzRnNzRlZBVWZCUG5o?=
 =?utf-8?B?NGQxTzRpRDJGc1ZrVDZ5TVVRTVV5NFBKbkIvOFJxY3kraGQzNWxldlVJNzZD?=
 =?utf-8?B?aDFweE8rL2J3MXNYSzVQMlFYeUN4UGoyZUxrQlkyNVZaSWNDVFZqcUY2KzZ6?=
 =?utf-8?B?VzIySU9SbkJ3b1FJQkNFdzg1ekZWU3BwWm9YT3hqWEd4Ris2Zm1scVJJeVVP?=
 =?utf-8?B?eWVFbVFuamhwWUlUQjVzL3B3aG53VVJLUXNUVjRydGYwMytzdTNCelNqZHl3?=
 =?utf-8?B?Z1l6MkRudTA2WStHOGp3VXprKzhHeXBPK3hJNE5MWTIxYlNlWENkQ2N4L3Av?=
 =?utf-8?B?QnNZL0RUbjZ1bWl2aXhGaDRyTzZjS0thYnNSNGwyd2RsNVI1bER1bnIwMFVP?=
 =?utf-8?B?M0pQRFUxcUF3a21MU2lub0gvVlVycnc0OTBpSGs1aExSZEd3czFhemFnMmJR?=
 =?utf-8?B?ZnU4eDVTOUphU21OeDBaOEFFU2JWSUxPWmdObWttV2xwUmVMYisyOGpYd2lv?=
 =?utf-8?B?TnRhQXVJa2pNM3VXVlBtRnc1MDd4S3NCNzAzcUFDblZUanBKa0czMS9GSnFE?=
 =?utf-8?B?eEE9PQ==?=
X-MS-Exchange-CrossTenant-Network-Message-Id: fda95faf-8653-4560-a981-08db41018ca3
X-MS-Exchange-CrossTenant-AuthSource: CY4PR11MB1862.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 19 Apr 2023 18:11:51.6627
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: ZIADQ8na3ulHjE/AZj9QjCcdqm/Fo/cqhArn+JdDRiXMHxbD/Q1uk0AG857UNYzUq4wZ7jnkjiXM9RkMsVhoi/T11h6DSaFEFRfMUAvgt6A=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA1PR11MB7109
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-7.0 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,NICE_REPLY_A,
        RCVD_IN_DNSWL_MED,SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE,
        URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

Hi Alex,

On 4/18/2023 3:38 PM, Alex Williamson wrote:
> On Tue, 18 Apr 2023 10:29:19 -0700
> Reinette Chatre <reinette.chatre@intel.com> wrote:
> 

...

>> diff --git a/include/linux/vfio_pci_core.h b/include/linux/vfio_pci_core.h
>> index 148fd1ae6c1c..4f070f2d6fde 100644
>> --- a/include/linux/vfio_pci_core.h
>> +++ b/include/linux/vfio_pci_core.h
>> @@ -67,6 +67,7 @@ struct vfio_pci_core_device {
>>  	u8			msix_bar;
>>  	u16			msix_size;
>>  	u32			msix_offset;
>> +	bool			has_dyn_msix;
>>  	u32			rbar[7];
>>  	bool			pci_2_3;
>>  	bool			virq_disabled;
> 
> Nit, the whole data structure probably needs to be sorted with pahole,
> but creating a hole here for locality to other msix fields should
> probably be secondary to keeping the structure well packed, which
> suggests including this new field among the bools below.  Thanks,

Thanks for catching this. Moving it one field lower as shown in the
delta patch below seems to improve the layout:

diff --git a/include/linux/vfio_pci_core.h b/include/linux/vfio_pci_core.h
index 4f070f2d6fde..d730d78754a2 100644
--- a/include/linux/vfio_pci_core.h
+++ b/include/linux/vfio_pci_core.h
@@ -67,8 +67,8 @@ struct vfio_pci_core_device {
 	u8			msix_bar;
 	u16			msix_size;
 	u32			msix_offset;
-	bool			has_dyn_msix;
 	u32			rbar[7];
+	bool			has_dyn_msix;
 	bool			pci_2_3;
 	bool			virq_disabled;
 	bool			reset_works;


Combined with the other changes to this struct (new struct xarray
for the context, and removing int num_ctx) the bools are no longer
together on a single cache line. Placing has_dyn_msix as shown above
keeps it on the same cache line as the other msix_* fields.

After this change the layout of this struct appears to be improved.
Before this patch series (v6.3-rc7):
        /* size: 2496, cachelines: 39, members: 46 */
        /* sum members: 2485, holes: 4, sum holes: 11 */
        /* paddings: 2, sum paddings: 11 */
        /* forced alignments: 1 */

After this patch series (v6.3-rc7 + V3 + delta patch):
        /* size: 2568, cachelines: 41, members: 46 */
        /* sum members: 2562, holes: 2, sum holes: 6 */
        /* paddings: 2, sum paddings: 11 */
        /* forced alignments: 1 */
        /* last cacheline: 8 bytes */

Reinette