Message-ID: <96bb2b52-27b2-655e-2a2f-157efb98826c@nvidia.com>
Date:   Thu, 18 Nov 2021 20:54:17 +0530
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:91.0) Gecko/20100101
 Thunderbird/91.3.0
Subject: Re: [RFC 2/3] vfio/pci: virtualize PME related registers bits and
 initialize to zero
Content-Language: en-US
To:     Alex Williamson <alex.williamson@redhat.com>
Cc:     kvm@vger.kernel.org, Cornelia Huck <cohuck@redhat.com>,
        Max Gurtovoy <mgurtovoy@nvidia.com>,
        Yishai Hadas <yishaih@nvidia.com>,
        Zhen Lei <thunder.leizhen@huawei.com>,
        Jason Gunthorpe <jgg@nvidia.com>, linux-kernel@vger.kernel.org
References: <20211115133640.2231-1-abhsahu@nvidia.com>
 <20211115133640.2231-3-abhsahu@nvidia.com>
 <20211117105304.5f9f9d72.alex.williamson@redhat.com>
From:   Abhishek Sahu <abhsahu@nvidia.com>
In-Reply-To: <20211117105304.5f9f9d72.alex.williamson@redhat.com>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?ekR3Y1JsczIrcHd1cnNtNWtEK2x2RjFXcDJ1MmdIek4wRTVFdXh5dDZhaGR4?=
 =?utf-8?B?NGcySlFtWXpYdVEzRWUwQkZvNWE5bWpBSnNwK2h0cXVMcFlQVllhYWxOVUI5?=
 =?utf-8?B?NW14SkpvbEl4MFprMkFnSGJwRUpYZm5keXBXSmF1SThDV1RRYzBEZGtWQmdw?=
 =?utf-8?B?NzRQNlo5UnNoRkFnNzVudmdmNjBuUHNpMXgrOTRCaHppQWFVQjkvS29vTElN?=
 =?utf-8?B?amt1V2pqTFRMc1orYzR1QnZYUmVjbTl5M0R1QTRFQW1mbWhlMlhnSVB6WFVZ?=
 =?utf-8?B?UWFiMmIvT0lTVGhNNi9pbnVZVHlBOEtGeVdZelBPZXRMOTc5MmV0RSs1T24w?=
 =?utf-8?B?SjlJcTJZVmNNTHA2cHVRN28vK3d4R0RHb3VXTFduM1V1bG54VnV4Z2Qxd091?=
 =?utf-8?B?RGgyNDRGR09XMVcwSnZick9LZmRvKzdiZjlwQ0xQM1Q3bEM4eDZ5MGQ0N2lS?=
 =?utf-8?B?N3BkMXlrM2JXdjV2ZFU3OU9lUDNhNFBlMDY4ZzBvOC9ORnYrL2RKdTlzdExS?=
 =?utf-8?B?V201cTQ4UmZLZmp6dVdqQUUrNUNobVhkaUhuUWJYaDBYR3hEUk11OENYYkF2?=
 =?utf-8?B?dXRqL0cycDZJQjV0cWZCcDQvOG4wRG1nSnRwM3ZxbVNJMy9YWC9kWnIzR3RO?=
 =?utf-8?B?MGRzVW94NC96dENwLzlJVlZUT0pIekk0aThsYmU3YUlVNUZta3BjemVGN245?=
 =?utf-8?B?eEd1K3pFcFRSZGtIQ202eDMyRXp3OGhVS1ZvSkVjSkNOc2M0RjRCRGt3QnM2?=
 =?utf-8?B?ODVtVHJtdTZVSXNMYzdhdlhtNnd2c2Vra2w2eEZmQUZrTFhDc0d3YmtxWlBW?=
 =?utf-8?B?eU5PWTkyNXZJSDhxMUkxRWJRSk9za1RYemNUY2J2TGtsT3JoVXpnMDZ3WVBM?=
 =?utf-8?B?RUNTTFJCbmFXYmUyL0d1RTVZV0NieFV5VFRhbGtWbytOenV1T1p3dG15elVx?=
 =?utf-8?B?K1hINERyNnQzR1lzYTh5UmNFbTVuazlpT2llUkZtcXhrT3p4dkdXOHNOVWgr?=
 =?utf-8?B?cWhsdFIyejJseDNadmt3L24xVTR3b0dYQytBWm51L0M2OXF1OGcvRHVhZ1lS?=
 =?utf-8?B?c3JOcGJHL1NpbHVMV0lRbzU0Z2RrblVvN1A2bndIWnRHZmhoTE1tLzBIY3dz?=
 =?utf-8?B?SXdxc2VHR1AwaDNZenhCYlVwMmdBdkR4dCtoVWYxUi9TZEVxbWg4VTI4d2Fz?=
 =?utf-8?B?ZkRHVFZiMjdXTFZmeHprWkpzZENuUlJMeDgyZ0ZzbkkwZ3RsT0NVUjUzVkNz?=
 =?utf-8?B?UThqWUY1dFFaeDU2SUpPKzNwTWlUQmJSMHlBclp5NHcvMjhPR3AyYWVEVm51?=
 =?utf-8?B?ajFQVTk4V0JKekhzTUtFUWtQU2JFcnBLdU5BMEZzNWhQeHBFenRKeXZFYzVB?=
 =?utf-8?B?QzhkTGFickF2dk91ei9mbHM4Sklab04xOWlsUUFXOHgzWVkwaFMvOXNYaU1X?=
 =?utf-8?B?SzVEbk9kZEFsRVgwcTJrODEwQ2FsNU1LK25NR1JmdHgraDJLUWVwbEYxU2d4?=
 =?utf-8?B?VlljYlRQWlc2RW1rbUpPbE4xYTh0d0xtYXczNmN3azZ0eUVpcmlseS9DajVk?=
 =?utf-8?B?a3hSNm1Tc3FLL1pUTjBlTjUxZUtPL2tWNlAwYjVCbVVaSFcrYzlicitrMElu?=
 =?utf-8?B?aVVqMlc2VXhqaUd1dTdqVU9BZXp6TFNJekF5YzFyS24xaDlTUlljRTdjS3pS?=
 =?utf-8?B?Ni9NUVpkVU42SnYvdDhrUE5tZWJqM3Y5SCtPSWlORXhDSnRNNDVCWEk0Y3Ex?=
 =?utf-8?B?K0ZFVi84elF1VEJUM0FWUmZ2M08zaGtCQXpVZGZXV1lBcThBZDd0c0FxZzN3?=
 =?utf-8?B?WjBDN0xuSW0yRmIvS0tib3Y0cTJ4aXFTaW5LdUZMZjVGQ2gyK1FaVjQ5ZFpE?=
 =?utf-8?B?UlhXSEc3ZUZTWW4vQzUrSUplMlRUT0JLOUtDSGhkVUlpNkJnTisxSjdjWmxv?=
 =?utf-8?B?TFR3MHpwREFndkYvWXJrODZSSVQrTFBwRWxSQ0R4WmV3aHBRTGlvdm16QUkv?=
 =?utf-8?B?NlEyWS8yZFNZYVV3cmVQRDlZZHpRL051QW5TM2Q5Rm5COG1sTGZHTHBUazJZ?=
 =?utf-8?B?UURiTDYyeVZ0d1BrREV0UGM3dytUbVBSWERTOHJpZThXUnZFZVI2MmR4eEVn?=
 =?utf-8?B?Wm5ZdjRyblNDdVpjdFZsY1J6RVFtK1kwOFozc2t0amZIZlkycVI4OUZhRTNQ?=
 =?utf-8?Q?P1uVMtr58uEmpdRJvg8NNKw=3D?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 9a9448bc-1bcd-4bed-25ab-08d9aaa78529
X-MS-Exchange-CrossTenant-AuthSource: DM6PR12MB2937.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 18 Nov 2021 15:24:32.3740
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: cItxVlGjctEVMbzIrcQOriLbvBjMVq6NU0WjC/r7u1eDMVyJE2G43HHfDPxF8OKLDlWL4w4zJ0OM+Dlpi3HggQ==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM6PR12MB3305
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On 11/17/2021 11:23 PM, Alex Williamson wrote:
> On Mon, 15 Nov 2021 19:06:39 +0530
> <abhsahu@nvidia.com> wrote:
> 
>> From: Abhishek Sahu <abhsahu@nvidia.com>
>>
>> If any PME event will be generated by PCI, then it will be mostly
>> handled in the host by the root port PME code. For example, in the case
>> of PCIe, the PME event will be sent to the root port and then the PME
>> interrupt will be generated. This will be handled in
>> drivers/pci/pcie/pme.c at the host side. Inside this, the
>> pci_check_pme_status() will be called where PME_Status and PME_En bits
>> will be cleared. So, the guest OS which is using vfio-pci device will
>> not come to know about this PME event.
>>
>> To handle these PME events inside guests, we need some framework so
>> that if any PME events will happen, then it needs to be forwarded to
>> virtual machine monitor. We can virtualize PME related registers bits
>> and initialize these bits to zero so vfio-pci device user will assume
>> that it is not capable of asserting the PME# signal from any power state.
>>
>> Signed-off-by: Abhishek Sahu <abhsahu@nvidia.com>
>> ---
>>  drivers/vfio/pci/vfio_pci_config.c | 32 +++++++++++++++++++++++++++++-
>>  1 file changed, 31 insertions(+), 1 deletion(-)
>>
>> diff --git a/drivers/vfio/pci/vfio_pci_config.c b/drivers/vfio/pci/vfio_pci_config.c
>> index 6e58b4bf7a60..fb3a503a5b99 100644
>> --- a/drivers/vfio/pci/vfio_pci_config.c
>> +++ b/drivers/vfio/pci/vfio_pci_config.c
>> @@ -738,12 +738,27 @@ static int __init init_pci_cap_pm_perm(struct perm_bits *perm)
>>        */
>>       p_setb(perm, PCI_CAP_LIST_NEXT, (u8)ALL_VIRT, NO_WRITE);
>>
>> +     /*
>> +      * The guests can't process PME events. If any PME event will be
>> +      * generated, then it will be mostly handled in the host and the
>> +      * host will clear the PME_STATUS. So virtualize PME_Support bits.
>> +      * It will be initialized to zero later on.
>> +      */
>> +     p_setw(perm, PCI_PM_PMC, PCI_PM_CAP_PME_MASK, NO_WRITE);
>> +
>>       /*
>>        * Power management is defined *per function*, so we can let
>>        * the user change power state, but we trap and initiate the
>>        * change ourselves, so the state bits are read-only.
>> +      *
>> +      * The guest can't process PME from D3cold so virtualize PME_Status
>> +      * and PME_En bits. It will be initialized to zero later on.
>>        */
>> -     p_setd(perm, PCI_PM_CTRL, NO_VIRT, ~PCI_PM_CTRL_STATE_MASK);
>> +     p_setd(perm, PCI_PM_CTRL,
>> +            PCI_PM_CTRL_PME_ENABLE | PCI_PM_CTRL_PME_STATUS,
>> +            ~(PCI_PM_CTRL_PME_ENABLE | PCI_PM_CTRL_PME_STATUS |
>> +              PCI_PM_CTRL_STATE_MASK));
>> +
>>       return 0;
>>  }
>>
>> @@ -1412,6 +1427,18 @@ static int vfio_ext_cap_len(struct vfio_pci_core_device *vdev, u16 ecap, u16 epo
>>       return 0;
>>  }
>>
>> +static void vfio_update_pm_vconfig_bytes(struct vfio_pci_core_device *vdev,
>> +                                      int offset)
>> +{
>> +      /* initialize virtualized PME_Support bits to zero */
>> +     *(__le16 *)&vdev->vconfig[offset + PCI_PM_PMC] &=
>> +             ~cpu_to_le16(PCI_PM_CAP_PME_MASK);
>> +
>> +      /* initialize virtualized PME_Status and PME_En bits to zero */
> 
>         ^ Extra space here and above.
> 
> 
>> +     *(__le16 *)&vdev->vconfig[offset + PCI_PM_CTRL] &=
>> +             ~cpu_to_le16(PCI_PM_CTRL_PME_ENABLE | PCI_PM_CTRL_PME_STATUS);
> 
> Perhaps more readable and consistent with elsewhere as:
> 
>         __le16 *pmc = (__le16 *)&vdev->vconfig[offset + PCI_PM_PMC];
>         __le16 *ctrl = (__le16 *)&vdev->vconfig[offset + PCI_PM_CTRL];
> 
>         /* Clear vconfig PME_Support, PME_Status, and PME_En bits */
>         *pmc &= ~cpu_to_le16(PCI_PM_CAP_PME_MASK);
>         *ctrl &= ~cpu_to_le16(PCI_PM_CTRL_PME_ENABLE | PCI_PM_CTRL_PME_STATUS);
> 
> Thanks,
> Alex
> 

 Thanks Alex. I will fix this.

 Regards,
 Abhishek

>> +}
>> +
>>  static int vfio_fill_vconfig_bytes(struct vfio_pci_core_device *vdev,
>>                                  int offset, int size)
>>  {
>> @@ -1535,6 +1562,9 @@ static int vfio_cap_init(struct vfio_pci_core_device *vdev)
>>               if (ret)
>>                       return ret;
>>
>> +             if (cap == PCI_CAP_ID_PM)
>> +                     vfio_update_pm_vconfig_bytes(vdev, pos);
>> +
>>               prev = &vdev->vconfig[pos + PCI_CAP_LIST_NEXT];
>>               pos = next;
>>               caps++;
>