Message-ID: <44100bee-81a0-079a-7247-5c1580142bac@intel.com>
Date:   Fri, 17 Dec 2021 11:58:58 +0100
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101
 Firefox/91.0 Thunderbird/91.4.0
Subject: Re: [PATCH 1/2] PCI: Add support for VF Resizable Bar extended cap
Content-Language: en-US
To:     =?UTF-8?Q?Christian_K=c3=b6nig?= <christian.koenig@amd.com>,
        <linux-pci@vger.kernel.org>, <linux-kernel@vger.kernel.org>,
        Bjorn Helgaas <bhelgaas@google.com>
CC:     Ard Biesheuvel <ardb@kernel.org>,
        "Michael J . Ruhl" <michael.j.ruhl@intel.com>,
        Rodrigo Vivi <rodrigo.vivi@intel.com>
References: <20211215141626.3090807-1-michal.winiarski@intel.com>
 <20211215141626.3090807-2-michal.winiarski@intel.com>
 <7a09b5c5-9c7c-106a-65c6-8c25c4918960@amd.com>
From:   =?UTF-8?Q?Micha=c5=82_Winiarski?= <michal.winiarski@intel.com>
In-Reply-To: <7a09b5c5-9c7c-106a-65c6-8c25c4918960@amd.com>
Content-Type: text/plain; charset="UTF-8"; format=flowed
Content-Transfer-Encoding: 8bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?S3A0K0RYNHZNbUwrVXRiVWVKZjlZUjVxcDk3SHdGaUtOem1rUkF0MTl3QXk5?=
 =?utf-8?B?d0VhOTZNOVJlb1YzR2xFRkxCWG9UL1pyUGVMNklhRGU1RmhrTUhuYW1qRm1s?=
 =?utf-8?B?c2x5anBGVFFKdWYrV09HSkRxcWhRTkNIbmVZN3ZWaFRaTmZBSG9hM2FxQ2oy?=
 =?utf-8?B?aWxsT1pZZ2UwWEZTc25ILytaTnlPRGx3d3p6OStFKzA4ZGt5UnhSSUduNTVp?=
 =?utf-8?B?WDgvZmRKbWxOVDh0NjNJL1hwa0k4TjBsQjZlek1FbGgwdFkvSkRMeG5UV3l4?=
 =?utf-8?B?d2syZVlkZUpDTUNaU0IrN2ZVRis4M3pxVlYwak9ib2RBc3J0VERwNzZJZTVk?=
 =?utf-8?B?VlhMOTl1Y3hQbCtwYW1mUlowWTdKbE51OWdKd0EvWW9NVlZEaktSTmhZd2Fw?=
 =?utf-8?B?ODVVeVVQODRQOHhYT3RaYlh1Q3hZcnM3bEVpZHJJVGlPZHp4SSs1WEtURUhj?=
 =?utf-8?B?eUI3OHJPUk94WEJ3K3pRQVlUNXVjdUpvaWxLZU4rTk1aRjJaMkN5OVRXMXZG?=
 =?utf-8?B?TE5ick1aK3NFK0VIMVFyT2xJT0RoQSt3V1RIWWxBY1c4NzNjaVphc0hhdXJu?=
 =?utf-8?B?TitOY2VtM2N2aUhNMlpoUVlEZk1adUhDV0ZrMjM3WVArbVQ3Z3NsL3RYTlZi?=
 =?utf-8?B?ajdUYUJiWlowdzdZK1dzcFN6Z2JJaEhUc21lVHgvR1NSUllaRW5uRm5Nb3Ry?=
 =?utf-8?B?ZEpZUHEybCtTZ3Y4SFNGVmNraUlLTUN0V2tiTWQ5ME44WjQwRThLdUM5U09j?=
 =?utf-8?B?Y080ZnNPN0tBMmZQV21HeUR4MU9LQlkrZFVmVTBicCtwVGNkWDV4aUtINTJr?=
 =?utf-8?B?dmNyRmh4ckRxR0ZyRDdaVXNnV1RXaU1lTEJLSmxBNm9kLzJNcHQ2cFlWTHJB?=
 =?utf-8?B?SFJpbkdFMjBFSkVWWkpONFhrdDhMa3M4L2taU2dzWUgwOUQrcnkrN3dGd2Zq?=
 =?utf-8?B?eFI5MVFvd1JsWjNnYUFNZXRPWmlsM28zWHdBT21icFFOdnpqbTFIeUtseVlE?=
 =?utf-8?B?N29oQnFFSWpVUk1YczgwbGh3MDNoV0dnL0IvSk8yNmh1WnZBMm43ZU9qSWZ0?=
 =?utf-8?B?Y0VGYUQ5bGVzOEh4Z1pwZklqMk9BTVl5SFh2WkFja0lzSzYwRkp2amhNT3V5?=
 =?utf-8?B?STBibjZkMGpkOXNscTRYdm51SUdXTFM1NG1neHluRllVdm93U2xaL3hXMFFs?=
 =?utf-8?B?eHJvbW9zbzZ0d01udVU3V2lZNnU5VG5pRUlqSXJYUXk2MjBiNnpYSXB5cVF1?=
 =?utf-8?B?bm0wcC9HNGxTcUFUaTFzZ3VkeW9UWUpHbDJ1L1ZrdnJ5T1Nqajh3djZsUC96?=
 =?utf-8?B?cXNuWU1QZFdKWEx6SFovU3RiMkNkeTNyQjBqM1U2dWJBUjF2ZjJEbTJmK2tL?=
 =?utf-8?B?R3hkRHN5L0lvZEVTalM1bHpRbHp5RUp2alpQS1M2YnYvRDdKNkNtY3I0YVh4?=
 =?utf-8?B?U3J6RG9laUhBQ0FVNSt6bERYaE0ySjN1c3UxZXJZVGx1ZFV3OXpVeEZMdXcv?=
 =?utf-8?B?dXc0M0xTM2lralNMK1BGMlZXcGppcDNoRmtIaFcxOWVtRGxGRU92ZG1oMk9R?=
 =?utf-8?B?TGtKTGx6ZUNoSHFJbGVKYThGeThveUs1RmxtaXpXK2tuamlPTDRrNitFd3BI?=
 =?utf-8?B?K0RyVlBOUUZzeFZkeThBakdxVUZlR2hPTEEyVnZSYjcxY3NFK29KR2RWa0VU?=
 =?utf-8?B?RTRQeTlwRlBiV2hXNys3NGkrT1dzTzlvQnlYWG4vRTgybWFsY0duTTZoVS8z?=
 =?utf-8?B?bjQ4ODY3a1U4ZFhiTXh0SCs0b2JpY0c1TnR1b1R5VGpWdVhUU1RaYXFHZDdJ?=
 =?utf-8?B?MzNQMXBmL0hsSTk3VmZ6MXc0SEVsZTBsZzBuQzJjWkNqeVE0M3pNRW43dGxv?=
 =?utf-8?B?Y1YyY0hEcTQrUmdKdHdpVU5ZcHExK3I1M2x5TkJKWFVtUkl6d3pNbXdORWxm?=
 =?utf-8?B?bjJoWVk3djZkNmFwQmREU25GSytoWG03RlZKNW5oY0RQc2w1MEpWbVNMdzdO?=
 =?utf-8?B?S3V4d2w1Vm01QjJ1WUhmTEl3WTBNcy9wMFdDTzhUQUtEWUtmbGVkMnI4VzZQ?=
 =?utf-8?B?Z3pQMzJ4b1cxTElmZ1BnZlJtTUJVcEd0clB4dFZEdE5hSllOejZCeHRnbSt4?=
 =?utf-8?B?ZUxWMGFRVG5lQWZMRldETnFjTHpZdEsrTFJYeEx4M25ZSmV0TWsrQzA5bHlQ?=
 =?utf-8?B?NkJORWV4b2N5Ulkxb0xJOGx6ZGVNa2t1WWwvV0ZGVHVoemV3WTE5QWVKZjBJ?=
 =?utf-8?B?VUJ0aW9TVjRSSXRWeWdDRlFyV0RpOE5waWVZZGcvT0hjMTVpTTY3S055SHhC?=
 =?utf-8?Q?38+Mf1+8EqpfUW9Kpc?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 901258db-2d16-4222-fee8-08d9c14c3d9b
X-MS-Exchange-CrossTenant-AuthSource: DM4PR11MB5373.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 17 Dec 2021 10:59:04.6818
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: QwCdKEebc4zzii1SyTca3wosOMUPwyby7phIViND2jyge1W0HLTR5SeLrfMH7CcjifHLRy3c94oyutBnB/CQsh8g+8hoQ22Qz3a7CF43bB8=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM8PR11MB5688
X-OriginatorOrg: intel.com
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On 16.12.2021 08:50, Christian König wrote:
> 
> 
> Am 15.12.21 um 15:16 schrieb Michał Winiarski:
>> Similar to regular resizable BAR, VF BAR can also be resized.
>> The structures are very similar, which means we can reuse most of the
>> implementation. See PCIe r4.0, sec 9.3.7.4.
>>
>> Signed-off-by: Michał Winiarski <michal.winiarski@intel.com>
>> ---
>>   drivers/pci/pci.c             | 25 +++++++++++++++--
>>   drivers/pci/setup-res.c       | 53 ++++++++++++++++++++++++++++++++---
>>   include/uapi/linux/pci_regs.h |  1 +
>>   3 files changed, 72 insertions(+), 7 deletions(-)
>>
>> diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c
>> index 3d2fb394986a4..89448c5104e46 100644
>> --- a/drivers/pci/pci.c
>> +++ b/drivers/pci/pci.c
>> @@ -1682,12 +1682,15 @@ static void pci_restore_config_space(struct 
>> pci_dev *pdev)
>>       }
>>   }
>> -static void pci_restore_rebar_state(struct pci_dev *pdev)
>> +static void __pci_restore_rebar_state(struct pci_dev *pdev, int cap)
>>   {
>>       unsigned int pos, nbars, i;
>>       u32 ctrl;
>> -    pos = pci_find_ext_capability(pdev, PCI_EXT_CAP_ID_REBAR);
>> +    if (WARN_ON(cap != PCI_EXT_CAP_ID_REBAR && cap != 
>> PCI_EXT_CAP_ID_VF_REBAR))
>> +        return;
>> +
>> +    pos = pci_find_ext_capability(pdev, cap);
>>       if (!pos)
>>           return;
>> @@ -1709,6 +1712,14 @@ static void pci_restore_rebar_state(struct 
>> pci_dev *pdev)
>>       }
>>   }
>> +static void pci_restore_rebar_state(struct pci_dev *pdev)
>> +{
>> +    __pci_restore_rebar_state(pdev, PCI_EXT_CAP_ID_REBAR);
>> +#ifdef CONFIG_PCI_IOV
>> +    __pci_restore_rebar_state(pdev, PCI_EXT_CAP_ID_VF_REBAR);
>> +#endif
>> +}
>> +
> 
> It's probably cleaner to let the caller specify the capability to 
> restore directly.

Ok - I'll go with:
pci_restore_rebar_state(dev);
pci_restore_vf_rebar_state(dev);

in pci_restore_state() (where pci_restore_vf_rebar_state() will be 
"no-op" if there's no CONFIG_PCI_IOV).

> 
>>   /**
>>    * pci_restore_state - Restore the saved state of a PCI device
>>    * @dev: PCI device that we're dealing with
>> @@ -3639,10 +3650,18 @@ void pci_acs_init(struct pci_dev *dev)
>>    */
>>   static int pci_rebar_find_pos(struct pci_dev *pdev, int bar)
>>   {
>> +    int cap = PCI_EXT_CAP_ID_REBAR;
>>       unsigned int pos, nbars, i;
>>       u32 ctrl;
>> -    pos = pci_find_ext_capability(pdev, PCI_EXT_CAP_ID_REBAR);
>> +#ifdef CONFIG_PCI_IOV
>> +    if (bar >= PCI_IOV_RESOURCES) {
>> +        cap = PCI_EXT_CAP_ID_VF_REBAR;
>> +        bar -= PCI_IOV_RESOURCES;
>> +    }
>> +#endif
>> +
>> +    pos = pci_find_ext_capability(pdev, cap);
>>       if (!pos)
>>           return -ENOTSUPP;
>> diff --git a/drivers/pci/setup-res.c b/drivers/pci/setup-res.c
>> index 7f1acb3918d0c..1946e52e7678a 100644
>> --- a/drivers/pci/setup-res.c
>> +++ b/drivers/pci/setup-res.c
>> @@ -407,13 +407,36 @@ void pci_release_resource(struct pci_dev *dev, 
>> int resno)
>>   }
>>   EXPORT_SYMBOL(pci_release_resource);
>> +static int pci_memory_decoding(struct pci_dev *dev)
>> +{
>> +    u16 cmd;
>> +
>> +    pci_read_config_word(dev, PCI_COMMAND, &cmd);
>> +    if (cmd & PCI_COMMAND_MEMORY)
>> +        return -EBUSY;
>> +
>> +    return 0;
>> +}
>> +
>> +#ifdef CONFIG_PCI_IOV
>> +static int pci_vf_memory_decoding(struct pci_dev *dev)
>> +{
>> +    u16 cmd;
>> +
>> +    pci_read_config_word(dev, dev->sriov->pos + PCI_SRIOV_CTRL, &cmd);
>> +    if (cmd & PCI_SRIOV_CTRL_MSE)
>> +        return -EBUSY;
>> +
>> +    return 0;
>> +}
>> +#endif
>> +
>>   int pci_resize_resource(struct pci_dev *dev, int resno, int size)
>>   {
>>       struct resource *res = dev->resource + resno;
>>       struct pci_host_bridge *host;
>>       int old, ret;
>>       u32 sizes;
>> -    u16 cmd;
>>       /* Check if we must preserve the firmware's resource assignment */
>>       host = pci_find_host_bridge(dev->bus);
>> @@ -424,9 +447,14 @@ int pci_resize_resource(struct pci_dev *dev, int 
>> resno, int size)
>>       if (!(res->flags & IORESOURCE_UNSET))
>>           return -EBUSY;
>> -    pci_read_config_word(dev, PCI_COMMAND, &cmd);
>> -    if (cmd & PCI_COMMAND_MEMORY)
>> -        return -EBUSY;
>> +#ifdef CONFIG_PCI_IOV
>> +    if (resno >= PCI_IOV_RESOURCES)
>> +        ret = pci_vf_memory_decoding(dev);
>> +    else
>> +#endif
>> +    ret = pci_memory_decoding(dev);
>> +    if (ret)
>> +        return ret;
> 
> Way to many #ifdef spread around inside the code, please restructure that.
> 
> For example concentrating the logic in a single function should help:
> 
> static int pci_check_decoding_disabled(..., inr resno)
> {
> 
> #ifdef CONFIG_PCI_IOV
>      if (resno...
>          return -EBUSY;
>      else
>          return 0;
> #endif
> 
>      if (...)
>          return -EBUSY;
>      return 0;
> 
> }
> 

Ok.

>>       sizes = pci_rebar_get_possible_sizes(dev, resno);
>>       if (!sizes)
>> @@ -445,6 +473,14 @@ int pci_resize_resource(struct pci_dev *dev, int 
>> resno, int size)
>>       res->end = res->start + pci_rebar_size_to_bytes(size) - 1;
>> +#ifdef CONFIG_PCI_IOV
>> +    if (resno >= PCI_IOV_RESOURCES) {
>> +        dev->sriov->barsz[resno - PCI_IOV_RESOURCES] = 
>> pci_rebar_size_to_bytes(size);
>> +        res->end = res->start +
>> +            resource_size(res) * pci_sriov_get_totalvfs(dev) - 1;
>> +    }
>> +#endif
>> +
>>       /* Check if the new config works by trying to assign everything. */
>>       if (dev->bus->self) {
>>           ret = pci_reassign_bridge_resources(dev->bus->self, 
>> res->flags);
>> @@ -456,6 +492,15 @@ int pci_resize_resource(struct pci_dev *dev, int 
>> resno, int size)
>>   error_resize:
>>       pci_rebar_set_size(dev, resno, old);
>>       res->end = res->start + pci_rebar_size_to_bytes(old) - 1;
>> +
>> +#ifdef CONFIG_PCI_IOV
>> +    if (resno >= PCI_IOV_RESOURCES) {
>> +        dev->sriov->barsz[resno - PCI_IOV_RESOURCES] = 
>> pci_rebar_size_to_bytes(old);
>> +        res->end = res->start +
>> +            pci_rebar_size_to_bytes(old) * 
>> pci_sriov_get_totalvfs(dev) - 1;
>> +    }
>> +#endif
>> +
> 
> Looks like this deserves it's own function.

I'll extract the whole "restore after failed resize" to separate 
function (for both native and IOV).

Thanks!
-Michał

> 
> Regards,
> Christian.
> 
>>       return ret;
>>   }
>>   EXPORT_SYMBOL(pci_resize_resource);
>> diff --git a/include/uapi/linux/pci_regs.h 
>> b/include/uapi/linux/pci_regs.h
>> index ff6ccbc6efe96..7f5726d23b038 100644
>> --- a/include/uapi/linux/pci_regs.h
>> +++ b/include/uapi/linux/pci_regs.h
>> @@ -734,6 +734,7 @@
>>   #define PCI_EXT_CAP_ID_L1SS    0x1E    /* L1 PM Substates */
>>   #define PCI_EXT_CAP_ID_PTM    0x1F    /* Precision Time Measurement */
>>   #define PCI_EXT_CAP_ID_DVSEC    0x23    /* Designated 
>> Vendor-Specific */
>> +#define PCI_EXT_CAP_ID_VF_REBAR    0x24    /* VF Resizable BAR */
>>   #define PCI_EXT_CAP_ID_DLF    0x25    /* Data Link Feature */
>>   #define PCI_EXT_CAP_ID_PL_16GT    0x26    /* Physical Layer 16.0 
>> GT/s */
>>   #define PCI_EXT_CAP_ID_MAX    PCI_EXT_CAP_ID_PL_16GT
>