Message-ID: <7a82ddb8-5cb9-aeb8-df20-9f0e2c6fccaa@intel.com>
Date:   Fri, 17 Dec 2021 12:23:40 +0100
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101
 Firefox/91.0 Thunderbird/91.4.0
Subject: Re: [PATCH 2/2] PCI: Don't fail BAR resize if nothing is reassigned
Content-Language: en-US
To:     =?UTF-8?Q?Christian_K=c3=b6nig?= <christian.koenig@amd.com>,
        <linux-pci@vger.kernel.org>, <linux-kernel@vger.kernel.org>,
        Bjorn Helgaas <bhelgaas@google.com>
CC:     Ard Biesheuvel <ardb@kernel.org>,
        "Michael J . Ruhl" <michael.j.ruhl@intel.com>,
        Rodrigo Vivi <rodrigo.vivi@intel.com>
References: <20211215141626.3090807-1-michal.winiarski@intel.com>
 <20211215141626.3090807-3-michal.winiarski@intel.com>
 <60ff4820-5dd9-bb3b-9baa-55ac53c0646c@amd.com>
From:   =?UTF-8?Q?Micha=c5=82_Winiarski?= <michal.winiarski@intel.com>
In-Reply-To: <60ff4820-5dd9-bb3b-9baa-55ac53c0646c@amd.com>
Content-Type: text/plain; charset="UTF-8"; format=flowed
Content-Transfer-Encoding: 8bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?a3VUOHB3eU1XRGtFc3VNaHVMdzd2TDlQV1ZCT2Z5dTRMbzdLdjNYU0VzZEhQ?=
 =?utf-8?B?NXlTYkw2QlNSVnJiVHlRblNyN2VsZGF5eTRRY29McTEwVzJGd085dFZlWW1j?=
 =?utf-8?B?RmdvbEJJMjcwWExiRnJwWlhLcm51WG1ZVmhrdENucnRGK3hrVXJWcDVPK293?=
 =?utf-8?B?aStMSEllV2FETXRwaFRnMHVXcFRsLzh0OWVwV2FmOHZVSFlXL2tOTDA0ZFNU?=
 =?utf-8?B?ZVZhcXNGd1E1SWNkY0puWlFkcWZBTUZBQ2c5R0hqNXQ4UkhYTUhLOGNvMml2?=
 =?utf-8?B?a09SK1Q0bWtiYjc2Ry9UOTRaN0xVWldCTVJQVG5JbVZYZjRLRGlGUkx5azcv?=
 =?utf-8?B?cTY3VUF1MFcxdmcrQnZIR21HeTRlYzYreVJsMmVNWlQrSVg5T09HYjNhUmJs?=
 =?utf-8?B?akxQaWU3V2d2b0J2ekhkUml0K2RLR0pmSmE1aGI5Z1dDOS9odk9uZEhoUFJy?=
 =?utf-8?B?K1pGM0syZWRFdGxZU2dmQTV2WTgrNVRxQnI0eVR6TWlQZFhka2NyQWR2WmFh?=
 =?utf-8?B?VlJkVndQcXRVbmwwSFp2U3ZtM1dDTnRFUU9SQkkydG5XUmNrVzhINTlWdFVh?=
 =?utf-8?B?SWdUTEhFTzFUMWNRdWk0aHZTL1BVSzdvaVVhUWhYYW80aS9wdi82NC81MTgz?=
 =?utf-8?B?SjJrc0ZIZEJOK2VzR1ExQ1Q2TjlrL1dsNUxTc1ZFYnNrMUV0blRBcjBPMXl2?=
 =?utf-8?B?QlNiRTJhOUxFSFRES3oxdncvT1pkMzh4OXp6bUhscmI5Y21iK1NIZ2JmZnkr?=
 =?utf-8?B?TzFMVmdaa04wd3VPdDFucnVlMGNVRlNWeGg5YkEybHJmTXlaKy82RXNWR2VD?=
 =?utf-8?B?ZWdjMFpMLys3NlV2dFpyMDBuWjBvTWNFdFBWM1NnM2xDd2Y4eVZpbWxjaSt3?=
 =?utf-8?B?TC9rOTJ2eS9UYUVFK0krdDJlRWJVblhaeC9IdU5iT3hKbHlzaFNpRjYydmoy?=
 =?utf-8?B?S2VxdllEakhEL29WajR1dUU1YmQ3WHJ2TDFTWmJGVzA0YVBwV2QzUUlEWlVh?=
 =?utf-8?B?RGxoSFI4d25PN2RYTEtmdFRJb2FFbnZoWFJlNU9YcVIrYVUzb3dIVUtVVjVH?=
 =?utf-8?B?M0xFQ3NYYk5iRkQreTAvblh0VTlkVkcxUXQ5VTFMTXByK1RMNmd2RHdwOEhD?=
 =?utf-8?B?RGNLSGZROXJxLzZGb1lNVVJNTFVFUFlEbGlWRjduNnFzUTFkSmp1Ylp5M3Fw?=
 =?utf-8?B?YzlyRlE0a2VaSlJjWVdIN2szTnRJRS9Gb1E2U1RNMnBQRTlkOVRBcTlvMmxm?=
 =?utf-8?B?RXU3SWtDRjZiNUR2MVo0YW9XVVd3K0NxbmlNbzNaTENaODVxRG4wTXcxOENQ?=
 =?utf-8?B?dWplNm5TMjA5ZWMzdjdkNmF6MDh0U2xqYkJ1RDg0VlhPU2E5REEwc21HMGtm?=
 =?utf-8?B?anZob2V4azkzM2Zwckx1aGp5RGs0c1hYaEhYSGNUWW9BVUZIak8ydUs5dU1s?=
 =?utf-8?B?d1U3UVl4TEY2VTVHYnhEQndvbDJOMTVQQzV3UkRTSUxHa096RUhkazRFNnpu?=
 =?utf-8?B?a3lOaXZQWmpWNHJYazU1LzB4SnRjbkRpcnVDMHlKdm4yb2huUmlwQkxBUzZX?=
 =?utf-8?B?TGVWelpEQkY3eVhqaHU4dUt2UWdISDV5Z21KMXVVMENDN292NkpVY0puSTEx?=
 =?utf-8?B?T1l5TTU0MTFJMmsyVkZiZStiR0w5OWNzSU9QZWV3eW9BWmlHa05FY0NyQnRp?=
 =?utf-8?B?VWdseklaNTVnSGRvRWdJY3VNdFY1d2dxcC9UMFJLTmxtQ284M1BoZkJSK3V3?=
 =?utf-8?B?MFVWYWNZWjl3cldXbkF6WjNuWXhkNDNkL3g3ZEVmNm5mTGRjTkJSNDJlZmla?=
 =?utf-8?B?VEFCeXlTVTBQT0FNWCt6YUhiTjMwQWJ6S3N0cVlvS3dsVC85RHdLQm9DS0JJ?=
 =?utf-8?B?RFFsNmhadW55NEZ1dWgvT1BMSXVkWXIrQUZaaHpoWXpQSHJ0c09OY25SUU0y?=
 =?utf-8?B?aXJCWXlOTnhWaVJMTStkQkRwVjJ2MUlTKzB4TTJQQUljYWRUM0oxSk42M3Vw?=
 =?utf-8?B?K1U5LzdGanZDcHg1SUJKSjNQbHlzVEFsNXd3Zjc1S01FdEFPZHdqejVrUGM3?=
 =?utf-8?B?RjRJSDlrMGJtMnd6bTlCM2phUDZYTWcvU0ZtUXpyNnphWkZueVV0WFJNVi9E?=
 =?utf-8?B?K3cwaGNOa1hDciszSStpeUc5ZDVPTEhZemx4QVhjc3poNXdxTm9BbUlTYVZr?=
 =?utf-8?B?bGZSZ2NZRWJEYUd3ZER5Um5ka2FxU285bWZBMmV6QytGRTlJY2NXSUs1Qkhw?=
 =?utf-8?B?Z1k4cEVzMHlBbDN1YTdxejFZR3pnaURWSEMrWS9KSmxPTnd3eUJNWlI1MG9P?=
 =?utf-8?Q?cQv541dAyzrGIqSviA?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 8105b605-ac97-49c3-b48a-08d9c14fb0fd
X-MS-Exchange-CrossTenant-AuthSource: DM4PR11MB5373.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 17 Dec 2021 11:23:46.7348
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: B/wijtwAWlgvsIQ9RzWWs8ZeZ3haF3mh2MqvoVgRyAt9iFJFmn7n9gl0wusQXqHpT2J6levPMRu04/6C/bVegPy8IWOBuxqKlg08qrGTy9U=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM6PR11MB4187
X-OriginatorOrg: intel.com
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On 16.12.2021 08:12, Christian König wrote:
> Am 15.12.21 um 15:16 schrieb Michał Winiarski:
>> When pci_reassign_bridge_resources returns -ENOENT, it means that no
>> resources needed to be "moved". This can happen when the resource was
>> resized to be smaller, and it's completely fine - there's no need to 
>> treat
>> this as an error and go back to the original BAR size.
> 
> Well that doesn't make much sense as far as I can see.
> 
> Drivers mandatory need to call pci_release_resource() on all resources 
> which might need to move for a resize, including the one which is about 
> to be resized.

Since IOV BARs have their own memory-decoding enabled bit, which is 
usually tied to the lifetime of virtual functions, the PF driver could 
do IOV BAR resize during its lifetime (without releasing its own resources).

> 
> When you get -ENOENT from pci_reassign_bridge_resources() it just means 
> that the function was not able to do it's work because the driver failed 
> to release it's resources before the resize.
> 
> Technically we could indeed skip this step if the new size is smaller 
> than the old size, but then the question is why would somebody resize in 
> the first place? The freed up address space is not usable if you don't 
> do this.

With regular BAR, the size of MMIO resource is equal to bar_size.
With IOV BAR, the size of MMIO resource is equal to iov_bar_size * 
total_vfs.

It means that the driver could use the pci_resize_resource in two ways, 
it could just call it like for the native BAR - overall MMIO resource is 
going to grow, or it could limit its total_vfs (overall MMIO resource is 
going to shrink, but from VF perspective, its individual BAR is going to 
be larger).

To ilustrate:

Native:

  1G    2G
+--+  +--+
|  |  |  |
+--+  |  |
       |  |
       +--+

Resource grows from 1G to 2G. No surprises.


IOV 4 VFs:

  1G    2G
+--+  +--+
|  |  |  |
+--+  |  |
|  |  |  |
+--+  +--+
|  |  |  |
+--+  |  |
|  |  |  |
+--+  +--+
       |  |
       |  |
       |  |
       +--+
       |  |
       |  |
       |  |
       +--+

Resource grows from 4G to 8G. But for larger number of VFs, and larger 
BAR sizes, finding MMIO space to accomodate may end up being tricky on 
some platforms.


IOV (limited to 2 VFs):

  1G    2G
+--+  +--+
|  |  |  |
+--+  |  |
|  |  |  |
+--+  +--+
|  |  |  |
+--+  |  |
|  |  |  |
+--+  +--+

No changes in resource size, we started with 4G and we end up with 4G 
after resize (but those 2 VFs can now use 2G BAR).

Does that make sense?

Thanks
-Michał

> 
> Regards,
> Christian.
> 
>>
>> Signed-off-by: Michał Winiarski <michal.winiarski@intel.com>
>> ---
>>   drivers/pci/setup-res.c | 2 +-
>>   1 file changed, 1 insertion(+), 1 deletion(-)
>>
>> diff --git a/drivers/pci/setup-res.c b/drivers/pci/setup-res.c
>> index 1946e52e7678a..5de5129055e0a 100644
>> --- a/drivers/pci/setup-res.c
>> +++ b/drivers/pci/setup-res.c
>> @@ -484,7 +484,7 @@ int pci_resize_resource(struct pci_dev *dev, int 
>> resno, int size)
>>       /* Check if the new config works by trying to assign everything. */
>>       if (dev->bus->self) {
>>           ret = pci_reassign_bridge_resources(dev->bus->self, 
>> res->flags);
>> -        if (ret)
>> +        if (ret && ret != -ENOENT)
>>               goto error_resize;
>>       }
>>       return 0;
>