Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <d34c7fca-1510-4565-3c23-4923174a66eb@amd.com>
Date:   Thu, 6 Apr 2023 11:42:25 +0530
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:91.0) Gecko/20100101
 Thunderbird/91.8.1
Subject: Re: [PATCH] iommu: amd: Set page size bitmap during V2 domain
 allocation
Content-Language: en-US
To:     Jerry Snitselaar <jsnitsel@redhat.com>,
        linux-kernel@vger.kernel.org, iommu@lists.linux.dev
Cc:     Suravee Suthikulpanit <suravee.suthikulpanit@amd.com>,
        Robin Murphy <robin.murphy@arm.com>,
        Will Deacon <will@kernel.org>, Joerg Roedel <joro@8bytes.org>
References: <20230404072742.1895252-1-jsnitsel@redhat.com>
 <k6cz5w66mfse7ztgtfy6ux4bjasg64dhbjnliybybcvmoywrs3@gehaolzeh3ph>
From:   Vasant Hegde <vasant.hegde@amd.com>
In-Reply-To: <k6cz5w66mfse7ztgtfy6ux4bjasg64dhbjnliybybcvmoywrs3@gehaolzeh3ph>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?Y1FmS1A0MFkwWW4zcWxjMGsxUGpvbnR4SThhSk4yRURsV0hBNnhFMWMwM3VS?=
 =?utf-8?B?L2NsMUZTWjFyNkQybS95RTB0WkZwTlFsZElaWXlxOE9QbW9jRDlCcWRlTHZT?=
 =?utf-8?B?M1dNWFRCS3F3ajVCUnFpQnB1ajJlQXhvUit2WnFadGhlUG5PNVFqN21QZjR1?=
 =?utf-8?B?SGhYK1pUdlBONmNQNEhlL1NmK0pDQVV0blBSZlduK1J0SkxGUzdRV1IvUlV3?=
 =?utf-8?B?VTZ6N1JaVmZZYWxobDJwRHV2YWkzbTJMUUxOWlczSHJORWZ4QXBaSHBpanVD?=
 =?utf-8?B?Y2UvTndNc3BoK1FkbTdaamFSSHBzSjlyRStDY21DdnhJY3VMcVVvM3dtWElo?=
 =?utf-8?B?eEd4NldubTdpSE5OaG0rRisxaXJlb1gwVWVkdUFxamJKQjdSMFBuK2Q3RnR2?=
 =?utf-8?B?SUw5anN3dTJsa1RTc0d5dTcrOGdTNmVXSW9JZW45SElGWlhRZjJuckRRcUhX?=
 =?utf-8?B?MVlKUUJVS0ZaYkpEbmJvV0J3cktYRTFTWjRVNTg4aFJIQzdoM2FSY25Ca1JB?=
 =?utf-8?B?S2dPeEplTWpldnczK3RGZXk0V3BxcnZWT2pqc2RYbThFbWs0R0pHZUgvWFBo?=
 =?utf-8?B?TUZiaVY2M01XMm9NSm9YbnpSOGp1aE1YK3FVU0pmYmlCa0ozaFJtekYva2tK?=
 =?utf-8?B?djk5VzRyYXJiSFBGZmlkNWdoQVZaVXdQQzhiNERaMDd5T2xWUndCUklPUktR?=
 =?utf-8?B?cUl3dms5S1pHdThONVpOVWhwVjRkZVdDYi9IK3E2bjhPWnZWQlFQM2NQOTl5?=
 =?utf-8?B?OXdlTTh1VUtFcEp1eVQ3OHcrMWZVbTBkSEdDalRVWi9pcVdqaUJqYUdWYzJ2?=
 =?utf-8?B?c3pJYThaVHJtTCtKRWxYMGhNSlNxMHl4QWlLcWRzY2xzK3h1Rm1LdWp4UXlr?=
 =?utf-8?B?UkRoQ0N5MGUwb2xNNDlrUzFGOUY0cWR5RThqNjd5dEJsN28yaGllMm1wMkRy?=
 =?utf-8?B?SmltOEhTLzJpV09zMkw5dkVjbE54TnFrYzV1QnZsVzlBYVdSUXZ6Yks5TTNP?=
 =?utf-8?B?Y2NrQVEyeUt6WkQxbjIvSG5MVG5KVVJmeEpmVlNSZkhmRUZlTEtnQjBYOElY?=
 =?utf-8?B?N29qMlhxYVBqQjY5T25ZU3lwQjBaeS9YNHJmYmcwYlBITFVmRi9GczFLVms5?=
 =?utf-8?B?aTdZZ2NZVlhHY01sUkJ6b0gzZ050M1dIYTZxd09mU05USDZUSlh3ZHNQb1pF?=
 =?utf-8?B?dzc1MWdic3o3VHlRSC9HTVZqeVM2cjZ3WWIwczhUQUtsSkNNNmlPMkFINnhx?=
 =?utf-8?B?M3RlS3I1S1laQXJkNE9qYkhjNGVmd3ZYUTl1eDNGVmFhbytkSWVMQ1BLd2RW?=
 =?utf-8?B?L2hmV2tIay9aY2tvZ2hUVmR0S2UyRWxKSnFCZDRTTVNYWEUrd3Ard213aUdE?=
 =?utf-8?B?alFGWHVac2lNOEdIcXhxOGZKY2tXTkVMUnprN2tNeCtkS1BRMlJvQTBmN0ow?=
 =?utf-8?B?R2I3eGFMQ1JocFozWmxMYnU3NUlKdnp3bG1qbCtHVTVSSWgzMmd2OWFRTngz?=
 =?utf-8?B?NjNjSlR3SkM0TW9yR0w4QUlobC9Hd20rT3VZclcxNnNwaVJxbVhqN2xlU0ZV?=
 =?utf-8?B?VkpwYkx0WjJVazk3ZXl0T2lCczkyMXFPOFp1UEgxWGxna1J4OGdraUFJNG51?=
 =?utf-8?B?NlBDeTZUNGxqN2I4clF5NGxNTFAydTRvZVV1ZUx0YW5PV1RhSmNDZElQblRO?=
 =?utf-8?B?TkJsN2FPcXVhZjREOTByUGRFSVpGV0dPczVUV2xEbnQxOE5taXpvZGg3em5T?=
 =?utf-8?B?WVcyM01PeXZDTVlSaXppNkpvYk1JMGY4bnRkRnMydDdPY0RLa2FwdjJYeEM4?=
 =?utf-8?B?VFJZY1JaM29wbWsvN3lJS1p3S2NCZEpXV25acDdVUkhja2pqNDdMakZFbmlh?=
 =?utf-8?B?a3c1d3NBMGJpN3o4UDd6VHorZUg5ZEF3ZnVQTHRGcDN4KzZCTXZ1MC8zODBu?=
 =?utf-8?B?VkZyaWlMZHNyYXlYQXJPaUhwSkJyVGZQUURKK1RkdUtheXdJRmhwL0dHSEJm?=
 =?utf-8?B?cGN4Y1ROTU11cG00Vi9uVlNDcDlzdVJUeDc5NTBJTVpHcGdKeFU4OVBaUDN5?=
 =?utf-8?B?dWM4aDk3cFdCbnZ5czVHYTFpQWUzL3djNWM5Ky9hUktWUjVNaS9lOE5VamVr?=
 =?utf-8?Q?iEGu8wlhfKoMqVI6cjE2tsakR?=
X-OriginatorOrg: amd.com
X-MS-Exchange-CrossTenant-Network-Message-Id: c446253a-8dec-485e-1063-08db3665ed0a
X-MS-Exchange-CrossTenant-AuthSource: DS7SPRMB0009.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 06 Apr 2023 06:12:40.3118
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: hR+xH+kdzzrDT6l8fZ8dHx+nh4FegruHkwQoLC3atgihf+b4L8+JDoWOKlpM8flDm0TfZRoF3h/mibY6BpwlDQ==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH0PR12MB8029
X-Spam-Status: No, score=-0.6 required=5.0 tests=DKIM_SIGNED,DKIM_VALID,
        DKIM_VALID_AU,DKIM_VALID_EF,FORGED_SPF_HELO,NICE_REPLY_A,
        RCVD_IN_DNSWL_NONE,RCVD_IN_MSPIKE_H2,SPF_HELO_PASS,SPF_NONE
        autolearn=unavailable autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

Jerry,

On 4/4/2023 9:58 PM, Jerry Snitselaar wrote:
> On Tue, Apr 04, 2023 at 12:27:42AM -0700, Jerry Snitselaar wrote:
>> With the addition of the V2 page table support, the domain page size
>> bitmap needs to be set prior to iommu core setting up direct mappings
>> for reserved regions. When reserved regions are mapped, if this is not
>> done, it will be looking at the V1 page size bitmap when determining
>> the page size to use in iommu_pgsize(). When it gets into the actual
>> amd mapping code, a check of see if the page size is supported can
>> fail, because at that point it is checking it against the V2 page size
>> bitmap which only supports 4K, 2M, and 1G.
>>
>> Add a check to __iommu_domain_alloc() to not override the
>> bitmap if it was already set by the iommu ops domain_alloc() code path.
>>
>> Cc: Vasant Hegde <vasant.hegde@amd.com>
>> Cc: Suravee Suthikulpanit <suravee.suthikulpanit@amd.com>
>> Cc: Robin Murphy <robin.murphy@arm.com>
>> Cc: Will Deacon <will@kernel.org>
>> Cc: Joerg Roedel <joro@8bytes.org>
>> Fixes: 4db6c41f0946 ("iommu/amd: Add support for using AMD IOMMU v2 page table for DMA-API")
>> Signed-off-by: Jerry Snitselaar <jsnitsel@redhat.com>
> 
> 
> 
> I'm still not sure this is the best solution. Feels odd with adding a
> check to core code to handle something one of the drivers has
> done. Another thought was something like arm does, with amd_iommu_ops
> dropping the const and setting the default page size bitmap in
> iommu_init_pci(), but I think that would still require adding
> something in the protection domain/init code to deal with it forcing
> v1, so it would still require a check in the core code.
> 

I am not familiar with how arm paging works. But in AMD case we decide the page
table during protection domain allocation. Also we can have different page size
for different domains. (like one domain in v1 and another in v2 page mode).

> Would adding an op make more sense, with a generic op doing what the
> core code currently does for setting the default? Or am I overthinking
> this?

IMO adding another ops is not required. I think current fix is good enough.
Thanks for debugging/fixing this issue.

Reviewed-by: Vasant Hegde <vasant.hegde@amd.com>


-Vasant

> 
> snits
> 
>> ---
>>  drivers/iommu/amd/iommu.c | 6 ++----
>>  drivers/iommu/iommu.c     | 9 +++++++--
>>  2 files changed, 9 insertions(+), 6 deletions(-)
>>
>> diff --git a/drivers/iommu/amd/iommu.c b/drivers/iommu/amd/iommu.c
>> index 5a505ba5467e..167da5b1a5e3 100644
>> --- a/drivers/iommu/amd/iommu.c
>> +++ b/drivers/iommu/amd/iommu.c
>> @@ -1666,10 +1666,6 @@ static void do_attach(struct iommu_dev_data *dev_data,
>>  	domain->dev_iommu[iommu->index] += 1;
>>  	domain->dev_cnt                 += 1;
>>  
>> -	/* Override supported page sizes */
>> -	if (domain->flags & PD_GIOV_MASK)
>> -		domain->domain.pgsize_bitmap = AMD_IOMMU_PGSIZES_V2;
>> -
>>  	/* Update device table */
>>  	set_dte_entry(iommu, dev_data->devid, domain,
>>  		      ats, dev_data->iommu_v2);
>> @@ -2048,6 +2044,8 @@ static int protection_domain_init_v2(struct protection_domain *domain)
>>  
>>  	domain->flags |= PD_GIOV_MASK;
>>  
>> +	domain->domain.pgsize_bitmap = AMD_IOMMU_PGSIZES_V2;
>> +
>>  	if (domain_enable_v2(domain, 1)) {
>>  		domain_id_free(domain->id);
>>  		return -ENOMEM;
>> diff --git a/drivers/iommu/iommu.c b/drivers/iommu/iommu.c
>> index 10db680acaed..256a38371120 100644
>> --- a/drivers/iommu/iommu.c
>> +++ b/drivers/iommu/iommu.c
>> @@ -1964,8 +1964,13 @@ static struct iommu_domain *__iommu_domain_alloc(struct bus_type *bus,
>>  		return NULL;
>>  
>>  	domain->type = type;
>> -	/* Assume all sizes by default; the driver may override this later */
>> -	domain->pgsize_bitmap = bus->iommu_ops->pgsize_bitmap;
>> +	/*
>> +	 * If not already set, assume all sizes by default; the driver
>> +	 * may override this later
>> +	 */
>> +	if (!domain->pgsize_bitmap)
>> +		domain->pgsize_bitmap = bus->iommu_ops->pgsize_bitmap;
>> +
>>  	if (!domain->ops)
>>  		domain->ops = bus->iommu_ops->default_domain_ops;
>>  
>> -- 
>> 2.38.1
>>
>