Message-ID: <d7ce665c-8fd9-30b4-bf86-8d6907a2c17a@suse.com>
Date:   Wed, 15 Mar 2023 07:52:21 +0100
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:102.0) Gecko/20100101
 Thunderbird/102.8.0
Subject: Re: [RFC PATCH 1/5] x86/xen: disable swiotlb for xen pvh
Content-Language: en-US
To:     Huang Rui <ray.huang@amd.com>
Cc:     Juergen Gross <jgross@suse.com>,
        "Huang, Honglei1" <Honglei1.Huang@amd.com>,
        "amd-gfx@lists.freedesktop.org" <amd-gfx@lists.freedesktop.org>,
        "dri-devel@lists.freedesktop.org" <dri-devel@lists.freedesktop.org>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        "Hildebrand, Stewart" <Stewart.Hildebrand@amd.com>,
        Oleksandr Tyshchenko <oleksandr_tyshchenko@epam.com>,
        "Chen, Jiqian" <Jiqian.Chen@amd.com>,
        Xenia Ragiadakou <burzalodowa@gmail.com>,
        "Deucher, Alexander" <Alexander.Deucher@amd.com>,
        "xen-devel@lists.xenproject.org" <xen-devel@lists.xenproject.org>,
        Boris Ostrovsky <boris.ostrovsky@oracle.com>,
        "Zhang, Julia" <Julia.Zhang@amd.com>,
        =?UTF-8?Q?Roger_Pau_Monn=ef=bf=bd?= <roger.pau@citrix.com>,
        Stefano Stabellini <sstabellini@kernel.org>,
        "Koenig, Christian" <Christian.Koenig@amd.com>
References: <20230312120157.452859-1-ray.huang@amd.com>
 <20230312120157.452859-2-ray.huang@amd.com>
 <ea0e3852-87ba-984b-4010-5eeac3d6c507@suse.com>
 <alpine.DEB.2.22.394.2303141747350.863724@ubuntu-linux-20-04-desktop>
 <ZBFGJUE0ahA4R9gr@amd.com>
From:   Jan Beulich <jbeulich@suse.com>
In-Reply-To: <ZBFGJUE0ahA4R9gr@amd.com>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?UFZxOElXbFVHMU5FOWlIcVBmdWJBK2dSKzl0TisvdFhZS2dUVXZpRnlMSktt?=
 =?utf-8?B?ZXJYRlZQSUo5OWhBS0ExQysyaXkxdHdWZmFCT1MrRXNwdm5ydHRSU2lkSC9O?=
 =?utf-8?B?cGxwNUNPNHpDajY0a1UwcTlFWkRvWk9taE92RStzK1RYT2ZSVDBreVhNK3ZX?=
 =?utf-8?B?aW5XWTNQYmFFaWNLcmkreEE4OVoyMFA1MnMvMy9LNnkxOVBTWndFd2tHckxm?=
 =?utf-8?B?RDhYQ0lRejh6cjNXVjNOYTF2VUwrbGhxQXUzVWxsN1F6ZDYrcUJMWm5TMjdx?=
 =?utf-8?B?YUE3Wko5MU5LUHF6TEg1UkxRemVZRDYzM01wU09Dc3VsTVlJODM5dS9pQUp3?=
 =?utf-8?B?Q1FIQUoxRUFHR05yUmpIeFdVWDJSaFFpc3A2SlYxZXNOTS85VUV2RE1DRmpr?=
 =?utf-8?B?TlVDWS9wcXFFKy9FalhMMjI4dWpFOXJ5elpPaHg2NDJ5U1BpdURldWtSRmNV?=
 =?utf-8?B?cmNuUHpvOEtCWmd5WEN4WXNjT2hkVStKRGtIUTV5a0xnTVBsZVhKMVJVU3N0?=
 =?utf-8?B?bksySmpVU2YrSkJHSzdFQVJuTVc1RlJ6VmtHVXJLcFovbVdOaGgwbElhdzlz?=
 =?utf-8?B?RWFoeWVsZ0RkakVIcjNNQlp0bllOZHdIek1nZVVNK1NhZnlRMEwxZUlISEpy?=
 =?utf-8?B?OXBaYThJaFp1bnVJQitMaklEb0htRHQ3dHBlbXR1OXdsUGVEY1BQUkMrRGpt?=
 =?utf-8?B?cm8renJudU9oOUttWWd1ZkZDMUpmdC9OSThFSzJaaW40YTVRQ1lKdzhVbDFz?=
 =?utf-8?B?QlNNcWxEcXByejFEUGJGb0sweFBjL2pHWEtTTzFSU3VTa1Rlc1RrUTNUS0k2?=
 =?utf-8?B?dnRkdDBKM2svcE9ieHIzUlBqbFMwdXBHWUhkenc5R2tnL2NyT1FhODV4TVlU?=
 =?utf-8?B?UzgrZHhvSy9DOGhoR0U3aGI4VXpJSW53ZWVKazRYNC9mcjhsWHJuTXorNjAv?=
 =?utf-8?B?TC9ndW1hZktmRXVIMmp2R0FTbTRSUEF3M09pei9HeExQZ1IvUEVCdzhZTkxD?=
 =?utf-8?B?REQ1dzMwRy80M2RsSzFpcndwMlZqeW95VFlRNE9BL1Vmd1d0S0lNWGRLNExB?=
 =?utf-8?B?MndjN1FXS2hkS0pjL1VXUldWOEp6ZGY5UUxxQ2g1bGthcTFvanBNdmJTaG85?=
 =?utf-8?B?S0JBc1JGVG9IeGozQUlNNkxmSXpxbVJxRUt0VjZKV2hCdmJKd21vTUgxRTIz?=
 =?utf-8?B?VHFYckhLQ3RkT0ZKN3N1YVRvaEVCN3lNSVFJL1dnTHVlYnNDeWJPNVdFanRT?=
 =?utf-8?B?TmVuTGU4a0tycGNMSDdpZE9IMnBTWjhQZXNqOCs4OFlZWjFyUTJLWldPQVA5?=
 =?utf-8?B?QUpST3pQQWVXNDBiUlVKUGdBWHFKcHJ5VFZHWldqOFNXL3BNcUN6SkpMUW9V?=
 =?utf-8?B?bmc3UzBhNjI3UFpWYlVaK1IwRUJISzhYcld0SzBqcGZpdGNzeHVKQUVoMTJ0?=
 =?utf-8?B?d1hEYjByNmFoR2w4T2VxdDlJd1NITWNyZHFKL2lNMjFZNTU5eDR5eExGSDJE?=
 =?utf-8?B?bmc1R3lGdEdTSDl6OEgvdHVUT0NNVEpUNDlsRzk3VGoxbWdUeGhiS3RRZzVv?=
 =?utf-8?B?SnhReWxHbzdybFg4b3BPNG00Nk9wSmRqbWNHNW1DeGhmdjZ1NEpoaE82QVJj?=
 =?utf-8?B?UDkvRk5qcGRiS0ExY1laemJLYmlBcStaK01JNFdRSkdxYmVPWG00bEZqa2tp?=
 =?utf-8?B?ZC9kRTdoL1JXVUw5dHI2ZFQ4dGdLeFZTZzNBSVFrREFHbUlGTjQ0U3VoaktK?=
 =?utf-8?B?VFBwUEZ3d0tQYmJFSkZiZzkvVmxUamtQRVBmY3pJeHJ5YTk1cy82RWkxcm9W?=
 =?utf-8?B?b2gxeE5ad1d3RGdnUnc0YytEMDh3eFRTVG91ckpwRVdTZXVubkg1ZCtBVERT?=
 =?utf-8?B?bTNqTUdrTjNzTVM1UFZrV2hCZnlDSEorcHA4UFZDUjZDRGpQbHVZSFlMZ1NQ?=
 =?utf-8?B?WThkdFRYM09hTW1zYXhxWTdBZVNPcENKMDQ5cG1NbUJwa3VjeDJ5SG9DSFB6?=
 =?utf-8?B?a3ZPbEt4U0U0ejFtZ1BLckNqQWxyVk4zaU5wUCtrdDB3VHVudithQWFKS25T?=
 =?utf-8?B?dmhVejZQcDdxaEhOdjZUZlZGdkYwTG15S1RFVE9NNXJCYnBaVnh3OTk4emQ3?=
 =?utf-8?Q?P2WZO67KXr6oqNfqShydK7MC7?=
X-OriginatorOrg: suse.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 5ab30859-5335-404a-e1e7-08db2521d48f
X-MS-Exchange-CrossTenant-AuthSource: VE1PR04MB6560.eurprd04.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 15 Mar 2023 06:52:23.6373
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: f7a17af6-1c5c-4a36-aa8b-f5be247aa4ba
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: wr4FUz3W7QSC9tI9ic70Pzda7tkaVKP1LzrYWRXKqav1dVtehgFBqtJQDjjYRCAdjixh+btX8ceg8rXymBYJFw==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DBAPR04MB7445
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On 15.03.2023 05:14, Huang Rui wrote:
> On Wed, Mar 15, 2023 at 08:52:30AM +0800, Stefano Stabellini wrote:
>> On Mon, 13 Mar 2023, Jan Beulich wrote:
>>> On 12.03.2023 13:01, Huang Rui wrote:
>>>> Xen PVH is the paravirtualized mode and takes advantage of hardware
>>>> virtualization support when possible. It will using the hardware IOMMU
>>>> support instead of xen-swiotlb, so disable swiotlb if current domain is
>>>> Xen PVH.
>>>
>>> But the kernel has no way (yet) to drive the IOMMU, so how can it get
>>> away without resorting to swiotlb in certain cases (like I/O to an
>>> address-restricted device)?
>>
>> I think Ray meant that, thanks to the IOMMU setup by Xen, there is no
>> need for swiotlb-xen in Dom0. Address translations are done by the IOMMU
>> so we can use guest physical addresses instead of machine addresses for
>> DMA. This is a similar case to Dom0 on ARM when the IOMMU is available
>> (see include/xen/arm/swiotlb-xen.h:xen_swiotlb_detect, the corresponding
>> case is XENFEAT_not_direct_mapped).
> 
> Hi Jan, sorry to late reply. We are using the native kernel amdgpu and ttm
> driver on Dom0, amdgpu/ttm would like to use IOMMU to allocate coherent
> buffers for userptr that map the user space memory to gpu access, however,
> swiotlb doesn't support this. In other words, with swiotlb, we only can
> handle the buffer page by page.

But how does outright disabling swiotlb help with this? There still wouldn't
be an IOMMU that your kernel has control over. Looks like you want something
like pvIOMMU, but that work was never completed. And even then the swiotlb
may continue to be needed for other purposes.

Jan