Message-ID: <5e22a45d-6f12-da9b-94f6-3112a30e8574@suse.com>
Date:   Thu, 16 Mar 2023 08:50:11 +0100
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:102.0) Gecko/20100101
 Thunderbird/102.8.0
Subject: Re: [RFC PATCH 1/5] x86/xen: disable swiotlb for xen pvh
Content-Language: en-US
To:     Stefano Stabellini <sstabellini@kernel.org>
Cc:     Huang Rui <ray.huang@amd.com>,
        Alex Deucher <alexander.deucher@amd.com>,
        =?UTF-8?Q?Christian_K=c3=b6nig?= <christian.koenig@amd.com>,
        Stewart Hildebrand <Stewart.Hildebrand@amd.com>,
        Xenia Ragiadakou <burzalodowa@gmail.com>,
        Honglei Huang <honglei1.huang@amd.com>,
        Julia Zhang <julia.zhang@amd.com>,
        Chen Jiqian <Jiqian.Chen@amd.com>,
        Juergen Gross <jgross@suse.com>,
        Oleksandr Tyshchenko <oleksandr_tyshchenko@epam.com>,
        Boris Ostrovsky <boris.ostrovsky@oracle.com>,
        =?UTF-8?Q?Roger_Pau_Monn=c3=a9?= <roger.pau@citrix.com>,
        xen-devel@lists.xenproject.org, linux-kernel@vger.kernel.org,
        dri-devel@lists.freedesktop.org, amd-gfx@lists.freedesktop.org
References: <20230312120157.452859-1-ray.huang@amd.com>
 <20230312120157.452859-2-ray.huang@amd.com>
 <ea0e3852-87ba-984b-4010-5eeac3d6c507@suse.com>
 <alpine.DEB.2.22.394.2303141747350.863724@ubuntu-linux-20-04-desktop>
 <f5e03f2a-8176-528f-e885-9a97940367c0@suse.com>
 <alpine.DEB.2.22.394.2303151616200.3462@ubuntu-linux-20-04-desktop>
From:   Jan Beulich <jbeulich@suse.com>
In-Reply-To: <alpine.DEB.2.22.394.2303151616200.3462@ubuntu-linux-20-04-desktop>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?a2VQMEF6WFppZDJKL3Y4MEtMdzFFNys3MjlzRVVQU1lzSTBISWFtR290RmtC?=
 =?utf-8?B?RkZZMUxFa2hOcWhza0NpQ2NKREhKTzlLOU1nTFFuTCtaWHVyYTZTbldEOTk4?=
 =?utf-8?B?YzZYWDNncWl4UXR2alVMZ0J4b3pSUWttbUU3UmRqMVJoemJkRVovcWJuVCtz?=
 =?utf-8?B?eVVVejM2MVdKYmUvK2xRcHNpUmw2clhhTWtsQVdHTlIwNURNV1Rqd3cvL2Z0?=
 =?utf-8?B?V2hNdVJzWGdTdEZUa3VPZnYvaVlJaFdJaThVaGkwbnM4SXgvQVdUdE0zeVVr?=
 =?utf-8?B?YXRrcFRuQTFrMDcrTDhiUzY5a0FORFZFWlVWNDhYOWdpNGVsWWdVUDFDRXVa?=
 =?utf-8?B?K2ZXSDVxMGRZUjVCZXBtTC9LSXJDeVFsZmxUV0lVSE1ZTVdlYXBHR1ZhSUYx?=
 =?utf-8?B?bXlWTGU2UWJXSFprYVMvSTNWQXBQbFgzL0o4Q2F4K0x1V0lFdis5bG16VTly?=
 =?utf-8?B?L1QwOHBYRFFQMG1VSTh6NE1Zby9va0VJME5ONEJZcjZLQVQwQytjY0NrejRi?=
 =?utf-8?B?emFSZVNmbHN5d1p4QldOWkxLOU1mdXJnVVo3bDhlaGJ3TnpUWFQySDRPOUtS?=
 =?utf-8?B?S0Rtb2xrK2tCYis0TDR5cDBXSVNua2Zwd0V0OUUwaytOdVlpMlpqMEFYSlNR?=
 =?utf-8?B?RVNidXl6aUVpeFF0SWkxM1VreEE4QkQ1dnBoZVJLVjBrV3FQd3grRFZNaUUv?=
 =?utf-8?B?VzBxZ0JGM05heFZhanBLU2l4VHF0YUovS1BONTVNbVJNOVdpbU5WN3F1bklI?=
 =?utf-8?B?c0FSRWd4NWRiZGRBNnY4KzBtVzIvZVBPUUd6NnRsaEd4emQ0VVI0bS9pQ3JL?=
 =?utf-8?B?Q0dUVXd1dnBMaUxjRWtyR0ovci94MTNoMVpWYTNxdWpHRCt0amRxV2IwSXZ5?=
 =?utf-8?B?MFl2VjduVzA1NUtIbS9hWUFxdVk1OWNuK0xkdDd2TWQ0MEZEeVRMdmFSNk1D?=
 =?utf-8?B?ck04Mk5ubHVET05BRjNwU25TRVJES1dyazFVZTFoR1F4cVBIZjF4cEFPbW5P?=
 =?utf-8?B?c3NVT3FUYndMUGIwZWFiWXVERzVCQWJLeU9pSm5zSXJ4VHN1NFo0Qmp5OWNP?=
 =?utf-8?B?V3lNRzg3NmUxRU9sc0xBc3lLa0ZITXdzY0FmY3NIc2J2emZzSi8reHc3Uzcx?=
 =?utf-8?B?aXA1VFNLM2R5UTJ5eVlJMTc3NWVPZmMraUx3dG1xWHd1Snp1Q2VOTTYxaHhX?=
 =?utf-8?B?UnJiV29XVlhNTUprWklPbWo4Wm5FZmdKR092S094aS9QRVh3QWY1ODRCTWc1?=
 =?utf-8?B?N0pFM3hTMzRlVHhVRlZYRC9IYVVlV0lYdkpoUDhEeTkwRE9ZellMNERGOWhY?=
 =?utf-8?B?SzNCb1R0emxWeDlLQjVQajc2bW5lTExmOEhyS29aS2htOFZGYjN5b1EwV2Jw?=
 =?utf-8?B?cmZ0OVR0cFRzR3JwY3JsQ1h0QWdDQTl3RXdaZlNxa1gvM3ZRb0VlWVByWjNZ?=
 =?utf-8?B?aWxGWWV3UnVhSnhhVXc1ZDBBekpXaXlORDBWRjQxcVEzQk5KRmtoWmYwMkc2?=
 =?utf-8?B?ZytXVXh3N01hNmhXdnYxV0JPQ01mZ25kbDlwRFNzOWpkVkt5Tnp3Q2doZGZ3?=
 =?utf-8?B?MjIxK1R2b2hKdnNZL3BRVHQ2eXpYWnRWSGtMamp3SElHM29TdXFEWFpLbE9J?=
 =?utf-8?B?aVRsV2gwTXZKWklYam9NUnhPZTVTcU9rMHovQlA3Sks0SVZnVG85MDgrdWtX?=
 =?utf-8?B?cUxJNFNNa3hHY0pVV2lac3IvYy9na2ZWMUN6RC9UWm41YUxJNzhtenhSSEY4?=
 =?utf-8?B?bHV2a2JzN1EzTDhoRUxuQ216ckJqd1V5RE40bC90K3ErY241NG1nY2t5Q01H?=
 =?utf-8?B?RFBmdkltN0ZXaWYyYjIyZlhuZkpJakU0WmNlSVlQVGNwU0xvenhXbXFzdWlU?=
 =?utf-8?B?Rm9pM0oxeHkxRjdIRWhGamdTQTl2VEF3TlJZU1RGcUozYTh3aEpXQ3p4ZUp4?=
 =?utf-8?B?Y0huN25qRlVnSWdSa04wLzhYOE1VOXEwK2JOUG9XOU1WR2dmdkJwc3lYMU9s?=
 =?utf-8?B?QmdnZ1lBN0hUakJFYUhXODkzY0FHR0FOZTF4dXFLai9BamROWmV0aVpCbE1U?=
 =?utf-8?B?MW9SbFVPMnRKeXU0SEdTOS9LZGdCUkhyWjcxNE5Pck1OWHFINyt2ZjVMOVRy?=
 =?utf-8?Q?mf/cSfr6cC6h35ttcyEv6+UNO?=
X-OriginatorOrg: suse.com
X-MS-Exchange-CrossTenant-Network-Message-Id: c70dcd48-1648-4c1e-196f-08db25f3135c
X-MS-Exchange-CrossTenant-AuthSource: VE1PR04MB6560.eurprd04.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 16 Mar 2023 07:50:13.7015
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: f7a17af6-1c5c-4a36-aa8b-f5be247aa4ba
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: kVzglH9PIFd0ZVz/JCljg7wZ45aIbpShfJ4hjz534e3JtfFrgVmEs9l0sorxHf9GQ0NuMbkMUejzZ56aQ/RBlw==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: AS8PR04MB8341
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On 16.03.2023 00:25, Stefano Stabellini wrote:
> On Wed, 15 Mar 2023, Jan Beulich wrote:
>> On 15.03.2023 01:52, Stefano Stabellini wrote:
>>> On Mon, 13 Mar 2023, Jan Beulich wrote:
>>>> On 12.03.2023 13:01, Huang Rui wrote:
>>>>> Xen PVH is the paravirtualized mode and takes advantage of hardware
>>>>> virtualization support when possible. It will using the hardware IOMMU
>>>>> support instead of xen-swiotlb, so disable swiotlb if current domain is
>>>>> Xen PVH.
>>>>
>>>> But the kernel has no way (yet) to drive the IOMMU, so how can it get
>>>> away without resorting to swiotlb in certain cases (like I/O to an
>>>> address-restricted device)?
>>>
>>> I think Ray meant that, thanks to the IOMMU setup by Xen, there is no
>>> need for swiotlb-xen in Dom0. Address translations are done by the IOMMU
>>> so we can use guest physical addresses instead of machine addresses for
>>> DMA. This is a similar case to Dom0 on ARM when the IOMMU is available
>>> (see include/xen/arm/swiotlb-xen.h:xen_swiotlb_detect, the corresponding
>>> case is XENFEAT_not_direct_mapped).
>>
>> But how does Xen using an IOMMU help with, as said, address-restricted
>> devices? They may still need e.g. a 32-bit address to be programmed in,
>> and if the kernel has memory beyond the 4G boundary not all I/O buffers
>> may fulfill this requirement.
> 
> In short, it is going to work as long as Linux has guest physical
> addresses (not machine addresses, those could be anything) lower than
> 4GB.
> 
> If the address-restricted device does DMA via an IOMMU, then the device
> gets programmed by Linux using its guest physical addresses (not machine
> addresses).
> 
> The 32-bit restriction would be applied by Linux to its choice of guest
> physical address to use to program the device, the same way it does on
> native. The device would be fine as it always uses Linux-provided <4GB
> addresses. After the IOMMU translation (pagetable setup by Xen), we
> could get any address, including >4GB addresses, and that is expected to
> work.

I understand that's the "normal" way of working. But whatever the swiotlb
is used for in baremetal Linux, that would similarly require its use in
PVH (or HVM) aiui. So unconditionally disabling it in PVH would look to
me like an incomplete attempt to disable its use altogether on x86. What
difference of PVH vs baremetal am I missing here?

Jan