Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.36 as permitted sender) client-ip=23.128.96.36;
Date:   Wed, 6 Dec 2023 11:38:09 -0400
From:   Jason Gunthorpe <jgg@nvidia.com>
To:     Lorenzo Pieralisi <lpieralisi@kernel.org>
Cc:     Catalin Marinas <catalin.marinas@arm.com>,
        Marc Zyngier <maz@kernel.org>, ankita@nvidia.com,
        Shameerali Kolothum Thodi 
        <shameerali.kolothum.thodi@huawei.com>, oliver.upton@linux.dev,
        suzuki.poulose@arm.com, yuzenghui@huawei.com, will@kernel.org,
        ardb@kernel.org, akpm@linux-foundation.org, gshan@redhat.com,
        aniketa@nvidia.com, cjia@nvidia.com, kwankhede@nvidia.com,
        targupta@nvidia.com, vsethi@nvidia.com, acurrid@nvidia.com,
        apopple@nvidia.com, jhubbard@nvidia.com, danw@nvidia.com,
        mochs@nvidia.com, kvmarm@lists.linux.dev, kvm@vger.kernel.org,
        linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org
Subject: Re: [PATCH v2 1/1] KVM: arm64: allow the VM to select DEVICE_* and
 NORMAL_NC for IO memory
Message-ID: <20231206153809.GS2692119@nvidia.com>
References: <86fs0hatt3.wl-maz@kernel.org>
 <ZW8MP2tDt4_9ROBz@arm.com>
 <20231205130517.GD2692119@nvidia.com>
 <ZW9OSe8Z9gAmM7My@arm.com>
 <20231205164318.GG2692119@nvidia.com>
 <ZW949Tl3VmQfPk0L@arm.com>
 <20231205194822.GL2692119@nvidia.com>
 <ZXCJ3pVbKuHJ3LTz@arm.com>
 <20231206150556.GQ2692119@nvidia.com>
 <ZXCQrTbf6q0BIhSw@lpieralisi>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <ZXCQrTbf6q0BIhSw@lpieralisi>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?JORLi2ffC8OvRHrCOQsfR48giCBrxTh1nAXoh8q3ZyTRf5k4hDVnF/G0+/A9?=
 =?us-ascii?Q?HAvSkw6viNhy7XYISxzrSang2DG/tP+2zFy9bncxngzTrPB5WE7hDK6GKQ5i?=
 =?us-ascii?Q?UbJeXi4APCexxy8K3uQnKKpRRuCDSn8UUKFyEFxVn+Hatn5WCf3RiDJyaYXh?=
 =?us-ascii?Q?eeM5Q04FwTA35LQl+GlYbfBBYiqzkp77Jpv4Hnh91nP56yVmb+oyCjSrR+An?=
 =?us-ascii?Q?4Aw92wI9tD1XmZYiq1EtnVD6k52HiY3ZJFY+5/E/eTDS2m2x8v9a8w2buwGu?=
 =?us-ascii?Q?EXhZOqPkLK/+HPxLLskiX7X0S616B4ONaqxSUEQrU5eWmn7yfhHFgKDLsOt4?=
 =?us-ascii?Q?/Vp5qtqLuey4qf29zsyM5jDy4hZmJDbdbuUZxWovQ6SKGGC/CqCtRkcrdWFs?=
 =?us-ascii?Q?fduiAXKJVja2aiuzCB2PxhSIoH9dxQy3/MN2el4dUlqMB2WNfsliX8DBou6f?=
 =?us-ascii?Q?u/i/HNpXGksI03JdcJ2lYRwbk7ZcT7t5h11mcIwCBfnx8kTplPdM3il7vndC?=
 =?us-ascii?Q?paHNh7Ekj5qVGxLYcSibJL8ZlEK4GpDTdbvbzSFnTc12rdLq2OKOwYgEYm6o?=
 =?us-ascii?Q?tBldmo0BC+zVpjBbLLmySXkaCQbFmnkTmb50h63jP3yoOTr9tKRX3cmpVJxV?=
 =?us-ascii?Q?9DAXLkBlAQXhGnW2vXAGB5PxXI42rlOuqU0H5tpXplebCuWxHryhaMp15Vln?=
 =?us-ascii?Q?4x1wNYao8hMgZs+yBmRWV309S7D7oSfbhM4wSYWEWZxVgz3jW3f1hk4z4XgO?=
 =?us-ascii?Q?K4FUodSdDwP1+E1vYYpqa8AcMbdySGKJh4/gSFP/Fv8ft0+9PhuIh3Bqh1ud?=
 =?us-ascii?Q?OWtItS9pFvAHeT/KC83K/57pu+zyfgJxn6xeK3H/uCcBHwWKhrtgSkqAuA+5?=
 =?us-ascii?Q?ch4e0Pcwx41H4PLV5XqTA7oNrPpnYYWzrG6cSp6Kgj+ZtQbpcLe3KQjVhP3k?=
 =?us-ascii?Q?K7ks+/yjL3sjNuwqiBFYMT+xuSaBIPlSHqdG1DXaE3P2qvwlGcfRAWBLcurF?=
 =?us-ascii?Q?sa2MO5o7NC/lC33xqT678ImCa0L3zlrKI2Y+ynKTykwjW6FUb0/9AaG2s0Ux?=
 =?us-ascii?Q?aZ0nDWf2LKol0HFOeHMRPekHIYI++r1r1zj2Z3bPxS8YSYhuY+RCwrNUbGv1?=
 =?us-ascii?Q?kaHicVoqoQY1QcDzgyf5nsxL1Cx15h1EQYd0jbarfBOSrcoBF+20WBQM1xFE?=
 =?us-ascii?Q?07/l4BVf8h8ldqscWImlSxARuM/SNdPdgWehroQEh8tqbEPHcFH98fCQvpDD?=
 =?us-ascii?Q?yFXueTMiD5q619XC5Ccvyv2FBEPPagkE+e/eEO3hsUhf6G1SODziJ//mwWHd?=
 =?us-ascii?Q?UuY0KrgJxyBRuUL6sRxiSQS+ryv6pziz/Ro8jYiyfru6tAD0z5uU+pQH0t1V?=
 =?us-ascii?Q?FQ3ZEbUkAOZNq7Zm/QR5+mNpa5FBQ9jsXQPeTD39obOBbfSatnW+pjuSbd4s?=
 =?us-ascii?Q?SeW2RQ3NNZ5FUgaXXRkzHs7t0S3PvNtf3t+kv9aH1iplaiznNy0m0wiro+zc?=
 =?us-ascii?Q?REbtpwgusQBjrwKNGk8tTVHTBPRYI1A52RIZLy3bVeX/xvMDXt0KNo+5k7nH?=
 =?us-ascii?Q?biCeO0f6+vhaJajFHr5tqEvrLjkoSr2AXwIPbX8y?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 2e0269d6-9c44-45df-ac23-08dbf67159db
X-MS-Exchange-CrossTenant-AuthSource: LV2PR12MB5869.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 06 Dec 2023 15:38:10.4246
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: 7Gfi/qvtE5jYIiXmCtzMfLFxaDxHTpVOmSKDAn+7jGPPQg0l0wW0SAbPVTUyJ4kh
X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN0PR12MB5980
X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED,
	DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,
	MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE
	autolearn=unavailable autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on pete.vger.email
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org
X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (pete.vger.email [0.0.0.0]); Wed, 06 Dec 2023 07:38:29 -0800 (PST)

On Wed, Dec 06, 2023 at 04:18:05PM +0100, Lorenzo Pieralisi wrote:
> On Wed, Dec 06, 2023 at 11:05:56AM -0400, Jason Gunthorpe wrote:
> > On Wed, Dec 06, 2023 at 02:49:02PM +0000, Catalin Marinas wrote:
> > > On Tue, Dec 05, 2023 at 03:48:22PM -0400, Jason Gunthorpe wrote:
> > > > On Tue, Dec 05, 2023 at 07:24:37PM +0000, Catalin Marinas wrote:
> > > > > On Tue, Dec 05, 2023 at 12:43:18PM -0400, Jason Gunthorpe wrote:
> > > > > > What if we change vfio-pci to use pgprot_device() like it already
> > > > > > really should and say the pgprot_noncached() is enforced as
> > > > > > DEVICE_nGnRnE and pgprot_device() may be DEVICE_nGnRE or NORMAL_NC?
> > > > > > Would that be acceptable?
> > > > > 
> > > > > pgprot_device() needs to stay as Device, otherwise you'd get speculative
> > > > > reads with potential side-effects.
> > > > 
> > > > I do not mean to change pgprot_device() I mean to detect the
> > > > difference via pgprot_device() vs pgprot_noncached(). They put a
> > > > different value in the PTE that we can sense. It is very hacky.
> > > 
> > > Ah, ok, it does look hacky though (as is the alternative of coming up
> > > with a new specific pgprot_*() that KVM can treat differently).
> > > 
> > > BTW, on those Mellanox devices that require different attributes within
> > > a BAR, do they have a problem with speculative reads causing
> > > side-effects? 
> > 
> > Yes. We definitely have had that problem in the past on older
> > devices. VFIO must map the BAR using pgprot_device/noncached() into
> > the VMM, no other choice is functionally OK.
> 
> Were those BARs tagged as prefetchable or non-prefetchable ? I assume the
> latter but please let me know if I am guessing wrong.

I don't know it was quite old HW. Probably.

Just because a BAR is not marked as prefetchable doesn't mean that the
device can't use NORMAL_NC on subsets of it.

Jason