Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:4 as permitted sender) client-ip=2620:137:e000::3:4;
Date:   Wed, 6 Dec 2023 11:05:56 -0400
From:   Jason Gunthorpe <jgg@nvidia.com>
To:     Catalin Marinas <catalin.marinas@arm.com>
Cc:     Marc Zyngier <maz@kernel.org>, ankita@nvidia.com,
        Shameerali Kolothum Thodi 
        <shameerali.kolothum.thodi@huawei.com>, oliver.upton@linux.dev,
        suzuki.poulose@arm.com, yuzenghui@huawei.com, will@kernel.org,
        ardb@kernel.org, akpm@linux-foundation.org, gshan@redhat.com,
        aniketa@nvidia.com, cjia@nvidia.com, kwankhede@nvidia.com,
        targupta@nvidia.com, vsethi@nvidia.com, acurrid@nvidia.com,
        apopple@nvidia.com, jhubbard@nvidia.com, danw@nvidia.com,
        mochs@nvidia.com, kvmarm@lists.linux.dev, kvm@vger.kernel.org,
        lpieralisi@kernel.org, linux-kernel@vger.kernel.org,
        linux-arm-kernel@lists.infradead.org
Subject: Re: [PATCH v2 1/1] KVM: arm64: allow the VM to select DEVICE_* and
 NORMAL_NC for IO memory
Message-ID: <20231206150556.GQ2692119@nvidia.com>
References: <20231205033015.10044-1-ankita@nvidia.com>
 <86fs0hatt3.wl-maz@kernel.org>
 <ZW8MP2tDt4_9ROBz@arm.com>
 <20231205130517.GD2692119@nvidia.com>
 <ZW9OSe8Z9gAmM7My@arm.com>
 <20231205164318.GG2692119@nvidia.com>
 <ZW949Tl3VmQfPk0L@arm.com>
 <20231205194822.GL2692119@nvidia.com>
 <ZXCJ3pVbKuHJ3LTz@arm.com>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <ZXCJ3pVbKuHJ3LTz@arm.com>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?kLC+1oeo4TLaUFGqY5IPaK1oCQuN+nnyVk+5FlHQRexhlYKd84bjyoCXfJpU?=
 =?us-ascii?Q?9JkMQySWZHExPUx2rF4Lema1Q0y7o13QvhctvBxjntrlxyoJNBT846IVnbkz?=
 =?us-ascii?Q?KGiy71zPvKewTsJdNErOkjh4x69iTcOXwmRtzpv07CxFmDE7Ri2uSujGQtGT?=
 =?us-ascii?Q?lHKXGX+Ysm7T05Cudj6cixC4JD0nO/CcbgGKZVFk+T+kgDceaXBJADnaKIyi?=
 =?us-ascii?Q?8+McFFVbOo5AqYZMqvm59/GWIovylGfg/ygsLwBim009PYf1hWThoQKzSJ91?=
 =?us-ascii?Q?K6Y3DRW/PUAV1dp24UXSbLBCWo+rN2rYD09BH+EACPXxlBJYbfCnp/NNrnnY?=
 =?us-ascii?Q?C4cxRnJKLjfdRieRO1mmrEikCWCMfYs4tRBAWdlKpzAmX2OxKqoIsR9rwhNt?=
 =?us-ascii?Q?whvpSPB0jyqcP0+Huurcc9vLFnslAIQAYT7N6I6J+FbmwmG7iN7A0/5zecmv?=
 =?us-ascii?Q?JASI3Q6EIxXm2PQy22EmzFLTrlqJ1vJ9MlZscNpF37BEnK43HaAx9b9DQ+HQ?=
 =?us-ascii?Q?U0qy9X6gB/DG3N5X6bSYDq0N/vhysCP3bsWIBW808W0WzGSAQGzhkAtiB0bl?=
 =?us-ascii?Q?VxUXsH8y01/iLQIKzfH43z+YGeEo4jsLvf/bl5ricwFROEiRfhHvELX3Lpea?=
 =?us-ascii?Q?7KunnJzL3MidsBC/RACfxSfPBiq/WfkN6K90mtMkhdOTk1BkDhNxDpqz6gMc?=
 =?us-ascii?Q?f6qbTOXdzcBYnqbpdJzrq3kiYRxMKfKyqjg1glTTmbwu7uu80P8FVE3all2h?=
 =?us-ascii?Q?5NOGAPmR0UamUd0cCQkpt22B+1BI+7Qy1FcW2MNrBhFsdGKN9PQfRVpGqrG0?=
 =?us-ascii?Q?UE3r6iUI5bTfCgqxsamvOna4CRjI9BtamUJF4SkeBD9kQFQc6W+yFKU1WIPv?=
 =?us-ascii?Q?mj92RZ8+WRg45yn/1od4CB+ULdXbvjOMIJSeua++cyZlh/ViZCw42M2vQyY2?=
 =?us-ascii?Q?bJ2lHHf21qGt64IkaIsmZrohoMt3WpgsacxzI0hwAAAmwag8QnqzHvKEsaMr?=
 =?us-ascii?Q?OQx3dz/s6jfY85IQYLD3zHgxEel0MzOlwEKwN7V9XfxQw3qZaLZVd1BZzznY?=
 =?us-ascii?Q?yZyt4QZ4IM9tNcc18a5+w/mP9gvHagP7ehWUqhOgB2vMZg2+g4sMV3iHNoox?=
 =?us-ascii?Q?yq56eB/rwHolXl1BAEyHdyiExPGcK24hpTRyYAHAh/nDkMYvsdpfTwy6hzv2?=
 =?us-ascii?Q?16FuIPGI5uDN69kzXQOiYc5xsRlJWTefCBjzXSywJrE4Exr268Gn0CklJrVE?=
 =?us-ascii?Q?oC3O/oHqtjFs9zVAhBHAYcA7+DBEpjupbhU/17oiqDAIfyK6tQuhKwK6AH4I?=
 =?us-ascii?Q?sROUUGRf64p+jgWieozq2KFUcfTb7PqCIhNKwbED2oIyRKBnJA1orvZR3guf?=
 =?us-ascii?Q?udPJiNl+eyCkTsfcXE2d4S1rX0D6CVY53RDk9qZWTYEghDriI9HDXRSg3Kcr?=
 =?us-ascii?Q?4a6HkZGMjhhfBCXKpg8NEfa2Xa5fYNMu7ekSMR/vjaTg1dHa8Bae9Kw7vhy7?=
 =?us-ascii?Q?C+mhSGaMN67L1xuV61ClAXhpHIo4Nef64Rtd6o73pvkJkaXsFd4uGeMv47Td?=
 =?us-ascii?Q?gjRi4bEXzRID9WLRI5ON2TmvKHSSig0qB7E8b3Ig?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 0f0c161b-0545-49fd-7d85-08dbf66cd986
X-MS-Exchange-CrossTenant-AuthSource: LV2PR12MB5869.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 06 Dec 2023 15:05:57.1507
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: 4z7HUl+FBfEz+aF1H4b5MiGV63kGqlXEJgftgXZMI5AqJyulYWdKQPxBPVOeI8jo
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS7PR12MB5816
X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED,
	DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,
	MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE
	autolearn=unavailable autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on howler.vger.email
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org
X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (howler.vger.email [0.0.0.0]); Wed, 06 Dec 2023 07:06:18 -0800 (PST)

On Wed, Dec 06, 2023 at 02:49:02PM +0000, Catalin Marinas wrote:
> On Tue, Dec 05, 2023 at 03:48:22PM -0400, Jason Gunthorpe wrote:
> > On Tue, Dec 05, 2023 at 07:24:37PM +0000, Catalin Marinas wrote:
> > > On Tue, Dec 05, 2023 at 12:43:18PM -0400, Jason Gunthorpe wrote:
> > > > What if we change vfio-pci to use pgprot_device() like it already
> > > > really should and say the pgprot_noncached() is enforced as
> > > > DEVICE_nGnRnE and pgprot_device() may be DEVICE_nGnRE or NORMAL_NC?
> > > > Would that be acceptable?
> > > 
> > > pgprot_device() needs to stay as Device, otherwise you'd get speculative
> > > reads with potential side-effects.
> > 
> > I do not mean to change pgprot_device() I mean to detect the
> > difference via pgprot_device() vs pgprot_noncached(). They put a
> > different value in the PTE that we can sense. It is very hacky.
> 
> Ah, ok, it does look hacky though (as is the alternative of coming up
> with a new specific pgprot_*() that KVM can treat differently).
> 
> BTW, on those Mellanox devices that require different attributes within
> a BAR, do they have a problem with speculative reads causing
> side-effects? 

Yes. We definitely have had that problem in the past on older
devices. VFIO must map the BAR using pgprot_device/noncached() into
the VMM, no other choice is functionally OK.

Only some pages can safely tolerate speculative reads and the guest
driver knows which they are.

Jason