Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:3 as permitted sender) client-ip=2620:137:e000::3:3;
Date:   Tue, 5 Dec 2023 13:10:24 -0400
From:   Jason Gunthorpe <jgg@nvidia.com>
To:     Catalin Marinas <catalin.marinas@arm.com>
Cc:     Lorenzo Pieralisi <lpieralisi@kernel.org>,
        Marc Zyngier <maz@kernel.org>, ankita@nvidia.com,
        Shameerali Kolothum Thodi 
        <shameerali.kolothum.thodi@huawei.com>, oliver.upton@linux.dev,
        suzuki.poulose@arm.com, yuzenghui@huawei.com, will@kernel.org,
        ardb@kernel.org, akpm@linux-foundation.org, gshan@redhat.com,
        aniketa@nvidia.com, cjia@nvidia.com, kwankhede@nvidia.com,
        targupta@nvidia.com, vsethi@nvidia.com, acurrid@nvidia.com,
        apopple@nvidia.com, jhubbard@nvidia.com, danw@nvidia.com,
        mochs@nvidia.com, kvmarm@lists.linux.dev, kvm@vger.kernel.org,
        linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org
Subject: Re: [PATCH v2 1/1] KVM: arm64: allow the VM to select DEVICE_* and
 NORMAL_NC for IO memory
Message-ID: <20231205171024.GH2692119@nvidia.com>
References: <20231205033015.10044-1-ankita@nvidia.com>
 <86fs0hatt3.wl-maz@kernel.org>
 <ZW8MP2tDt4_9ROBz@arm.com>
 <20231205130517.GD2692119@nvidia.com>
 <ZW81mT4WqKqtLnid@lpieralisi>
 <20231205144417.GE2692119@nvidia.com>
 <ZW9OtrUa_fPA0_Ns@arm.com>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <ZW9OtrUa_fPA0_Ns@arm.com>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?jzd4bGJuGshU8IvY0HrtoljGc2QoebCTiDfkcHofytqPkuRv8ZqwJ+WmtEzU?=
 =?us-ascii?Q?iB4znFA/Px1WGCVrWUgMvoZthWWCn9A40PYN9PAERDL+jKQX/huJkXJtgEdW?=
 =?us-ascii?Q?jDKGq6cJDQKMg05WWl5wRNhvWE7c+vocG1LxQam3tpYTbNWyLt9pPMHk1Rw6?=
 =?us-ascii?Q?FMYFsLZqr3IOVKVLHJpVWRhnVM3hLX1hmLB9Id4gZRSheScN6LOFL7htcjRv?=
 =?us-ascii?Q?sYSlT4pwWjSuYVzsuk0FZ9zpKI11QnyvMwslf7+PNxAJ9MyAisjlG+NLY3XQ?=
 =?us-ascii?Q?n5xvFLDA+ydEPbmdtWR61OrQLJgYSjRgM66L333bV3NBg++7gjtxqRukfiNm?=
 =?us-ascii?Q?DN1dnPiXOe2/n1hteUoFyJWEnoCVUHA3AKnH21bmQchTlkxKZ2p/MP1AFLWj?=
 =?us-ascii?Q?0NOBlRNDf156tRJvQ9PB3dt9JiHsbr+5Ev0Guv2T/+Vpx8ep+Q6SlQH6dObL?=
 =?us-ascii?Q?/hzeCg1eFDb8N5FLcmj4b91FR59T53vXyQqdvAQwJqnQr+iH/GKoQUMei4cf?=
 =?us-ascii?Q?YQNivYAFF3h1dV+Gpkh8IQNHuNy1dQiuebIqYwydwB1puP/OX4Dk26xTHlwN?=
 =?us-ascii?Q?hkww+JJLgVSdXFadtVYh9QhAvT/z7xYPbm750eAOJ3U6K0l+j3LBkUE+ierD?=
 =?us-ascii?Q?2sDsiW91IziUKq3hE5i0reLney5q6rYTcC15Vbl6K1KtIluz5SgZ2AvOF3hm?=
 =?us-ascii?Q?UnbnFCkPiQfLgolg77V/oa2P0tL2luvnqe6wnewQUY/UY6kof1EpHoY1KOk2?=
 =?us-ascii?Q?BfVnvc3s7teuxxpoFS00CCiMnbE0PidB4mc2uDPK7Mcec7mOwT7d4FwOJhYT?=
 =?us-ascii?Q?cnRwxPMcpT2zJkR04PpCH+mseOfrbdTz636gh+fLH367Z8S2j914e4mrMvDI?=
 =?us-ascii?Q?maIaXH33A7rXZCR5xqx4Iqbc5i2lbSRMvWjHxsSYpAgaeJOXjWgWRpi1MJWv?=
 =?us-ascii?Q?braRTPkUDx4ehJBsH6uVu0l2w6Wj10Ds0BQzri/GR7HGE7uOzbJp33seqvzE?=
 =?us-ascii?Q?3YWOktNF1ApOQkb0KNdFb80l4nMVEI0wqnRg13i9eGCPnBpQKdZZrc2J9uBV?=
 =?us-ascii?Q?3EtdKVNVSVCyGAMDRd1dZXt/EpT9Nn0CY9tc3ZVZpVS/eWUUJFjwPPEARp9D?=
 =?us-ascii?Q?T1riRI2wn+lCUBeepVvw9fJ1ywQk8i9+jR1Bb1DkD6lQNJCdj9gLIsDoI0Sn?=
 =?us-ascii?Q?j/TS+Aks3OKUxpjYuybuMufKT21ruUz2Pg+K7ov/WzJd6HDahSfp5xtyLZx2?=
 =?us-ascii?Q?IePoks+f2fA7mNPyCC/AmZwA81P1dQCwAVuEZMAIJOKjLgdDsuwuBWnAeFG1?=
 =?us-ascii?Q?cWNyj2mf1NfTCZm7CNGzh0AK2xNV5jn0U/YnzUpHA9smX95b7vvtztFOe+xW?=
 =?us-ascii?Q?luVDWVru5GHNXygX6nTR50HSOCer0OTL2Q2q6LV2T1Di1mMBvhwB5s76UASJ?=
 =?us-ascii?Q?E/82bhRHRIcATbxk01U4HS9Di33jFK6RKVGEiEWntaeQKq6LBojuAoGJeu2f?=
 =?us-ascii?Q?jiAbwuCAIuN2NuMEQaL1gVMMtABvzs6Cuar3Vgvl9jTzdpHHvQEv1LIbyRtO?=
 =?us-ascii?Q?6mkBJf0UEyQmG4iz4vFI3B0DJbxWXSF3XdU0MnzM?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 7596ba29-f654-4a3c-c878-08dbf5b512ba
X-MS-Exchange-CrossTenant-AuthSource: LV2PR12MB5869.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 05 Dec 2023 17:10:25.6759
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: EdCBCxa6RbQfuECbb+OYKgRHc1im79+jH/Ogn3KaGF4+YHRfc9EUSzO91koZq8QO
X-MS-Exchange-Transport-CrossTenantHeadersStamped: MW4PR12MB7286
X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED,
	DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,
	MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE
	autolearn=unavailable autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lipwig.vger.email
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org
X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (lipwig.vger.email [0.0.0.0]); Tue, 05 Dec 2023 09:10:47 -0800 (PST)

On Tue, Dec 05, 2023 at 04:24:22PM +0000, Catalin Marinas wrote:
> On Tue, Dec 05, 2023 at 10:44:17AM -0400, Jason Gunthorpe wrote:
> > On Tue, Dec 05, 2023 at 03:37:13PM +0100, Lorenzo Pieralisi wrote:
> > > On Tue, Dec 05, 2023 at 09:05:17AM -0400, Jason Gunthorpe wrote:
> > > > On Tue, Dec 05, 2023 at 11:40:47AM +0000, Catalin Marinas wrote:
> > > > > > - Will had unanswered questions in another part of the thread:
> > > > > > 
> > > > > >   https://lore.kernel.org/all/20231013092954.GB13524@willie-the-truck/
> > > > > > 
> > > > > >   Can someone please help concluding it?
> > > > > 
> > > > > Is this about reclaiming the device? I think we concluded that we can't
> > > > > generalise this beyond PCIe, though not sure there was any formal
> > > > > statement to that thread. The other point Will had was around stating
> > > > > in the commit message why we only relax this to Normal NC. I haven't
> > > > > checked the commit message yet, it needs careful reading ;).
> > > > 
> > > > Not quite, we said reclaiming is VFIO's problem and if VFIO can't
> > > > reliably reclaim a device it shouldn't create it in the first place.
> > > 
> > > I think that as far as device reclaiming was concerned the question
> > > posed was related to memory attributes of transactions for guest
> > > mappings and the related grouping/ordering with device reset MMIO
> > > transactions - it was not (or wasn't only) about error containment.
> > 
> > Yes. It is VFIO that issues the reset, it is VFIO that must provide
> > the ordering under the assumption that NORMAL_NC was used.
> 
> And does it? Because VFIO so far only assumes Device-nGnRnE. Do we need
> to address this first before attempting to change KVM? Sorry, just
> questions, trying to clear the roadblocks.

There is no way to know. It is SOC specific what would be needed.

Could people have implemented their platform devices with a multi-path
bus architecture for the reset? Yes, definately. In fact, I've built
things like that. Low speed stuff like reset gets its own low speed
bus.

If that was done will NORMAL_NC vs DEVICE_nGnRnE make a difference?
I'm not sure. It depends a lot on how the SOC was designed and how
transactions flow on the high speed side. Posting writes, like PCIe
does, would destroy any practical ordering difference between the two
memory types. If the writes are not posted then the barriers in the
TLBI sequence should order it.

Fortunately, if some SOC has this issue we know how to solve it - you
must do flushing reads on all the multi-path interconnect segments to
serialize everything around the reset.

Regardless, getting this wrong is not a functional issue, it causes a
subtle time sensitive security race with VFIO close() that would be
hard to actually hit, and would already require privilege to open a
VFIO device to exploit. IOW, we don't care.

Jason