Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:5 as permitted sender) client-ip=2620:137:e000::3:5;
Date:   Tue, 5 Dec 2023 15:48:22 -0400
From:   Jason Gunthorpe <jgg@nvidia.com>
To:     Catalin Marinas <catalin.marinas@arm.com>
Cc:     Marc Zyngier <maz@kernel.org>, ankita@nvidia.com,
        Shameerali Kolothum Thodi 
        <shameerali.kolothum.thodi@huawei.com>, oliver.upton@linux.dev,
        suzuki.poulose@arm.com, yuzenghui@huawei.com, will@kernel.org,
        ardb@kernel.org, akpm@linux-foundation.org, gshan@redhat.com,
        aniketa@nvidia.com, cjia@nvidia.com, kwankhede@nvidia.com,
        targupta@nvidia.com, vsethi@nvidia.com, acurrid@nvidia.com,
        apopple@nvidia.com, jhubbard@nvidia.com, danw@nvidia.com,
        mochs@nvidia.com, kvmarm@lists.linux.dev, kvm@vger.kernel.org,
        lpieralisi@kernel.org, linux-kernel@vger.kernel.org,
        linux-arm-kernel@lists.infradead.org
Subject: Re: [PATCH v2 1/1] KVM: arm64: allow the VM to select DEVICE_* and
 NORMAL_NC for IO memory
Message-ID: <20231205194822.GL2692119@nvidia.com>
References: <20231205033015.10044-1-ankita@nvidia.com>
 <86fs0hatt3.wl-maz@kernel.org>
 <ZW8MP2tDt4_9ROBz@arm.com>
 <20231205130517.GD2692119@nvidia.com>
 <ZW9OSe8Z9gAmM7My@arm.com>
 <20231205164318.GG2692119@nvidia.com>
 <ZW949Tl3VmQfPk0L@arm.com>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <ZW949Tl3VmQfPk0L@arm.com>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?lKa6v73Zw8jv66PsdfFRlMglbHrXfNIKk5c4NdbfC/yaqVFz1OhIemsu+WaE?=
 =?us-ascii?Q?AWlLWGoLKPzKmRVCOxBZ1LiprAW+o7AAXiexNCd4NsSAidG64rjyW9dPRldr?=
 =?us-ascii?Q?ZX32wUjbkAKYHTtEJOvWDzn38x5tBAoO9ZvbQ1S6ykDOp0FvgBEPcJDho/0P?=
 =?us-ascii?Q?HqSs7oEvzvgRoe2vQr3y5rr8lO39IYxiMMf0jhH6+eu98JQCFvE97tDAoUHv?=
 =?us-ascii?Q?9Ep1NlBpGg7tYiGK5dO0gxR6bluwgagFmf7qEBhroZnApZ9/BDN09HAHQZ8/?=
 =?us-ascii?Q?beHPKcC6juyOtQt0bMFoSR+09ESiBRZE3jzkIfEfst1J6X8A4XMnj/yDyOUo?=
 =?us-ascii?Q?IBcXxYzjKwboAFJC1CtcnoEOj6t6YFPkKylAzckJA7VG7S9nrTsoA/Th/QfV?=
 =?us-ascii?Q?JiPm/Czms3mcXS0MRBC3vflapvlkquO62OZqWUAB6MRdE4g5EjgwAwXqPHT8?=
 =?us-ascii?Q?r/E/hG3xgeWanJ6MbLFa5ngQ76WEISGTCNNQCty8GOmiySIP5usreTNMZ9n1?=
 =?us-ascii?Q?NPVH2xW5gawjdDr5FFC+EDeAS3nyt85jTdDjUhAZukZu1hOF4Lv2kQiigucR?=
 =?us-ascii?Q?RM2MpSxKCRMsUbcjGtEZXW1ZJiKSF2ngPov0G3upGcYO5sigbSYFiE378yKK?=
 =?us-ascii?Q?c4/DOUQdXVoRjIQDKmtdw/2HZAYHU23RyPGId/dTNtfzir+BBYJHr33bnb4B?=
 =?us-ascii?Q?kyH3QwTzndH1E/iEXAe245PG4CjiQJ/rvXSLRwrFZC7CEasKsb/SpTGIluE1?=
 =?us-ascii?Q?EOcvR5TbsBVD+vIxvp99zxC0goTjxz+K8TWfYIASVczwBt4PJselp3eN7RrR?=
 =?us-ascii?Q?NVuzM7YvHDrQLrqL+SfSILEQL/eHWznPULYgEQws7Z+Ejc0TB7GVtNqiRqPT?=
 =?us-ascii?Q?LjFOoNOvFxr5Le9/AYQ5obJsjjHilN7VhDBZ36BpQ9+7wFxUGpQMMCyXzqR3?=
 =?us-ascii?Q?D/2v8dd1Rl6AD0ciG62SU0Pyz0qDUQMhhYBh9heVcNtr7sbgGIijm7Xul1Uk?=
 =?us-ascii?Q?4sGEcyJ0FP1EOwv3tvWN+WB5/9Vc7zpdh3WIongLQneJKopwglnAO20QcIVP?=
 =?us-ascii?Q?XWiIuqh4GqczJjxpKVDCcaCTyqnWd1f228L0XUzR66kEpyb7siqoo57EZrMx?=
 =?us-ascii?Q?p4MyogLCqDyqrFFEc+r+dS1SrcSL6BshDWhd4Fg99bfKo3o5VWD/dWej+tqe?=
 =?us-ascii?Q?yrG2xV2RxipOGli0pBVej04GsEaySqy57fVYn2UxHfemf0hqoJk+fszEskv9?=
 =?us-ascii?Q?UprKevKl1fwE9cvGIzUqaXXn+v4QFD+max3ZxfZnt6eKHadAqHGR8IwRo8dU?=
 =?us-ascii?Q?M18vrN/r1WtfGWWZN7kmdKOIC3zNoFjo78rptdQsSOQN1Jbb0UWAoTerDvQW?=
 =?us-ascii?Q?RAxCrdjHnKY+Xw3SaoXMztOyOsWj/gYeJ2rXnKSQMYsFa+BYRdbdRCf0/O4s?=
 =?us-ascii?Q?eeWI0n0tElxir1VK1QYuJhHjNLXls0xBigYcg1oTy+CfcA0Zyci7RYyqiYlq?=
 =?us-ascii?Q?yLrITvMiegjoOq3BkQhVroqxE9bBdG1krztvF3KAM3z3OY5GveUH3vLkXUGU?=
 =?us-ascii?Q?Cboup1MxRIRlzx0hB8BO/z0qmcvITSwc2ctfiX5J?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 39a6521c-a71a-4a11-5fd5-08dbf5cb23a8
X-MS-Exchange-CrossTenant-AuthSource: LV2PR12MB5869.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 05 Dec 2023 19:48:23.1337
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: qTc5DakpwSKg8B0HEYKswQbqOGlRrK1+HMzkjxVgQLUmjPj3Tc2sfHMsR3q9KYEL
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BL1PR12MB5779
X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED,
	DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,
	MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE
	autolearn=unavailable autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on groat.vger.email
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org
X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (groat.vger.email [0.0.0.0]); Tue, 05 Dec 2023 11:48:42 -0800 (PST)

On Tue, Dec 05, 2023 at 07:24:37PM +0000, Catalin Marinas wrote:
> On Tue, Dec 05, 2023 at 12:43:18PM -0400, Jason Gunthorpe wrote:
> > On Tue, Dec 05, 2023 at 04:22:33PM +0000, Catalin Marinas wrote:
> > > Yeah, I made this argument in the past. But it's a fair question to ask
> > > since the Arm world is different from x86. Just reusing an existing
> > > driver in a different context may break its expectations. Does Normal NC
> > > access complete by the time a TLBI (for Stage 2) and DSB (DVMsync) is
> > > completed? It does reach some point of serialisation with subsequent
> > > accesses to the same address but not sure how it is ordered with an
> > > access to a different location like the config space used for reset.
> > > Maybe it's not a problem at all or it is safe only for PCIe but it would
> > > be good to get to the bottom of this.
> > 
> > IMHO, the answer is you can't know architecturally. The specific
> > vfio-platform driver must do an analysis of it's specific SOC and
> > determine what exactly is required to order the reset. The primary
> > purpose of the vfio-platform drivers is to provide this reset!
> > 
> > In most cases I would expect some reads from the device to be required
> > before the reset.
> 
> I can see in the vfio_platform_common.c code that the reset is either
> handled by an ACPI _RST method or some custom function in case of DT.
> Let's consider the ACPI method for now, I assume the AML code pokes some
> device registers but we can't say much about the ordering it expects
> without knowing the details. The AML may assume that the ioaddr mapped
> as Device-nRnRE (ioremap()) in the kernel has the same attributes
> wherever else is mapped in user or guests. Note that currently the
> vfio_platform and vfio_pci drivers only allow pgprot_noncached() in
> user, so they wouldn't worry about other mismatched aliases.

AML is OS agnostic. It technically shouldn't assume the OS never used
NORMAL_NC to access the device.

By the time the AML is invoked the VMAs are all revoked and all TLBs
are flushed so I think the mismatched alias problem is gone??

> It can be argued that it's the responsibility of whoever grants device
> access to know the details. However, it would help if we give some
> guidance, any expectations broken if an alias is Normal-NC? It's easier
> to start with PCIe first until we get some concrete request for other
> types of devices.

The issue was about NORMAL_NC access prior to the TLBI continuing to
float in the interconnect and not be ordered strictly before the reset
write.

Which, IMHO, is basically arguing that DSB doesn't order these
operations on some specific SOC - which I view with some doubt.

> > The complexity is my concern, and the disruption to the ecosystem with
> > some of the ideas given.
> > 
> > If there was a trivial way to convey in the VMA that it is safe then
> > sure, no objection from me.
> 
> I suggested a new VM_* flag or some way to probe the iomem_resources for
> PCIe ranges (if they are described in there, not sure). We can invent
> other tree searching for ranges that get registers from the vfio driver,
> I don't think it's that difficult.

I do not like iomem_resource probing on principle :(

A VM_ flag would be fine, but seems wasteful.

> A more complex way is to change vfio to allow Normal mappings and KVM
> would mimic them. You were actually planning to do this for Cacheable
> anyway.

This needs qemu changes, so I very much don't like it.

Cachable is always cachable, it is different thing.
 
> > I would turn it around and ask we find a way to restrict platform
> > devices when someone comes with a platform device that wants to use
> > secure kvm and has a single well defined HW problem that is solved by
> > this work.
> 
> We end up with similar search/validation mechanism, so not sure we gain
> much.

We gain by not doing it, ever. I'm expecting nobody will ever bring it
up.

The vfio-platform drivers all look to be rather old to me, and the
manifestation is, at worst, that a hostile VM that didn't crash the
machine before does crash it now.

I have serious doubts anyone will run into an issue.

> > What if we change vfio-pci to use pgprot_device() like it already
> > really should and say the pgprot_noncached() is enforced as
> > DEVICE_nGnRnE and pgprot_device() may be DEVICE_nGnRE or NORMAL_NC?
> > Would that be acceptable?
> 
> pgprot_device() needs to stay as Device, otherwise you'd get speculative
> reads with potential side-effects.

I do not mean to change pgprot_device() I mean to detect the
difference via pgprot_device() vs pgprot_noncached(). They put a
different value in the PTE that we can sense. It is very hacky.

Jason