Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:5 as permitted sender) client-ip=2620:137:e000::3:5;
Date:   Wed, 6 Dec 2023 13:20:35 -0400
From:   Jason Gunthorpe <jgg@nvidia.com>
To:     Catalin Marinas <catalin.marinas@arm.com>
Cc:     Marc Zyngier <maz@kernel.org>, ankita@nvidia.com,
        Shameerali Kolothum Thodi 
        <shameerali.kolothum.thodi@huawei.com>, oliver.upton@linux.dev,
        suzuki.poulose@arm.com, yuzenghui@huawei.com, will@kernel.org,
        ardb@kernel.org, akpm@linux-foundation.org, gshan@redhat.com,
        aniketa@nvidia.com, cjia@nvidia.com, kwankhede@nvidia.com,
        targupta@nvidia.com, vsethi@nvidia.com, acurrid@nvidia.com,
        apopple@nvidia.com, jhubbard@nvidia.com, danw@nvidia.com,
        mochs@nvidia.com, kvmarm@lists.linux.dev, kvm@vger.kernel.org,
        lpieralisi@kernel.org, linux-kernel@vger.kernel.org,
        linux-arm-kernel@lists.infradead.org
Subject: Re: [PATCH v2 1/1] KVM: arm64: allow the VM to select DEVICE_* and
 NORMAL_NC for IO memory
Message-ID: <20231206172035.GU2692119@nvidia.com>
References: <ZW9OSe8Z9gAmM7My@arm.com>
 <20231205164318.GG2692119@nvidia.com>
 <86bkb4bn2v.wl-maz@kernel.org>
 <ZW9ezSGSDIvv5MsQ@arm.com>
 <86a5qobkt8.wl-maz@kernel.org>
 <ZW9uqu7yOtyZfmvC@arm.com>
 <868r67blwo.wl-maz@kernel.org>
 <ZXBlmt88dKmZLCU9@arm.com>
 <20231206151603.GR2692119@nvidia.com>
 <ZXCh9N2xp0efHcpE@arm.com>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <ZXCh9N2xp0efHcpE@arm.com>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?kBVnosITWAeDzgtSXO9TxTToBRB30CpIaElW/4IeCcn18Pex0eicT+qF3hqG?=
 =?us-ascii?Q?B0MCnJK88LZXw62bWoq7dRzzQT5liNnBpyO7oeMNiJnpeQO37r7mKHTYBBt+?=
 =?us-ascii?Q?MwgvTRT4ZcBnYI+K/sXUAA1nfj4BjVgU2WO3FRk7IEW6Xg1Qr2OffwvSSUu3?=
 =?us-ascii?Q?Tn8A7z2A/C+YDsxqPp6vinSzn9gSbijxvQrgcNSQjy9Tlw2G/pEupwnQ6qCC?=
 =?us-ascii?Q?7Pfxc1lYEKEXQL7lJF3eDcC9REYNKfL8RNb1fvw2378Yj7h/jrehyw7mKRWQ?=
 =?us-ascii?Q?YmrM8UHTF3hjg5r1Bkig0/Sdi17O91vUGB8NJyyYoD+hEkUBil8rRfvVEM4x?=
 =?us-ascii?Q?LmCjMfOfPSWiEopdbfoQ12ny+d1PSmT/MaB42G1+DdoxhqJrp8o++AFXGFHJ?=
 =?us-ascii?Q?jX8mNkTLlKXy+w2T3A72W7oe5vENdEz4t6tMb2NuW9icJFTFPNO9f6gbnJVt?=
 =?us-ascii?Q?y+EAVAke5Oz/aQVb5gxcYyLaY23D13iAbBlgs8mzwsj+vj/S3rqyaicCqV34?=
 =?us-ascii?Q?fiPqTsDbO5/sAGnv0z7CSKBPeKS2UPtlCEhflR6zIrqNN0sBskvLvCwI2Duv?=
 =?us-ascii?Q?P3fonk0QzwD/Ha0VnAWaccosswnB5FvIOZLLrddSIn2lLOu1J6kzuaR6Gqvl?=
 =?us-ascii?Q?OsYPqW6ogNOW1GZBYi/pk4GSh+WIBKVJd8Xwe0PSEaBf5LNokr/J7qmjbCpn?=
 =?us-ascii?Q?LkqrHI6+R/6aQlc7UWiTMwSMlVLGCwlQpL2iXBBufpaEucft853AmZz09CzE?=
 =?us-ascii?Q?u629WKTHdlK3ZgaHzNrev9P74r9AQuJVMlU4AbhSXn+i2bzNOGZ2Wv7BN6vM?=
 =?us-ascii?Q?hnjR4zsnTC26yIecBJG9NO3M+PGw+Q/PBZqj9ydGpZxbWBk1SxjbUkTWh1fK?=
 =?us-ascii?Q?bgs2sZwvnZk1PA8D5WzHm890meBYs4MmD0PT7RlXqHvMwXR/W/dHmjSVP5NR?=
 =?us-ascii?Q?EY2Fip+M8rNEpMbiso81s5udxlsp4pqVJny7EpwFON05603V63ljP1kAHkZ7?=
 =?us-ascii?Q?lanIoi2vhXm44e88Hl3S1U4E/J9elhyMRs+NJOFqUKCuWbYot3oRPvyIg1KA?=
 =?us-ascii?Q?Hu9fdlz8vbVDTT5Svp5OQqlgyUl9zpP9KhHUGbyDqvUBsmTeKJsjQcySmaPs?=
 =?us-ascii?Q?D5dygoyVdk/f8mv0OQzZhsD8a5vP5bG9ZaqtyQsFX3qKuSDCFFAy+lkdc1br?=
 =?us-ascii?Q?7xHrlBbLqNPct3k54xaCvAzpN/3x50DIA4/avfsESxF41wMkhfFm5Peq6LNb?=
 =?us-ascii?Q?EQaoFlCPp4mtTEgCgF5JcOj78mxnyV26DRwx4X1OsySceixGY61XIFJwU9so?=
 =?us-ascii?Q?sRmvSTX1zJvfTDsUe7agpy8iM5K6o1ji20RWXB1Qc+kHO/1vpaun6kmq4zJQ?=
 =?us-ascii?Q?1KFG+wzrk2ApBF9JhdgQ7fGjRuAiPJQIavaCCP16U6gjcGxfvhhbMBAHCkNP?=
 =?us-ascii?Q?vwjndTQOCRreL+RvRf+MSHI0wotNA25lHSKdW0929tqisTj6PZA3+zYNSGGt?=
 =?us-ascii?Q?g1OIR28G5vp0BsvPbitBIzbBX/tStzjmVSANVOLRwx5ry2zcDYZJy6cuWdue?=
 =?us-ascii?Q?iCnyt8PvIlrUvI6EA1BgFnBFvTJpbr5rACiBd5YU?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 085ca220-403a-46e2-6cb2-08dbf67fa9a4
X-MS-Exchange-CrossTenant-AuthSource: LV2PR12MB5869.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 06 Dec 2023 17:20:37.2119
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: zDs2OKmci2tMSCfZ8bA5Lyc4+ta19BSfiU3OF/OMxVJf8FpcQ+W1n/b3fXod96Rp
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM6PR12MB4057
X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED,
	DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,
	MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE
	autolearn=unavailable autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on groat.vger.email
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org
X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (groat.vger.email [0.0.0.0]); Wed, 06 Dec 2023 09:21:17 -0800 (PST)

On Wed, Dec 06, 2023 at 04:31:48PM +0000, Catalin Marinas wrote:

> > This would be fine, as would a VMA flag. Please pick one :)
> > 
> > I think a VMA flag is simpler than messing with pgprot.
> 
> I guess one could write a patch and see how it goes ;).

A lot of patches have been sent on this already :(

> > > If we want the VMM to drive this entirely, we could add a new mmap()
> > > flag like MAP_WRITECOMBINE or PROT_WRITECOMBINE. They do feel a bit
> > 
> > As in the other thread, we cannot unconditionally map NORMAL_NC into
> > the VMM.
> 
> I'm not suggesting this but rather the VMM map portions of the BAR with
> either Device or Normal-NC, concatenate them (MAP_FIXED) and pass this
> range as a memory slot (or multiple if a slot doesn't allow multiple
> vmas).

The VMM can't know what to do. We already talked about this. The VMM
cannot be involved in the decision to make pages NORMAL_NC or
not. That idea ignores how actual devices work.

Either the VM decides directly as this patch proposes or the VM does
some new generic trap/hypercall to ask the VMM to change it on its
behalf. The VMM cannot do it independently.

AFAIK nobody wants to see a trap/hypercall solution.

That is why we have been exclusively focused on this approach.

> > > The latter has some benefits for DPDK but it's a lot more involved
> > > with
> > 
> > DPDK WC support will be solved with some VFIO-only change if anyone
> > ever cares to make it, if that is what you mean.
> 
> Yeah. Some arguments I've heard in private and public discussions is
> that the KVM device pass-through shouldn't be different from the DPDK
> case. 

I strongly disagree with this.

The KVM case should be solved without the VMM being aware of what
mappings the VM is doing.

DPDK is in control and can directly ask VFIO to make the correct
pgprot with an ioctl.

You can hear Alex also articulate this position in that video.

> There was some statement in there that for x86, the guests are
> allowed to do WC without other KVM restrictions (not sure whether
> that's the case, not familiar with it).

x86 has a similar issue (Sean was talking about this and how he wants
to fix it) where the VMM can restrict things and on x86 there are
configurations where WC does and doesn't work in VM's too. Depends on
who made the hypervisor. :(

Nobody has pushed hard enough to see it resolved in upstream, but I
understand some of the cloud operator set have their own solutions.

> > We talked about this already, the guest must decide, the VMM doesn't
> > have the information to pre-predict which pages the guest will want to
> > use WC on.
> 
> Are the Device/Normal offsets within a BAR fixed, documented in e.g. the
> spec or this is something configurable via some MMIO that the guest
> does.

No, it is fully dynamic on demand with firmware RPCs.

Jason