Received: by 2002:a05:6358:c692:b0:131:369:b2a3 with SMTP id fe18csp431309rwb; Tue, 25 Jul 2023 18:57:32 -0700 (PDT) X-Google-Smtp-Source: APBJJlE55gNzP/5wRLgssFyrHJlCt3wrOonaqgD/hpiNtmDy4SU7Vhag2Qj8xyCpPV4ok6gUB0W5 X-Received: by 2002:ac2:5f48:0:b0:4f1:4cdc:ec03 with SMTP id 8-20020ac25f48000000b004f14cdcec03mr325774lfz.18.1690336652540; Tue, 25 Jul 2023 18:57:32 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1690336652; cv=none; d=google.com; s=arc-20160816; b=X8cSMopFniRD1dhBowRAsEuH2S4IC2/RqVqlxK+LQ+tExv4UfkNZT3COknFyHWlMZK mQlpl7c3nLylGDD4QPJT1DqHSJu2rVkipATGTmaR3JwVM2xMwAS/cxjsF4O94BkGJtVP FZu/a4uoW0WUY+92vOUnoqUbWcnnJvi92H+nkMfWhWHERIxhR7cNrFLDcy7KASLHLQ9S v7SVEd5xPv/qAeWrnm8TRlAZ8v5gv/MpDVJWGS3B5XRtC08Lng+uFxeUK4KuSsLuEV0K EKJtxUKpoxerbRZ1h9RE10d03og+ndd62RZQP4AP0HaDvj2U2o9tuKFYAepSbwFWcdQS 9DWg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:mime-version:user-agent :content-transfer-encoding:references:in-reply-to:date:cc:to:from :subject:message-id; bh=9TyciI+WV8hmLCqU/zk86fxg7ohgh6xr9bDmNGQYwUc=; fh=eTKlslMHj565mO788fwNOA3fr9sxhJ7Z9panm/H5c0g=; b=Swmy7lUuaYzpX3MyneiLq2CkUGaCiPsrBn+AmCQdsYhwPpdeYubht+fOo9RQoT7uZ3 uL0hqfL9qkYTSv0y3akZ0UgqX1kFw+J6eaFt2+i2Uymc5lUbl6NrXjfOqXKP8CX3t30Q ArGaDK9emOkyo5aHPcghZHYpPwa+2eCdu6v5RqWy4LjmQQML6a3ifDC47xX1elCXo6k0 YY3HHhcUyH36ke6YCt420KE5zQvqc6GrvvkEHbfk9pmMrX8YvbX9bvo/u0Mif7STIMJd cx6ZbVGe2ghdNgQ4596Fz5d5vrm29/A0vPA18G4kUbYg1ZY9ygiz2gtOflZwnupFwWGc TOkg== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id p8-20020aa7d308000000b0052229e19114si4633708edq.471.2023.07.25.18.57.08; Tue, 25 Jul 2023 18:57:32 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229654AbjGZBUj convert rfc822-to-8bit (ORCPT + 99 others); Tue, 25 Jul 2023 21:20:39 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37928 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229483AbjGZBUi (ORCPT ); Tue, 25 Jul 2023 21:20:38 -0400 Received: from gate.crashing.org (gate.crashing.org [63.228.1.57]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id DF84AB0; Tue, 25 Jul 2023 18:20:36 -0700 (PDT) Received: from [IPv6:::1] (localhost.localdomain [127.0.0.1]) by gate.crashing.org (8.14.1/8.14.1) with ESMTP id 36Q1JIxL008573; Tue, 25 Jul 2023 20:19:19 -0500 Message-ID: <012cb915ae0ce3a32f1608400cf6389a03f2aa7d.camel@kernel.crashing.org> Subject: Re: VFIO (PCI) and write combine mapping of BARs From: Benjamin Herrenschmidt To: Jason Gunthorpe Cc: Lorenzo Pieralisi , kvm@vger.kernel.org, linux-kernel@vger.kernel.org, alex.williamson@redhat.com, osamaabb@amazon.com, linux-pci@vger.kernel.org, Clint Sbisa , catalin.marinas@arm.com, maz@kernel.org Date: Wed, 26 Jul 2023 11:19:18 +1000 In-Reply-To: References: <2838d716b08c78ed24fdd3fe392e21222ee70067.camel@kernel.crashing.org> Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 8BIT User-Agent: Evolution 3.44.4-0ubuntu1 MIME-Version: 1.0 X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,SPF_HELO_PASS, SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Tue, 2023-07-25 at 08:39 -0300, Jason Gunthorpe wrote: > On Tue, Jul 25, 2023 at 04:15:39PM +1000, Benjamin Herrenschmidt wrote: > > > Assuming this is for #2, I think VFIO has fallen into a bit of a trap > > > by allowing userspace to form the mmap offset. I've seen this happen > > > in other subsystems too. It seems like a good idea then you realize > > > you need more stuff in the mmap space and become sad. > > > > > > Typically the way out is to covert the mmap offset into a cookie where > > > userspace issues some ioctl and then the ioctl returns an opaque mmap > > > offset to use. > > > > > > eg in the vfio context you'd do some 'prepare region for mmap' ioctl > > > where you could specify flags. The kernel would encode the flags in > > > the cookie and then mmap would do the right thing. Adding more stuff > > > is done by enhancing the prepare ioctl. > > > > > > Legacy mmap offsets are kept working. > > > > This indeed what I have in mind. IE. VFIO has legacy regions and add-on > > regions though the latter is currently only exploited by some drivers > > that create their own add-on regions. My proposal is to add an ioctl to > > create them from userspace as "children" of an existing driver-provided > > region, allowing to set different attributes for mmap. > > I wouldn't call it children, you are just getting a different mmap > cookie for the same region object. I though they could be subsets but that might be overkill. > > In the current VFIO the implementation is *entirely* in vfio_pci_core > > for PCI and entirely in vfio_platform_common.c for platform, so while > > the same ioctls could be imagined to create sub-regions, it would have > > to be completely implemented twice unless we do a lot of heavy lifting > > to move some of that region stuff into common code. > > The machinery for managing the mmap cookies should be in common code Ok. I'll whip up a POC within vfio_pci only intially to test the concept and to agree on the API, then look at how we can clean all that up. Cheers, Ben.