Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Tue, 25 Jul 2023 08:39:51 -0300
From:   Jason Gunthorpe <jgg@nvidia.com>
To:     Benjamin Herrenschmidt <benh@kernel.crashing.org>
Cc:     Lorenzo Pieralisi <lpieralisi@kernel.org>, kvm@vger.kernel.org,
        linux-kernel@vger.kernel.org, alex.williamson@redhat.com,
        osamaabb@amazon.com, linux-pci@vger.kernel.org,
        Clint Sbisa <csbisa@amazon.com>, catalin.marinas@arm.com,
        maz@kernel.org
Subject: Re: VFIO (PCI) and write combine mapping of BARs
Message-ID: <ZL+0h9gvJGTyWKZX@nvidia.com>
References: <2838d716b08c78ed24fdd3fe392e21222ee70067.camel@kernel.crashing.org>
 <ZLD1l1274hQQ54RT@lpieralisi>
 <ZLFBnACjoTbDmKuU@nvidia.com>
 <bc6c3a08e3d0a343fe8317218106609ba159dfe2.camel@kernel.crashing.org>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <bc6c3a08e3d0a343fe8317218106609ba159dfe2.camel@kernel.crashing.org>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?CBcXgRrjIDB4Gppv7rz0tF0WJyBS5v6v2Mair4S3RsUC6wqspUi03n5Czu/R?=
 =?us-ascii?Q?Lk877nlq9WI1h4m951yA7B+JVSoRDSHXopnSvoYLNwNOpMLtfuXgMIB+4D8O?=
 =?us-ascii?Q?rY+P8v4mJ93Hrm4S3UsxZaw3PrxoT3WebqRSWIU1wjhtlTMngS60CyDXtoiF?=
 =?us-ascii?Q?1+JiBx+lIU/z0J0kH9PCL+DgYuAt/lhyPd2llgcxlNs66QoMO6ikVfsVqbxI?=
 =?us-ascii?Q?MgSm/vFzOHWSzDtJyolc/jxQFHbwkker+lkoMTH7kNMx7ni9wqPhBKffRT5w?=
 =?us-ascii?Q?CfSS67sfMpAQJqFraDFgE7y0Se+4qEeXcD6/fFXG6xzFCXhAI8rfXnqhE/Tt?=
 =?us-ascii?Q?j2mw/G9LOQe1St6TNFSKIL8pp4T97LK6bDAI3wUTh7Go9npB4sHQJuvD7ywM?=
 =?us-ascii?Q?D1tLSBaj7sbHs0NU55PAd0vowaKLj7Ynu5OwAWRvy99XoZZF+h3v/QmSaJWV?=
 =?us-ascii?Q?Rn/HoBkpy/Nht9fynuj341+IA/afflA1XVrwaJCUDlWOfPGnioO/yoszofgJ?=
 =?us-ascii?Q?iAzbp40Ne/X4tFg5o+LKReVjyN8ayTlUl+bLO9qUeX6EDskn0RE0VGypgLbr?=
 =?us-ascii?Q?119zQNeLTmPWwRnTOXTG5E5Gx940YZTR1vvfd3YqFJ0ZW+X9DVBrUH8+f2Jo?=
 =?us-ascii?Q?NcWwx1f849lZheqMyM/7+c6WT39809yKEd0CsRGn8BaVnY6lp2hnOYhyiUiU?=
 =?us-ascii?Q?MaVwd3l65rOFw7rHTtuq/5aYtGPw8hbjAZNS7hnFDBNqw8EDt5DG84nnJF42?=
 =?us-ascii?Q?xGX8PQcZ7vCt8mMSZPIDUKUFxZ5MhVHnVLHHIrfFwMafwlXn15Q/1fNYXUwD?=
 =?us-ascii?Q?IbvtOK08XpXpMHTVM2Cq6/gtN0buoUD/Zcw1zVTG4cvnOtWES55/cvVQ5UyC?=
 =?us-ascii?Q?7Ok1I5mLU5h8XejIbTLhG0AXvQnPNUU2n2BUhoyMPq6+9PR1rvQoYU+Tf2tV?=
 =?us-ascii?Q?w9BgAcPSZNfAIPngFpyuuL7EJhF/UXSuZxY00mNpqGl3B94JbTeNER1QpoSL?=
 =?us-ascii?Q?E9a2rBvev6Su4x5w6L0a2GEp47/0lH0Eupj1/v3wEaX2MV5QUjZEzpOiXosw?=
 =?us-ascii?Q?hzgxCz4hZSoK/DqGB/RSIW+DgiVqT+9Tn/8az4BIPUG5axFwjThE6SNdOGKq?=
 =?us-ascii?Q?x4ceK3bSgOdxYJ18XNC8vApgD5eZB+NbZpEORdqhlCxLdq6YNzYtCTOCA78O?=
 =?us-ascii?Q?C5GCsq9dGsA3Pu0Fxd+xbHBix1cAZaeiUkLPFrJAbwTCsfOM7GG+uAL7H7Ec?=
 =?us-ascii?Q?/W3yQabg1JtaaylV6CB/8p24K91YlFHxYJcFnuSmR12kBh03/EfEdM9rVEB4?=
 =?us-ascii?Q?ZL4q+ZA14fQGWfWBoGRm9X2N4XERCmiskI09qlL6b/RzQyBHEKhPRuhw+hXv?=
 =?us-ascii?Q?faOxLu1h7h90TpP1KmbSSb3us1pfBFgCqSlGupBe99yF5qU5p1FGRffvkJZ1?=
 =?us-ascii?Q?Le3LSBL7++8XYqZiiUMRXMbMPZjFA3LivGXfMfX/sUP+LA03ig365DnemrlZ?=
 =?us-ascii?Q?9qbRPUp6pamZxbdIIGCXGjI3n5AAB28hULuBSkPXgBwgs+vjbCUYkbG9MWHB?=
 =?us-ascii?Q?X3tqGzN1OH+gmZQI5oOqN1XQwbb3J8JUMYJMaZ5L?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: ca050d60-a27d-475e-9ac9-08db8d03dc46
X-MS-Exchange-CrossTenant-AuthSource: LV2PR12MB5869.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 25 Jul 2023 11:39:52.4458
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: nxKk3DwDh3suPpYy1D5aLGcqiKAS3ycBERNS90Rp37KgQwBu/x+oyVdq0ayBFMNI
X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH0PR12MB8030
X-Spam-Status: No, score=-1.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FORGED_SPF_HELO,
        RCVD_IN_DNSWL_BLOCKED,RCVD_IN_MSPIKE_H2,SPF_HELO_PASS,SPF_NONE,
        T_SCC_BODY_TEXT_LINE,URIBL_BLOCKED autolearn=no autolearn_force=no
        version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On Tue, Jul 25, 2023 at 04:15:39PM +1000, Benjamin Herrenschmidt wrote:
> > Assuming this is for #2, I think VFIO has fallen into a bit of a trap
> > by allowing userspace to form the mmap offset. I've seen this happen
> > in other subsystems too. It seems like a good idea then you realize
> > you need more stuff in the mmap space and become sad.
> > 
> > Typically the way out is to covert the mmap offset into a cookie where
> > userspace issues some ioctl and then the ioctl returns an opaque mmap
> > offset to use.
> > 
> > eg in the vfio context you'd do some 'prepare region for mmap' ioctl
> > where you could specify flags. The kernel would encode the flags in
> > the cookie and then mmap would do the right thing. Adding more stuff
> > is done by enhancing the prepare ioctl.
> > 
> > Legacy mmap offsets are kept working.
> 
> This indeed what I have in mind. IE. VFIO has legacy regions and add-on
> regions though the latter is currently only exploited by some drivers
> that create their own add-on regions. My proposal is to add an ioctl to
> create them from userspace as "children" of an existing driver-provided
> region, allowing to set different attributes for mmap.

I wouldn't call it children, you are just getting a different mmap
cookie for the same region object.
 
> In the current VFIO the implementation is *entirely* in vfio_pci_core
> for PCI and entirely in vfio_platform_common.c for platform, so while
> the same ioctls could be imagined to create sub-regions, it would have
> to be completely implemented twice unless we do a lot of heavy lifting
> to move some of that region stuff into common code.

The machinery for managing the mmap cookies should be in common code

Jason