Received-SPF: pass (google.com: domain of linux-kernel+bounces-156044-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.199.223 as permitted sender) client-ip=147.75.199.223;
Date: Tue, 23 Apr 2024 17:07:58 -0700
From: Dan Williams <dan.j.williams@intel.com>
To: Jonathan Cameron <Jonathan.Cameron@huawei.com>, Dan Williams
	<dan.j.williams@intel.com>
CC: <linux-cxl@vger.kernel.org>, Sreenivas Bagalkote
	<sreenivas.bagalkote@broadcom.com>, Brett Henning
	<brett.henning@broadcom.com>, Harold Johnson <harold.johnson@broadcom.com>,
	Sumanesh Samanta <sumanesh.samanta@broadcom.com>,
	<linux-kernel@vger.kernel.org>, Davidlohr Bueso <dave@stgolabs.net>, "Dave
 Jiang" <dave.jiang@intel.com>, Alison Schofield <alison.schofield@intel.com>,
	Vishal Verma <vishal.l.verma@intel.com>, Ira Weiny <ira.weiny@intel.com>,
	<linuxarm@huawei.com>, <linux-api@vger.kernel.org>, Lorenzo Pieralisi
	<lpieralisi@kernel.org>, "Natu, Mahesh" <mahesh.natu@intel.com>
Subject: Re: RFC: Restricting userspace interfaces for CXL fabric management
Message-ID: <66284d5e8ac01_690a29431@dwillia2-xfh.jf.intel.com.notmuch>
References: <20240321174423.00007e0d@Huawei.com>
 <66109191f3d6d_2583ad29468@dwillia2-xfh.jf.intel.com.notmuch>
 <20240410124517.000075f2@Huawei.com>
Content-Type: text/plain; charset="us-ascii"
Content-Disposition: inline
In-Reply-To: <20240410124517.000075f2@Huawei.com>
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?fKguvXecjOTkm1aVJykoJszKLni+sXnJQh9KDM1RDoulLVwmp1S/Cd6FjwtQ?=
 =?us-ascii?Q?SBfSPXdvzzX6wyjbsl7aNFCxSH5MivQ1xYldCy8FLK7WMxf4vC7oS3gjxr2w?=
 =?us-ascii?Q?4s+kiyZIwSZooNszFWwOLUt5gQw3InrBW1EPRmRTIiP149D8rCEzbeFWdLPt?=
 =?us-ascii?Q?xGeUCEOQKLewkTUPh0lpHp9OeeAZZoA1SN3+JmDrpvypurTCMnTUn/PpkyEG?=
 =?us-ascii?Q?fOM+oGRkFPDSLFzsHOLdJjpMTyes5kksaaMyuDdg7rV63IdjGVMh2HDREGus?=
 =?us-ascii?Q?XERLFh864Tc3mdwdyEvUaUUFfBpaQZHeB/kvwZh/x1iKI2JHLpiWZLjZ3ID8?=
 =?us-ascii?Q?zF6S8ji+9+Blp9qhZZtJaZIyvXxIePjJ2P08SUsUAuaANbT3ajeQGG0IFRK1?=
 =?us-ascii?Q?khiHCFWXhFln5srS4qmmJOG297JEjqXS5WE7qdiSTs+TNkDgRdFaIcatIUW6?=
 =?us-ascii?Q?ugqxWcj10o7gICXXqnd3oKh2UvmUerQHAw+d4hpkPE0XZpIO2Hrq1V5kDIHo?=
 =?us-ascii?Q?3ubZmLmCZZFVlWlql/lRrT74d0NK/QD9zcFYUI2tOTNZb/UPs7ntFEliEzEj?=
 =?us-ascii?Q?W0fFgOiq1hGfmsWqEnvnitGwkku/qznEie5PG4WQo626sx3Fk+Tckz+TwbqA?=
 =?us-ascii?Q?WyrZt1yAE/h2PbVZj412K+vVR6gxJtX2KcNffdiMzMVyexZKr+SV0MhZHSw5?=
 =?us-ascii?Q?Ww2sf8ycM9UoT0BL2nVoglEf7dtMv8B8v6NlSyMFGM2uxCYB/L9Sz3LOVxGh?=
 =?us-ascii?Q?esNwFTRm8plFQrsiM2/vWP9AGfps9mWj+HrA1b5Yh9+X36sPgPXXdT2PhIZi?=
 =?us-ascii?Q?L4Iy2FTjHz0a4fmFkcQIhAKDFTSurcfQemk/j2tZCAf8ctHM2Pce+OoMcccT?=
 =?us-ascii?Q?WTCWfg8Ar5YOtE/sE27Kv1UnPGPS3FDAz4+3s/CRYb67W6faqGRberjz9Uox?=
 =?us-ascii?Q?BM32ZZ+gyhGCOfzdyfFTm8RVtenTnN77WHEpECriGlOPTLqnduFYR7rJHRvi?=
 =?us-ascii?Q?p0PAfZN28Fo7NrMWXSrOtx8Y+6b7LLg7gZPnbikhHr2e3AQvt37Mwj0eXZhe?=
 =?us-ascii?Q?iC7UPqzHfLxkrmG2mme6ANAuuddq0w62kNZKlPLLwgG2FxFFDLem7E/hj2e/?=
 =?us-ascii?Q?hQT51kL+iyd1tdSFTG7GvM2JmQw2XDIUgrQV8vokug/9mdYiRjMBuOwdZJeW?=
 =?us-ascii?Q?cjubGRWWlSy2xeMY0Frfb04N2/60DekfXsMINs/HDQuM4vrlTgRljJbHIeC0?=
 =?us-ascii?Q?wGMjHCE9k8yQcZOukMwLYbjUo8KL/SgjOXsOgUOIDn0hmoiwKDkLnOSr00Zw?=
 =?us-ascii?Q?RRckxFUqooRRglGM13b6LKVF2EZmhmpBh97KFvPVvg6EjuoZAI50z5yoQQ9K?=
 =?us-ascii?Q?hau9+n1mTCZeVRf+OhPIlKd+FIW+Cq9T72dWxcnVT1P79a6AojLAOSBeR4DF?=
 =?us-ascii?Q?J+QPPknrlv/2Rb8x5l0+5aC6QHSM4C9hKciy95lAWC/03CuLSW1VEwAqFHMx?=
 =?us-ascii?Q?CLsoD0JSSTzSwdrSxzLDapdvxL6/D6hXiCAe0t/bKOg00vC1kW3PzDR4l7YG?=
 =?us-ascii?Q?GDaVOhh+phbdzfyb9My2wpqH0fSNIJTvEK6otd7gWQkUahdk6tJX6HcSd9BL?=
 =?us-ascii?Q?6Q=3D=3D?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 9d7d5b4d-1897-468a-d03d-08dc63f29afa
X-MS-Exchange-CrossTenant-AuthSource: PH8PR11MB8107.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 24 Apr 2024 00:08:01.4922
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: 6xi01a8J3KjUzV8PKh46mKI2f5/oFfBYstVbfhYjHfBD98o813L0/WVdE9a0OEPyKMSlEhuBAIKXCS9U9mC+V8ePyd0FS7repq64nMI/SXs=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA0PR11MB4558
X-OriginatorOrg: intel.com

Jonathan Cameron wrote:
[..]
> > It is not clear to me that this material makes sense to house in
> > drivers/ vs tools/ or even out-of-tree just for maintenance burden
> > relief of keeping the universes separated. What does the Linux kernel
> > project get out of carrying this in mainline alongside the inband code?
> 
> I'm not sure what you mean by in band.  Aim here was to discuss
> in-band drivers for switch CCI etc. Same reason from a kernel point of
> view for why we include embedded drivers.  I'll interpret in band
> as host driven and not inband as FM-API stuff.
>  
> > I do think the mailbox refactoring to support non-CXL use cases is
> > interesting, but only so far as refactoring is consumed for inband use
> > cases like RAS API.
> 
> If I read this right, I disagree with the 'only so far' bit.
> 
> In all substantial ways we should support BMC use case of the Linux Kernel
> at a similar level to how we support forms of Linux Distros.

I think we need to talk in terms of specifics, because in the general
case I do not see the blockage. OpenBMC currently is based on v6.6.28
and carries 136 patches. An additional patch to turn off raw commands
restrictions over there would not even be noticed.

> It may not be our target market as developers for particular parts of
> our companies, but we should not block those who want to support it.

It is also the case that there is a responsibility to build maintainable
kernel interfaces that can be reasoned about, especially with devices as
powerful as CXL that are trusted to host system memory and be caching
agents. For example, I do not want to be in the position of auditing
whether proposed tunnels and passthroughs violate lockdown expectations.

Also, the assertion that these kernels will be built with
CONFIG_SECURITY_LOCKDOWN_LSM=n and likely CONFIG_STRICT_DEVMEM=n, then
the entire user-mode driver ABI is available for use. CXL commands are
simple polled mmio, does Linux really benefit from carrying drivers in
the kernel that the kernel itself does not care about?

[..]
> Switch CCI Driver: PCI driver doing everything beyond the CXL mbox specific bit.
> Type 3 Stack: All the normal stack just with the CXL Mailbox specific stuff factored
>               out. Note we can move different amounts of shared logic in here, but
>               in essence it deals with the extra layer on top of the raw MMPT mbox.
> MMPT Mbox: Mailbox as per the PCI spec.
> RAS API:   Shared RAS API specific infrastructure used by other drivers.

Once the CXL mailbox core is turned into a library for kernel internal
consumers, like RAS API, or CXL accelerators, then it becomes easier to
add a Switch CCI consumer (perhaps as an out-of-tree module in tools/),
but it is still not clear why the kernel benefits from that arrangement.

This is less about blocking developers that have different goals it is
about finding the right projects / places to solve the problem
especially when disjoint design goals are in play and user space drivers
might be in reach.

[..]
> > > The various CXL upstream developers and maintainers may have
> > > differing views of course, but my current understanding is we want
> > > to support 1 and 2, but are very resistant to 3!  
> > 
> > 1, yes, 2, need to see the patches, and agree on 3.
> 
> If we end up with top architecture of the diagrams above, 2 will look pretty
> similar to last version of the switch-cci patches.  So raw commands only + taint.
> Factoring out MMPT is another layer that doesn't make that much difference in
> practice to this discussion. Good to have, but the reuse here would be one layer
> above that.
> 
> Or we just say go for second proposed architecture and 0 impact on the
> CXL specific code, just reuse of the MMPT layer.  I'd imagine people will get
> grumpy on code duplication (and we'll spend years rejecting patch sets that
> try to share the cdoe) but there should be no maintenance burden as
> a result.

I am assuming that the shared code between MMPT and CXL will happen and
that all of the command infrastructure is where centralized policy can
not keep up. If OpenBMC wants to land a driver that consumes the MMPT
core in tools/ that would seem to satisfy both the concerns of mainline
not shipping ABI that host kernels need to strictly reason about while
letting OpenBMC not need to carry out-of-tree patches indefinitely.