Received-SPF: pass (google.com: domain of linux-kernel+bounces-159076-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) client-ip=147.75.80.249;
Date: Thu, 25 Apr 2024 12:25:35 -0700
From: Dan Williams <dan.j.williams@intel.com>
To: Jonathan Cameron <Jonathan.Cameron@huawei.com>, Dan Williams
	<dan.j.williams@intel.com>
CC: <linux-cxl@vger.kernel.org>, Sreenivas Bagalkote
	<sreenivas.bagalkote@broadcom.com>, Brett Henning
	<brett.henning@broadcom.com>, Harold Johnson <harold.johnson@broadcom.com>,
	Sumanesh Samanta <sumanesh.samanta@broadcom.com>,
	<linux-kernel@vger.kernel.org>, Davidlohr Bueso <dave@stgolabs.net>, "Dave
 Jiang" <dave.jiang@intel.com>, Alison Schofield <alison.schofield@intel.com>,
	Vishal Verma <vishal.l.verma@intel.com>, Ira Weiny <ira.weiny@intel.com>,
	<linuxarm@huawei.com>, <linux-api@vger.kernel.org>, Lorenzo Pieralisi
	<lpieralisi@kernel.org>, "Natu, Mahesh" <mahesh.natu@intel.com>,
	<gregkh@linuxfoundation.org>
Subject: Re: RFC: Restricting userspace interfaces for CXL fabric management
Message-ID: <662aae2fe4887_a96f294bf@dwillia2-mobl3.amr.corp.intel.com.notmuch>
References: <20240321174423.00007e0d@Huawei.com>
 <66109191f3d6d_2583ad29468@dwillia2-xfh.jf.intel.com.notmuch>
 <20240410124517.000075f2@Huawei.com>
 <66284d5e8ac01_690a29431@dwillia2-xfh.jf.intel.com.notmuch>
 <20240425123344.000001a9@Huawei.com>
 <662a826dbeeff_b6e02943c@dwillia2-mobl3.amr.corp.intel.com.notmuch>
 <20240425182605.00005f22@Huawei.com>
Content-Type: text/plain; charset="us-ascii"
Content-Disposition: inline
In-Reply-To: <20240425182605.00005f22@Huawei.com>
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?kBuZ+2NBu+82GQZkDXbzmNXg0NuKcW3QS7ntLRuPtY1OoJgnCLbPQ0qjq+qu?=
 =?us-ascii?Q?PLkVkc2LvcxTRFQRnaWB9eM+D1sFxzpaXCcp56fAK6vUnQFDpRWFTkqhMgHY?=
 =?us-ascii?Q?yJlCk76o6VdvWGeMHDPJphj/csXjwznbsPIlR7TNIdlLgIX9ck/DJwlJ3gLI?=
 =?us-ascii?Q?+mgTIsVVcitwT4M2cO42mwAAr15VVfLJ3gdCyR8bfAEf8aZaqEFo2RXM4hpo?=
 =?us-ascii?Q?NbZkBZ59I+ly3AtSY7bXB046PDGpQGw/K8DDrdNOkzPzSzOECu7/p3Xsm6yH?=
 =?us-ascii?Q?1H0V4Gl8es5zpsspgjSlSOhRFE4XmjiiMj3kk1P6fiX1EvuiI+tILGmC+eVW?=
 =?us-ascii?Q?MPr6Jhl9NfS2H+TSqHag+vEHune6oRVEXMtozUCC9hNd1OHGhplqqJSfh1mo?=
 =?us-ascii?Q?1AjwOzvEwF/rZc+mWM5SuI6jn8CLywfMuLi2EFe+Dyko3U6ju9r0nA+WW3iL?=
 =?us-ascii?Q?ga7kQuBzzyG3PtghdHWWBxE+uksfMmJF60Vg5cqN98G0BR0Y31nto736sj2K?=
 =?us-ascii?Q?H7sWpv6EtwXoHWJ8adXlcaJ8xo3DXAX1GceevNKp7IUEd/Rizfc4LNpVFUHD?=
 =?us-ascii?Q?09GXRwDenDFaVUEhY5Oo/zOk9tIAYG17dxWOaZkvoW0cU/fw7o9dScAkrsQ+?=
 =?us-ascii?Q?GX2TctoWC2h1KedBlFv9bJQWsc4JjVv/dVKHjta3QNnZM3HUugs61U2Oxg2D?=
 =?us-ascii?Q?/4O4vDeMAZfHP4MMX2HXsx+ihdOZg6815XyRK2+YBwYnxEekDUK5+BKoaXoO?=
 =?us-ascii?Q?HOs4fzUb6cToJRZKpfyBhiSyLYL/G+EJUIfOeb2rZDPFSFuCnf4Jw2VD6FWg?=
 =?us-ascii?Q?4aXNkstgUJA2BBOJJaDQ1wfmtI4qMKFkeEljLTWOr1mkWI2wM3NfP2q8FGEU?=
 =?us-ascii?Q?W6yS2YhKvjgVR2ow61gmCe8C0W1Gv6+WRKECSD1YtoH5WJIji1Z8KnrNMmMC?=
 =?us-ascii?Q?eo9EEaxfw5fT/wrdXuC6L9kuPMKObxo8EXOcoh73hLAWWc0J7sU+UTGcnK1u?=
 =?us-ascii?Q?bp2ILVBOre8s4cuM+v2CscznhWV2Di4e1waYmjYsoaGmw9NVAtVkmNH2EgLt?=
 =?us-ascii?Q?KizJAxp6EN7WwESHc9tEJMdqnOTup2x2eQWClryrXc7NlDgrpCKC3n023Dxe?=
 =?us-ascii?Q?hfbNZojRsRc2jKStKDFkgymPJnJgechzqElHRJiXN2g2HxtNxCHe0bBKuqp4?=
 =?us-ascii?Q?yuW76uftu8x9TZmfpyV7gdVpHzLiwRKqi2Ct9x//MYZ4R+oaCJ+DxVVJXBiH?=
 =?us-ascii?Q?9k9gDDDma6jktH9oMxPIt7vd8/s3mKatJbPoo/enPSjB4T89lBoklyZ5ApTg?=
 =?us-ascii?Q?BwP062ZQwEXL+c75MvU02qRzYiSogf32Hyuav0hIhRtdWc9OH0LSLT/z+24B?=
 =?us-ascii?Q?LWFa5IVxO0AxmLZxdswkFde51X2VDYVYjD/+0jBOhcqVdp1vcKl/2fCl07rJ?=
 =?us-ascii?Q?cAGFCpdXO7OkztSzrybcMn+ondQ14CK3+4V+cKIxPvvVVM0Uj51CMPS94L07?=
 =?us-ascii?Q?0VaZ9Pz+hC8oe4sc2BJXRHvRIfmKj1wy5YKepqtVVgZe9yuu2sqi8p5Fw44L?=
 =?us-ascii?Q?kTo7B4ALZxvs3nbtgdpg8ShbPUyedBJkB5ilcbTdDPaQzO1ZZ2tO9lozheFN?=
 =?us-ascii?Q?lw=3D=3D?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 223fa650-020a-4836-78ff-08dc655d7d53
X-MS-Exchange-CrossTenant-AuthSource: PH8PR11MB8107.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 25 Apr 2024 19:25:39.0493
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: yuSPI6HdvWCOsYiyJJEaGB/7v0TZvcelqtq54rdeeV6Cy9CpoOxDFC1jLzL3lnwCGy+LsRTqJcM5QHibuq3fCqKHds2z1Jqz3DC2WP2g6lM=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM4PR11MB8228
X-OriginatorOrg: intel.com

Jonathan Cameron wrote:
> Dan Williams <dan.j.williams@intel.com> wrote:
> > The minimum requirement for justifying an in kernel driver is that
> > something else in the kernel consumes that facility. So, again, I want
> > to get back to specifics what else in the kernel is going to leverage
> > the Switch CCI mailbox?
> 
> Why?  I've never heard of such as a requirement and numerous drivers
> provide fairly direct access to hardware. Sometimes there is a subsystem
> aiding the data formatting etc, but fundamentally that's a convenience.
> 
> Taking this to a silly level, on this basis all networking drivers would
> not be in the kernel.  They are there mainly to provide userspace access to
> a network.

Networking is an odd choice to bring into this discussion because that
subsystem has a long history of wrestling with the "kernel bypass"
concern. It has largely been able to weather the storm of calls to get
out of the way and let vendor drivers have free reign.

The AF_XDP socket family was the result of finding a path to let
userspace networking stacks build functionality without forfeiting the
relevance and ongoing collaboration on the in-kernel stack.

> Any of the hardware access subsystems such hwmon, input, IIO
> etc are primarily about providing a convenient way to get data to/from
> a device.  They are kernel drivers because that is the cleaner path
> for data marshaling, interrupt handling etc.

Those are drivers supporting a subsystem to bring a sane kernel
interface to front potenitally multiple vendor implementations of
similar functionality.

They are not asking for kernel bypass facilities that defeat the purpose
of ever talking to the kernel community again for potentially
system-integrity violating functionality behind disparate vendor
interfaces.

> In kernel users are a perfectly valid reason to have a kernel driver,
> but it's far from the only one.  None of the AI accelerators have in kernel
> users today (maybe they will in future). Sure there are other arguments
> that mean only a few such devices have been upstreamed, but it's not
> that they need in kernel users. If it's really an issue I'll just submit
> it to driver/misc and Greg can take a view on whether it's an acceptable
> device to have driver for... (after he's asked the obvious question of
> why aren't the CXL folk taking it!) +cc Greg to save providing info later.

AI accelerators are heavy consumers of the core-mm you can not
reasonably coordinate with the core-mm from userspace.

If the proposal is to build a new CXL Fabric Management subsystem with
proper ABIs and openly defined command sets that will sit behind thought
out kernel interfaces then I can get on board with that.

Where I am stuck currently is the assertion that step 1 is "build ioctl
passthrough tunnels with 'do anything you want and get away with it'
semantics".

Recall that the current restriction for raw commands was to encourage
vendor collaboration and building sane kernel interfaces, and that
distros would enable it in their "debug" kernels to enable hardware
validation test benches. If the assertion is "that's too restrictive,
enable a vendor ecosystem based on kernel bypass" that goes too far.

> For background this is a PCI function with a mailbox used for switch
> configuration. The mailbox is identical to the one found on CXL type3
> devices. Whole thing defined in the CXL spec. It gets a little complex
> because you can tunnel commands to devices connected to the switch,
> potentially affecting other hosts.  Typical Linux device doing this
> would be a BMC, but there have been repeated questions about providing
> a subset of access to any Linux system (avoiding the foot guns)
> Whole thing fully discoverable - proposal is a standard PCI driver.
> 
> > The generic-Type-3-device mailbox has an in kernel driver because the
> > kernel has need to send mailbox commands internally and it is
> > fundamental to RAS and provisioning flows that the kernel have this
> > coordination. What are the motivations for an in-band Switch CCI command
> > submission path?
> > 
> > It could be the case that you have a self-evident example in mind that I
> > have thus far failed to realize.
> > 
> 
> There are possibilities, but for now it's a transport driver just like
> MCTP etc with a well defined chardev interface, with documented ioctl
> interface etc (which I'd keep inline with one the CXL mailbox uses
> just to avoid reinventing the wheel - I'd prefer to use that directly
> to avoid divergence but I don't care that much).
> 
> As far as I can see, with the security / blast radius concern alleviated
> by disabling this if lockdown is in use + taint for unaudited commands
> (and a nasty sounding config similar to the cxl mailbox one)
> there is little reason not to take such a driver into the kernel.
> It has next to no maintenance impact outside of itself and a bit of
> library code which I've proposed pushing down to the level of MMPT
> (so PCI not CXL) if you think that is necessary. 
> 
> We want interrupt handling and basic access controls / command
> interface to userspace.
> 
> Apologies if I'm grumpy - several long days of battling cpu hotplug code.

Again, can we please get back to the specifics of the commands to be
enabled here? I am open to CXL Fabric Management as a first class
citizen, I am not currently open to CXL Fabric Management gets to live
in the corner of the kernel that is unreviewable because all it does is
take opaque ioctl blobs and marshal them to hardware.