Received-SPF: pass (google.com: domain of linux-kernel+bounces-176041-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.199.223 as permitted sender) client-ip=147.75.199.223;
Date: Fri, 10 May 2024 10:13:41 -0700
From: Dan Williams <dan.j.williams@intel.com>
To: Borislav Petkov <bp@alien8.de>, Dan Williams <dan.j.williams@intel.com>
CC: Jonathan Cameron <Jonathan.Cameron@huawei.com>, Shiju Jose
	<shiju.jose@huawei.com>, "linux-cxl@vger.kernel.org"
	<linux-cxl@vger.kernel.org>, "linux-acpi@vger.kernel.org"
	<linux-acpi@vger.kernel.org>, "linux-mm@kvack.org" <linux-mm@kvack.org>,
	"dave@stgolabs.net" <dave@stgolabs.net>, "dave.jiang@intel.com"
	<dave.jiang@intel.com>, "alison.schofield@intel.com"
	<alison.schofield@intel.com>, "vishal.l.verma@intel.com"
	<vishal.l.verma@intel.com>, "ira.weiny@intel.com" <ira.weiny@intel.com>,
	"linux-edac@vger.kernel.org" <linux-edac@vger.kernel.org>,
	"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
	"david@redhat.com" <david@redhat.com>, "Vilas.Sridharan@amd.com"
	<Vilas.Sridharan@amd.com>, "leo.duran@amd.com" <leo.duran@amd.com>,
	"Yazen.Ghannam@amd.com" <Yazen.Ghannam@amd.com>, "rientjes@google.com"
	<rientjes@google.com>, "jiaqiyan@google.com" <jiaqiyan@google.com>,
	"tony.luck@intel.com" <tony.luck@intel.com>, "Jon.Grimm@amd.com"
	<Jon.Grimm@amd.com>, "dave.hansen@linux.intel.com"
	<dave.hansen@linux.intel.com>, "rafael@kernel.org" <rafael@kernel.org>,
	"lenb@kernel.org" <lenb@kernel.org>, "naoya.horiguchi@nec.com"
	<naoya.horiguchi@nec.com>, "james.morse@arm.com" <james.morse@arm.com>,
	"jthoughton@google.com" <jthoughton@google.com>, "somasundaram.a@hpe.com"
	<somasundaram.a@hpe.com>, "erdemaktas@google.com" <erdemaktas@google.com>,
	"pgonda@google.com" <pgonda@google.com>, "duenwen@google.com"
	<duenwen@google.com>, "mike.malvestuto@intel.com"
	<mike.malvestuto@intel.com>, "gthelen@google.com" <gthelen@google.com>,
	"wschwartz@amperecomputing.com" <wschwartz@amperecomputing.com>,
	"dferguson@amperecomputing.com" <dferguson@amperecomputing.com>,
	"wbs@os.amperecomputing.com" <wbs@os.amperecomputing.com>,
	"nifan.cxl@gmail.com" <nifan.cxl@gmail.com>, tanxiaofei
	<tanxiaofei@huawei.com>, "Zengtao (B)" <prime.zeng@hisilicon.com>,
	"kangkang.shen@futurewei.com" <kangkang.shen@futurewei.com>, wanghuiqiang
	<wanghuiqiang@huawei.com>, Linuxarm <linuxarm@huawei.com>, Greg Kroah-Hartman
	<gregkh@linuxfoundation.org>, Jean Delvare <jdelvare@suse.com>, Guenter Roeck
	<linux@roeck-us.net>, Dmitry Torokhov <dmitry.torokhov@gmail.com>
Subject: Re: [RFC PATCH v8 01/10] ras: scrub: Add scrub subsystem
Message-ID: <663e55c59d9d_3d7b429475@dwillia2-mobl3.amr.corp.intel.com.notmuch>
References: <20240508172002.GGZju0QvNfjB7Xm6qL@fat_crate.local>
 <4ceb38897d854cc095fca1220d49a4d2@huawei.com>
 <20240508192546.GHZjvRuvtu0XSJbkmz@fat_crate.local>
 <20240509101939.0000263a@Huawei.com>
 <D9511DC1-1566-473A-A426-111BB1F7F9F0@alien8.de>
 <20240509200306.GAZj0r-h5Tnc0ecIOz@fat_crate.local>
 <663d3e58a0f73_1c0a1929487@dwillia2-xfh.jf.intel.com.notmuch>
 <20240509215147.GBZj1Fc06Ieg8EQfnR@fat_crate.local>
 <663d55515a2d9_db82d2941e@dwillia2-xfh.jf.intel.com.notmuch>
 <20240510092511.GBZj3n9ye_BCSepFZy@fat_crate.local>
Content-Type: text/plain; charset="us-ascii"
Content-Disposition: inline
In-Reply-To: <20240510092511.GBZj3n9ye_BCSepFZy@fat_crate.local>
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?/sP6xoCxtPRFoqXOEmP6yZb5dz0+930KeVuKuwaXXlzuEnwRczPZCjmbGflw?=
 =?us-ascii?Q?7cxUXa8z3zZaWcQ2A94N6DX3Ko16caY1Imn76DNEpWxgbmFFYOpb1Laatp24?=
 =?us-ascii?Q?GBgbVmYHsTNHCerMcNkvBBVQ6eu14TpJ8Xz+MK41ZiA84rB8fmSeNPruZrsF?=
 =?us-ascii?Q?9RqJ67Lu7EHwm0VuZkCdOzS1hap8gsNCClcRAoNr3oEyqh3KPlS5uO176VL/?=
 =?us-ascii?Q?/b39o13ggTXO1jZ1Fe0D8bIBAGwyjA4XN6fndx6y1olMXsl8D37rQznDD918?=
 =?us-ascii?Q?PcqQdNPmrNlApzYQKbH4GSafo/5zmPiM6rEqxsXgXmiFn5/lRhYVeMuHyLwz?=
 =?us-ascii?Q?S1Q+apj6kU6lRNCtyk4zSGdIOw5rm8oi3fUtb672ul9oEs3hVjaj6T2mQg63?=
 =?us-ascii?Q?Kul0xIa7Mj85rjZPFOIw2JoxBqMqO75BOGdMNVcE7oecjInRzvoZRMG8iu/e?=
 =?us-ascii?Q?zUBq7oYl1HcYeoJCtM6FiDk009PtzgbB1UttvaqlaccQ5jYb0tzZ/TEGUJZh?=
 =?us-ascii?Q?qEDBULznzl9YxGa9UlzFYZjRvCbdDy7woRyk3oFZ3yK1CFiRxQTpvNFI2Jas?=
 =?us-ascii?Q?c4xGWkq/OTEzuDGtC3uFANH27CXDdh7AXNH4Xuylt0nRYJXR1t7N6z8UXqxb?=
 =?us-ascii?Q?HuQhKWPWHNx01UTksN+rzz7PsqDe53W1YwcSTeP55DC0Tkj+vYNjL8uWQZzO?=
 =?us-ascii?Q?oj56FMKQmmg2gSc39pwuJ/pZRVfLiEkaalIQzYLw6mDUjQMg00xXU3RHAy8T?=
 =?us-ascii?Q?utl3UNGkVBwFtMNOgzFZumVukgerKQucUVVWwWFlA1JI5vZDd5RVyLm3tmJQ?=
 =?us-ascii?Q?apAxK4U2le20tRhXNYIsOXdj4nfiyLBGWx4KifmC4uV1hzxArfRsXoJ/DnaI?=
 =?us-ascii?Q?zZeOgmwChLvz9I+NkC0enoWYT+RCGZp+8zoK3phDG3ukoaJX9DcDg6+LwcoY?=
 =?us-ascii?Q?X+sJPG282bZnWFqBDoXG4WuXmFw4kv4hPO5eFxiDkxPx257mnKFc9HrJsCI5?=
 =?us-ascii?Q?EibwPskCwNW8NiKF0C1UHmVrmYOEDLa4dJ5Or5reTrf+1Ugw5Spaf7gmpOv0?=
 =?us-ascii?Q?hW7lQr5ZSUm4wU7wuCyrWo9UB2IMtkNQKaPAijmSvxDcMxYoVlRwXPj+l8nk?=
 =?us-ascii?Q?KmKFE3vho3KmXc0rRC+il7wSRGwzyj8JjWZ69Z9WyTnt5G2NuZhDymZDZDvW?=
 =?us-ascii?Q?RdCMMnbhiDf27751H4PZdzEJ2sGDaY03vVJuPQS2n05b5dY0Z0OsGPoLYCUl?=
 =?us-ascii?Q?K4xQ0OR+2G4ioehZ1QwM1GJyhvCU6hMTWkfe0vWEcrFwRCkf5PrW/KgaENZN?=
 =?us-ascii?Q?uH6BVTw4hDTysFOM+UN/o/XWUHFiM5TU3hrAzZT5olQX2RaXYG0wEtr5pm4+?=
 =?us-ascii?Q?qRhKVt1ltnn9kFedIR6F8w6FRtZ4XUOeztACCeM3qWJ1226st0ToVtz4qXwd?=
 =?us-ascii?Q?3+iMqIyjkQIep+jo8nur+ikdkMvc8HgmsTD5qSw+OyZN2oyMo6//kIgHFGkE?=
 =?us-ascii?Q?RznwUWqRuCFNsa5J/qunIi/zNLFwMCbgepSnrQD7eKjOTKUGsf6ppyMVhgSW?=
 =?us-ascii?Q?4PjBCu0RQBxq/3m7R4VMeh3KV4uYDwdhi6fOtbypLZ2KcdOddsdZX6xSBjl9?=
 =?us-ascii?Q?Cw=3D=3D?=
X-MS-Exchange-CrossTenant-Network-Message-Id: f632bc09-df05-49b0-802a-08dc71148cae
X-MS-Exchange-CrossTenant-AuthSource: PH8PR11MB8107.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 10 May 2024 17:13:45.5219
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: rb9hopLlaylbhNY5hZ0EzWk/ZAHGmGJ+X9RRPcYncNN50cgn3RWKAK40szLD84wr/Hs6zZQHaKsA0FxfTdgEg5OJli8z8alf8ljJaRosois=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BL3PR11MB6457
X-OriginatorOrg: intel.com

Borislav Petkov wrote:
> On Thu, May 09, 2024 at 03:59:29PM -0700, Dan Williams wrote:
> > No, at a minimum that's a layering violation. This is a generic library
> > facility that should not care if it is being called for a CXL device or
> > an ACPI device.
> 
> Really?

Yes.

> Because this looks like creating a subsystem for those two newfangled
> scrubbing functionalities which will be present in CXL devices and
> by that ACPI RAS2 thing.
> 
> Because we have a *lot* of hw scrubbing functionality already. Just do:
> 
> git grep "scrub"
> 
> Some of it controls hw scrubbing. If this is a generic facility, does
> this mean that all those older scrubbers should be converted to it?
> 
> Or is this thing going to support only new stuff? I.e., we want to
> disable our scrubbers when doing performance-sensitive workloads and
> reenable them after that but we don't care about old systems?
> 
> And all that other bla about controlling scrubbers from userspace.
> 
> So which is it?

In fact this question matches my reaction to the last posting [1], and
led to a much improved cover letter and the "Comparison of scrubbing
features". To your point there are scrub capabilities already in the
kernel and we would need to make a decision about what to do about them.
I called out NVDIMM ARS as one example and am open to exploring
converting that to the common scrub ABI, but not block this proposal in
the meantime.

For me the proposal can be boiled down to, "here we (kernel community)
are again with 2 new scrub interfaces to add to the kernel. Lets step
back, define a common ABI for ACPI RAS 2 and CXL to stop the
proliferation of scrub ABIs, and then make a decision about when/whether
to integrate legacy scrub facilities into this new interface".

[1]: http://lore.kernel.org/r/65d6936952764_1138c7294e@dwillia2-xfh.jf.intel.com.notmuch

> > I think it works for x86 drivers because the functionality in those
> > modules is wholly contained within that one module. This scrub module is
> > a service library for other modules.
> 
> Well, you have that thing in EDAC. edac_core.ko is that service module
> and the chipset-specific drivers - at least on x86 - use a match_id to
> load only on the systems they should load on.

Which is exactly the same mechanism being defined here. scrub_core.ko is
a service module that would only be requested by an ACPI module or a CXL
module after one of those loads based on their match_id.

> If I had a BIOS table which had "EDAC" in it, I won't load edac_core.ko
> either but there isn't one.
> 
> > It is functionally the wrong place to do the check. When module_init()
> > fails it causes not only the current module to be unloaded but any
> > dependent modules will also fail to load.
> 
> See above. I'm under the assumption that this is using two methods to
> detect scrubbing functionality.

The scrub_core, like edac_core, has no method to detect scrubbing
facility, it is simply a passive library waiting for the first
scrub_device_register() call.

> > Let's take an example of the CXL driver wanting to register with this
> > scrub interface to support the capability that *might* be available on
> > some CXL devices. The cxl_pci.ko module, that houses cxl_pci_driver,
> > grows a call to scrub_device_register(). That scrub_device_register()
> > call is statically present in cxl_pci.ko so that when cxl_pci.ko loads
> > symbol resolution requires scrub.ko to load.
> > 
> > Neither of those modules (cxl_pci.ko or scrub.ko) load automatically.
> > Either udev loads cxl_pci.ko because it sees a device that matches
> > cxl_mem_pci_tbl, or the user manually insmods those modules because they
> > think they know better. No memory wasted unless the user explicitly asks
> > for memory to be wasted.
> 
> The scrub.ko goes and asks the system: "Do you have a CXL device with
> scrubbing functionality?" "Yes: load." "No: ok, won't."

Yeah, that's backwards. CXL enumeration belongs in the CXL driver and
the CXL driver is fully responsible for deciding when to incur the costs
of loading scrub_core.

> > If no CXL devices in the system have scrub capabilities, great, then
> > scrub_device_register() will never be called.
> > 
> > Now, if memory_scrub_control_init() did its own awkward and redundant
> > CXL scan, and fails with "no CXL scrub capable devices found" it would
> > not only block scrub.ko from loading, but also cxl_pci.ko since
> > cxl_pci.ko needs to resolve that symbol to load.
> 
> Why would it fail the scan?

cxl_pci.ko loads based on match_id and cxl_pci_probe() enumerates device
capabilities. My interpretation of your feedback is that
memory_scrub_control_init() should duplicate that cxl_pci_probe()
enumeration?

Assume that it does and memory_scrub_control_init() finds no scrub
facilities in any CXL devices and fails memory_scrub_control_init(). Any
module that links to scrub_device_register() will also fail to load
because module symbol resolution depends on all modules completing init.

> Isn't this fancy GET_SUPPORTED_FEATURES command giving you all info you
> need?

Sure, but that's a driver-probe-time facility, not a module_init-time
facility.

> > Lastly I think drivers based on ACPI tables are awkward. They really
> > need to have an ACPI device to attach so that typical automatic Linux
> > module loading machinery can be used. The fact this function is a
> > subsys_initcall() is a red-flag since nothing should be depending on the
> > load order of a little driver to control scrub parameters.
> 
> Yeah, it is becoming a mess before it has even started.

I assume you do not consider edac_core a mess?

Now, the question of how many legacy scrub interfaces should be
considered in this design out of the gate is a worthwhile discussion. I
am encouraged that this ABI is at least trying to handle more than 1
backend, which makes me feel better that adding a 3rd and 4th might not
be prohibitive.

> So I don't mind if such drivers get loaded as long as doing this is the
> best we can do given the situation. What gets me up the palms, as they
> say in .de, is "just because" and "look, the others do it too."

Which matches what I reacted to on the last posting:

   "Maybe it is self evident to others, but for me there is little in these
    changelogs besides 'mechanism exists, enable it'"

..and to me that feedback was taken to heart with much improved
changelogs in this new posting.

This init time feature probing discussion feels like it was born from a
micommunication / misunderstanding.