Received-SPF: pass (google.com: domain of linux-kernel+bounces-165861-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) client-ip=2604:1380:45e3:2400::1;
Date: Wed, 1 May 2024 16:49:24 -0700
From: Ira Weiny <ira.weiny@intel.com>
To: Jonathan Cameron <Jonathan.Cameron@huawei.com>, <ira.weiny@intel.com>
CC: Dave Jiang <dave.jiang@intel.com>, Fan Ni <fan.ni@samsung.com>, "Navneet
 Singh" <navneet.singh@intel.com>, Dan Williams <dan.j.williams@intel.com>,
	Davidlohr Bueso <dave@stgolabs.net>, Alison Schofield
	<alison.schofield@intel.com>, Vishal Verma <vishal.l.verma@intel.com>,
	<linux-btrfs@vger.kernel.org>, <linux-cxl@vger.kernel.org>,
	<linux-kernel@vger.kernel.org>, Chris Mason <clm@fb.com>, Josef Bacik
	<josef@toxicpanda.com>, David Sterba <dsterba@suse.com>
Subject: Re: [PATCH 00/26] DCD: Add support for Dynamic Capacity Devices (DCD)
Message-ID: <6632d503f3ae5_e1f58294df@iweiny-mobl.notmuch>
References: <20240324-dcd-type2-upstream-v1-0-b7b00d623625@intel.com>
 <20240404184901.00002104@Huawei.com>
Content-Type: text/plain; charset="us-ascii"
Content-Disposition: inline
In-Reply-To: <20240404184901.00002104@Huawei.com>
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?bEBpGVKvz7Zoax7bIMS5XUfW+XYlvLEKmNQW/AqZhOTmgMAIBiGR1l9H/Tze?=
 =?us-ascii?Q?lusCTqPDUXuwM5qRfjUxUW/rrwPuaKVMAJHX3/3S+UXZtAiln4I4JGJQF9GA?=
 =?us-ascii?Q?tXZQb0G/j0+HshcAt8ekgiP2C9Mj3ZdG1dnxXGxdBrtUAqGfcJWuXDIf20Kd?=
 =?us-ascii?Q?43r23K7X5DfS+ArtVVBAFN0qBtOaSZ1Mp5FaGcixqGTWYuSFfTjz/uWiIq9A?=
 =?us-ascii?Q?so9U8/r8VscJUOhK1BXV9Knt44JqiMy0awVYBO2kbdz3mPBkXvFKv4ko900B?=
 =?us-ascii?Q?URTUVCSXdFp6sR5Ki6bTB8NYz0q8DEvQ5W1NPC8gz+mQ3fImjarawRY7Yiht?=
 =?us-ascii?Q?biz2XgB8gU7h8g0iFgV50EyBkmsx7cix2RVkliqBrUyBioPS2b4A9zfmTOtX?=
 =?us-ascii?Q?UqTqwDquF5TXYDR2tYaL+fLgh6aPqlm/6ImhnFah9/0UfZ3QHKOMt/bRDNhK?=
 =?us-ascii?Q?k5h8GOp8PwXKhsH3dsl8lIZeUyyrztDnsx9tTkWY47BsLC4IkrdAHX06kWkk?=
 =?us-ascii?Q?Uiu7iAXFjnVf7vscHmt1ftvpSot5nXeEtRZJAESXTEh7c+Rv/gzQk/N4BtTU?=
 =?us-ascii?Q?orL9eZat0lzrDc1PVxp+0RUOGhYKeH/OrrpzxwEtYHR9A/12owuiZfdDcIRi?=
 =?us-ascii?Q?jW9eBo4xmRGnJu/Ey6ojugYJxk0hRF1XPO2iZLjYWFlKE8y88z2uHK81jQ1D?=
 =?us-ascii?Q?vPQBdEl0k1DvZTXhhefEg3snjirBq+i+ob7IAP308rRWAmzAGBQAZfKoFXW0?=
 =?us-ascii?Q?nzI3pZxJmZsvTQy4ILCrqIuRmleIavLlJWKaKlwHu8rb4PH8Sp7Q/f8hXvPL?=
 =?us-ascii?Q?AQYYgDa8nlbrd9Y85VMIdFk0xFD+tO0PQJXoaBlXNzOlC9etSNJPhhYt8QL7?=
 =?us-ascii?Q?b/0FinkBj8pgaEGQI5PlNMDDkvD+xNoG6J5UEKc71LgCPtogV0duAUtua/V6?=
 =?us-ascii?Q?mXBHx5iAKiHy97O8JLAH5QNOo/GGl/FioKKrilXBU+hSOdcezhjhe8+1tf+a?=
 =?us-ascii?Q?TZhqd7dTC9DY0IoUxqHf5/clSD5uh6Sxpymfwci0rmbpesqbMT4VfbCXifmQ?=
 =?us-ascii?Q?aXszs5KfEro+Gx8O6RfN7CWF0cEiQQDh2I2iQPSkgBWNwO4JTcD1dxLJEiRu?=
 =?us-ascii?Q?Jd4pbiVc2vCKP42+UxYfcF+c+T8cZt0YFWcA4SPkYIl9WR2aXiDrOezks+3Z?=
 =?us-ascii?Q?Ix5NYlFdrOjLebEvT6gw7ZVNAMukOpJFY97eLK99Dx8kAreP1CuMOGLLxLto?=
 =?us-ascii?Q?svGgRnaJBj+3qAcEAa5au9Jsa5ggqe6Y1m+TIJBv9inQqrczMEnyPaSi+bKY?=
 =?us-ascii?Q?obz+CNsstSYI8s7dT22jPl5W6kivrk8k8LdJ3+fQyQCQ8ukVtNT84yim2vk7?=
 =?us-ascii?Q?eR01RECI/ZNDsD8RAQlZLZilLfcb1zh6pxbXZVJyyUd+y3VDwNxtOYQoRLic?=
 =?us-ascii?Q?5vMDiZ1LaOFtRMxgjBX5OuQCtqAHMzQNVN10v0RVdMmify4+3PDarBB1cxPt?=
 =?us-ascii?Q?QFaSTjpDMnIteuQJYmWGsjJorqgvJqGEAoRQOGSlRJ7HgHLE+7gp02zAcAlo?=
 =?us-ascii?Q?GFuQCspMdeIKu1c3/f7PA7FwVNOpRYTtdVauKScq?=
X-MS-Exchange-CrossTenant-Network-Message-Id: b7f206a8-b4c5-445b-1c47-08dc6a3956d6
X-MS-Exchange-CrossTenant-AuthSource: SA1PR11MB6733.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 01 May 2024 23:49:28.4270
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: dWFNn8fKD5iuuAgGFM/jH5SiKKcHrqtoxJupo82d2RZ/p+cCeWT12MEpXmw6es7eiK0qPph5cUo38HJmtlRTQA==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA3PR11MB7653
X-OriginatorOrg: intel.com

Jonathan Cameron wrote:
> 
> > 
> > Fan Ni's latest v5 of Qemu DCD was used for testing.[2]
> Hi Ira, Navneet.
> > 
> > Remaining work:
> > 
> > 	1) Integrate the QoS work from Dave Jiang
> > 	2) Interleave support
> 
> 
> More flag.  This one I think is potentially important and don't
> see any handling in here.

Nope I admit I missed the spec requirement.

> 
> Whilst an FM could in theory be careful to avoid sending a
> sparse set of extents, if the device is managing the memory range
> (which is possible all it supports) and the FM issues an Initiate Dynamic
> Capacity Add with Free (again may be all device supports) then we
> can't stop the device issuing a bunch of sparse extents.
> 
> Now it won't be broken as such without this, but every time we
> accept the first extent that will implicitly reject the rest.
> That will look very ugly to an FM which has to poke potentially many
> times to successfully allocate memory to a host.

This helps me to see see why the more bit is useful.

> 
> I also don't think it will be that hard to support, but maybe I'm
> missing something? 

Just a bunch of code and refactoring busy work.  ;-)  It's not rocket
science but does fundamentally change the arch again.

> 
> My first thought is it's just a loop in cxl_handle_dcd_add_extent()
> over a list of extents passed in then slightly more complex response
> generation.

Not exactly 'just a loop'.  No matter how I work this out there is the
possibility that some extents get surfaced and then the kernel tries to
remove them because it should not have.

To be most safe the cxl core is going to have to make 2 round trips to the
cxl region layer for each extent.  The first determines if the extent is
valid and creates the extent as much as possible.  The second actually
surfaces the extents.  However, if the surface fails then you might not
get the extents back.  So now we are in an invalid state.  :-/  WARN and
continue I guess?!??!

I think the safest way to handle this is add a new kernel notify event
called 'extent create' which stops short of surfacing the extent.  [I'm
not 100% sure how this is going to affect interleave.]

I think the safest logic for add is something like:

	cxl_handle_dcd_add_event()
		add_extent(squirl_list, extent);

		if (more bit) /* wait for more */
			return;

		/* Create extents to hedge the bets against failure */
		for_each(squirl_list)
			if (notify 'extent create' != ok)
				send_response(fail);
				return;
		
		for_each(squirl_list)
			if (notify 'surface' != ok)
				/*
				 * If the more bit was set, some extents
				 * have been surfaced and now need to be
				 * removed...
				 *
				 * Try to remove them and hope...
				 */
				WARN_ON('surface extents failed');
				for_each(squirl_list)
					notify 'remove without response'
				send_response(fail);
				return;

		send_response(squirl_list, accept);

The logic for remove is not changed AFAICS because the device must allow
for memory to be released at any time so the host is free to release each
of the extents individually despite the 'more' bit???

> 
> I don't want this to block getting initial DCD support in but it
> will be a bit ugly if we quickly support the more flag and then end
> up with just one kernel that an FM has to be careful with...

I'm not sure which is worse.  Given your use case above it seems like the
more bit may be more important for 'dumb' devices which want to add
extents in blocks before responding to the FM.  Thus complicating the FM.

It seems 'smarter' devices which could figure this out (not requiring the
more bit) are the ones which will be developed later.  So it seems the use
case time line is the opposite of what we need right now.

For that reason I'm inclined to try and get this in.

Ira