Received-SPF: pass (google.com: domain of linux-kernel+bounces-169302-linux.lists.archive=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) client-ip=139.178.88.99;
Date: Sun, 5 May 2024 21:35:39 -0700
From: Ira Weiny <ira.weiny@intel.com>
To: Dan Williams <dan.j.williams@intel.com>, Ira Weiny <ira.weiny@intel.com>,
	Jonathan Cameron <Jonathan.Cameron@huawei.com>
CC: Dave Jiang <dave.jiang@intel.com>, Fan Ni <fan.ni@samsung.com>, "Navneet
 Singh" <navneet.singh@intel.com>, Dan Williams <dan.j.williams@intel.com>,
	Davidlohr Bueso <dave@stgolabs.net>, Alison Schofield
	<alison.schofield@intel.com>, Vishal Verma <vishal.l.verma@intel.com>,
	<linux-btrfs@vger.kernel.org>, <linux-cxl@vger.kernel.org>,
	<linux-kernel@vger.kernel.org>
Subject: Re: [PATCH 16/26] cxl/extent: Realize extent devices
Message-ID: <66385e1b63173_25842129417@iweiny-mobl.notmuch>
References: <20240324-dcd-type2-upstream-v1-0-b7b00d623625@intel.com>
 <20240324-dcd-type2-upstream-v1-16-b7b00d623625@intel.com>
 <20240404173241.00003a6f@Huawei.com>
 <6630641e5238e_e1f5829431@iweiny-mobl.notmuch>
 <663401ba2e237_1384629438@dwillia2-mobl3.amr.corp.intel.com.notmuch>
Content-Type: text/plain; charset="us-ascii"
Content-Disposition: inline
In-Reply-To: <663401ba2e237_1384629438@dwillia2-mobl3.amr.corp.intel.com.notmuch>
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?nG9qRyJaou7ar+qXHNLyuX9trYCc8USk3lcV6WvFF8/Q3QInNvJa6t/1QgA8?=
 =?us-ascii?Q?5skLiEFM0XSMmF5QmQ0h+Bs+h4FxtDKMbyQwf+SlT3zcUtVJhiBOXRjgaddB?=
 =?us-ascii?Q?12DvvQWZpgGDKLUVdNhKcp6qlV0Q1HlmllQvQn+7NksnlGASDIUfU759rMDp?=
 =?us-ascii?Q?fToS3aHxMbFbNqd1EE0ZB50OZjWsAIAJqS4NQJFXn0pNN/0ca9pIfmojmvhF?=
 =?us-ascii?Q?haCjcWhoPFsqQvr0HI/K11WUfgIem2u5WXEx5pEvvAe7KQKLKmWxn+NWHk6l?=
 =?us-ascii?Q?zt4IXNqNaLnOPsqyPlxv4+1QET5Zegmf1ETlI+x9C/D3g9WxrrIj2zSwY3R6?=
 =?us-ascii?Q?zl5BFIYpEmw4nVeJot4JcG79c1tIzmXF71EykNgyI6np4vJLxqBGX4xLeVtB?=
 =?us-ascii?Q?Jxyoo1+w5kz51aAnv6E7E2MJRyOoApREu3utafhDlYbQfatVMiu8kgnDCEo+?=
 =?us-ascii?Q?tp8oor3zy1D9GTij1x+VXnbaDARH37MF92ufHUJSA3/4dlHpuTlFQSHgWTUy?=
 =?us-ascii?Q?sTDsYn6fhJlQmGzmeU0BzI/lV8wF+GUb/wnA/NWk6uzH5RV6F3D6lTUrPOEI?=
 =?us-ascii?Q?8RTxVKfA9Bo+3XEawQsXeP4KVKgUbpNOJPMHBfEw1b9YVQKux5FHA1IGznCz?=
 =?us-ascii?Q?t/CT5bYNY+Ukj2u8Q2NRDh1WCQ8DOvxMp70fDFHxSxkPJ/7ibrWdsBSYO1S0?=
 =?us-ascii?Q?oNtkDiBcyjaySutFJnGiDbm8GhO5hjqLZ1/jL/4FBKFmhnoPJb/sHHnY+Pxe?=
 =?us-ascii?Q?KPnzzO3y3J1j5elUAG2/imS8ziLPf99vgFdtvPQ8azCSbgip7ccMwURs38AL?=
 =?us-ascii?Q?3pGWszrHXyW2Ihn/FUSG3T3kX8NV+tA3eM6g+2Unra2UU79VoB6Hzc51myXE?=
 =?us-ascii?Q?oAATd9b+5Mep9DAVKp7oR5FY18N32jYvWi09V4tCGN1zoBFwmGBx1laIL6xu?=
 =?us-ascii?Q?8e4Wrtv26GqAkcF7dGOQPhd7SYyM04pCYICEDYPgTHojdj4r/UmzZ1ExOwJx?=
 =?us-ascii?Q?g0rUAoRckKguyenjyYl9jMyfw5zxYSo6MFMp2CE8fSecPubLRHTANfegcA0B?=
 =?us-ascii?Q?zxxQk1EliN1DnWGVLRFjiMS4Zpei6z/HnFwi+qn2z3mpUHM/VMSuKVxDM3xV?=
 =?us-ascii?Q?o1yW+TGURoP/sbaxrR7z6J1paUBN5FhvRDwokbahesb92yQd44HSMP3ocTfo?=
 =?us-ascii?Q?S9EuDO4o+JYrAHQ8F0qLju/nEIpms5uU0b4Vfdf2mvd/goAYCANbdMaThji+?=
 =?us-ascii?Q?cxPW2fcW0iZw0AGc7X8TpI/ppTIz1mcoQZqSiWE+y3yTdyoL8Vbt/a3Gk+/f?=
 =?us-ascii?Q?u9KL93topSyKRN9dYvkBaN2vCdGjZMRrL5wQ8phAbMidpmtmwv7lgvIa4vLi?=
 =?us-ascii?Q?xqXgNIucvHBmuxFLuMdYQRdruKtKyCGeaA/Mfp7ZJ6TbCWsWpll+TUd+XnXf?=
 =?us-ascii?Q?9rPRISQJj2iTIskZps55x8j1PDh8juZlOEHv9Vr9G/5zJWz7SCMuRGMD1H8A?=
 =?us-ascii?Q?2qVs5GH7tDjjzxH6Wx/KNuMAIa1RFHTC+ASR3rXVTXzQvUPu6ffSVrzIfjED?=
 =?us-ascii?Q?zfrPpcTIeJzhagAe8DG+uw1L7JOLtUvG4ZWQEXfo?=
X-MS-Exchange-CrossTenant-Network-Message-Id: c4199748-39a3-480a-f51c-08dc6d85fd0d
X-MS-Exchange-CrossTenant-AuthSource: SA1PR11MB6733.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 06 May 2024 04:35:42.4817
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: 4qG0/TxZ+LKoCyXgtDIPx75xFiFle49ngt+zKgbIXy/uXJ9TrJEzP8Ms1yNez1xjPV9lmVO9bHLqdN9mrVQXPA==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA1PR11MB6823
X-OriginatorOrg: intel.com

Dan Williams wrote:
> Ira Weiny wrote:
> > Jonathan Cameron wrote:
> > > On Sun, 24 Mar 2024 16:18:19 -0700
> > > ira.weiny@intel.com wrote:
> > > 
> > > > From: Navneet Singh <navneet.singh@intel.com>
> > > > 
> > > > Once all extents of an interleave set are present a region must
> > > > surface an extent to the region.
> > > > 
> > > > Without interleaving; endpoint decoder and region extents have a 1:1
> > > > relationship.  Future support for IW > 1 will maintain a N:1
> > > > relationship between the device extents and region extents.
> > > > 
> > > > Create a region extent device for every device extent found.  Release of
> > > > the extent device triggers a response to the underlying hardware extent.
> > > > 
> > > > There is no strong use case to support the addition of extents which
> > > > overlap previously accepted extent ranges.  Reject such new extents
> > > > until such time as a good use case emerges.
> > > > 
> > > > Expose the necessary details of region extents by creating the following
> > > > sysfs entries.
> > > > 
> > > > 	/sys/bus/cxl/devices/dax_regionX/extentY
> > > > 	/sys/bus/cxl/devices/dax_regionX/extentY/offset
> > > > 	/sys/bus/cxl/devices/dax_regionX/extentY/length
> > > > 	/sys/bus/cxl/devices/dax_regionX/extentY/label
> > > 
> > > Docs?
> > 
> > That is a good idea.
> > 
> > > The label in particular worries me a little as I'm not sure what
> > > is in it.
> > 
> > I envisioned a pass through of the tag.
> > 
> > >
> > > If it's the tag one possible format is a uuid (not a coincidence
> > > that it is the same length) and interpreting that as characters isn't
> > > going to get us far.  I wonder if we have to treat it as a binary attr
> > > given we have no idea what it is.
> > 
> > In thinking about this more (and running some experiments): none of these are
> > strictly necessary in this initial implementation.  No code currently uses
> > them directly.
> > 
> > I questioned these in the past and I've done so again over the weekend.
> > 
> > I was about to rip them out entirely when I remembered Gregory Price's
> > comments on Discord.  There he indicating a desire to very carefully place
> > dax devices.  Without at least the offset and length above (and to a
> > lesser extent the label) this can't be done.
> 
> Careful placement of dax-devices physically requires an entirely new
> allocation ABI. There is the mapping_store() interface that was added
> for a specific kexec / VMM fast restore use case, but that never
> envisioned the sparse region case. So I do think it is worthwhile to
> punt on that question to a later add-on feature.

Agreed.

>  
> [..]
> > I don't think this is exactly what the user is going to expect.  This can
> > be resolved by by looking at the dax device mappings though.[0]  So I'm going
> > to leave this for now.  But I expect some additional porcelain is going to
> > be required to fully meet Gregory's requirements.
> 
> Not sure what the exact requirement is, but if it's the typical, "I want
> to allocate by tag",

I'm extrapolating that it will be.  I want to allocate on the first extent.
Tags were removed from the series a while ago.

> then I think there is another potential coarse
> grained solution that probably covers most cases. Allow multiple
> dax_regions per cxl_dcd_regions, where each dax_region manages an
> exclusive set of tags.

I'll have to think on that because don't dax regions map to specific dpas on
creation?

> 
> The host negotiates a dax_region tag layout with the orchestrator and
> can then trust that all of the extents that show up in a given dax_region
> belong to a given tag or set of tags.
> 
> This is not something that needs to be considered in the initial
> enabling, but is potentially a way to avoid bolting-on a new fine grained
> allocation api after the fact.
> 

The point is I'm not trying to bolt anything on.  Just trying to explain what
can and can't be done.   The purpose of these entries was to give the user the
ability to see what extents existed and by correlating the dax mappings could
see where their dax mappings landed.  Careful allocation of dax devices could
result in the use of some extents and not others.  But this was __not__ at all
intended to be done initially.  Just use the space as it come available without
any tag use at all.

> 
> > [0]
> > 	/sys/bus/dax/devices/daxX.Y/mapping[0..N]/start
> > 	/sys/bus/dax/devices/daxX.Y/mapping[0..N]/end
> > 
> > Back to the label field:  It is currently just the 'tag' of the individual
> > extent (because no interleaving).  My vision for the interleave case would
> > be for the kernel to assemble device extents into a region extent only if
> > the tags match and export that.
> > 
> > Thinking on it more though we should leave label out for now.  This is the
> > second time it has been questioned.
> 
> I don't understand the issue. That is a critical piece of information
> and it is at the cxl device level
> 
> /sys/bus/cxl/devices/dax_regionX/extentY/label
> 
> ...now I would just call that "tag" and UUID format it (to Jonathan's
> point), but I see no rationale to hide what is most likely the most
> useful information about an extent.

The rationale is that the user was not going to use it.  So no use case == no
reason to have it...  yet.  I had code a while back to allocate dax devices on
specific tags.  But that was deemed to different from the current dax
allocation mechanism so it was scrapped...  for now.

I can add it back...  But I'm just getting a bit testy about who wants what and
how this is all going to get used.

Ira