Received-SPF: pass (google.com: domain of linux-kernel+bounces-137880-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.199.223 as permitted sender) client-ip=147.75.199.223;
Date: Tue, 9 Apr 2024 21:25:29 -0700
From: Ira Weiny <ira.weiny@intel.com>
To: fan <nifan.cxl@gmail.com>, <ira.weiny@intel.com>
CC: Dave Jiang <dave.jiang@intel.com>, Fan Ni <fan.ni@samsung.com>, "Jonathan
 Cameron" <Jonathan.Cameron@huawei.com>, Navneet Singh
	<navneet.singh@intel.com>, Dan Williams <dan.j.williams@intel.com>, Davidlohr
 Bueso <dave@stgolabs.net>, Alison Schofield <alison.schofield@intel.com>,
	Vishal Verma <vishal.l.verma@intel.com>, <linux-btrfs@vger.kernel.org>,
	<linux-cxl@vger.kernel.org>, <linux-kernel@vger.kernel.org>
Subject: Re: [PATCH 09/26] cxl/region: Add Dynamic Capacity CXL region support
Message-ID: <661614b92b6bf_e9f9f29420@iweiny-mobl.notmuch>
References: <20240324-dcd-type2-upstream-v1-0-b7b00d623625@intel.com>
 <20240324-dcd-type2-upstream-v1-9-b7b00d623625@intel.com>
 <ZgNMz9doYO0KlgKU@debian>
Content-Type: text/plain; charset="us-ascii"
Content-Disposition: inline
In-Reply-To: <ZgNMz9doYO0KlgKU@debian>
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?Rbx2jBSkBdnDKj0wMg6hJr1BSdCT68V/0MSR4CvwyW8AD2og3ELvFuM56mt1?=
 =?us-ascii?Q?bXM+/3XiOtya4DBvKNJa/056/omJgxNeN7exP1ZlzOfSwNuV4SFwkTv4M19X?=
 =?us-ascii?Q?651DPFeDqF3AkMbCffNG/0OBRdlU77J3ayyvFvqDz2s8nzimFM0R2Tv/5tvS?=
 =?us-ascii?Q?iOB1zJM7l13649gF0dJTuCULtGnTWBhLzIO7BF6lQFjP9uu293pH3G3SfNiH?=
 =?us-ascii?Q?6ggPm7oy+tE1X3CdNo0rRpXMZpqc4idBPcEYzNxj5+wpfPU2bLGHL4UqjvUp?=
 =?us-ascii?Q?d15OqpH29C4V9LeQhpooKl8Gwf5g80ViyM35+89Nx7Gr675DKDPqdcVbKv+I?=
 =?us-ascii?Q?jy9+sLfdF2A09NCZCZHxkOIyT5TP99lLgfYDi9RAtuvjyzFhq6kBKKL4wBjv?=
 =?us-ascii?Q?Wx0eWEc53A5RbTdAmcP5dWE6O4DkNUAoNBgD5sdxU/L/FtZHfo4KtyYXq1jf?=
 =?us-ascii?Q?5dDCmjyhsTle6OBAZUDPqHvyFn3awHTN3CNlXtAu9gbJsOHhhUQzmwRb9Z1F?=
 =?us-ascii?Q?GrJIWPNzw3akSoCQX6ne3GvOBHaXPcG0ztfFR0JfLKmIImK0INIu1kBcw5AP?=
 =?us-ascii?Q?Nop/fMMUqpvlXoVu1wdrTW41kAYCPlBpyfVex4e30Xw2b5MbxrTwx9wscKHa?=
 =?us-ascii?Q?3rXEMNI2sYLW5TJPeHt0GKUrpRID1QOm3Y3HfVX/AM/jlOhdJlxE3LbQ4/UA?=
 =?us-ascii?Q?j1AThtPv0lLDeLJ/WDbB9Q+WOxBXywWK4LLYoUWfFmyFLmuxNwkPpYw7uoIt?=
 =?us-ascii?Q?3BHEyyUIb9PJ3g2ekHDuhYeL57/3Di4X67J4BdNA63VAJC+yQLdAV/pAo39R?=
 =?us-ascii?Q?QSFqY4EpgYR6Kb4iq+10ANQoYArkFPbfIxJnLOnyVhP6gmETfmsMgKC1xCLd?=
 =?us-ascii?Q?nugW4TEuVx7QxLaFRHi+fdbR8bCsThj/azEtP7rO5XaxBDGWCaTC4j5//qFZ?=
 =?us-ascii?Q?u/AUWmOlw/9VqMklHEQlLkoxsx7xAi+6HHVD4wSbQxdkRPY1iiCFLn44gkDz?=
 =?us-ascii?Q?rnd0H1Zwiq2YoHqDj23N4l644cRHY4GcEtRAwWCzhiea4EquHKNKwevmKGJs?=
 =?us-ascii?Q?ZNT3cy1Grj577awoVvkbZSPrBoNkHpEZapSRSmYVQFYiJT1XG9d+COkp98eL?=
 =?us-ascii?Q?omS+0uRRterNei7m2+fU0qitxU2OxgG49bpDgHGOS7yZD0LPRfrVWYHgrotE?=
 =?us-ascii?Q?O5D+f3hwI5WImP6IBDf/mf4z3b11EVbfCl5PUrMynnTAMVTx+RQ4vP5vzZlC?=
 =?us-ascii?Q?LRuI2weKI99WQhBnYHMVAYEZeAdObofJUibzJIok3mwR4COeLdtMxj3U+ZUD?=
 =?us-ascii?Q?DUJ95tKemS+LQK6Q70qxz2s41/pZUM9iU3YIDpOHffcfDQPB4wvU06/R4vRK?=
 =?us-ascii?Q?vtoVSPR/qnOrcnOjVT5DvgbDqG6UGHVWU9wE+1pGq6yfosMwoOPpnTXw5kYb?=
 =?us-ascii?Q?xzLroikmqRHohZ4wvnoF+jOS72Tlk7CzsF8uS/qKBfK+pqTECv+L7vfWkiTv?=
 =?us-ascii?Q?8sOMYkUblG8/IspXU7Vz00psNgaWFhinQaDM1F13EYulzRqu0dKBcfkuCn+n?=
 =?us-ascii?Q?xvTPSIGfYbZwYqqiLp+yaFqU7/RNBjpYRuXtghPD?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 1c67c247-d223-46ab-8441-08dc591642ba
X-MS-Exchange-CrossTenant-AuthSource: SA1PR11MB6733.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 10 Apr 2024 04:25:32.5083
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: ICDSWXLPB6mVGL4wvDTspRI+27DL8pwY6cp5UnNm+Tla92nv3cYi0GI6arRmdbzbKXUVJUUY2tkOiu2fyXdl/Q==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ0PR11MB6790
X-OriginatorOrg: intel.com

fan wrote:
> On Sun, Mar 24, 2024 at 04:18:12PM -0700, ira.weiny@intel.com wrote:
> > From: Navneet Singh <navneet.singh@intel.com>
> > 
> > CXL devices optionally support dynamic capacity.  CXL Regions must be
> > configured correctly to access this capacity.  Similar to ram and pmem
> > partitions, DC Regions, as they are called in CXL 3.1, represent
> > different partitions of the DPA space.
> > 
> > Introduce the concept of a sparse DAX region.  Add the create_dc_region
> > sysfs entry to create sparse DC DAX regions.  Special case DC capable
> > regions to create a 0 sized seed DAX device to maintain backwards
> > compatibility with older software which needs a default DAX device to
> > hold the region reference.
> > 
> > Flag sparse DAX regions to indicate 0 capacity available until such time
> > as DC capacity is added.
> > 
> > Interleaving is deferred in this series.  Add an early check.
> > 
> > Signed-off-by: Navneet Singh <navneet.singh@intel.com>
> > Co-developed-by: Ira Weiny <ira.weiny@intel.com>
> > Signed-off-by: Ira Weiny <ira.weiny@intel.com>
> > 
> > ---
> > Changes for v1:
> > [djiang: mark sysfs entries to be in 6.10 kernel including date]
> > [djbw: change dax region typing to be 'sparse' rather than 'dynamic']
> > [iweiny: rebase changes to master instead of type2 patches]
> > ---
> >  Documentation/ABI/testing/sysfs-bus-cxl | 22 +++++++++++-----------
> >  drivers/cxl/core/core.h                 |  1 +
> >  drivers/cxl/core/port.c                 |  1 +
> >  drivers/cxl/core/region.c               | 33 +++++++++++++++++++++++++++++++++
> >  drivers/dax/bus.c                       |  8 ++++++++
> >  drivers/dax/bus.h                       |  1 +
> >  drivers/dax/cxl.c                       | 15 +++++++++++++--
> >  7 files changed, 68 insertions(+), 13 deletions(-)
> > 
> > diff --git a/Documentation/ABI/testing/sysfs-bus-cxl b/Documentation/ABI/testing/sysfs-bus-cxl
> > index 8a4f572c8498..f0cf52fff9fa 100644
> > --- a/Documentation/ABI/testing/sysfs-bus-cxl
> > +++ b/Documentation/ABI/testing/sysfs-bus-cxl
> > @@ -411,20 +411,20 @@ Description:
> >  		interleave_granularity).
> >  
> >  
> > -What:		/sys/bus/cxl/devices/decoderX.Y/create_{pmem,ram}_region
> > -Date:		May, 2022, January, 2023
> > -KernelVersion:	v6.0 (pmem), v6.3 (ram)
> > +What:		/sys/bus/cxl/devices/decoderX.Y/create_{pmem,ram,dc}_region
> > +Date:		May, 2022, January, 2023, June 2024
> > +KernelVersion:	v6.0 (pmem), v6.3 (ram), v6.10 (dc)
> >  Contact:	linux-cxl@vger.kernel.org
> >  Description:
> >  		(RW) Write a string in the form 'regionZ' to start the process
> > -		of defining a new persistent, or volatile memory region
> > -		(interleave-set) within the decode range bounded by root decoder
> > -		'decoderX.Y'. The value written must match the current value
> > -		returned from reading this attribute. An atomic compare exchange
> > -		operation is done on write to assign the requested id to a
> > -		region and allocate the region-id for the next creation attempt.
> > -		EBUSY is returned if the region name written does not match the
> > -		current cached value.
> > +		of defining a new persistent, volatile, or Dynamic Capacity
> > +		(DC) memory region (interleave-set) within the decode range
> > +		bounded by root decoder 'decoderX.Y'. The value written must
> > +		match the current value returned from reading this attribute.
> > +		An atomic compare exchange operation is done on write to assign
> > +		the requested id to a region and allocate the region-id for the
> > +		next creation attempt.  EBUSY is returned if the region name
> > +		written does not match the current cached value.
> >  
> >  
> >  What:		/sys/bus/cxl/devices/decoderX.Y/delete_region
> > diff --git a/drivers/cxl/core/core.h b/drivers/cxl/core/core.h
> > index 3b64fb1b9ed0..91abeffbe985 100644
> > --- a/drivers/cxl/core/core.h
> > +++ b/drivers/cxl/core/core.h
> > @@ -13,6 +13,7 @@ extern struct attribute_group cxl_base_attribute_group;
> >  #ifdef CONFIG_CXL_REGION
> >  extern struct device_attribute dev_attr_create_pmem_region;
> >  extern struct device_attribute dev_attr_create_ram_region;
> > +extern struct device_attribute dev_attr_create_dc_region;
> >  extern struct device_attribute dev_attr_delete_region;
> >  extern struct device_attribute dev_attr_region;
> >  extern const struct device_type cxl_pmem_region_type;
> > diff --git a/drivers/cxl/core/port.c b/drivers/cxl/core/port.c
> > index 036b61cb3007..661177b575f7 100644
> > --- a/drivers/cxl/core/port.c
> > +++ b/drivers/cxl/core/port.c
> > @@ -335,6 +335,7 @@ static struct attribute *cxl_decoder_root_attrs[] = {
> >  	&dev_attr_qos_class.attr,
> >  	SET_CXL_REGION_ATTR(create_pmem_region)
> >  	SET_CXL_REGION_ATTR(create_ram_region)
> > +	SET_CXL_REGION_ATTR(create_dc_region)
> >  	SET_CXL_REGION_ATTR(delete_region)
> >  	NULL,
> >  };
> > diff --git a/drivers/cxl/core/region.c b/drivers/cxl/core/region.c
> > index ec3b8c6948e9..0d7b09a49dcf 100644
> > --- a/drivers/cxl/core/region.c
> > +++ b/drivers/cxl/core/region.c
> > @@ -2205,6 +2205,7 @@ static struct cxl_region *devm_cxl_add_region(struct cxl_root_decoder *cxlrd,
> >  	switch (mode) {
> >  	case CXL_REGION_RAM:
> >  	case CXL_REGION_PMEM:
> > +	case CXL_REGION_DC:
> >  		break;
> >  	default:
> >  		dev_err(&cxlrd->cxlsd.cxld.dev, "unsupported mode %s\n",
> > @@ -2314,6 +2315,32 @@ static ssize_t create_ram_region_store(struct device *dev,
> >  }
> >  DEVICE_ATTR_RW(create_ram_region);
> >  
> > +static ssize_t create_dc_region_show(struct device *dev,
> > +				     struct device_attribute *attr, char *buf)
> > +{
> > +	return __create_region_show(to_cxl_root_decoder(dev), buf);
> > +}
> > +
> > +static ssize_t create_dc_region_store(struct device *dev,
> > +				      struct device_attribute *attr,
> > +				      const char *buf, size_t len)
> > +{
> > +	struct cxl_root_decoder *cxlrd = to_cxl_root_decoder(dev);
> > +	struct cxl_region *cxlr;
> > +	int rc, id;
> > +
> > +	rc = sscanf(buf, "region%d\n", &id);
> > +	if (rc != 1)
> > +		return -EINVAL;
> > +
> > +	cxlr = __create_region(cxlrd, CXL_REGION_DC, id);
> > +	if (IS_ERR(cxlr))
> > +		return PTR_ERR(cxlr);
> > +
> > +	return len;
> > +}
> > +DEVICE_ATTR_RW(create_dc_region);
> 
> create_ram_region_store, create_pmem_region_store and
> create_dc_region_store have mostly duplicate code, should we consider
> extracting out as a helper function and pass region type for ram/pmem/dc region
> store?

That is mostly done with __create_region().  But the following is a nice cleanup.

I'll test it.

Ira

21:24:37 > git di
diff --git a/drivers/cxl/core/region.c b/drivers/cxl/core/region.c
index e8549af47da7..8a83a415fd0b 100644
--- a/drivers/cxl/core/region.c
+++ b/drivers/cxl/core/region.c
@@ -2638,9 +2638,8 @@ static struct cxl_region *__create_region(struct cxl_root_decoder *cxlrd,
        return devm_cxl_add_region(cxlrd, id, mode, CXL_DECODER_HOSTONLYMEM);
 }
 
-static ssize_t create_pmem_region_store(struct device *dev,
-                                       struct device_attribute *attr,
-                                       const char *buf, size_t len)
+static ssize_t create_region_store(struct device *dev, const char *buf,
+                                  size_t len, enum cxl_region_mode mode)
 {
        struct cxl_root_decoder *cxlrd = to_cxl_root_decoder(dev);
        struct cxl_region *cxlr;
@@ -2650,31 +2649,26 @@ static ssize_t create_pmem_region_store(struct device *dev,
        if (rc != 1)
                return -EINVAL;
 
-       cxlr = __create_region(cxlrd, CXL_REGION_PMEM, id);
+       cxlr = __create_region(cxlrd, mode, id);
        if (IS_ERR(cxlr))
                return PTR_ERR(cxlr);
 
        return len;
 }
+
+static ssize_t create_pmem_region_store(struct device *dev,
+                                       struct device_attribute *attr,
+                                       const char *buf, size_t len)
+{
+       return create_region_store(dev, buf, len, CXL_REGION_PMEM);
+}
 DEVICE_ATTR_RW(create_pmem_region);
 
 static ssize_t create_ram_region_store(struct device *dev,
                                       struct device_attribute *attr,
                                       const char *buf, size_t len)
 {
-       struct cxl_root_decoder *cxlrd = to_cxl_root_decoder(dev);
-       struct cxl_region *cxlr;
-       int rc, id;
-
-       rc = sscanf(buf, "region%d\n", &id);
-       if (rc != 1)
-               return -EINVAL;
-
-       cxlr = __create_region(cxlrd, CXL_REGION_RAM, id);
-       if (IS_ERR(cxlr))
-               return PTR_ERR(cxlr);
-
-       return len;
+       return create_region_store(dev, buf, len, CXL_REGION_RAM);
 }
 DEVICE_ATTR_RW(create_ram_region);
 
@@ -2688,19 +2682,7 @@ static ssize_t create_dc_region_store(struct device *dev,
                                      struct device_attribute *attr,
                                      const char *buf, size_t len)
 {
-       struct cxl_root_decoder *cxlrd = to_cxl_root_decoder(dev);
-       struct cxl_region *cxlr;
-       int rc, id;
-
-       rc = sscanf(buf, "region%d\n", &id);
-       if (rc != 1)
-               return -EINVAL;
-
-       cxlr = __create_region(cxlrd, CXL_REGION_DC, id);
-       if (IS_ERR(cxlr))
-               return PTR_ERR(cxlr);
-
-       return len;
+       return create_region_store(dev, buf, len, CXL_REGION_DC);
 }
 DEVICE_ATTR_RW(create_dc_region);