Received-SPF: pass (google.com: domain of linux-kernel+bounces-170208-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) client-ip=2604:1380:4601:e00::3;
Date: Mon, 6 May 2024 09:56:40 -0700
From: Dan Williams <dan.j.williams@intel.com>
To: <ira.weiny@intel.com>, Dave Jiang <dave.jiang@intel.com>, Fan Ni
	<fan.ni@samsung.com>, Jonathan Cameron <Jonathan.Cameron@huawei.com>, Navneet
 Singh <navneet.singh@intel.com>
CC: Dan Williams <dan.j.williams@intel.com>, Davidlohr Bueso
	<dave@stgolabs.net>, Alison Schofield <alison.schofield@intel.com>, "Vishal
 Verma" <vishal.l.verma@intel.com>, Ira Weiny <ira.weiny@intel.com>,
	<linux-btrfs@vger.kernel.org>, <linux-cxl@vger.kernel.org>,
	<linux-kernel@vger.kernel.org>
Subject: Re: [PATCH 13/26] cxl/mem: Configure dynamic capacity interrupts
Message-ID: <66390bc876ad7_2e2d22947e@dwillia2-mobl3.amr.corp.intel.com.notmuch>
References: <20240324-dcd-type2-upstream-v1-0-b7b00d623625@intel.com>
 <20240324-dcd-type2-upstream-v1-13-b7b00d623625@intel.com>
Content-Type: text/plain; charset="us-ascii"
Content-Disposition: inline
In-Reply-To: <20240324-dcd-type2-upstream-v1-13-b7b00d623625@intel.com>
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?NJuU/JoCHMGlMvmbdek8Zd/5eHDaZLh1uaRa6wpxiFb8TDZ36KInEgKpbHTO?=
 =?us-ascii?Q?JRi/K7XKqNtgGc2Vh/E9JtLex0g0lDF50Mr53mAYEAPhxGfeHqJqLMoC/Ao3?=
 =?us-ascii?Q?nTe5yUSeATc+LiueMk0Lglgs1Rtho+imTq1M8me33KeBVqtLHmJVR86ecnKp?=
 =?us-ascii?Q?ooiThTJlmWlejr/7TOw8r+HE1Iqn+w5WKniEr0mfWUJHJYCOYHh/DnUEoyzw?=
 =?us-ascii?Q?j/njvQGjPP2pGewzCfH8GYrXrXT0NPQRN7SUO/gsM5HL15c5Dj2yn2/B2Be7?=
 =?us-ascii?Q?v3SKevl+DmMR+HPuGThx6DPWXbeefSqbywCSHe4mJPsXQtgvpZXWlwvW+szB?=
 =?us-ascii?Q?o7Zqh0NJ+qE2qn9GKCUN7P+8Y78e6NE8ZrCa2W1eNo7CQEfbOQig/pid/mQj?=
 =?us-ascii?Q?aGqqCdyb94NCML58ZFm199X2cgHGHbwOA9D9pJppCqxZtR2l9/aPy+09Uczm?=
 =?us-ascii?Q?ii+COGn0woTK9Ap1tm5LW0/2vBLBWEKpaUr9weF47I4GREYNdPixu37YXtzr?=
 =?us-ascii?Q?SxweFo5nPkqce1LgoQm9VPkfprCD80+IN7A/hZAV1wu1x+dt42vydKwsHCCn?=
 =?us-ascii?Q?upHlXCss608UZ9M7r8s/Qir53z4JuZMjhitFyWBPOaZHUJfXVs/H6cUx/cSX?=
 =?us-ascii?Q?lqzKKhVOlOCizbNReQaaZUprGo6e8A1x+2KHcwpQghaZPP96Uf4Oepy1BWuh?=
 =?us-ascii?Q?6ssfyuZW6MN07rG6jbNK+bD37ZzUEEu0DzimHyPhNuuzHx9ZRQAaRKTYBfbX?=
 =?us-ascii?Q?HfzZFvq+2akg9WoYmegpjwSPg3lbRX/7fchUHzn5pRnl5gWX7rLPYkS9p+mM?=
 =?us-ascii?Q?7dwosik3bMDzSYu4kuSx3a3TWF2FHkr3gRPPEviao5BBJRT80mgcawK6XzgK?=
 =?us-ascii?Q?P3nnOOTfYhESF0pY0TtDj0O2zAHZkifrlcm9dk0Fhia4vDlziml1mkfnvaCK?=
 =?us-ascii?Q?FyNTvSkH3ICJ6lsmieOQvH7sknW0+fjGyQRE0qvefPcFw7wDRycpiWWBdFmc?=
 =?us-ascii?Q?D6aPdRDaUoEvbpMhAEF3TE6Qswk8JXrQA6cm5ETXZEEDaOanNJDTSnx8p0EJ?=
 =?us-ascii?Q?PAKFdu9g/wq7QNCAfs2iZw8u1Dy9Uek3qlvfu6iL1hkztVTpGAShbWTmbe2k?=
 =?us-ascii?Q?c8kRD3TH66QSqConUopNKLbLiB7Xodn8CjsRugqb9VQvh+DzcOgeleqjPXfE?=
 =?us-ascii?Q?8NN3+YSNjkylND7XEZ0qYjcqq/Hzxa4lSBSAAOjKrEeByB7HiJbUDjReKniD?=
 =?us-ascii?Q?AG1nKu/PxcgXLdUQGjit9sHbOcW1/3EoSRcC6ar0sAIxO7z6wmsn9c2HwGrB?=
 =?us-ascii?Q?Oel8qGCsyRJ7Rtu72dbCQLXCzOZq74lKvJDCWbRQA9SpvVwx0frzosuf3X7V?=
 =?us-ascii?Q?sv6T/tKB4UZQsbS+JjGEOvDWHl2Sc4EVKU/4MpW1vwF/YXfAduxx4TpJRZt6?=
 =?us-ascii?Q?5xQgyviVpGGDlzSt0XPFb9cY/9UttORBx5nqNs/T8PlvUa0Zr7Zp8j1B5Nb7?=
 =?us-ascii?Q?dr+mmRglWGV4E2P8H8yEYZIp4cG2oL2az8MTbehhY7f0SyRGXf9qp9XP8EAT?=
 =?us-ascii?Q?3wff3UmAYiCDn1UgrAORHQaocJLebt5yN1JIy1iolh5XiKXSM0El7q0T6Vk+?=
 =?us-ascii?Q?5Q=3D=3D?=
X-MS-Exchange-CrossTenant-Network-Message-Id: f87e2ad1-6070-4a44-c0b1-08dc6ded81ce
X-MS-Exchange-CrossTenant-AuthSource: PH8PR11MB8107.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 06 May 2024 16:56:43.3831
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: f4F/ML9HOwa7Zc7sxDPuI4/BKcy84CkfYcDNy9INwVdLQMBAk6RPWCHwbgdeJixiInEptt4voS2StoyaoKxOs7gxLw62zAZiKJySwsadRnI=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS7PR11MB7691
X-OriginatorOrg: intel.com

ira.weiny@ wrote:
> From: Navneet Singh <navneet.singh@intel.com>
> 
> Dynamic Capacity Devices (DCD) support extent change notifications
> through the event log mechanism.  The interrupt mailbox commands were
> extended in CXL 3.1 to support these notifications.
> 
> Firmware can't configure DCD events to be FW controlled but can retain
> control of memory events.  Split irq configuration of memory events and
> DCD events to allow for FW control of memory events while DCD is host
> controlled.
> 
> Configure DCD event log interrupts on devices supporting dynamic
> capacity.  Disable DCD if interrupts are not supported.
> 
> Signed-off-by: Navneet Singh <navneet.singh@intel.com>
> Co-developed-by: Ira Weiny <ira.weiny@intel.com>
> Signed-off-by: Ira Weiny <ira.weiny@intel.com>
> 
> ---
> Changes for v1
> [iweiny: rebase to upstream irq code]
> [iweiny: disable DCD if irqs not supported]
> ---
>  drivers/cxl/core/mbox.c |  9 ++++++-
>  drivers/cxl/cxl.h       |  4 ++-
>  drivers/cxl/cxlmem.h    |  4 +++
>  drivers/cxl/pci.c       | 71 ++++++++++++++++++++++++++++++++++++++++---------
>  4 files changed, 74 insertions(+), 14 deletions(-)
> 
> diff --git a/drivers/cxl/core/mbox.c b/drivers/cxl/core/mbox.c
> index 14e8a7528a8b..58b31fa47b93 100644
> --- a/drivers/cxl/core/mbox.c
> +++ b/drivers/cxl/core/mbox.c
> @@ -1323,10 +1323,17 @@ static int cxl_get_dc_config(struct cxl_memdev_state *mds, u8 start_region,
>  	return rc;
>  }
>  
> -static bool cxl_dcd_supported(struct cxl_memdev_state *mds)
> +bool cxl_dcd_supported(struct cxl_memdev_state *mds)
>  {
>  	return test_bit(CXL_DCD_ENABLED_GET_CONFIG, mds->dcd_cmds);
>  }
> +EXPORT_SYMBOL_NS_GPL(cxl_dcd_supported, CXL);
> +
> +void cxl_disable_dcd(struct cxl_memdev_state *mds)
> +{
> +	clear_bit(CXL_DCD_ENABLED_GET_CONFIG, mds->dcd_cmds);
> +}
> +EXPORT_SYMBOL_NS_GPL(cxl_disable_dcd, CXL);

Just use the open-coded bit ops, or local / static helpers because these
helpers do not consume any other infra from core/mbox.c.

>  /**
>   * cxl_dev_dynamic_capacity_identify() - Reads the dynamic capacity
> diff --git a/drivers/cxl/cxl.h b/drivers/cxl/cxl.h
> index 15d418b3bc9b..d585f5fdd3ae 100644
> --- a/drivers/cxl/cxl.h
> +++ b/drivers/cxl/cxl.h
> @@ -164,11 +164,13 @@ static inline int ways_to_eiw(unsigned int ways, u8 *eiw)
>  #define CXLDEV_EVENT_STATUS_WARN		BIT(1)
>  #define CXLDEV_EVENT_STATUS_FAIL		BIT(2)
>  #define CXLDEV_EVENT_STATUS_FATAL		BIT(3)
> +#define CXLDEV_EVENT_STATUS_DCD			BIT(4)
>  
>  #define CXLDEV_EVENT_STATUS_ALL (CXLDEV_EVENT_STATUS_INFO |	\
>  				 CXLDEV_EVENT_STATUS_WARN |	\
>  				 CXLDEV_EVENT_STATUS_FAIL |	\
> -				 CXLDEV_EVENT_STATUS_FATAL)
> +				 CXLDEV_EVENT_STATUS_FATAL|	\
> +				 CXLDEV_EVENT_STATUS_DCD)
>  
>  /* CXL rev 3.0 section 8.2.9.2.4; Table 8-52 */
>  #define CXLDEV_EVENT_INT_MODE_MASK	GENMASK(1, 0)
> diff --git a/drivers/cxl/cxlmem.h b/drivers/cxl/cxlmem.h
> index 4624cf612c1e..01bee6eedff3 100644
> --- a/drivers/cxl/cxlmem.h
> +++ b/drivers/cxl/cxlmem.h
> @@ -225,7 +225,9 @@ struct cxl_event_interrupt_policy {
>  	u8 warn_settings;
>  	u8 failure_settings;
>  	u8 fatal_settings;
> +	u8 dcd_settings;
>  } __packed;
> +#define CXL_EVENT_INT_POLICY_BASE_SIZE 4 /* info, warn, failure, fatal */
>  
>  /**
>   * struct cxl_event_state - Event log driver state
> @@ -890,6 +892,8 @@ void cxl_event_trace_record(const struct cxl_memdev *cxlmd,
>  			    enum cxl_event_log_type type,
>  			    enum cxl_event_type event_type,
>  			    const uuid_t *uuid, union cxl_event *evt);
> +bool cxl_dcd_supported(struct cxl_memdev_state *mds);
> +void cxl_disable_dcd(struct cxl_memdev_state *mds);
>  int cxl_set_timestamp(struct cxl_memdev_state *mds);
>  int cxl_poison_state_init(struct cxl_memdev_state *mds);
>  int cxl_mem_get_poison(struct cxl_memdev *cxlmd, u64 offset, u64 len,
> diff --git a/drivers/cxl/pci.c b/drivers/cxl/pci.c
> index 12cd5d399230..ef482eae09e9 100644
> --- a/drivers/cxl/pci.c
> +++ b/drivers/cxl/pci.c
> @@ -669,22 +669,33 @@ static int cxl_event_get_int_policy(struct cxl_memdev_state *mds,
>  }
>  
>  static int cxl_event_config_msgnums(struct cxl_memdev_state *mds,
> -				    struct cxl_event_interrupt_policy *policy)
> +				    struct cxl_event_interrupt_policy *policy,
> +				    bool native_cxl)
>  {
>  	struct cxl_mbox_cmd mbox_cmd;
> +	size_t size_in;
>  	int rc;
>  
> -	*policy = (struct cxl_event_interrupt_policy) {
> -		.info_settings = CXL_INT_MSI_MSIX,
> -		.warn_settings = CXL_INT_MSI_MSIX,
> -		.failure_settings = CXL_INT_MSI_MSIX,
> -		.fatal_settings = CXL_INT_MSI_MSIX,
> -	};
> +	if (native_cxl) {
> +		*policy = (struct cxl_event_interrupt_policy) {
> +			.info_settings = CXL_INT_MSI_MSIX,
> +			.warn_settings = CXL_INT_MSI_MSIX,
> +			.failure_settings = CXL_INT_MSI_MSIX,
> +			.fatal_settings = CXL_INT_MSI_MSIX,
> +			.dcd_settings = 0,

No need to initialize dcd_settings.

> +		};
> +	}
> +	size_in = CXL_EVENT_INT_POLICY_BASE_SIZE;

Let's skip adding this new #define and make it explicit, i.e. wihtout
needing to crack open the spec, that the dcd settings are incremental to
the original payload size:

    size_in = offsetof(typeof(*policy), dcd_settings);

> +
> +	if (cxl_dcd_supported(mds)) {
> +		policy->dcd_settings = CXL_INT_MSI_MSIX;
> +		size_in += sizeof(policy->dcd_settings);

..and then this can just be:

	size_in = sizeof(*policy);

> +	}
>  
>  	mbox_cmd = (struct cxl_mbox_cmd) {
>  		.opcode = CXL_MBOX_OP_SET_EVT_INT_POLICY,
>  		.payload_in = policy,
> -		.size_in = sizeof(*policy),
> +		.size_in = size_in,
>  	};
>  
>  	rc = cxl_internal_send_cmd(mds, &mbox_cmd);
> @@ -731,6 +742,31 @@ static int cxl_event_irqsetup(struct cxl_memdev_state *mds,
>  	return 0;
>  }
>  
> +static int cxl_irqsetup(struct cxl_memdev_state *mds,
> +			struct cxl_event_interrupt_policy *policy,
> +			bool native_cxl)
> +{
> +	struct cxl_dev_state *cxlds = &mds->cxlds;
> +	int rc;
> +
> +	if (native_cxl) {
> +		rc = cxl_event_irqsetup(mds, policy);
> +		if (rc)
> +			return rc;
> +	}
> +
> +	if (cxl_dcd_supported(mds)) {
> +		rc = cxl_event_req_irq(cxlds, policy->dcd_settings);
> +		if (rc) {
> +			dev_err(cxlds->dev, "Failed to get interrupt for DCD event log\n");
> +			cxl_disable_dcd(mds);
> +			return rc;
> +		}
> +	}

I think this could be simplified if cxl_event_req_irq() simply skipped
the non CXL_INT_MSI_MSIX modes. I.e. cxl_event_req_irq() is being too
strict after the policy settings have already run the gauntlet.