Received-SPF: pass (google.com: domain of linux-kernel+bounces-163185-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) client-ip=2604:1380:45e3:2400::1;
Date: Mon, 29 Apr 2024 17:34:36 -0700
From: Dan Williams <dan.j.williams@intel.com>
To: Dongsheng Yang <dongsheng.yang@easystack.cn>, Gregory Price
	<gregory.price@memverge.com>, Dan Williams <dan.j.williams@intel.com>, "John
 Groves" <John@groves.net>
CC: <axboe@kernel.dk>, <linux-block@vger.kernel.org>,
	<linux-kernel@vger.kernel.org>, <linux-cxl@vger.kernel.org>,
	<nvdimm@lists.linux.dev>
Subject: Re: [PATCH RFC 0/7] block: Introduce CBD (CXL Block Device)
Message-ID: <66303c9c98f2_148729450@dwillia2-mobl3.amr.corp.intel.com.notmuch>
References: <20240422071606.52637-1-dongsheng.yang@easystack.cn>
 <66288ac38b770_a96f294c6@dwillia2-mobl3.amr.corp.intel.com.notmuch>
 <ef34808b-d25d-c953-3407-aa833ad58e61@easystack.cn>
 <ZikhwAAIGFG0UU23@memverge.com>
 <bbf692ec-2109-baf2-aaae-7859a8315025@easystack.cn>
 <ZiuwyIVaKJq8aC6g@memverge.com>
 <98ae27ff-b01a-761d-c1c6-39911a000268@easystack.cn>
 <ZivS86BrfPHopkru@memverge.com>
 <8f373165-dd2b-906f-96da-41be9f27c208@easystack.cn>
Content-Type: text/plain; charset="utf-8"
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
In-Reply-To: <8f373165-dd2b-906f-96da-41be9f27c208@easystack.cn>
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?TE5hODVBOUFKVjJZeXB5WUwveXBxUTF4aTBzbnhDa0NHOGdDd2JFTVVGNVdK?=
 =?utf-8?B?UTVpUnpMN0JxdUtMRlpYWER6U3NhVkJUREJqcG5GR1hyWlN2NFVtTkFBTVhu?=
 =?utf-8?B?MkpNS0pxeThXc1VCVXhKUFdCMnpmcS80ZG9pVHNDa1dwdzFOS2M3NUlKc0hk?=
 =?utf-8?B?MVgxcTlSTEpqNlc0TnZVNHlsR0svN0lvTzBFV0tTN0V3dUNmSjVDZWVLbGkv?=
 =?utf-8?B?R2lzSzRFYTA5YlQyMklvRC9ZWDlsZHoyakgwWldlaElpNnZFQ3RaTlFtbWQ3?=
 =?utf-8?B?ZXYvSG4ySDlMOEZTYXFtMG9yTGsvdGlydFE1RUpmejFEenhMSi91RGxNaFRx?=
 =?utf-8?B?QTFpOHZZNkRtbWhXUEZVdXF0VXp5QmJtK2M0T1Vtd0Joa2w4QlA4cTVEVUhL?=
 =?utf-8?B?aTZYZTlFbGs4TEZGWmRYeENpOFFNMkpoL1F5eENVWXQyUkRrYlZuZG1CTEVl?=
 =?utf-8?B?OWlLS05aclNzZVBIODNydkRiNnlwRzNhYllxMVljNmdUMXV4djU3dHVRVkVH?=
 =?utf-8?B?eWpJNHpLa2xCSk9ZZHNBeG1FRHBjOEFXUlNuMHBnRXJkSG8yZnV0Qm5uU20w?=
 =?utf-8?B?VkhlMkRKOWlsZURtZkJTWEVBYk5OczdJaG90NjV5eEo2N2JLdU5XeWJENFFs?=
 =?utf-8?B?MHY4OEVKNUZuRjU1eFRYNkoyOHdiN1BhZUN2K1JOZ3pZWllud2VyVnV2ejha?=
 =?utf-8?B?Y1pIU2tzcEttbTN4bXdsV1BEMS9PZUxERHljVVNQNWpXRFRWdS9rU1IzOXYv?=
 =?utf-8?B?QSt2bzhVTzJkYURaWElHTEZObWdxYnhPYjRINm5aTTI0MFROeXUvM0FVVC9q?=
 =?utf-8?B?WDNqK2o3TGp0Z2svVmtOV2pVMmVJWHhtU0pzOUZweXFyU3V4Y2ZhWlM5dk1X?=
 =?utf-8?B?ZG5SSUkyRnZ5MzdlM3BNVHdhcXpTMWFGSWRTR29QcE5aYVlRN2duMDBOMmtz?=
 =?utf-8?B?djhrRzhGSTFuczFXWEdZMXdDQTdIa2pEVFZsRWRqczIxc29wK2NSNGNiK3da?=
 =?utf-8?B?cEtnSGhRemc0N3ptYlJOM20xQXVldjQ2clBDcWg3UkFLY1NEWjZsTm5wUzl2?=
 =?utf-8?B?bkVtVE5UV2kxQWVwOTdFRzBpMi85cnRIZzM4akJRVTAyUFJQdE5xUjVPS09S?=
 =?utf-8?B?UzNHQ1EwRWM5Z3pnVWFBVXlhWFpjVE0vdklFQzlwQThMTmtXQXRrM01aVGZz?=
 =?utf-8?B?dlRaZlczVi83Z2t4VUlSczFHSzBhUHpMOERDK3g5d01HeHpNQnpUNlRGWHk3?=
 =?utf-8?B?QzZZT2U0Vkw3SHRuUTloYjhFRnZYRmtyQ0VTMnB6eDU5RTk1Vkh5Y0UzRlgw?=
 =?utf-8?B?bnlNYlZyeFNtaUF2ai8yM3lEb1c5a2tXY1hoMFFpWVhmM0k2M3FPaDl6dWRu?=
 =?utf-8?B?TERUOCswd1JHSkpQQnFGditlN1RWaEptTm51SWh1RnZISlRUTXNwbW1ONHRp?=
 =?utf-8?B?VExpYXgvbERDWlZieTFXT1FFaU5BbkJxNW0wMFhpbjV5WW5xSkpnTUFKeG9Y?=
 =?utf-8?B?OTh6NTVtQ2p6MTVOb1Bsek5TSXQ0VnR2VDE3ZzdSbmx0anQwRUI5N3ZEVjFz?=
 =?utf-8?B?WXhnZk9HUERLWElMaEZxVm5aNk5nMW9Qam9XSHpSbjlTZjJDTEo3NVY4aHlw?=
 =?utf-8?B?TWlVQmJtQ1hMejgrbm4zVjM5OGc4azloVC9ZQkRQQ1VMM3owZ0tJdzVPNTVO?=
 =?utf-8?B?UmFRQ1ZnSHdld3lsdzJIeHNtQ2ZFV1FpdlBRd0RxQ2g1YVNEZUJZcDZqRXpw?=
 =?utf-8?B?ZUpRNlNncjUzOGIrcjBxWHh2SWQ0bVRUbmJSMVVtVm9FTU5KcVVRallOWDVO?=
 =?utf-8?B?VWhDZWFTSUx0S05WWDd1cTVWMDk2R0poTElkaWRLNkxwa2txMEhuZ2xxdDMz?=
 =?utf-8?B?VnFrR2lTOFVlT3pZSXZhTnJIYmpiUHU2TG1MOURtWXRlOWIrRnhzT1Q4RUo5?=
 =?utf-8?B?d290QiszdmpKNVdMT3RIY3l0LzQ2aEN4RC8rYlp2cDRXb3BhL0lVWGJmSUZ5?=
 =?utf-8?B?YjdXWEZ1RnlYcGszdCs2Myt3R1pLRkdBT3pDY1NtcHFKdi9EZ1VmMUlSWEZT?=
 =?utf-8?B?eWtGdForWWVVdXJBRnlocW1SZEUxVzl2SDdGZ1dvQ3drYVZETzg3Yk9HMXlo?=
 =?utf-8?B?bXRQV0JHUnI0dHJ3UmxneHpNNzEySWZyelJGdXRuUDNkRkV5NlB0ZEh0OUtQ?=
 =?utf-8?B?WGc9PQ==?=
X-MS-Exchange-CrossTenant-Network-Message-Id: b325a2ea-1001-4579-cb78-08dc68ad51e0
X-MS-Exchange-CrossTenant-AuthSource: PH8PR11MB8107.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 30 Apr 2024 00:34:39.2915
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: O0568Q0h1mUvWZkjfsSPppXKxCXqzTGY5qaHMNGprPmkt2ouorL9kDwedrAOKPMbvGNrBexuuM8hNumFVY3KncXKF2MQHnUUlhKZs8D3iew=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH0PR11MB5127
X-OriginatorOrg: intel.com

Dongsheng Yang wrote:
> 
> 
> 在 2024/4/27 星期六 上午 12:14, Gregory Price 写道:
> > On Fri, Apr 26, 2024 at 10:53:43PM +0800, Dongsheng Yang wrote:
> >>
> >>
> >> 在 2024/4/26 星期五 下午 9:48, Gregory Price 写道:
> >>>
> >>
> >> In (5) of the cover letter, I mentioned that cbd addresses cache coherence
> >> at the software level:
> >>
> >> (5) How do blkdev and backend interact through the channel?
> >> 	a) For reader side, before reading the data, if the data in this channel
> >> may be modified by the other party, then I need to flush the cache before
> >> reading to ensure that I get the latest data. For example, the blkdev needs
> >> to flush the cache before obtaining compr_head because compr_head will be
> >> updated by the backend handler.
> >> 	b) For writter side, if the written information will be read by others,
> >> then after writing, I need to flush the cache to let the other party see it
> >> immediately. For example, after blkdev submits cbd_se, it needs to update
> >> cmd_head to let the handler have a new cbd_se. Therefore, after updating
> >> cmd_head, I need to flush the cache to let the backend see it.
> >>
> > 
> > Flushing the cache is insufficient.  All that cache flushing guarantees
> > is that the memory has left the writer's CPU cache.  There are potentially
> > many write buffers between the CPU and the actual backing media that the
> > CPU has no visibility of and cannot pierce through to force a full
> > guaranteed flush back to the media.
> > 
> > for example:
> > 
> > memcpy(some_cacheline, data, 64);
> > mfence();
> > 
> > Will not guarantee that after mfence() completes that the remote host
> > will have visibility of the data.  mfence() does not guarantee a full
> > flush back down to the device, it only guarantees it has been pushed out
> > of the CPU's cache.
> > 
> > similarly:
> > 
> > memcpy(some_cacheline, data, 64);
> > mfence();
> > memcpy(some_other_cacheline, data, 64);
> > mfence()
> > 
> > Will not guarantee that some_cacheline reaches the backing media prior
> > to some_other_cacheline, as there is no guarantee of write-ordering in
> > CXL controllers (with the exception of writes to the same cacheline).
> > 
> > So this statement:
> > 
> >> I need to flush the cache to let the other party see it immediately.
> > 
> > Is misleading.  They will not see is "immediately", they will see it
> > "eventually at some completely unknowable time in the future".
> 
> This is indeed one of the issues I wanted to discuss at the RFC stage. 
> Thank you for pointing it out.
> 
> In my opinion, using "nvdimm_flush" might be one way to address this 
> issue, but it seems to flush the entire nd_region, which might be too 
> heavy. Moreover, it only applies to non-volatile memory.
> 
> This should be a general problem for cxl shared memory. In theory, FAMFS 
> should also encounter this issue.
> 
> Gregory, John, and Dan, Any suggestion about it?

The CXL equivalent is GPF (Global Persistence Flush), not be confused
with "General Protection Fault" which is likely what will happen if
software needs to manage cache coherency for this solution. CXL GPF was
not designed to be triggered by software. It is hardware response to a
power supply indicating loss of input power.

I do not think you want to spend community resources reviewing software
cache coherency considerations, and instead "just" mandate that this
solution requires inter-host hardware cache coherence. I understand that
is a difficult requirement to mandate, but it is likely less difficult
than getting Linux to carry a software cache coherence mitigation.

In some ways this reminds me of SMR drives and the problems those posed
to software where ultimately the programming difficulties needed to be
solved in hardware, not exported to the Linux kernel to solve.