Received-SPF: pass (google.com: domain of linux-kernel+bounces-160365-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) client-ip=2604:1380:4601:e00::3;
Date: Fri, 26 Apr 2024 12:14:43 -0400
From: Gregory Price <gregory.price@memverge.com>
To: Dongsheng Yang <dongsheng.yang@easystack.cn>
Cc: Dan Williams <dan.j.williams@intel.com>, axboe@kernel.dk,
	John Groves <John@groves.net>, linux-block@vger.kernel.org,
	linux-kernel@vger.kernel.org, linux-cxl@vger.kernel.org
Subject: Re: [PATCH RFC 0/7] block: Introduce CBD (CXL Block Device)
Message-ID: <ZivS86BrfPHopkru@memverge.com>
References: <20240422071606.52637-1-dongsheng.yang@easystack.cn>
 <66288ac38b770_a96f294c6@dwillia2-mobl3.amr.corp.intel.com.notmuch>
 <ef34808b-d25d-c953-3407-aa833ad58e61@easystack.cn>
 <ZikhwAAIGFG0UU23@memverge.com>
 <bbf692ec-2109-baf2-aaae-7859a8315025@easystack.cn>
 <ZiuwyIVaKJq8aC6g@memverge.com>
 <98ae27ff-b01a-761d-c1c6-39911a000268@easystack.cn>
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
In-Reply-To: <98ae27ff-b01a-761d-c1c6-39911a000268@easystack.cn>
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0:
	=?utf-8?B?SlZwTzRlNnNHQWt0WnpmRkNGQ3oyMmQ4YmJnWFlzVFFhV1EyQm85dEVmMWdy?=
 =?utf-8?B?cm5MVENXa0dHcHFmSm9lV2s5YzFDTmI2VjUrd2xzNU5NVzB2ZENsME53WW9x?=
 =?utf-8?B?VUNRZDE2QTFEZENHSFZpUkt2ZDA4ZnlBZGNvUGQyWGsyM0RtaTVWV1dsbGUw?=
 =?utf-8?B?SXFBWU5vUFlzVldqYlYzdWlsd2IyWUdUU1M3Sk1nNFJMUWRzRFJBWUo1b2NG?=
 =?utf-8?B?T2h2bFpTYVpQd29ZbXpCR0M3SHp3RFdzNXJFdmt6SjhnSGw5c1c5YSttajUr?=
 =?utf-8?B?RnYwQmJVZHBWQitGNUY0bW0vdGhMd2lZRmwrL016MGZQVldUMlQ1c2hOUEkx?=
 =?utf-8?B?L2E3b2R4elB5QnlXYjNlK2R4MHVQSndQWk1OUVY1djNhK0RHOUkxMWI4V0tq?=
 =?utf-8?B?U1JzS1FHZENzdVhLV0NJYlZZZFNPQzJKaWkxcXVKSmhVTXNSY1lFVFFjK1Vn?=
 =?utf-8?B?VmtBSmhYQTUxSkplejh4MFZJZEVNM3piYU9nVzRkSy9MdWt3T3g4SnBZaWsy?=
 =?utf-8?B?VWh6RWQ0a2VURkMyT2xYNjFFdkVTc3RaakpHVWlIeWtvWkNwenRYOEdSemtR?=
 =?utf-8?B?ZCs2aHhFTXRWZ25uOG9vci9IWE90MFRDYys3a1BSQlhhdzN3ellicFhsSmRS?=
 =?utf-8?B?OVg2VkNLNXlJY0xFMlhKYnJEM2MrZlc0WTBUMGJGTE9obE5FcXA4UDB4eHpm?=
 =?utf-8?B?OU1ocWU0UDhuVGtxRWs0dFhMdjJROWRib1FrWk8wM1Avc0N2WmJUbDVNRzAr?=
 =?utf-8?B?aFMraVU2d1RxalpNRGNIb2c3Y3IzRnlPU2F6WEZsN3lrQWJFMWtSTW5YYWZn?=
 =?utf-8?B?UlY5SkZjcVdza0dYVGt1TTZVZ0dnV003MUlyNVhLd3I3RzhSTVZFako0cnlY?=
 =?utf-8?B?bjhPeVJLak1WclMzaUdmR1FaTGRocTFsWkw5NW92bHFJNDZrSU4wOFBCNTRl?=
 =?utf-8?B?R2V6UUlyZU5yajNBUzFkckxMelppS0tnNjdzYWhJU0xPNzNsUHY1SjBlOXkw?=
 =?utf-8?B?SFErZWRleUV1TkgxV3Y4ejh3Wk45WGErSlo1anVTem13OUczbktBc3pmdkFo?=
 =?utf-8?B?Yjk2Z0xMdTFOSVo2c3pYdGFlTWtOQkRacXZHbkl4NWpRMlJMZVlaOTI5N1Fv?=
 =?utf-8?B?dnYwRWc0VGZJZVFWcjBwd25FMC82U0hvQmZJQlQraXZrQm5Jems4VlUyNDdz?=
 =?utf-8?B?aXhPc3IremVrOUlsaThxcG9ZUFpUSTZCdnFsbklnYnpaSW5JemRmeW9DMkR5?=
 =?utf-8?B?bDdJa2hCeTA0NkVNSjJxdDlWVUFKTG45M0czRnJ5R2ZDeDRXdDJBYjY5c2th?=
 =?utf-8?B?algya1B1VGxNbWZJaEp4d0k3VTNqb1psV20xcEgwczVQRGVCS1pBZVpQSlNT?=
 =?utf-8?B?dFJOUjVwbjJ1aVhINXhPUy9TUDFtcmFXeGxkZ0dQeWsvekpINTFKMHc0Vzhs?=
 =?utf-8?B?c2REWVVLeDM1T3lUdHlic1pyVTFVd2h3NDVSSDlSdzVBTkxyYzdVdnd0OEtj?=
 =?utf-8?B?Tm1LL2ZtcFd1bms3eUQ3NmdWWnJFSHhmbXV2WGJuUVNPTjN4eU43Yk9qcFE5?=
 =?utf-8?B?d1h3THdMaE42RVFVMFlmbU1MbmFvK2hWMUlPTVYzZFJjQ1FhNUh6dDhiaGZD?=
 =?utf-8?B?WG1CL3o0LzcySVZnRGRTNE1hQ3ptOTBBK1BxRkhZWjRpQ0tvQi9DQ1BiZ0tM?=
 =?utf-8?B?ZW5jVk9rZ2MzU1ZpSlVTWGJGWXptalQvTkQzQUpNL3o4S0xzdGhKUCtGU2xj?=
 =?utf-8?B?d2NvSk9ieTlaQ3RmMDlSM05XYVR5d1k3Y3FrMWVjZkMrdWdRR2NzN240SVpy?=
 =?utf-8?B?RiszVU92Z1cwajIzUTQxWCtKcWlDMUxQUHluZkVVQzdGc0l0bFRNbi9rbzI5?=
 =?utf-8?B?ZFVJWmtLTzBvSjNwd0c5eFN2OXFHVmE5a3huVWxjY1laWE4xSFpHV2FRNytB?=
 =?utf-8?B?WlppMTRZMEZERnhNMjYzQllneFdBNVZQT0dHbFZTcFIxcUpBK2hXT1l5UUxq?=
 =?utf-8?B?WGJmcWpBelhHSVdLb2ordWZtOEVpbUJINDFicWFuWDdzSUVqSzIyY2UxNmcr?=
 =?utf-8?B?VTFpdndlQzZrN3ZiNWxMMG1FSlE1ampMOG5Rb1g5VFFkc0JKUWRscUxBcnIr?=
 =?utf-8?B?Zm1WaUdHMk9RRVJ5YXVSa2hvZkdzbDN3Z09EVXFSUWt0T3kvTEVZMlZWbWVH?=
 =?utf-8?B?WXc9PQ==?=
X-OriginatorOrg: memverge.com
X-MS-Exchange-CrossTenant-Network-Message-Id: dff0d860-f073-4b58-18a9-08dc660bfde5
X-MS-Exchange-CrossTenant-AuthSource: SJ0PR17MB5512.namprd17.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 26 Apr 2024 16:14:47.2017
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 5c90cb59-37e7-4c81-9c07-00473d5fb682
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: H5pxphtB5vs5yo1TPRZyWCBtIEndtLPBnKaEtcWTwSsSqR7wazqbbjmSg+PfujttWy1GLvn6EZ4MrkK0azKHm9wvFoS8sf7CJzoiSa223Mg=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: MW5PR17MB6009

On Fri, Apr 26, 2024 at 10:53:43PM +0800, Dongsheng Yang wrote:
> 
> 
> 在 2024/4/26 星期五 下午 9:48, Gregory Price 写道:
> > 
> > Also, on a tangential note, you're using pmem/qemu to emulate the
> > behavior of shared CXL memory.  You should probably explain the
> > coherence implications of the system more explicitly.
> > 
> > The emulated system implements what amounts to hardware-coherent
> > memory (i.e. the two QEMU machines run on the same physical machine,
> > so coherency is managed within the same coherence domain).
> > 
> > If there is no explicit coherence control in software, then it is
> > important to state that this system relies on hardware that implements
> > snoop back-invalidate (which is not a requirement of a CXL 3.x device,
> > just a feature described by the spec that may be implemented).
> 
> In (5) of the cover letter, I mentioned that cbd addresses cache coherence
> at the software level:
> 
> (5) How do blkdev and backend interact through the channel?
> 	a) For reader side, before reading the data, if the data in this channel
> may be modified by the other party, then I need to flush the cache before
> reading to ensure that I get the latest data. For example, the blkdev needs
> to flush the cache before obtaining compr_head because compr_head will be
> updated by the backend handler.
> 	b) For writter side, if the written information will be read by others,
> then after writing, I need to flush the cache to let the other party see it
> immediately. For example, after blkdev submits cbd_se, it needs to update
> cmd_head to let the handler have a new cbd_se. Therefore, after updating
> cmd_head, I need to flush the cache to let the backend see it.
> 

Flushing the cache is insufficient.  All that cache flushing guarantees
is that the memory has left the writer's CPU cache.  There are potentially
many write buffers between the CPU and the actual backing media that the
CPU has no visibility of and cannot pierce through to force a full
guaranteed flush back to the media.

for example:

memcpy(some_cacheline, data, 64);
mfence();

Will not guarantee that after mfence() completes that the remote host
will have visibility of the data.  mfence() does not guarantee a full
flush back down to the device, it only guarantees it has been pushed out
of the CPU's cache.

similarly:

memcpy(some_cacheline, data, 64);
mfence();
memcpy(some_other_cacheline, data, 64);
mfence()

Will not guarantee that some_cacheline reaches the backing media prior
to some_other_cacheline, as there is no guarantee of write-ordering in
CXL controllers (with the exception of writes to the same cacheline).

So this statement:

> I need to flush the cache to let the other party see it immediately.

Is misleading.  They will not see is "immediately", they will see it
"eventually at some completely unknowable time in the future".

~Gregory