Received-SPF: pass (google.com: domain of linux-kernel+bounces-194358-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) client-ip=2604:1380:45d1:ec00::1;
Date: Wed, 29 May 2024 11:25:02 -0400
From: Gregory Price <gregory.price@memverge.com>
To: Dongsheng Yang <dongsheng.yang@easystack.cn>
Cc: Dan Williams <dan.j.williams@intel.com>,
	Jonathan Cameron <Jonathan.Cameron@huawei.com>,
	John Groves <John@groves.net>, axboe@kernel.dk,
	linux-block@vger.kernel.org, linux-kernel@vger.kernel.org,
	linux-cxl@vger.kernel.org, nvdimm@lists.linux.dev
Subject: Re: [PATCH RFC 0/7] block: Introduce CBD (CXL Block Device)
Message-ID: <ZldIzp0ncsRX5BZE@memverge.com>
References: <20240503105245.00003676@Huawei.com>
 <5b7f3700-aeee-15af-59a7-8e271a89c850@easystack.cn>
 <20240508131125.00003d2b@Huawei.com>
 <ef0ee621-a2d2-e59a-f601-e072e8790f06@easystack.cn>
 <20240508164417.00006c69@Huawei.com>
 <3d547577-e8f2-8765-0f63-07d1700fcefc@easystack.cn>
 <20240509132134.00000ae9@Huawei.com>
 <a571be12-2fd3-e0ee-a914-0a6e2c46bdbc@easystack.cn>
 <664cead8eb0b6_add32947d@dwillia2-mobl3.amr.corp.intel.com.notmuch>
 <8f161b2d-eacd-ad35-8959-0f44c8d132b3@easystack.cn>
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
In-Reply-To: <8f161b2d-eacd-ad35-8959-0f44c8d132b3@easystack.cn>
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0:
	=?utf-8?B?WEtrRHh1bzhkSitPc3BJM1AvWGh5ZHpCZkgxOGFiUmxGc1lydy9LUE50Ulp5?=
 =?utf-8?B?VVpjakxGdFBHTDVwM2E2ZzR0VkxkSVlRbWlKSll4ZS9YeFdBVmpHWG94QVNB?=
 =?utf-8?B?RkU5d2kya3IvZ054VnZwZ1VLdGQvTU9taXEwZHNGTzQ0eGFzdFdncThGcHEw?=
 =?utf-8?B?eTM3WnVRRFZ5Z0M0ajZ3N1h2THBDVWRtQ25aMHp5VWV0enR5TjBtSXNhTDcw?=
 =?utf-8?B?bU9HQ3RzYmppSHZjY3pxT3dob2NUcmZEc1lrYVNSa1VZRzZWTjhId0x5UGxu?=
 =?utf-8?B?alZwMU1rSFE0MGJJemwvUElwWWNtTWR5cjhEZGs2VEdVYWlXcjNHN0dIVE54?=
 =?utf-8?B?Qm9mR2Z5UG0wOVBtZ09BYWdNMlVRbi9BaG4xdkIwNVhQdG9ZZVlFdVRjbERx?=
 =?utf-8?B?L3hiZkg5S0ZaUFpscTZmbXhMOW1ScVQydTUrSkc3eE5DRE5ZdTRmZ0RSQ2VR?=
 =?utf-8?B?L2h3b2tJaWppVU5rOTdXV0FNZVFyVnBaY0xodGVVc0k1emFUaXNGaWRmd2o3?=
 =?utf-8?B?NUViS3p5eVMwYVlzNzJXN3ExdHZyYTJvdWFPVTBIWDVQOW5HR043SVVDY1Nz?=
 =?utf-8?B?K1V5WFFnMmR5TGRLa2ZEbVYyNVJ6eXBlb3haS0NEQTFOL2plWE92Q003b2gy?=
 =?utf-8?B?SVF1VGtHZFdWOTBhOHZJclUvM3RWREZleS83dHVFQ3MzOFdueVNvc3JJVkpm?=
 =?utf-8?B?Z2pEU2ZFYjRKeFBYcGRCV3MzMVBxWVQ0YkxXdDJ1eE9raFJiMkZPMGxxSWx6?=
 =?utf-8?B?eEEwTTVDR2RtZTh6Yk5zelQzYkFmTURGakhLdXRUb2xpOVQyNjgrRVQvK3hV?=
 =?utf-8?B?Y0k4b2hYVnVQTWM3S29Ca0JQUDV5Zi9ZeHhzdVNITXFwUVlBWUVaWEs0aHR0?=
 =?utf-8?B?LzVrQ2xLY25vTWVvUC90RUlyaVhmdjBGWDlDRDFPb2hrOTNxS1djMHJiWGpr?=
 =?utf-8?B?Qk5xNXpzUy9qK0pOWnNPeWR0T01jVFQ1UkY0dzdTM1B6bjlodWtuU1Z4cUQx?=
 =?utf-8?B?bHlDenpuNkdKU3l2bWl6NHNXS0hSR2pHbzN1THlFQk5CbEhKaW5wRUZNckFv?=
 =?utf-8?B?QzFUM3Z0NjIwL3hnaDBIdlM1bDg3Rzl3cFZlZGU4L2szVjY1MllxWXN6QW5r?=
 =?utf-8?B?QTA3TjNsT2JaWWFkOXFDajBLY21uK1Jwbm5OQnJRb0xkV2ZTMmtMd09WNnAw?=
 =?utf-8?B?bE9yYjkrV1grbDRTaFl1ZVZJTjh3ajZVcFRZOVJlekF5MDVCdk1FK0NiNnBr?=
 =?utf-8?B?aUNQK3gvL3NDSkNDR3hXc3FBY1ZwV2pVN1FNbUJveU0rS1VpTkx6cDNvMXly?=
 =?utf-8?B?NEYzQnZvUDl2cWttYjQ1UUlrbjNaZGFtYUZ4WnR2eENQTWdpdzFXK3pnb2JF?=
 =?utf-8?B?amFkcDlWVUlhbTVjVnhkUVd5R2RINTg0MWh3WHNqZ091bUd3dlNkMjUrdElP?=
 =?utf-8?B?d1o3cDZBbGIyQTdmQldETlprUUcrNFA2bFVGK0JrRnRpMWFKalVXdGpTdzAv?=
 =?utf-8?B?anNWYlpYN3ZlMS91ODJlRXlsdUIrWnpTY1REL1g0L2M3UWMvMnkxM1dWYlVi?=
 =?utf-8?B?aVZNVWNzcUU2TjBzOGpMaWdPOG5jc3BZSHoxOTFvZDliSDk4NFFIVFVZVXNj?=
 =?utf-8?B?WnNaeStBdU1EdVB5amFYSFRTYWdPdndZR0xnVEI1WG9Yb2ZaaHlFRXg4cWNP?=
 =?utf-8?B?WDFlY2U3RkpKM2V6cm16QlZVTzRXUjFTYm1CYW0zbDZTNDJSK2VLeTJZMldi?=
 =?utf-8?B?dUZPdXZKSDdTSW1TLzQ4SHZGQ2NrZ2lOVWRHbEhEZStZTTluRmVrSEc1Uk1N?=
 =?utf-8?B?SlVOVG16OFlveFQvL3F1YU9jaU12Nk02VjhKQ3JJQm41eTRLRG43dzJMWjAx?=
 =?utf-8?B?UTlGV0E2SlpQVENyRzJXZHcwa3E0YmVpVzZpY0ZZaUg0YmdGVnU4V0tvaUhr?=
 =?utf-8?B?WC94dEREY3FmeXVBdFNZUlovQlhGTGR4KzcwVVVQSThBWVdZc1MrdFd5d3NS?=
 =?utf-8?B?cXJPeEJRQXN4QWNsOVRkUklzVXZ3Yyt2LzEvbUJGRzBoRldrUjViUVc1NEhR?=
 =?utf-8?B?R1lBQkEyY3E4alJwY0VzQWZLYTNwOERENHU2cUpUKzlYQUU0a0RxVzFiZVUw?=
 =?utf-8?B?YnR2OW9pNlYydG43OTRPR3Fwb1hMS1Y0SGd5Wlp2S2p1SlZpcFRDR1FMeTN1?=
 =?utf-8?B?b3c9PQ==?=
X-OriginatorOrg: memverge.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 67490fab-22f1-4781-1a0c-08dc7ff386a3
X-MS-Exchange-CrossTenant-AuthSource: MW4PR17MB5515.namprd17.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 29 May 2024 15:25:09.3714
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 5c90cb59-37e7-4c81-9c07-00473d5fb682
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: Tdkv43Z5hizUY6haSuXiKQekXQjCozgMEhYOrROnJO1Mm0eAlMMr4ksYYR4ZkYeiJY1wxGXEwKzN7t7xPePVR5Ssjka0BczRZdfGHQYEADo=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH0PR17MB4623

On Wed, May 22, 2024 at 02:17:38PM +0800, Dongsheng Yang wrote:
> 
> 
> 在 2024/5/22 星期三 上午 2:41, Dan Williams 写道:
> > Dongsheng Yang wrote:
> > 
> > What guarantees this property? How does the reader know that its local
> > cache invalidation is sufficient for reading data that has only reached
> > global visibility on the remote peer? As far as I can see, there is
> > nothing that guarantees that local global visibility translates to
> > remote visibility. In fact, the GPF feature is counter-evidence of the
> > fact that writes can be pending in buffers that are only flushed on a
> > GPF event.
> 
> Sounds correct. From what I learned from GPF, ADR, and eADR, there would
> still be data in WPQ even though we perform a CPU cache line flush in the
> OS.
> 
> This means we don't have a explicit method to make data puncture all caches
> and land in the media after writing. also it seems there isn't a explicit
> method to invalidate all caches along the entire path.
> 
> > 
> > I remain skeptical that a software managed inter-host cache-coherency
> > scheme can be made reliable with current CXL defined mechanisms.
> 
> 
> I got your point now, acorrding current CXL Spec, it seems software managed
> cache-coherency for inter-host shared memory is not working. Will the next
> version of CXL spec consider it?
> > 

Sorry for missing the conversation, have been out of office for a bit.

It's not just a CXL spec issue, though that is part of it. I think the
CXL spec would have to expose some form of puncturing flush, and this
makes the assumption that such a flush doesn't cause some kind of
race/deadlock issue.  Certainly this needs to be discussed.

However, consider that the upstream processor actually has to generate
this flush.  This means adding the flush to existing coherence protocols,
or at the very least a new instruction to generate the flush explicitly.
The latter seems more likely than the former.

This flush would need to ensure the data is forced out of the local WPQ
AND all WPQs south of the PCIE complex - because what you really want to
know is that the data has actually made it back to a place where remote
viewers are capable of percieving the change.

So this means:
1) Spec revision with puncturing flush
2) Buy-in from CPU vendors to generate such a flush
3) A new instruction added to the architecture.

Call me in a decade or so.


But really, I think it likely we see hardware-coherence well before this.
For this reason, I have become skeptical of all but a few memory sharing
use cases that depend on software-controlled cache-coherency.

There are some (FAMFS, for example). The coherence state of these
systems tend to be less volatile (e.g. mappings are read-only), or
they have inherent design limitations (cacheline-sized message passing
via write-ahead logging only).

~Gregory