Received-SPF: pass (google.com: domain of linux-kernel+bounces-163173-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.199.223 as permitted sender) client-ip=147.75.199.223;
Date: Mon, 29 Apr 2024 17:10:31 -0700
From: Dan Williams <dan.j.williams@intel.com>
To: Dongsheng Yang <dongsheng.yang@easystack.cn>, Dan Williams
	<dan.j.williams@intel.com>, <axboe@kernel.dk>, John Groves <John@groves.net>
CC: <linux-block@vger.kernel.org>, <linux-kernel@vger.kernel.org>,
	<linux-cxl@vger.kernel.org>, Dongsheng Yang <dongsheng.yang.linux@gmail.com>
Subject: Re: [PATCH RFC 0/7] block: Introduce CBD (CXL Block Device)
Message-ID: <663036f72b643_148729418@dwillia2-mobl3.amr.corp.intel.com.notmuch>
References: <20240422071606.52637-1-dongsheng.yang@easystack.cn>
 <66288ac38b770_a96f294c6@dwillia2-mobl3.amr.corp.intel.com.notmuch>
 <ef34808b-d25d-c953-3407-aa833ad58e61@easystack.cn>
 <66294a9a8e9a8_b6e02947d@dwillia2-mobl3.amr.corp.intel.com.notmuch>
 <539c1323-68f9-d753-a102-692b69049c20@easystack.cn>
Content-Type: text/plain; charset="utf-8"
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
In-Reply-To: <539c1323-68f9-d753-a102-692b69049c20@easystack.cn>
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?cWE1SVhPbC85UFpqRE9qN2MxV20wL0I0dEFkR3BIQU1XRFUwanE5WmpabzBq?=
 =?utf-8?B?V2ovNjZRY21RdGVQRmJ5ZmRHNzJtWUpOL0Z5VlJIZFFtRFhDdkRMRUhHSU82?=
 =?utf-8?B?Nnlrb0JBY0RzdWp0bzZTMEtxMVlnc3hYRC9UMVkrUks4T3ZDNDJ1bkVCazZ6?=
 =?utf-8?B?TWNuQkVvVERxSzYvR3hPbVhUY2JKMVhFLzhZQzlYdDBEby9HYW9NU1VFMTU2?=
 =?utf-8?B?ZnhvdnljZGVPNTIvY0JFRmV0bnZUNjQ1S3FpMG5oK2VtdjUvOXZZNG15S0F4?=
 =?utf-8?B?UUFyUHAxUUd1TllYUC9DRDFmRkV3ckJlQ051KzRvNXhsbllKN2tweEF5UlY5?=
 =?utf-8?B?aFYreHBmMXVJd2N6OXlwUDgzeDZra2ZvQWErZEZTQVFjcWFXQTlpODBaNE14?=
 =?utf-8?B?TGJnS203WnFwTkErOExOdmJhODRVR1FZN1RGTCthajhGeGpaYzJ2MXZ2aDVB?=
 =?utf-8?B?WHlEZTlSbWhZbG14dHZMUVVYUmUzWi9jUXhLajVYeFdUUEpxQTMrdVBHcldX?=
 =?utf-8?B?OUtnL25JN0d5N3EzcWYzUEJ4ckR6SUU1Q0NKSlZYSVloRm5OSzRuRllOeS9y?=
 =?utf-8?B?RFk2TThWSVBsWFM1WkpKd2NXZC9aaEpka0hJWnM2Z3dEZWJJRkhpS2FiWFd3?=
 =?utf-8?B?UEVuSW5rMDF4K0lXNEtpeFBvZWNmRURPdDNmdkxodDR4V2VLUDFjWVN0MjRV?=
 =?utf-8?B?Nk0reGNmYnFRZEhjRHhyWjh0NVBpSlVkQTNHZURzMzVyUkFNcG9iS3YyRUFY?=
 =?utf-8?B?eEpyVWg3bUhuSnRCREVIbUxGeWhBSVowb0cwRHdmaXh1bzhwdnZtTVZOUGMz?=
 =?utf-8?B?Z2NWZlV5YmdweGs3dXRkdWtpdnZ2emUwY21yTGVyaXNjYk5VdjVjMlZwdjR5?=
 =?utf-8?B?eGoxNmRyT1pJeDRGSnNBb0p4MUJMaXNTbHNsV3RQR0tSV2d0Z3RXTzhQNUs0?=
 =?utf-8?B?VzNGT1pBWVVrZXl2bGxuSTFKRUpVSGlaREFyVXMvRVdvWjJhK1p1bU5vUDJK?=
 =?utf-8?B?UnA0T2FhRGhNS3MwTW83d0dCbWlDUWQxUVp0WnBwd3hEb3JuS085ejloZlM3?=
 =?utf-8?B?QktSMDNpLzZlZmkydHZiQUg4YXVTYURUVERuZTcyWEF1eDVIeXVPVmErVjBB?=
 =?utf-8?B?TDFtRGppVHZQUWVITTZicWFlMXRhUWJSNFBrVHU3bnpQZ21jZFZDQnA5OFpJ?=
 =?utf-8?B?SkQrOUM2K2p2YWRwZHZVQVRCbEtield3NkNYUXVMTk92eE1iMGlKOXFnNjhD?=
 =?utf-8?B?ZW9Kd3JoNzYzN2tYd3NGS2N0c1RDbGxxS1R1elorN1I0cEJqT1AyV3dIL09L?=
 =?utf-8?B?emJCUU1HSm9Nd3FDblE5eDBBbGYzZW45M1FhMTdQWElzYlBpTXhHMGFUd1hG?=
 =?utf-8?B?V2x5WEZocm5pZHM3b2pkOVdEcGV6Z3k3dFJ3RjZVYk1mOUZ3OXQxdkEyZ0Vo?=
 =?utf-8?B?QndXRGpDaSs3dnMxZjFuclZoTmxMUHhmc0pBYzZPd1VqOEFEa0FBWGtvbTg4?=
 =?utf-8?B?Q0pRNWRJdEl2cDA0VUpKZWsycVJHVXM3U3k3RzRBZndxYlVQdkw4elhicito?=
 =?utf-8?B?WGtZN05jekJhKzROeTdOMytYeENHTVBGeG1PbFBDVzdaM1ZwdDYzb0h4QzVp?=
 =?utf-8?B?anRySTBiSmhlOUFiN1poRXVOd1FBQXJOdzdTSXRXQ20wQzJjNnJxaklYNUF2?=
 =?utf-8?B?MmJMSHovSU15cmg5elkwMjh2TVhwRHVFajdObTZ0bmZXRlRjL05sOXAwZGlG?=
 =?utf-8?B?SXBKZkVlR204bGdsTUZjZG83TFl0QUR4VHpCSjJ6YmVEMGwyMmlvYzdPTSt4?=
 =?utf-8?B?Y1R4SXRpN2FkbStiQ3VLV1Z3MVR2eHNDVURxTkk3dW80a3Jpazc5czdvNDN2?=
 =?utf-8?B?Rmp5RXBOS2pnN3l3WG45MVBJa1F4MXI4dk1jaVVrYXlWYUYxSG11SUlYcGds?=
 =?utf-8?B?Y0FvM3NJekUzYTJtQXluN1N4RXllb3ZIYzNyVkl1dXhqaXRtMWVZc3YxOHZC?=
 =?utf-8?B?VkhhY09mKzYvV2ZJUEhRZWRmVytZcUlEcVovR0Z4ZkkxYmNEbjNuSEJEREQv?=
 =?utf-8?B?UnlqRjNPbnBkRzJvL1AyK0EzYytXcGYyK0UrQStSWnNWTEdUSk5Ec21wNUdZ?=
 =?utf-8?B?TUJwc0dJcVJyMXZEaVRsWEJOczBZNWpydkljOXVaSEtvbm5sUk9HMFFLV2t1?=
 =?utf-8?B?Y1E9PQ==?=
X-MS-Exchange-CrossTenant-Network-Message-Id: af90842d-1233-437d-f402-08dc68a9f42e
X-MS-Exchange-CrossTenant-AuthSource: PH8PR11MB8107.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 30 Apr 2024 00:10:33.7228
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: RkQfMpOs/AwSCjXm3DiSZXzmYyNih8DiI1fm3fQ7UvA+Bka5/+oSfmDSPnRcBeY9nnfys8ReFaYWKKRNee4IF3ziMIlsGUhVILXIEcTs6M8=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH7PR11MB6883
X-OriginatorOrg: intel.com

Dongsheng Yang wrote:
> 
> 
> 在 2024/4/25 星期四 上午 2:08, Dan Williams 写道:
> > Dongsheng Yang wrote:
> >>
> >>
> >> 在 2024/4/24 星期三 下午 12:29, Dan Williams 写道:
> >>> Dongsheng Yang wrote:
> >>>> From: Dongsheng Yang <dongsheng.yang.linux@gmail.com>
> >>>>
> >>>> Hi all,
> >>>> 	This patchset introduce cbd (CXL block device). It's based on linux 6.8, and available at:
> >>>> 	https://github.com/DataTravelGuide/linux
> >>>>
> >>> [..]
> >>>> (4) dax is not supported yet:
> >>>> 	same with famfs, dax device is not supported here, because dax device does not support
> >>>> dev_dax_iomap so far. Once dev_dax_iomap is supported, CBD can easily support DAX mode.
> >>>
> >>> I am glad that famfs is mentioned here, it demonstrates you know about
> >>> it. However, unfortunately this cover letter does not offer any analysis
> >>> of *why* the Linux project should consider this additional approach to
> >>> the inter-host shared-memory enabling problem.
> >>>
> >>> To be clear I am neutral at best on some of the initiatives around CXL
> >>> memory sharing vs pooling, but famfs at least jettisons block-devices
> >>> and gets closer to a purpose-built memory semantic.
> >>>
> >>> So my primary question is why would Linux need both famfs and cbd? I am
> >>> sure famfs would love feedback and help vs developing competing efforts.
> >>
> >> Hi,
> >> 	Thanks for your reply, IIUC about FAMfs, the data in famfs is stored in
> >> shared memory, and related nodes can share the data inside this file
> >> system; whereas cbd does not store data in shared memory, it uses shared
> >> memory as a channel for data transmission, and the actual data is stored
> >> in the backend block device of remote nodes. In cbd, shared memory works
> >> more like network to connect different hosts.
> >>
> >> That is to say, in my view, FAMfs and cbd do not conflict at all; they
> >> meet different scenario requirements. cbd simply uses shared memory to
> >> transmit data, shared memory plays the role of a data transmission
> >> channel, while in FAMfs, shared memory serves as a data store role.
> > 
> > If shared memory is just a communication transport then a block-device
> > abstraction does not seem a proper fit. From the above description this
> > sounds similar to what CONFIG_NTB_TRANSPORT offers which is a way for
> > two hosts to communicate over a shared memory channel.
> > 
> > So, I am not really looking for an analysis of famfs vs CBD I am looking
> > for CBD to clarify why Linux should consider it, and why the
> > architecture is fit for purpose.
> 
> Let me explain why we need cbd:
> 
> In cloud storage scenarios, we often need to expose block devices of 
> storage nodes to compute nodes. We have options like nbd, iscsi, nvmeof, 
> etc., but these all communicate over the network. cbd aims to address 
> the same scenario but using shared memory for data transfer instead of 
> the network, aiming for better performance and reduced network latency.
> 
> Furthermore, shared memory can not only transfer data but also implement 
> features like write-ahead logging (WAL) or read/write cache, further 
> improving performance, especially latency-sensitive business scenarios. 
> (If I understand correctly, this might not be achievable with the 
> previously mentioned ntb.)
> 
> To ensure we have a common understanding, I'd like to clarify one point: 
> the /dev/cbdX block device is not an abstraction of shared memory; it is 
> a mapping of a block device (such as /dev/sda) on the remote host. 
> Reading/writing to /dev/cbdX is equivalent to reading/writing to 
> /dev/sda on the remote host.
> 
> This is the design intention of cbd. I hope this clarifies things.

I does, thanks for the clarification. Let me go back and take a another
look now that I undertand that this is a "remote storage target over CXL
memory" solution.