Received: by 2002:ab2:6991:0:b0:1f7:f6c3:9cb1 with SMTP id v17csp1101364lqo; Thu, 9 May 2024 05:03:12 -0700 (PDT) X-Forwarded-Encrypted: i=3; AJvYcCVk9DSpiX8Hdzhwfegg8sbZM88SZTaE9NPJiXBuRfsgZ1sGh0JI+7aekY/4JSDmWl4yJ46ojvpUvFexPsGKvCiaPHA67k+4fIOlbN087g== X-Google-Smtp-Source: AGHT+IHhwJb471+/oPAS2HQQIDUPOaWC+pUjW1TVdojigKHu/Wu/qISotYm6mY/VsMcIvWFMbBq0 X-Received: by 2002:a05:6830:e0e:b0:6ee:636f:7e1d with SMTP id 46e09a7af769-6f0b793ff9amr5440207a34.9.1715256192200; Thu, 09 May 2024 05:03:12 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1715256192; cv=pass; d=google.com; s=arc-20160816; b=cSzX4DW2+XcoslFQT8rkehUJjM7gUmhh8NtoU9cJxTYCuKz9rqE5mx9dkmkrn6GWrG YJIMRBmmJs+EygmizwemniIAd55PrGD+6dXK5hssF067U2Mf2p3cvwqmVAYoagSZm7h2 oYK3x6bOE+6wVdoY8ew4adZ7nqDLrpHZjBn8n5mEua9cAUUICURqLa/Y+p/T+WnXeYSM qpvooCnYllRegdrWv0HFugbbfn0F2uJScZWJafIW0HxhU/6SAhXhLm9oE+0SnR3oSUHt KTxzF+KPn6SvRKA4V8pHpXcUVHXXKZvc4ggkDjvt0003HfmjiqAmsWUdxg0JJvSidzzv f8Dw== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:in-reply-to:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:user-agent:date:message-id:from :references:cc:to:subject; bh=Nq1FA0MIzq7zr8xDdroN+pZwnfx1ijW/A9Fm1jlv6X0=; fh=+jpsVRCZ3FEbhLPyPwEr4f65OZFumgBTFeeTOceFnP0=; b=yBqaGSM3fy+RIqxNM+ZUdtueuF9mW59QJoAXcVPFjm2wiPHypCHP/AUmVljQ006Q8t weHL0w1KiT/O2IgblaT/BUKsVBVEp8QMP6UdRnr6YkB4YshmXYedhXuMb/lZ9pgFXs6M ByzHQI5KEZkJ0V7eq7r9XDZQOzwM+ZMNHKmwY7qWT8hdRWJ6J1nsP0gfwV5u6X39+nA0 bSrOmU3zKZWsQaNbYeiCVkf24skfz32AyP9BYFWaT4p8kzEpjc9chsajmsO+noE0yIjg ZuLCB3thmNFZF62XMR0gv+f5J1CN4LqwfltAUfW++lj0d+4bi/xwESWjrpBkrzR3D2oe 96wQ==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; arc=pass (i=1 spf=pass spfdomain=easystack.cn dmarc=pass fromdomain=easystack.cn); spf=pass (google.com: domain of linux-kernel+bounces-174465-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-174465-linux.lists.archive=gmail.com@vger.kernel.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=easystack.cn Return-Path: Received: from sv.mirrors.kernel.org (sv.mirrors.kernel.org. [2604:1380:45e3:2400::1]) by mx.google.com with ESMTPS id 46e09a7af769-6f0e021bfdesi525525a34.201.2024.05.09.05.03.11 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 09 May 2024 05:03:12 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel+bounces-174465-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) client-ip=2604:1380:45e3:2400::1; Authentication-Results: mx.google.com; arc=pass (i=1 spf=pass spfdomain=easystack.cn dmarc=pass fromdomain=easystack.cn); spf=pass (google.com: domain of linux-kernel+bounces-174465-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-174465-linux.lists.archive=gmail.com@vger.kernel.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=easystack.cn Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sv.mirrors.kernel.org (Postfix) with ESMTPS id CF4EE282F89 for ; Thu, 9 May 2024 12:03:11 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 8753214B091; Thu, 9 May 2024 12:02:50 +0000 (UTC) Received: from mail-m17214.xmail.ntesmail.com (mail-m17214.xmail.ntesmail.com [45.195.17.214]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 53B8514AD32; Thu, 9 May 2024 12:02:34 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=45.195.17.214 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715256169; cv=none; b=NhFzEI+QmX3JH2Sk/GvosOJA7zoqvfPD10kKcmO/KQ8FYyyDFcUZ0oET1vD21rRoeibU+ZMEQLhInnlrwSunfB9zCkWWkZir6UFnkcMDUcNpLjugi94uulVAn8qPjpUr368tW4yKIu5HzkK6/+tYqvE1nnC2kt8PQBcNG6+1MOU= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715256169; c=relaxed/simple; bh=KnUfaCc3MpuPBR3EdwXghR15Y3buYrD1ThIZlSRopRE=; h=Subject:To:Cc:References:From:Message-ID:Date:MIME-Version: In-Reply-To:Content-Type; b=IZ0uQElDlYKSe9TGsWBnBCbsf/pA6yk69GOjetwHYvT3g4LzSsF7Wdu7P7sEyriCdz7M3g7F9Yq9ihGfrroveuYutYK/sO1lvtK6Hi8+Y+Y2P4JKfThsAdwOO9YrPcR0veh0GNuRHj4cSaZs1+V6o/kFcJrOBkIHp7UmyvwHXWs= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=easystack.cn; spf=pass smtp.mailfrom=easystack.cn; arc=none smtp.client-ip=45.195.17.214 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=easystack.cn Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=easystack.cn Received: from [192.168.122.189] (unknown [218.94.118.90]) by smtp.qiye.163.com (Hmail) with ESMTPA id D7F2986021B; Thu, 9 May 2024 19:24:29 +0800 (CST) Subject: Re: [PATCH RFC 0/7] block: Introduce CBD (CXL Block Device) To: Jonathan Cameron Cc: John Groves , Dan Williams , Gregory Price , axboe@kernel.dk, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-cxl@vger.kernel.org, nvdimm@lists.linux.dev References: <20240422071606.52637-1-dongsheng.yang@easystack.cn> <66288ac38b770_a96f294c6@dwillia2-mobl3.amr.corp.intel.com.notmuch> <98ae27ff-b01a-761d-c1c6-39911a000268@easystack.cn> <8f373165-dd2b-906f-96da-41be9f27c208@easystack.cn> <20240503105245.00003676@Huawei.com> <5b7f3700-aeee-15af-59a7-8e271a89c850@easystack.cn> <20240508131125.00003d2b@Huawei.com> <20240508164417.00006c69@Huawei.com> From: Dongsheng Yang Message-ID: <3d547577-e8f2-8765-0f63-07d1700fcefc@easystack.cn> Date: Thu, 9 May 2024 19:24:28 +0800 User-Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:78.0) Gecko/20100101 Thunderbird/78.4.0 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 In-Reply-To: <20240508164417.00006c69@Huawei.com> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit X-HM-Spam-Status: e1kfGhgUHx5ZQUpXWQgPGg8OCBgUHx5ZQUlOS1dZFg8aDwILHllBWSg2Ly tZV1koWUFJQjdXWS1ZQUlXWQ8JGhUIEh9ZQVlDHh4YVkxKHRpMGUtLSUxCTVUZERMWGhIXJBQOD1 lXWRgSC1lBWUlKQ1VCT1VKSkNVQktZV1kWGg8SFR0UWUFZT0tIVUpNT0lMTlVKS0tVSkJLS1kG X-HM-Tid: 0a8f5d18fe9a023ckunmd7f2986021b X-HM-MType: 1 X-HM-Sender-Digest: e1kMHhlZQR0aFwgeV1kSHx4VD1lBWUc6PBw6Dhw5Qzc1EUouDEoKEw8j LjhPFA5VSlVKTEpOSU5IQ0xLTUJCVTMWGhIXVR8UFRwIEx4VHFUCGhUcOx4aCAIIDxoYEFUYFUVZ V1kSC1lBWUlKQ1VCT1VKSkNVQktZV1kIAVlBSklDTk03Bg++ 在 2024/5/8 星期三 下午 11:44, Jonathan Cameron 写道: > On Wed, 8 May 2024 21:03:54 +0800 > Dongsheng Yang wrote: > >> 在 2024/5/8 星期三 下午 8:11, Jonathan Cameron 写道: >>> On Wed, 8 May 2024 19:39:23 +0800 >>> Dongsheng Yang wrote: >>> >>>> 在 2024/5/3 星期五 下午 5:52, Jonathan Cameron 写道: >>>>> On Sun, 28 Apr 2024 11:55:10 -0500 >>>>> John Groves wrote: >>>>> >>>>>> On 24/04/28 01:47PM, Dongsheng Yang wrote: >>>>>>> >>>>>>> >>>>>>> 在 2024/4/27 星期六 上午 12:14, Gregory Price 写道: >>>>>>>> On Fri, Apr 26, 2024 at 10:53:43PM +0800, Dongsheng Yang wrote: >>>>>>>>> >>>>>>>>> >>>>>>>>> 在 2024/4/26 星期五 下午 9:48, Gregory Price 写道: >>>>>>>>>> >>>>>>>>> >>>> >>>> ... >>>>>> >>>>>> Just to make things slightly gnarlier, the MESI cache coherency protocol >>>>>> allows a CPU to speculatively convert a line from exclusive to modified, >>>>>> meaning it's not clear as of now whether "occasional" clean write-backs >>>>>> can be avoided. Meaning those read-only mappings may be more important >>>>>> than one might think. (Clean write-backs basically make it >>>>>> impossible for software to manage cache coherency.) >>>>> >>>>> My understanding is that clean write backs are an implementation specific >>>>> issue that came as a surprise to some CPU arch folk I spoke to, we will >>>>> need some path for a host to say if they can ever do that. >>>>> >>>>> Given this definitely effects one CPU vendor, maybe solutions that >>>>> rely on this not happening are not suitable for upstream. >>>>> >>>>> Maybe this market will be important enough for that CPU vendor to stop >>>>> doing it but if they do it will take a while... >>>>> >>>>> Flushing in general is as CPU architecture problem where each of the >>>>> architectures needs to be clear what they do / specify that their >>>>> licensees do. >>>>> >>>>> I'm with Dan on encouraging all memory vendors to do hardware coherence! >>>> >>>> Hi Gregory, John, Jonathan and Dan: >>>> Thanx for your information, they help a lot, and sorry for the late reply. >>>> >>>> After some internal discussions, I think we can design it as follows: >>>> >>>> (1) If the hardware implements cache coherence, then the software layer >>>> doesn't need to consider this issue, and can perform read and write >>>> operations directly. >>> >>> Agreed - this is one easier case. >>> >>>> >>>> (2) If the hardware doesn't implement cache coherence, we can consider a >>>> DMA-like approach, where we check architectural features to determine if >>>> cache coherence is supported. This could be similar to >>>> `dev_is_dma_coherent`. >>> >>> Ok. So this would combine host support checks with checking if the shared >>> memory on the device is multi host cache coherent (it will be single host >>> cache coherent which is what makes this messy) >>>> >>>> Additionally, if the architecture supports flushing and invalidating CPU >>>> caches (`CONFIG_ARCH_HAS_SYNC_DMA_FOR_DEVICE`, >>>> `CONFIG_ARCH_HAS_SYNC_DMA_FOR_CPU`, >>>> `CONFIG_ARCH_HAS_SYNC_DMA_FOR_CPU_ALL`), >>> >>> Those particular calls won't tell you much at all. They indicate that a flush >>> can happen as far as a common point for DMA engines in the system. No >>> information on whether there are caches beyond that point. >>> >>>> >>>> then we can handle cache coherence at the software layer. >>>> (For the clean writeback issue, I think it may also require >>>> clarification from the architecture, and how DMA handles the clean >>>> writeback problem, which I haven't further checked.) >>> >>> I believe the relevant architecture only does IO coherent DMA so it is >>> never a problem (unlike with multihost cache coherence).Hi Jonathan, >> >> let me provide an example, >> In nvmeof-rdma, the `nvme_rdma_queue_rq` function places a request into >> `req->sqe.dma`. >> >> (1) First, it calls `ib_dma_sync_single_for_cpu()`, which invalidates >> the CPU cache: >> >> >> ib_dma_sync_single_for_cpu(dev, sqe->dma, >> sizeof(struct nvme_command), DMA_TO_DEVICE); >> >> >> For example, on ARM64, this would call `arch_sync_dma_for_cpu`, followed >> by `dcache_inval_poc(start, start + size)`. > > Key here is the POC. It's a flush to the point of coherence of the local > system. It has no idea about interhost coherency and is not necessarily > the DRAM (in CXL or otherwise). > > If you are doing software coherence, those devices will plug into today's > hosts and they have no idea that such a flush means pushing out into > the CXL fabric and to the type 3 device. > >> >> (2) Setting up data related to the NVMe request. >> >> (3) then Calls `ib_dma_sync_single_for_device` to flush the CPU cache to >> DMA memory: >> >> ib_dma_sync_single_for_device(dev, sqe->dma, >> sizeof(struct nvme_command), >> DMA_TO_DEVICE); >> >> Of course, if the hardware ensures cache coherency, the above operations >> are skipped. However, if the hardware does not guarantee cache >> coherency, RDMA appears to ensure cache coherency through this method. >> >> In the RDMA scenario, we also face the issue of multi-host cache >> coherence. so I'm thinking, can we adopt a similar approach in CXL >> shared memory to achieve data sharing? > > You don't face the same coherence issues, or at least not in the same way. > In that case the coherence guarantees are actually to the RDMA NIC. > It is guaranteed to see the clean data by the host - that may involve > flushes to PoC. A one time snapshot is then sent to readers on other > hosts. If writes occur they are also guarantee to replace cached copies > on this host - because there is well define guarantee of IO coherence > or explicit cache maintenance to the PoC right, the PoC is not point of cohenrence with other host. it sounds correct. thanx. > > >> >>>> >>>> (3) If the hardware doesn't implement cache coherence and the cpu >>>> doesn't support the required CPU cache operations, then we can run in >>>> nocache mode. >>> >>> I suspect that gets you no where either. Never believe an architecture >>> that provides a flag that says not to cache something. That just means >>> you should not be able to tell that it is cached - many many implementations >>> actually cache such accesses. >> >> Sigh, then that really makes thing difficult. > > Yes. I think we are going to have to wait on architecture specific clarifications > before any software coherent use case can be guaranteed to work beyond the 3.1 ones > for temporal sharing (only one accessing host at a time) and read only sharing where > writes are dropped anyway so clean write back is irrelevant beyond some noise in > logs possibly (if they do get logged it is considered so rare we don't care!). Hi Jonathan, Allow me to discuss further. As described in CXL 3.1: ``` Software-managed coherency schemes are complicated by any host or device whose caching agents generate clean writebacks. A “No Clean Writebacks” capability bit is available for a host in the CXL System Description Structure (CSDS; see Section 9.18.1.6) or for a device in the DVSEC CXL Capability2 register (see Section 8.1.3.7). ``` If we check and find that the "No clean writeback" bit in both CSDS and DVSEC is set, can we then assume that software cache-coherency is feasible, as outlined below: (1) Both the writer and reader ensure cache flushes. Since there are no clean writebacks, there will be no background data writes. (2) The writer writes data to shared memory and then executes a cache flush. If we trust the "No clean writeback" bit, we can assume that the data in shared memory is coherent. (3) Before reading the data, the reader performs cache invalidation. Since there are no clean writebacks, this invalidation operation will not destroy the data written by the writer. Therefore, the data read by the reader should be the data written by the writer, and since the writer's cache is clean, it will not write data to shared memory during the reader's reading process. Additionally, data integrity can be ensured. The first step for CBD should depend on hardware cache coherence, which is clearer and more feasible. Here, I am just exploring the possibility of software cache coherence, not insisting on implementing software cache-coherency right away. :) Thanx > >>> >>>> >>>> CBD can initially support (3), and then transition to (1) when hardware >>>> supports cache-coherency. If there's sufficient market demand, we can >>>> also consider supporting (2). >>> I'd assume only (3) works. The others rely on assumptions I don't think >> >> I guess you mean (1), the hardware cache-coherency way, right? > > Indeed - oops! > Hardware coherency is the way to go, or a well defined and clearly document > description of how to play with the various host architectures. > > Jonathan > > >> >> :) >> Thanx >> >>> you can rely on. >>> >>> Fun fun fun, >>> >>> Jonathan >>> >>>> >>>> How does this approach sound? >>>> >>>> Thanx >>>>> >>>>> J >>>>> >>>>>> >>>>>> Keep in mind that I don't think anybody has cxl 3 devices or CPUs yet, and >>>>>> shared memory is not explicitly legal in cxl 2, so there are things a cpu >>>>>> could do (or not do) in a cxl 2 environment that are not illegal because >>>>>> they should not be observable in a no-shared-memory environment. >>>>>> >>>>>> CBD is interesting work, though for some of the reasons above I'm somewhat >>>>>> skeptical of shared memory as an IPC mechanism. >>>>>> >>>>>> Regards, >>>>>> John >>>>>> >>>>>> >>>>>> >>>>> >>>>> . >>>>> >>> >>> . >>> > >