Received: by 2002:a05:6a10:1d13:0:0:0:0 with SMTP id pp19csp858168pxb; Mon, 16 Aug 2021 20:31:32 -0700 (PDT) X-Google-Smtp-Source: ABdhPJxRjuenw7JLI2xCcBSg6jAt/gbdrGYMfZ2Siks2yqUg7XkbmdbYToOzYVgsS8TcW91G6kgK X-Received: by 2002:a17:906:b104:: with SMTP id u4mr1436637ejy.201.1629171092743; Mon, 16 Aug 2021 20:31:32 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1629171092; cv=none; d=google.com; s=arc-20160816; b=oNrt/SruAPpGUQGoldhoHveF98lT1oq3KJWv6NUIQp5zzCuVViSiLwsCGbiKC7bCC8 rLGE7URiR2JZqwYhnZXrZQyswqOqhf7fpf/Urormvob6qxvF9T05Yc1tljgUzT81bICe qXopZAk+O9KEyChwJTFi1nwLpzjUV4c5CWjs9yFxQiZ66ZAKLOZ1xXboALzNQu9FyiM0 jYfZL/IFyzwkFHjky+ucAw5Ryy/7em+MnZDGMwM8IoI/+Ip9xwa57pZ4PUKcY96dKyvH xeQ+YvEyk7V95v2Fw/20wVNcgyqFgDO7Aqsdo6dOtkysOedyOMX7WOkvIlEoTeDDmU2M t6yA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:subject:message-id:date:from:in-reply-to :references:mime-version:dkim-signature; bh=Tzdzktn6eJFw4s49wHe5WL1ljzPbWd0BSnO2ChJenyg=; b=fRffbwZ3cnUtYIzHlnhch38CTthGhQ6uHyKwgEUetXFvA09wKsDkpNdn1VErKZhXit alM1tMw85OiLdA3CDsfhBZw5DrLTZvOE4zNsFzyJui/VRy6uf7D8cXHnmhjQ8Xm5gSJw k1a1VtPGNO9qraDTN+TQk7OiVofzDxD8j/l4WxA+iZ8H/zA7g4jWtvWM4/5iS7WgPZXV BI2btskg0rvtB/IgO69yconNqiJDoZZg4pB0CTds7FkwcUiGrevnsgQBUNYOEgSsah3A 91lP52+FyCbENYRi6wSwvNwKIC5OzKroXY42jz4azn27oTjgavKTGC6nZbhnFQm8pGfR NfoA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@atishpatra.org header.s=google header.b=iRigEkOU; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [23.128.96.18]) by mx.google.com with ESMTP id z23si944218edx.139.2021.08.16.20.31.07; Mon, 16 Aug 2021 20:31:32 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18; Authentication-Results: mx.google.com; dkim=pass header.i=@atishpatra.org header.s=google header.b=iRigEkOU; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236665AbhHQD33 (ORCPT + 99 others); Mon, 16 Aug 2021 23:29:29 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46490 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236662AbhHQD31 (ORCPT ); Mon, 16 Aug 2021 23:29:27 -0400 Received: from mail-yb1-xb2f.google.com (mail-yb1-xb2f.google.com [IPv6:2607:f8b0:4864:20::b2f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 231E5C0613C1 for ; Mon, 16 Aug 2021 20:28:55 -0700 (PDT) Received: by mail-yb1-xb2f.google.com with SMTP id m193so36867192ybf.9 for ; Mon, 16 Aug 2021 20:28:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=atishpatra.org; s=google; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=Tzdzktn6eJFw4s49wHe5WL1ljzPbWd0BSnO2ChJenyg=; b=iRigEkOU/lq2H7+3rXhuYnpJrvUTCZEtP+sT9oh7jzX60yWEA71GljsuFR2qdsTxk4 1D4Xgi/8lzLK3hT/eODEXv/fg27FhlfLLMBQv4c+Le/u+cRKoE+XNCuUFzAPVOZ3Cu5A 5mALnvxWLki1/8rU9dGV2+5xRSfzrb20HkgH4= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=Tzdzktn6eJFw4s49wHe5WL1ljzPbWd0BSnO2ChJenyg=; b=RvSEauHHul+rh8+9OuJHE4XuTjlJslAx1E1nMrBamaZ2bnCOylN/WDdWHVsTd/dHod vpGbc1sXuoOifRrzoty4ra/s0m855AnmuTQdWm0jgXYxVW0VwoOy75pFg8ua8iLbptyW YRdCWzbwAp/cAVcLfbVf9qf6CIlj5NmzJeoc9f9aADThXBWFWEBAAZo0hvIua6WSJxIU A+jEkero6tHx93bugjIKFWeSSPwA6/HqrQd1clgqBxZpEjtX0R3CoZK9XLLxBjek9gTY raXJikHPeNZbh/Y+DLFVUl+k9Rg+WN4SFpx9ynFbP+U2JViFo5xZWCFC6fFAyH7pOM+i CfsA== X-Gm-Message-State: AOAM532lYfLOhoKux/aeWoU/RbCzyfFfsdq6KKXqv3Aq3b/XFry2enfB 10DD1GaprGikY4/qRxQSJTd5n353LJlmHJhE6xfH X-Received: by 2002:a25:3bcb:: with SMTP id i194mr1729383yba.442.1629170934402; Mon, 16 Aug 2021 20:28:54 -0700 (PDT) MIME-Version: 1.0 References: <20210723214031.3251801-1-atish.patra@wdc.com> In-Reply-To: From: Atish Patra Date: Mon, 16 Aug 2021 20:28:43 -0700 Message-ID: Subject: Re: [RFC 0/5] Support non-coherent DMA on RISC-V using a global pool To: Guo Ren Cc: Palmer Dabbelt , devicetree , Albert Ou , Tobias Klauser , Robin Murphy , "linux-kernel@vger.kernel.org List" , Rob Herring , Atish Patra , iommu@lists.linux-foundation.org, Guo Ren , Paul Walmsley , linux-riscv , Frank Rowand , Christoph Hellwig , Dmitry Vyukov Content-Type: text/plain; charset="UTF-8" Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Mon, Aug 16, 2021 at 6:37 PM Guo Ren wrote: > > 1 > > On Thu, Jul 29, 2021 at 2:19 PM Atish Patra wrote: > > > > On Wed, Jul 28, 2021 at 9:30 PM Palmer Dabbelt wrote: > > > > > > On Fri, 23 Jul 2021 14:40:26 PDT (-0700), Atish Patra wrote: > > > > RISC-V privilege specification doesn't define an IOMMU or any method modify > > > > PMA attributes or PTE entries to allow non-coherent mappings yet. In > > > > the beginning, this approach was adopted assuming that most of the RISC-V > > > > platforms would support full cache-coherent IO. Here is excerpt from the > > > > priv spec section 3.6.5 > > > > > > > > "In RISC-V platforms, the use of hardware-incoherent regions is discouraged > > > > due to software complexity, performance, and energy impacts." > > > > > > > > While some of the RISC-V platforms adhere to the above suggestion, not all > > > > platforms can afford to build to fully cache-coherent I/O devices. To > > > > address DMA for non-coherent I/O devices, we need to mark a region of memory > > > > as non-cacheable. Some of the platforms rely on a fixed region of uncached > > > > memory that is remapped to the system memory while some other modify > > > > the PTE entries to achieve that. > > > > > > > > The patch3 solves the issue for the fist use case by using a global > > > > pool of memory that is reserved for DMA. The device access the reserved area > > > > of the region while corresponding CPU address will be from uncached region > > > > As the uncached region is remapped to the beginning of the system ram, both > > > > CPU and device get the same view. > > > > > > > > To facilitate streaming DMA APIs, patch 1 introduces a set of generic > > > > cache operations. Any platform can use the generic ops to provide platform > > > > specific cache management operations. Once the standard RISC-V CMO extension > > > > is available, it will also use these generic ops. > > > > > > > > To address the second use case, Page Based Memory Attribute (PBMT) extension > > > > is proposed. Once the extension is in good shape, we can leverage that > > > > using CONFIG_DIRECT_REMAP. Currently, it is selected via a compile time config > > > > option. We will probably need another arch specific hooks to know if the > > > > platform supports direct remap at runtime. For RISC-V, it will check the > > > > presence of PBMT extension while other arch function will simply return true > > > > > > IIUC this is another extension that's not yet frozen or implemented in > > > hardware? Is this one compatible with what's in the c906, or is it > > > doing things its own way? > > > > Hi Palmer, > > This series doesn't implement the PBMT extension which is still in discussion. > > It simply reuse the existing non-coherent dma mapping support in > > upstream kernel and enable > > it for RISC-V. The current version uses a non-coherent global pool. I > > will update it to use arch_set_uncached > > as per Christoph's suggestion. It solves the non-coherent DMA problem > > for beagleV and not c906. > > > > I briefly mentioned the PBMT extension just to provide an idea how the > > RISC-V Linux kernel > > can support both unached window and PBMT extension. PBMT extension is > > planned to be frozen > > by the end of this year and none of the hardware has implemented it. > > > > The implementation in c906 is a non-standard one and will not be > > supported by the default PBMT > > extension implementation. > The default PBMT & c908 PBMT are similar, only BIT definitions are > different. I propose to support default PBMT first and give the back > door to modify the PBMT definition during boot. > The "protection_map[] = (__P000, __P001 ..__S000, __S001)" in > mm/mmap.c has been modified by arch/mips, arm, sparc, x86, so I think > it's proper solution direction. > > The reset problem is how to passing custom PBMT at the very early boot > stage. Now I'm trying to use the DTS parsing instead of boot param hdr > which Anup objected to. > IIRC, c906 has a compatible mode that has the compliant PTE bit modifications. Can you use that mode in the Allwinner D1 board to boot Linux ? I am not sure if you have any fallback method for non-coherent DMA if custom DMA_COHERENT bits are not enabled through enhanced mode ? > ref: https://lore.kernel.org/linux-riscv/1623693067-53886-1-git-send-email-guoren@kernel.org/ > > Any comments are welcome. > > > > > > > > > > > > if DIRECT_REMAP is enabled. This is required as arious different config > > > > (DIRECT_REMAP, GLOBAL_POOL) will be enabled in the defconfig so that a > > > > unified kernel image can boot on all RISC-V platforms. > > > > > > > > This patch series depends on Christoph's global pool support series[1]. > > > > Tested on Qemu, HiFive unleashed and beagleV. This series is also available > > > > at [2]. > > > > This series also solves the non-coherent DMA support on beagleV but requires > > > > additional beagleV specific patches[3] which will be upstreamed soon. > > > > > > > > > > > > [1] https://lists.linuxfoundation.org/pipermail/iommu/2021-July/057266.html > > > > [2] https://github.com/atishp04/linux/tree/riscv_nc_global_pool > > > > [3] https://github.com/atishp04/linux/tree/wip_beaglev_dma_nc_global > > > > > > > > Atish Patra (5): > > > > RISC-V: Implement arch_sync_dma* functions > > > > of: Move of_dma_get_range to of_address.h > > > > dma-mapping: Enable global non-coherent pool support for RISC-V > > > > dma-direct: Allocate dma pages directly if global pool allocation > > > > fails > > > > RISC-V: Support a new config option for non-coherent DMA > > > > > > > > arch/riscv/Kconfig | 8 +++ > > > > arch/riscv/include/asm/dma-noncoherent.h | 19 +++++++ > > > > arch/riscv/mm/Makefile | 1 + > > > > arch/riscv/mm/dma-noncoherent.c | 66 ++++++++++++++++++++++++ > > > > drivers/of/of_private.h | 10 ---- > > > > include/linux/of_address.h | 12 +++++ > > > > kernel/dma/coherent.c | 49 +++++++++++++++--- > > > > kernel/dma/direct.c | 7 ++- > > > > 8 files changed, 152 insertions(+), 20 deletions(-) > > > > create mode 100644 arch/riscv/include/asm/dma-noncoherent.h > > > > create mode 100644 arch/riscv/mm/dma-noncoherent.c > > > > > > Can you guys please make up your minds about how this is going to be > > > supported at the ISA level? I get a different answer every day here: > > > sometimes it's that these systems are not compliant, sometimes that > > > Linux is the compliance suite, sometimes that we're doing an ISA > > > extension, and sometimes that we're doing the SBI stuff. > > > > > > > I am not sure whom you have talked to. I would be happy to set up a > > meeting so that everybody is on > > the same page if you are getting different answers every time. > > > > > I don't really care all that much about what the decision is, but it's > > > impossible to move forward without some semblance of a plan. > > > > > > _______________________________________________ > > > linux-riscv mailing list > > > linux-riscv@lists.infradead.org > > > http://lists.infradead.org/mailman/listinfo/linux-riscv > > > > > > > > -- > > Regards, > > Atish > > _______________________________________________ > > iommu mailing list > > iommu@lists.linux-foundation.org > > https://lists.linuxfoundation.org/mailman/listinfo/iommu > > > > -- > Best Regards > Guo Ren > > ML: https://lore.kernel.org/linux-csky/ -- Regards, Atish