Received: by 2002:a05:6359:c8b:b0:c7:702f:21d4 with SMTP id go11csp173417rwb; Mon, 26 Sep 2022 16:46:57 -0700 (PDT) X-Google-Smtp-Source: AMsMyM50JDcj9++fnMPI974Iazs5vETocslSPjhAHtVNcfcfZj8CrcCSmLp8gTmpB8YCYQ/UcOqE X-Received: by 2002:a17:906:cc18:b0:77b:9653:464d with SMTP id ml24-20020a170906cc1800b0077b9653464dmr1398081ejb.583.1664236017037; Mon, 26 Sep 2022 16:46:57 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1664236017; cv=none; d=google.com; s=arc-20160816; b=GAaf43SEmloxE1Fzoeq6ePT8y5wOEZAyiSSY17rhSo7aWZj/mYQt8EsOLFkK6VEbXZ oHriHMJghNlcJbBAOobT5Z0ncyaspzSvDQ16NsyPa2wOxMSUwvbmVlF76I8QambXExPB 8XHJoMAJ4bDg7ndv4ea5U8qjwmsibngI1skQ7j9YxVIDY3cM249b9VwE6BSwOHXLKbL7 TunYhqCGkAL3yjstSYYuOnlzMM7f3lCF/MVeNYn5+QF9+a1Zirmyu9AKnDhKzXX7enih 5EEmtTvhoXSnVBg9aDBSD5NrraIGHCQREuegktQY1JpDG7vmw5nORTiqM2k7C9utgAza YxCA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:in-reply-to:content-disposition:mime-version :references:message-id:subject:cc:to:from:date:dkim-signature; bh=2ZAUfIPdByJYOZmyUsHUmUEiuSWh8jC/LVnniqZNqUg=; b=s9W1OVHCeGrcZSDxihpMnch/HTID+uNLqo5wbhUKfVENcuUCZOpqe3kXrEkHK6zjat +TgbY/5qOcB4ZQqnVrJaJgJ2o5BUiwzTqw+js9IqYeZk07cl/hZPKq+gOIynKUmeR4e8 7ppWGl7QuZDi0DrBAPbi24NUd+q95F8SEOohgY4LLRa+r1C/79vs1AJ90L5Oy+XbQBiI yZL7FsB1iV7nhwy81KFip3jSjMDyj7Sl4fQbX3sJZupJ2yTTTtVeSV/3VkQkRr0Y/vMR 5njRwe85A1rk7gEcjZoA6itEUZeEeP0fLg7yuRQq5QSvrZ+xuDvtg/qPbbHmrVgGHYzx UJSQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=Bzbe+Dqn; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Return-Path: Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id y18-20020a056402359200b00451dfbbc983si26683edc.491.2022.09.26.16.46.31; Mon, 26 Sep 2022 16:46:57 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=Bzbe+Dqn; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229962AbiIZXI2 (ORCPT + 99 others); Mon, 26 Sep 2022 19:08:28 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42066 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230094AbiIZXIZ (ORCPT ); Mon, 26 Sep 2022 19:08:25 -0400 Received: from mail-pl1-x632.google.com (mail-pl1-x632.google.com [IPv6:2607:f8b0:4864:20::632]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 38FCEAA4C4 for ; Mon, 26 Sep 2022 16:08:23 -0700 (PDT) Received: by mail-pl1-x632.google.com with SMTP id t3so7590865ply.2 for ; Mon, 26 Sep 2022 16:08:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date; bh=2ZAUfIPdByJYOZmyUsHUmUEiuSWh8jC/LVnniqZNqUg=; b=Bzbe+DqnRQd9k01snesjOx+VUOIfxbjHBuxNxGzX47BL0dIH9+H6WLU2ypKFArCHWd o/zyvYwG0wRr8Ot9TXa5SE5wFTM9PH+xRYWvc1LZc8ekSN5fIQZt/7v3PB/xamUJz6l5 BxM1apc9ZYKlY/ECXH2aGXDTAGpUl3211gjPKeedVh8UidzjJZ8Lrl58oQVG9JbX2jy0 K8QLTe/UFY1UCcCZcM2t/Rtumm6EcVM7y39JRAQ8IPJn3XEr70SYcbhzKKQRzfpMh80r CDByO8m+U+bKVkvm0tz6s9vaYGDmb+pu7eSeP/tBTi+AIA3TavTrUQTRX4wydQ38H9Wd weNg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date; bh=2ZAUfIPdByJYOZmyUsHUmUEiuSWh8jC/LVnniqZNqUg=; b=cAX/RP+AK47VDS3dPgm04Lk6hvRUIgEEiqpBhlAat7N1OJi6/LbOPxuvVGqsFoP7JY H+8pm0xdb+MgrNWN0uNlRUl9qLUpQoN6v6mucMnB4VY0F4K/n2dko69wjzvy/8bSmuL9 JaC7sk5s3o4/vBGv4dG2syiC0EzylR07S6+CUAKYIrbDPfji6H+djyezz8wLVYQajCe7 DwdnGrYwypNO9qMJOT1gGDD6Ejbl/XVM2xoItZWQtvV6hZ1Tb46ClUQ+/Lrr3jqZbOI5 Q8bcGML4Lm/eXJHpG/oNPIpA4Li4GTnq4BZS1LiH+hjM9AkwjzKSO8UwEQWk8hfn5VAO xAXA== X-Gm-Message-State: ACrzQf1rm6f56vOvt3cfycCrT4csA1FEi92mGEnAJ4UsQ2vuqeBrR/uy zwE9ebDe8+U2/XvT+r6A5ocxzg== X-Received: by 2002:a17:903:1109:b0:179:d220:1f55 with SMTP id n9-20020a170903110900b00179d2201f55mr11548054plh.42.1664233702422; Mon, 26 Sep 2022 16:08:22 -0700 (PDT) Received: from google.com (201.215.168.34.bc.googleusercontent.com. [34.168.215.201]) by smtp.gmail.com with ESMTPSA id e13-20020a170902784d00b0017887d6aa1dsm11774170pln.146.2022.09.26.16.08.21 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 26 Sep 2022 16:08:21 -0700 (PDT) Date: Mon, 26 Sep 2022 23:08:18 +0000 From: William McVicker To: Serge Semin Cc: Christoph Hellwig , Lorenzo Pieralisi , Marek Szyprowski , Bjorn Helgaas , Rob Herring , Robin Murphy , Serge Semin , Rob Herring , Krzysztof Kozlowski , Lorenzo Pieralisi , Jingoo Han , Gustavo Pimentel , Krzysztof =?utf-8?Q?Wilczy=C5=84ski?= , Alexey Malahov , Pavel Parkhomenko , Frank Li , Manivannan Sadhasivam , linux-pci@vger.kernel.org, devicetree@vger.kernel.org, linux-kernel@vger.kernel.org Subject: Re: [PATCH v5 20/20] PCI: dwc: Add Baikal-T1 PCIe controller support Message-ID: References: <20220822184701.25246-1-Sergey.Semin@baikalelectronics.ru> <20220822184701.25246-21-Sergey.Semin@baikalelectronics.ru> <20220912000211.ct6asuhhmnatje5e@mobilestation> <20220926124924.4vodhncnuaorrlwj@mobilestation> <20220926143127.GB19031@lst.de> <20220926205333.qlhb5ojmx4sktzt5@mobilestation> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20220926205333.qlhb5ojmx4sktzt5@mobilestation> X-Spam-Status: No, score=-15.6 required=5.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF, ENV_AND_HDR_SPF_MATCH,HK_RANDOM_ENVFROM,HK_RANDOM_FROM, RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS,USER_IN_DEF_DKIM_WL, USER_IN_DEF_SPF_WL autolearn=no autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 09/26/2022, Serge Semin wrote: > On Mon, Sep 26, 2022 at 04:31:28PM +0200, Christoph Hellwig wrote: > > On Mon, Sep 26, 2022 at 03:49:24PM +0300, Serge Semin wrote: > > > @Christoph, @Marek, @Bjorn, @Rob could you please join to the > > > DMA-mask related discussion. @Lorenzo can't decide which driver should > > > initialize the device DMA-mask. > > > > > The driver that does the actual DMA mapping or allocation functions > > need to set it. But even with your comments on the questions I'm > > still confused what struct device you are even talking about. Can > > you explain this a bit better? > > We are talking about the DW PCIe Root Port controller with DW eDMA engine > embedded. It' simplified structure can be represented as follows: > > +---------------+ +--------+ > | System memory | | CPU(s) | > +---------------+ +--------+ > ^ | | ^ > | ... System bus ... | > ... | | ... > | v v | > +------------+------+--------+----------+------+ > | DW PCIe RP | AXI-m| | AXI-s/DBI| | > | +------+ +----------+ | > | ^ ^ | | > | +------+----+ | CSRs | > | v v v | > | +-------+ +---------+ +----------+ | > | | eDMA | | in-iATU | | out-iATU | | > | +-------+ +---------+ +----------+ | > | ^ ^ ^ | > | +--------+--+---+-------+ | > +------------------| PIPE |--------------------+ > +------+ > | ^ > v | > PCIe bus > > The DW PCIe controller device is instantiated as a platform device > defined in the system DT source file. The device is probed by the > DW PCIe low-level driver, which after the platform-specific setups > initiates the generic DW PCIe host-controller registration. On the way > of that procedure the DW PCIe core tries to auto-detect the DW eDMA > engine availability. If the engine is found, the DW eDMA probe method > is called in order to register the DMA-engine device. After that the > PCIe host bridge is registered. Both the PCIe host-bridge and > DMA-engine devices will have the DW PCIe platform device as parent. > > Getting back to the sketch above. Here is a short description of the > content: > 1. DW eDMA is capable of performing the data transfers from/to System > memory to/from PCIe bus memory. > 2. in-iATU is the Inbound Address Translation Unit, which is > responsible for the PCIe bus peripheral devices to access the system > memory. The "dma-ranges" DT-property is used to initialize the > PCIe<->Sys memory mapping. (@William note the In-iATU setup doesn't > affect the eDMA transfers.) > 3. out-iATU is responsible for the CPU(s) to access the PCIe bus > peripheral devices memory/cfg-space. > > So eDMA and in-iATU are using the same AXI-master interface to access > the system memory. Thus the DMAable memory capability is the same for > both of them (Though in-iATU may have some specific mapping based on > the "dma-ranges" DT-property setup). Neither DW eDMA nor DW PCIe Root > Port CSRs region have any register to auto-detect the AXI-m interface > address bus width. It's selected during the IP-core synthesize and is > platform-specific. The question is: "What driver/code is supposed to > set the DMA-mask of the DW PCIe platform device?" Seeing the parental > platform device is used to perform the memory-mapping for both DW eDMA > clients and PCIe-bus peripheral device drivers, and seeing the AXI-m > interface parameters aren't auto-detectable and are platform-specific, > the only place it should be done in is the DW PCIe low-level device > driver. I don't really see any alternative... What is your opinion? > > -Sergey I believe this eDMA implementation is new for an upstream DW PCIe device driver, right? If so, this will require some refactoring of the DMA mask code, but you need to also make sure you don't break the MSI target address use case that prompted this 32-bit DMA mask change -- [1]. My changes were directly related to allowing the DW PCIe device driver to fallback to a 64-bit DMA mask for the MSI target address if there are no 32-bit allocations available. For that use-case, using a 32-bit mask doesn't have any perf impact here since there is no actual DMAs happening. Would it be possible for the DW PCIe device driver to set a capabilities flag that the PCIe host controller can read and set the mask accordingly. This way you don't need to go fix up any drivers that require a 32-bit DMA'able address for the MSI target address. For example, I see several of the PCI capability features have 64-bit flags, e.g. PCI_MSI_FLAGS_64BIT and PCI_X_STATUS_64BIT. If not, then you're going to have to re-work the host controller driver and DW PCIe device drivers that require a 32-bit MSI target address. [1] https://lore.kernel.org/all/20201117165312.25847-1-vidyas@nvidia.com/ Thanks, Will