Received: by 2002:ab2:2441:0:b0:1f3:1f8c:d0c6 with SMTP id k1csp137273lqe; Thu, 4 Apr 2024 01:59:27 -0700 (PDT) X-Forwarded-Encrypted: i=3; AJvYcCXytdaocP5nST6zKlOQGdoYvSd6dtDSGTyZziGtpNzLoRU1t0wNTb/2y3KLO1K1lHTntRZs/RMXnMCc9s5UEyjaa+S89o8GSEjOaAA55A== X-Google-Smtp-Source: AGHT+IEb+hAKA8F8p3/y+RJeiEc8dL8JoaKJZCgSbHbyalll6HLb8ghtgWJDAKtRt1d1yuir7p5L X-Received: by 2002:a05:6359:4584:b0:17f:882e:67d4 with SMTP id no4-20020a056359458400b0017f882e67d4mr1894494rwb.3.1712221166840; Thu, 04 Apr 2024 01:59:26 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1712221166; cv=pass; d=google.com; s=arc-20160816; b=NsHDPGnogh+rN1rIr+gtHZbiCYluTwbQypPeB4XDgSIOvQuYyX/WTZRl++XzWDRxnq xzgi4Q/hdZdOAxtJpJaBwgaNRSWSbVTXfj13IlDSF+p/R/KPkfM1Iu7Qiz1TE7xfWhQD IkiohlAasv5a9XneVEebtIp08ejd+tRV3dN0iYmjVzkYaIaZtRe9fOi9Cs0uaJU8nLp9 Oi0Ad1SmHiwoVBnIEOewWV2ozANhXKsoHNnWT8s/ddk1vKZPc5ATlLGzz+16MsozVpBb ALmCxdG0wYBqHOprGaTZrxGAgvDNCRZBUNheD6+HITIy7bXQZF6UiGlFpulxxB5Y3hT0 h/eQ== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=in-reply-to:content-disposition:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:message-id:subject:cc :to:from:date:dkim-signature; bh=dDVp0WFyVk+ukoIWDZjb8fY7p06xBtHSOMVijvd0Eqs=; fh=DtRMUU1b30MPq4T58Em66SVMgrnor1lAiDwPZPo/jSo=; b=I5wswnC+gKeXnTogAelSt/7VGCGpBwB9GAGqqiyvhO55zJIFyF9LzkMQl8y7BOCPd5 arHJZxVSAzAHqrZ7464NwdDBPY4PrtwPO13l/5nGYITUduaRz+Kvb4YBnbWJtOwIQRhI 3QqYGz/vzNMjzZIGoFFEOKAJdjuEvbPTdf5bWUxJT+vR9Mex2BCVS9ExOizQKP/3ZACj p5MNFQg+ffuIdjdH6uo3bXFwFwakNz7z2bSxsKyGQ8qxRmGe+7J0EDYsvUbMXiZWSpqj 2n50Ssx3BYfP6WUc44R9HlHy9VjKup1fWxgL12xZ2GIfPmhtkWjYiZQBQ5qYvwu2dar6 5Eig==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=NIYXHpaf; arc=pass (i=1 dkim=pass dkdomain=kernel.org); spf=pass (google.com: domain of linux-kernel+bounces-131123-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:40f1:3f00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-131123-linux.lists.archive=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Return-Path: Received: from sy.mirrors.kernel.org (sy.mirrors.kernel.org. [2604:1380:40f1:3f00::1]) by mx.google.com with ESMTPS id v4-20020a634644000000b005dc9404f489si15112599pgk.338.2024.04.04.01.59.26 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 04 Apr 2024 01:59:26 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel+bounces-131123-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:40f1:3f00::1 as permitted sender) client-ip=2604:1380:40f1:3f00::1; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=NIYXHpaf; arc=pass (i=1 dkim=pass dkdomain=kernel.org); spf=pass (google.com: domain of linux-kernel+bounces-131123-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:40f1:3f00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-131123-linux.lists.archive=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sy.mirrors.kernel.org (Postfix) with ESMTPS id 08663B2888B for ; Thu, 4 Apr 2024 08:44:59 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 205A071B42; Thu, 4 Apr 2024 08:44:49 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="NIYXHpaf" Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 01D2359B61; Thu, 4 Apr 2024 08:44:47 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1712220288; cv=none; b=gF/5uG4qA4gB9Zm688AZfmug93JN4zJiXijm2rQo1cOWiyuv7iE4BWK68iMVy2LDXU+q9PFFvtUEB+Ql7u8WOwVVdS8z52/S1dVw4zT3Iz5RAt8mfvl/CUXcPE9LjPOLRvHL27AOwMamOU1rRtD4bgYWJWY5foHfSxQQlFLoYK4= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1712220288; c=relaxed/simple; bh=9D5gn2lk/QVStWbWFD9aEUrA60ETwIJ6u2rH2INzuts=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=ZndSoj0q8pGZ38UpYZWsIqwvfCNG+4klnGTnv+sO1ebWql+bfle7rWpsWGp6AFG1xB4VsmlJJTiFv8y+wgTR68x+Xn/KIg9fr5hsucCKl7g3gJexPaHSEAakBddJXRrOgZk3pEFkBA77lFvPqcic43UfgLqD1jsC2vTBvXTTKpk= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=NIYXHpaf; arc=none smtp.client-ip=10.30.226.201 Received: by smtp.kernel.org (Postfix) with ESMTPSA id B2A78C433C7; Thu, 4 Apr 2024 08:44:43 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1712220287; bh=9D5gn2lk/QVStWbWFD9aEUrA60ETwIJ6u2rH2INzuts=; h=Date:From:To:Cc:Subject:References:In-Reply-To:From; b=NIYXHpafLZH6KcJZnlAzlIjUXkgHbhS7H4WiQqA5B751qndGYod5lThl7gmT//Vq+ 4vZdzFQ7kKv0vxlO9ZojdCLzIHvYV6l1+gDhUBkpJdSYftYdbzts20ImMv01+BfP0n agzjEVIqRnOTjc1lC1XiCiMxswG+EyMraTNRS5Fk4JXKo7Us/BWCEL/dDvZjxsR3j6 rGPmx1sjJQ8u4jT3DA/iMwiEswgdRMX+GBuZC45YA67RPTDzJIlSUVk9emFzK1eUdH n7YRyo6TLT8yiatAbG68BNo3fkorjladuvQDPjqUZVRqnGUyOpDgPeu8ZrnWc9TphF 8wP1Suw4Pjxig== Date: Thu, 4 Apr 2024 10:44:40 +0200 From: Niklas Cassel To: Vinod Koul , Manivannan Sadhasivam Cc: Lorenzo Pieralisi , Krzysztof =?utf-8?Q?Wilczy=C5=84ski?= , Rob Herring , Bjorn Helgaas , Kishon Vijay Abraham I , Thierry Reding , Jonathan Hunter , Jingoo Han , Gustavo Pimentel , linux-pci@vger.kernel.org, linux-arm-msm@vger.kernel.org, linux-kernel@vger.kernel.org, mhi@lists.linux.dev, linux-tegra@vger.kernel.org, Damien Le Moal , Frank Li Subject: Re: [PATCH 05/11] PCI: epf-{mhi/test}: Move DMA initialization to EPC init callback Message-ID: References: <20240314-pci-epf-rework-v1-0-6134e6c1d491@linaro.org> <20240314-pci-epf-rework-v1-5-6134e6c1d491@linaro.org> <20240326082636.GG9565@thinkpad> <20240327055457.GA2742@thinkpad> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: On Fri, Mar 29, 2024 at 12:12:48AM +0530, Vinod Koul wrote: > On 27-03-24, 12:39, Niklas Cassel wrote: > > > > So my question is: > > Is the "dummy memcpy" DMA channel always available? > > That depends on the system, you may or maynot have such a system where > you have a generic memcpy dma controller which can provide you with > these channels I misunderstood DMA_MEMCPY then, I assumed that it was a "software emulated" DMA channel, which allowed the a driver to always use dmaengine + DMA API. It actually uses a real DMA controller. I don't have any DMA controller in the PCIe EP device tree node, but perhaps it can use any DMA controller that has been registered with dmaengine? > Based on my reading you might have this mechanism: > - eDMA provides dmaengine_prep_slave_single() which transfers data from > mem to pci ep device, so fasted > - dmaengine_prep_dma_memcpy: This will copy the data but treat it as > memory. I dont pci internals to figure out how both can work... So > cant really make out why it is slowed > - memcpy_xxx that is IO mem functions, so ofc they will be slowest > > I think the code is decent from fallback pov... chooses fastest path if > available on a system Indeed, it makes more sense to me now, thank you Vinod. > > I was wrong here, pci-epf-test always calls pci_epc_map_addr() > > regardless if FLAG_USE_DMA is set or not. > > > > (Even though this should be unnecessary when using the eDMA.) > > > > However, if we look at pci-epf-mhi.c we can see that it does > > NOT call pci_epc_map_addr() when using DMA API + dmaengine. > > > > Is it really safe to avoid pci_epc_map_addr() in all EPC controllers? > > I assume that it should be safe for all "real" DMA channels. > > We can see that it is not safe when using DMA API + "dummy" memcpy > > dma-channel. (That is why I was asking if we need a NEEDS_MAP, or > > MAP_NOT_NEEDED flag.) > > pci-epf-test.c: > > -Always calls pci_epc_map_addr() when using DMA API. > > > > pci-epf-mhi.c: > > -Never calls pci_epc_map_addr() when using DMA API. Mani, I still think that this part is inconsistent between PCI EPF drivers. Looking more at commit: 8353813c88ef ("PCI: endpoint: Enable DMA tests for endpoints with DMA capabilities") Adding Frank on CC, since he is the author of that commit. When the commit added support for eDMA to pci-epf-test, it added an extra parameter to pci_epf_test_data_transfer(), to pass the PCI/DMA address of the remote buffer, in addition to the already provided local physical address that pci_epc_map_addr() has mapped the PCI/DMA address to. So in the case of eDMA transfer, the pci_epc_map_addr() operation is still being performed, even though pci-epf-test never uses the result of the the mapping operation... This is just confusing and a waste of CPU cycles. What I would like is more consistency between the EPF drivers. I guess an if-statement that skips the pci_epc_map_addr() in pci-epf-test if using eDMA would make pci-epf-mhi and pci-epf-test most consistent. However, when reading the DWC databook: -The eDMA and HDMA always goes via the iATU table. If you do not want this, then you need to set the the appropriate bypass bit. For eDMA: "" When you do not want the iATU to translate outbound requests that are generated by the internal DMA module, then you must implement one of the following approaches: - Ensure that the combination of DMA channel programming and iATU control register programming, causes no translation of DMA traffic to be done in the iATU. or - Activate the DMA bypass mode to allow request TLPs which are initiated by the DMA controller to pass through the iATU untranslated. You can activate DMA bypass mode by setting the DMA Bypass field of the iATU Control 2 Register (IATU_REGION_C- TRL_OFF_2_OUTBOUND_0). "" For HDMA: "" When you do not want the iATU to translate outbound requests that are generated by the internal HDMA module, then you must implement one of the following approaches: - Ensure that the combination of HDMA channel programming and iATU control register programming, causes no translation of DMA traffic to be done in the iATU. or - Activate the HDMA bypass mode to allow request TLPs which are initiated by the HDMA controller to pass through the iATU untranslated. You can activate HDMA bypass mode by setting the HDMA Bypass field of the iATU Control 2 Register (IATU_REGION_C- TRL_OFF_2_OUTBOUND_0). "" We also know that, if there is no match in the iATU table: "" The default behavior of the ATU when there is no address match in the outbound direction or no TLP attribute match in the inbound direction, is to pass the transaction through. "" So even if we do not call pci_epc_map_addr(), the eDMA and HDMA will go via the iATU table, it will most likely not find a match, so it will go through untranslated. So I think we need to answer these questions: 1) Do we want to rely on the fact that hopefully none of the iATUs in the DWC controller has configured a mapping that might mess things up for us? I don't see why the PCI/DMA address of the remote buffer, supplied to pci-epf-test via test_reg BAR, might not fall within the physical iATU window on the local EP system. (As long as the PCI EPF driver has mapped any address using pci_epc_map_addr().) This is a big argument that EPF drivers running on a DWC-based EPC should definitely NOT call pci_epc_map_addr() needlessly when using eDMA, as it can be catastrophic. (pci-epf-test needs to be patched.) 2) Can we really assume that both pci-epf-test and pci-epf-mhi does not need to call pci_epc_map_addr() when using a DMA_SLAVE DMA controller? This seems to be designed only with DWC in mind. Other PCIe endpoint controllers might require this. (Yes, for DWC-based controllers, this definitely should be skipped, but EPF drivers are supposed to be independent from a specific EPC.) I'm fine with just avoiding the pci_epc_map_addr() call when using DMA_SLAVE DMA in pci-epf-test for now, as that is the only DMA controller that I'm familiar with. This second question was more a question for how EPF drivers are should be designed now and in the future. Kind regards, Niklas