Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Fri, 16 Dec 2022 02:52:18 +0300
From:   Serge Semin <fancer.lancer@gmail.com>
To:     Robin Murphy <robin.murphy@arm.com>
Cc:     Christoph Hellwig <hch@infradead.org>,
        Serge Semin <Sergey.Semin@baikalelectronics.ru>,
        Gustavo Pimentel <gustavo.pimentel@synopsys.com>,
        Vinod Koul <vkoul@kernel.org>, Rob Herring <robh@kernel.org>,
        Bjorn Helgaas <bhelgaas@google.com>,
        Lorenzo Pieralisi <lorenzo.pieralisi@arm.com>,
        Cai Huoqing <cai.huoqing@linux.dev>,
        Jingoo Han <jingoohan1@gmail.com>, Frank Li <Frank.Li@nxp.com>,
        Manivannan Sadhasivam <manivannan.sadhasivam@linaro.org>,
        Lorenzo Pieralisi <lpieralisi@kernel.org>,
        Krzysztof =?utf-8?Q?Wilczy=C5=84ski?= <kw@linux.com>,
        Alexey Malahov <Alexey.Malahov@baikalelectronics.ru>,
        Pavel Parkhomenko <Pavel.Parkhomenko@baikalelectronics.ru>,
        caihuoqing <caihuoqing@baidu.com>,
        Yoshihiro Shimoda <yoshihiro.shimoda.uh@renesas.com>,
        linux-pci@vger.kernel.org, dmaengine@vger.kernel.org,
        linux-kernel@vger.kernel.org
Subject: Re: [PATCH v7 23/25] PCI: dwc: Restore DMA-mask after MSI-data
 allocation
Message-ID: <20221215235218.wsuwy5uckqfxjnb6@mobilestation>
References: <20221214235305.31744-1-Sergey.Semin@baikalelectronics.ru>
 <20221214235305.31744-24-Sergey.Semin@baikalelectronics.ru>
 <Y5rJJfZeVqliA5Rg@infradead.org>
 <20221215092721.tvz3hpaql3kotgnu@mobilestation>
 <07ec7610-f1be-9b5c-416d-17781a22427d@arm.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <07ec7610-f1be-9b5c-416d-17781a22427d@arm.com>
Precedence: bulk

On Thu, Dec 15, 2022 at 10:26:08AM +0000, Robin Murphy wrote:
> On 2022-12-15 09:27, Serge Semin wrote:
> > Hi Christoph
> > 
> > On Wed, Dec 14, 2022 at 11:13:41PM -0800, Christoph Hellwig wrote:
> > > On Thu, Dec 15, 2022 at 02:53:03AM +0300, Serge Semin wrote:
> > > > DW PCIe Root Ports and End-points can be equipped with the DW eDMA engine.
> > > > In that case it is critical to have the platform device pre-initialized
> > > > with a valid DMA-mask so the drivers using the eDMA-engine would be able
> > > > to allocate the DMA-able buffers. The MSI-capable data requires to be
> > > > allocated from the lowest 4GB region. Since that procedure implies the
> > > > DMA-mask change we need to restore the mask set by the low-level drivers
> > > > after the MSI-data allocation is done.
> > > 
> > > You can't change the DMA mask when there are existing allocations.
> > 
> > Em, what do you guys suggest for the DW PCIe devices with the embedded
> > DMA-engine then? To live forever with the SWIOTLBs? I can't drop the
> > DMA-mask update due to this commit 423511ec23e2 ("PCI: dwc: Drop
> > dependency on ZONE_DMA32") and I can't change the mask after it's
> > updated. Note it's updated for the memory allocation to which actually
> > no DMA will be performed, see
> > https://lore.kernel.org/linux-pci/20220825185026.3816331-2-willmcvicker@google.com/.
> > My patches imply adding the real DMA operations support.
> > 
> > We've discussed this a lot with Robin in various threads and I thought
> > a workable solution was found. I was going to update the mask in
> > another place, but basically it would still mean to have first setting
> > the 32-bit mask here, and then change it to 64-bit one in the
> > framework of the DW eDMA driver.
> > 
> > So to speak I don't see a proper way out from the situation. Nothing I
> > suggested was accepted and now we'll have to live with the SWIOTLBs
> > used for the memory above 4GB. So please suggest a workable solution
> > then. We need the next things:
> > 1. Somehow preserve a single DWORD of the PCIe bus memory for the
> > iMSI-RX engine. (That's what is currently done the
> > dw_pcie_msi_host_init() method by allocating the coherent memory.)
> > 2. Set the actual DMA-mask to the DW PCIe platform device so the
> > DMA-engine clients would be able to allocate actually DMA-able memory.
> > 
> > @Robin, please join the discussion.
> 

> Basically just don't touch the coherent mask. The eDMA drivers can still set
> the streaming mask to something larger, and that's the one that's going to
> matter for most dmaengine clients anyway. Even if someone does call
> dma_alloc_coherent() for their eDMA channel, it's not going to make much
> practical difference if that has to come from a DMA zone, unless the system
> is under severe memory pressure anyway.

Got it. Thanks for clarification. I'll resubmit the series with only
the streaming DMA mask restoration.

-Serge(y)

> 
> Thanks,
> Robin.