Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1754272AbcKYMsI (ORCPT ); Fri, 25 Nov 2016 07:48:08 -0500 Received: from smtp2-g21.free.fr ([212.27.42.2]:61311 "EHLO smtp2-g21.free.fr" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1753973AbcKYMri (ORCPT ); Fri, 25 Nov 2016 07:47:38 -0500 Subject: Re: Tearing down DMA transfer setup after DMA client has finished To: Vinod Koul Cc: dmaengine@vger.kernel.org, Linus Walleij , Dan Williams , LKML , Linux ARM , Jon Mason , Mark Brown , Lars-Peter Clausen , Lee Jones , Laurent Pinchart , Arnd Bergmann , Maxime Ripard , Dave Jiang , Peter Ujfalusi , Mans Rullgard , Bartlomiej Zolnierkiewicz References: <58356EA8.2010806@free.fr> <20161125045549.GC2698@localhost> From: Mason Message-ID: <092f44ee-4560-be17-25f7-00948dba3cfa@free.fr> Date: Fri, 25 Nov 2016 13:46:26 +0100 User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:49.0) Gecko/20100101 Firefox/49.0 SeaMonkey/2.46 MIME-Version: 1.0 In-Reply-To: <20161125045549.GC2698@localhost> Content-Type: text/plain; charset=ISO-8859-15 Content-Transfer-Encoding: 7bit Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 4552 Lines: 126 On 25/11/2016 05:55, Vinod Koul wrote: > On Wed, Nov 23, 2016 at 11:25:44AM +0100, Mason wrote: > >> On my platform, setting up a DMA transfer is a two-step process: >> >> 1) configure the "switch box" to connect a device to a memory channel >> 2) configure the transfer details (address, size, command) >> >> When the transfer is done, the sbox setup can be torn down, >> and the DMA driver can start another transfer. >> >> The current software architecture for my NFC (NAND Flash controller) >> driver is as follows (for one DMA transfer). >> >> sg_init_one >> dma_map_sg >> dmaengine_prep_slave_sg >> dmaengine_submit >> dma_async_issue_pending >> configure_NFC_transfer >> wait_for_IRQ_from_DMA_engine // via DMA_PREP_INTERRUPT >> wait_for_NFC_idle >> dma_unmap_sg > > Looking at thread and discussion now, first thinking would be to ensure > the transaction is completed properly and then isr fired. You may need > to talk to your HW designers to find a way for that. It is quite common > that DMA controllers will fire and complete whereas the transaction is > still in flight. It seems there is a disconnect between what Linux expects - an IRQ when the transfer is complete - and the quirks of this HW :-( On this system, there are MBUS "agents" connected via a "switch box". An agent fires an IRQ when it has dealt with its *half* of the transfer. SOURCE_AGENT <---> SBOX <---> DESTINATION_AGENT Here are the steps for a transfer, in the general case: 1) setup the sbox to connect SOURCE TO DEST 2) configure source to send N bytes 3) configure dest to receive N bytes When SOURCE_AGENT has sent N bytes, it fires an IRQ When DEST_AGENT has received N bytes, it fires an IRQ The sbox connection can be torn down only when the destination agent has received all bytes. (And the twist is that some agents do not have an IRQ line.) The system provides 3 RAM-to-sbox agents (read channels) and 3 sbox-to-RAM agents (write channels). The NAND Flash controller read and write agents do not have IRQ lines. So for a NAND-to-memory transfer (read from device) - nothing happens when the NFC has finished sending N bytes to the sbox - the write channel fires an IRQ when it has received N bytes In that case, one IRQ fires when the transfer is complete, like Linux expects. For a memory-to-NAND transfer (write to device) - the read channel fires an IRQ when it has sent N bytes - the NFC driver is supposed to poll the NFC to determine when the controller has finished writing N bytes In that case, the IRQ does not indicate that the transfer is complete, merely that the sending half has finished its part. For a memory-to-memory transfer (memcpy) - the read channel fires an IRQ when it has sent N bytes - the write channel fires an IRQ when it has received N bytes So you actually get two IRQs in that case, which I don't think Linux (or the current DMA driver) expects. I'm not sure how we're supposed to handle this kind of HW in Linux? (That's why I started this thread.) > If that is not doable, then since you claim this is custom part which > other vendors won't use (hope we are wrong down the line), I'm not sure how to interpret "you claim this is custom part". Do you mean I may be wrong, that it is not custom? I don't know if other vendors may have HW with the same quirky behavior. What do you mean about being wrong down the line? > then we can have a custom api, > > foo_sbox_configure(bool enable, ...); > > This can be invoked from NFC driver when required for configuration and > teardown. For very specific cases where people need some specific > configuration we do allow custom APIs. I don't think that would work. The fundamental issue is that Linux expects a single IRQ to indicate "transfer complete". And the driver (as written) starts a new transfer as soon as the IRQ fires. But the HW may generate 0, 1, or even 2 IRQs for a single transfer. And when there is a single IRQ, it may not indicate "transfer complete" (as seen above). > Only problem with that would be it wont be a generic solution > and you seem to be fine with that. I think it is possible to have a generic solution: Right now, the callback is called from tasklet context. If we can have a new flag to have the callback invoked directly from the ISR, then the driver for the client device can do what is required. For example, the NFC driver waits for the IRQ from the memory agent, and then polls the controller itself. I can whip up a proof-of-concept if it's better to illustrate with a patch? Regards.