Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
From:   Sean Anderson <sean.anderson@seco.com>
Subject: Re: [BUG] ls1046a: eDMA does not transfer data from I2C
To:     Robin Murphy <robin.murphy@arm.com>,
        Oleksij Rempel <linux@rempel-privat.de>,
        Pengutronix Kernel Team <kernel@pengutronix.de>,
        linux-i2c@vger.kernel.org,
        linux-arm-kernel <linux-arm-kernel@lists.infradead.org>,
        Vinod Koul <vkoul@kernel.org>, dmaengine@vger.kernel.org
Cc:     Li Yang <leoyang.li@nxp.com>, Peng Ma <peng.ma@nxp.com>,
        Linux Kernel Mailing List <linux-kernel@vger.kernel.org>,
        dri-devel@lists.freedesktop.org,
        =?UTF-8?Q?Christian_K=c3=b6nig?= <christian.koenig@amd.com>,
        linaro-mm-sig@lists.linaro.org, Robin Gong <yibin.gong@nxp.com>,
        Shawn Guo <shawnguo@kernel.org>,
        Sumit Semwal <sumit.semwal@linaro.org>,
        Joy Zou <joy.zou@nxp.com>, linux-media@vger.kernel.org
References: <38974aab-06d0-f5ff-d359-5eedd2f3bb3e@seco.com>
 <5ef51421-e6b0-edd5-6b6e-439b47b794a8@arm.com>
Message-ID: <9b20569d-7855-0031-3552-090ff7880cec@seco.com>
Date:   Tue, 20 Sep 2022 11:24:17 -0400
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101
 Thunderbird/68.10.0
In-Reply-To: <5ef51421-e6b0-edd5-6b6e-439b47b794a8@arm.com>
Content-Type: text/plain; charset=utf-8
Content-Language: en-US
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?K0JHd3NlZ2x1L0NsQ3lWUzFNRUJYaXYxUlQ3VXMyRHZSK0d1UjZyZDNSajVS?=
 =?utf-8?B?UFVjODFPSW4wTnZudUk5VS92bSttdkdQODFkN0lXSnJERWdqWXZJR0R5UTds?=
 =?utf-8?B?RzNzYzh5N0RabndseklzcWIxNmIyVFRlN1hRYUZnb3cvNFUyK1Ezby9JU2My?=
 =?utf-8?B?bTBEbzlKcmFzU2dDN0pUNmtvcFFTck5JanZTZkJmM1RXZnB0MU1zdjczNFpB?=
 =?utf-8?B?SXdMME5UNDNUNndPVVc5UTdDQUZUZ1lwZllqYzVQRFRHMXRFUGFtRkRLMDZh?=
 =?utf-8?B?bDNuQzZjNjNWZUhhRE50WTJKVWtGNEh4Z2ZHcmlYTTFDaERGcFEwVmtJOEQr?=
 =?utf-8?B?WWNsei8xeXpFL3oyZmN5eXNzQUtMeXhyZzcya0J5d293Tkt3S0FaN21lYmgw?=
 =?utf-8?B?NWJHbHM0TExLazhsb3puVEt2aDN4VWdYMzU3S3dHZ2hMNENPaG9xZzRIRUpY?=
 =?utf-8?B?TnozZXJYRkYxZUJvaHpqVmhyZU13am8wUjUvY2c5MjFiNHd4OTVwdEtVTWhv?=
 =?utf-8?B?RWREcVRCYzRjaDBPN2wySWxyN2tOTTZ6Um42UzdRNUsxMzIxbS83YVdJcmVK?=
 =?utf-8?B?cTJ3VzgvdytyREhuWGxTVGg1UnBpWG1scERvSzR3M21VWm5VZjNXei9ieXdn?=
 =?utf-8?B?bEJRM2txV3VaOFdHUnVBcnl2U0Jod2ZmdDM2YXdpN211NWlnb0c2aUYzUU5K?=
 =?utf-8?B?QUwrUG1jV25TQTVBN2poUDVDYXozamo3elNScUFLL3l1aG5EOURaNWcrVmJE?=
 =?utf-8?B?dUlKQmdrbENRNUJlN1FXem9YUVFjenVMak9aZVVPa3V6QnZReTRkYkFkTVV2?=
 =?utf-8?B?N3ZsYlZHeUJaamNmRlJtZDFESlRoOFp4bTJrbTkwZmErOXBrbGMyeEVCT2Y3?=
 =?utf-8?B?THpqQittbFZoQmJWWUh4UzJML0lBVWdnOHBwVTdKanIvUE16UXFndmhzdGhD?=
 =?utf-8?B?b3FTdHkyNDAzdVlRM08wUEIwZSt1MjMvd3I0dWJRWEI0UG50UHFFMys2d0w3?=
 =?utf-8?B?aW5zaFZhZHlCTTY4RVlTd2JBbldmbGFLLzRUUmUxbldRa1IxQW53SFhUVzNK?=
 =?utf-8?B?MktpeVNZSlk1OTQ3TXg1OThQRG5YVWZpUEtXa2hpYnM5NnEyTkdPM3R1Tklh?=
 =?utf-8?B?UkJpWHFsamliNklFZncycWNRK3lKTnZ2NEt5RlpPUll1QnlTTlYwZVVzV0hR?=
 =?utf-8?B?bVFheEZIWDB0Ynh5UHBuRmNBY3M5ZjdHVjZoZ1J2cXhneFoyaTZINzlaMUFE?=
 =?utf-8?B?cjRUVFo4ZDVXOXpUM01XbHRhSksvVlc2c0hQNWc2VVB4Y0Ruekwwekt3RjVn?=
 =?utf-8?B?c1NvOGdIU0UvcTRadkdMMjNmY28xaHI5bXdyQTl1MUtEbUNmcG4rZ1loOE1t?=
 =?utf-8?B?VXBJdTZjUVZ5bVgzbEd0Z0o3cXI0ZGZVNGNpUTJVbmZkVllhdlFMQzczRWJG?=
 =?utf-8?B?ZTBLajZmOFFUVllKaTBRZVVrRmNwQXFYTzVhb2JYMVcxS2ZNVlhZL1p3dGhx?=
 =?utf-8?B?WXhMVW16ZFN6Uk94RXRYdjczZzdqVDg4SG1sUUpqRVV3NERTOW9oNml0enAw?=
 =?utf-8?B?MWptY2tERlJYYkJXMXdiNXdqODgyVEVrQmh4anhmbnFBbU9idUszZDU2RFdr?=
 =?utf-8?B?WStDM2k2aGlSZlJuYysyTzU0aTlKRGltQjVRRGhIR3lEYkUrbHVhSjVmZ2xR?=
 =?utf-8?B?ZFB0QVQrVG9lVXFlTU1GNGs3Rm82U3VFR1VyTU5xUDg1OWMzeVl1K0ZyRHhx?=
 =?utf-8?B?ZTd2OG5sNGVVN3h0NGV0ajBjWVhBWlhSUDBhUnNEcUZrSDczeERCS29DTEEw?=
 =?utf-8?B?VFdYZGJSb1VIVHhiZDUzQmx1dlE0Q0d2K2lEeXJWdG9KKy9TTWtpZFJncWpr?=
 =?utf-8?B?RU5iY3Q3WThtditKVTdmNE9vc1hQWmdLUmNmWkNUbzdib2k4NitOQzNlSmJL?=
 =?utf-8?B?clpxQi91OFowMDExUCtKVUNaS1gzS2lsM0prRkFhMk84WHhyb0l1SGwwUmx2?=
 =?utf-8?B?ODZtMXExSmpqT3FtejFCNHNldGNZbzZqWk1hZldodHBva1VHSEFvaVJaRFdq?=
 =?utf-8?B?QUU1dHpxU3VRTENSS2M4R0tqdVJxUWhlbXQ2RFY1cUl1S3pmWUt4Zm9yL0VK?=
 =?utf-8?B?RmtlWWV2SmNhVlhaODdUblplQThNNW52dHhiWTc0ZVBmL0RYNjZGMmIvOUpO?=
 =?utf-8?B?dmc9PQ==?=
X-OriginatorOrg: seco.com
X-MS-Exchange-CrossTenant-Network-Message-Id: c6420254-cf0e-47f0-295d-08da9b1c31db
X-MS-Exchange-CrossTenant-AuthSource: DB7PR03MB4972.eurprd03.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 20 Sep 2022 15:24:22.6057
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: bebe97c3-6438-442e-ade3-ff17aa50e733
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: kJH6ilmHfdqv9+tridiY1bM72uxXY/WzfDUkVkk4Tjy9Zx3Ywj51hrZk0KB4gigbfc9Wefrm4nyINL39kpPsaQ==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM7PR03MB6387
X-Spam-Status: No, score=-4.3 required=5.0 tests=BAYES_00,DKIM_SIGNED,
        DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,NICE_REPLY_A,RCVD_IN_DNSWL_NONE,
        RCVD_IN_MSPIKE_H2,SPF_HELO_PASS,SPF_PASS autolearn=ham
        autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org


On 9/20/22 6:07 AM, Robin Murphy wrote:
> On 2022-09-19 23:24, Sean Anderson wrote:
>> Hi all,
>>
>> I discovered a bug in either imx_i2c or fsl-edma on the LS1046A where no
>> data is read in i2c_imx_dma_read except for the last two bytes (which
>> are not read using DMA). This is perhaps best illustrated with the
>> following example:
>>
>> # hexdump -C /sys/bus/nvmem/devices/0-00540/nvmem
>> [  308.914884] i2c i2c-0: ffff000809380000 0x0000000889380000 0x00000000f5401000 ffff000075401000
>> [  308.923529] src= 2180004 dst=f5401000 attr=   0 soff=   0 nbytes=1 slast=       0
>> [  308.923529] citer=  7e biter=  7e doff=   1 dlast_sga=       0
>> [  308.923529] major_int=1 disable_req=1 enable_sg=0
>> [  308.942113] fsl-edma 2c00000.edma: vchan 000000001b4371fc: txd 00000000d9dd26c5[4]: submitted
>> [  308.974049] fsl-edma 2c00000.edma: txd 00000000d9dd26c5[4]: marked complete
>> [  308.981339] i2c i2c-0: ffff000809380000 = [2e 2e 2f 2e 2e 2f 2e 2e 2f 64 65 76 69 63 65 73 2f 70 6c 61 74 66 6f 72 6d 2f 73 6f 63 2f 32 31 38 30 30 30 30 2e 69 32 63 2f 69 32 63 2d 30 2f 30 2d 30 30 35 34 2f 30 2d 30 30 35 34 30 00 00]
>> [  309.002226] i2c i2c-0: ffff000075401000 = [2e 2e 2f 2e 2e 2f 2e 2e 2f 64 65 76 69 63 65 73 2f 70 6c 61 74 66 6f 72 6d 2f 73 6f 63 2f 32 31 38 30 30 30 30 2e 69 32 63 2f 69 32 63 2d 30 2f 30 2d 30 30 35 34 2f 30 2d 30 30 35 34 30 00 00]
>> [  309.024649] i2c i2c-0: ffff000809380080 0x0000000889380080 0x00000000f5401800 ffff000075401800
>> [  309.033270] src= 2180004 dst=f5401800 attr=   0 soff=   0 nbytes=1 slast=       0
>> [  309.033270] citer=  7e biter=  7e doff=   1 dlast_sga=       0
>> [  309.033270] major_int=1 disable_req=1 enable_sg=0
>> [  309.051633] fsl-edma 2c00000.edma: vchan 000000001b4371fc: txd 00000000d9dd26c5[5]: submitted
>> [  309.083526] fsl-edma 2c00000.edma: txd 00000000d9dd26c5[5]: marked complete
>> [  309.090807] i2c i2c-0: ffff000809380080 = [00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00]
>> [  309.111694] i2c i2c-0: ffff000075401800 = [00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00]
>> 00000000  2e 2e 2f 2e 2e 2f 2e 2e  2f 64 65 76 69 63 65 73  |../../../devices|
>> 00000010  2f 70 6c 61 74 66 6f 72  6d 2f 73 6f 63 2f 32 31  |/platform/soc/21|
>> 00000020  38 30 30 30 30 2e 69 32  63 2f 69 32 63 2d 30 2f  |80000.i2c/i2c-0/|
>> 00000030  30 2d 30 30 35 34 2f 30  2d 30 30 35 34 30 00 00  |0-0054/0-00540..|
>> 00000040  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
>> *
>> 00000070  00 00 00 00 00 00 00 00  00 00 00 00 00 00 ff ff  |................|
>> 00000080  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
>> *
>> 000000f0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 ff 5b  |...............[|
>> 00000100
>>
>> (patch with my debug prints appended below)
>>
>> Despite the DMA completing successfully, no data was copied into the
>> buffer, leaving the original (now junk) contents. I probed the I2C bus
>> with an oscilloscope, and I verified that the transfer did indeed occur.
>> The timing between submission and completion seems reasonable for the
>> bus speed (50 kHz for whatever reason).
>>
>> I had a look over the I2C driver, and nothing looked obviously
>> incorrect. If anyone has ideas on what to try, I'm more than willing.
> 
> Is the DMA controller cache-coherent? I see the mainline LS1046A DT doesn't have a "dma-coherent" property for it, but the behaviour is entirely consistent with that being wrong - dma_map_single() cleans the cache, coherent DMA write hits the still-present cache lines, dma_unmap_single() invalidates the cache, and boom, the data is gone and you read back the previous content of the buffer that was cleaned out to DRAM beforehand.

I've tried both with and without [1] applied. I also tried removing the
call to dma_unmap_single, but to no effect.

--Sean

[1] https://lore.kernel.org/linux-arm-kernel/20220915233432.31660-6-leoyang.li@nxp.com/

>> --Sean
>>
>> diff --git a/drivers/dma/fsl-edma-common.c b/drivers/dma/fsl-edma-common.c
>> index 15896e2413c4..1d9d4a55d2af 100644
>> --- a/drivers/dma/fsl-edma-common.c
>> +++ b/drivers/dma/fsl-edma-common.c
>> @@ -391,6 +391,12 @@ void fsl_edma_fill_tcd(struct fsl_edma_hw_tcd *tcd, u32 src, u32 dst,
>>   {
>>          u16 csr = 0;
>>   +       pr_info("src=%8x dst=%8x attr=%4x soff=%4x nbytes=%u slast=%8x\n"
>> +               "citer=%4x biter=%4x doff=%4x dlast_sga=%8x\n"
>> +               "major_int=%d disable_req=%d enable_sg=%d\n",
>> +               src, dst, attr, soff, nbytes, slast, citer, biter, doff,
>> +               dlast_sga, major_int, disable_req, enable_sg);
>> +
>>          /*
>>           * eDMA hardware SGs require the TCDs to be stored in little
>>           * endian format irrespective of the register endian model.
>> diff --git a/drivers/i2c/busses/i2c-imx.c b/drivers/i2c/busses/i2c-imx.c
>> index 3576b63a6c03..0217f0cb1331 100644
>> --- a/drivers/i2c/busses/i2c-imx.c
>> +++ b/drivers/i2c/busses/i2c-imx.c
>> @@ -402,6 +402,9 @@ static int i2c_imx_dma_xfer(struct imx_i2c_struct *i2c_imx,
>>                  dev_err(dev, "DMA mapping failed\n");
>>                  goto err_map;
>>          }
>> +       phys_addr_t bufp = virt_to_phys(msgs->buf);
>> +       dev_info(dev, "%px %pap %pad %px\n", msgs->buf, &bufp,
>> +                &dma->dma_buf, phys_to_virt(dma->dma_buf));
>>            txdesc = dmaengine_prep_slave_single(dma->chan_using, dma->dma_buf,
>>                                          dma->dma_len, dma->dma_transfer_dir,
>> @@ -965,6 +968,9 @@ static int i2c_imx_dma_read(struct imx_i2c_struct *i2c_imx,
>>                  }
>>                  schedule();
>>          }
>> +       dev_info(dev, "%px = [%*ph]\n", msgs->buf, msgs->len, msgs->buf);
>> +       dev_info(dev, "%px = [%*ph]\n", phys_to_virt(dma->dma_buf), msgs->len,
>> +                phys_to_virt(dma->dma_buf));
>>            temp = imx_i2c_read_reg(i2c_imx, IMX_I2C_I2CR);
>>          temp &= ~I2CR_DMAEN;