When I have KASAN enabled on my kernel and I start stressing the
touchscreen my system tends to hang. The touchscreen is one of the
only things that does a lot of big i2c transfers and ends up hitting
the DMA paths in the geni i2c driver. It appears that KASAN adds
enough delay in my system to tickle a race condition in the DMA setup
code.
When the system hangs, I found that it was running the geni_i2c_irq()
over and over again. It had these:
m_stat = 0x04000080
rx_st = 0x30000011
dm_tx_st = 0x00000000
dm_rx_st = 0x00000000
dma = 0x00000001
Notably we're in DMA mode but are getting M_RX_IRQ_EN and
M_RX_FIFO_WATERMARK_EN over and over again.
Putting some traces in geni_i2c_rx_one_msg() showed that when we
failed we were getting to the start of geni_i2c_rx_one_msg() but were
never executing geni_se_rx_dma_prep().
I believe that the problem here is that we are starting the geni
command before we run geni_se_rx_dma_prep(). If a transfer makes it
far enough before we do that then we get into the state I have
observed. Let's change the order, which seems to work fine.
Although problems were seen on the RX path, code inspection suggests
that the TX should be changed too. Change it as well.
Fixes: 37692de5d523 ("i2c: i2c-qcom-geni: Add bus driver for the Qualcomm GENI I2C controller")
Signed-off-by: Douglas Anderson <[email protected]>
Tested-by: Sai Prakash Ranjan <[email protected]>
Reviewed-by: Akash Asthana <[email protected]>
---
Even though this patch is slightly different than v1 I have kept tags.
Hopefully this is OK.
Changes in v2:
- Fix both TX and RX.
- Only move the setting up of the command, not the set of the length.
drivers/i2c/busses/i2c-qcom-geni.c | 6 ++++--
1 file changed, 4 insertions(+), 2 deletions(-)
diff --git a/drivers/i2c/busses/i2c-qcom-geni.c b/drivers/i2c/busses/i2c-qcom-geni.c
index 18d1e4fd4cf3..7f130829bf01 100644
--- a/drivers/i2c/busses/i2c-qcom-geni.c
+++ b/drivers/i2c/busses/i2c-qcom-geni.c
@@ -367,7 +367,6 @@ static int geni_i2c_rx_one_msg(struct geni_i2c_dev *gi2c, struct i2c_msg *msg,
geni_se_select_mode(se, GENI_SE_FIFO);
writel_relaxed(len, se->base + SE_I2C_RX_TRANS_LEN);
- geni_se_setup_m_cmd(se, I2C_READ, m_param);
if (dma_buf && geni_se_rx_dma_prep(se, dma_buf, len, &rx_dma)) {
geni_se_select_mode(se, GENI_SE_FIFO);
@@ -375,6 +374,8 @@ static int geni_i2c_rx_one_msg(struct geni_i2c_dev *gi2c, struct i2c_msg *msg,
dma_buf = NULL;
}
+ geni_se_setup_m_cmd(se, I2C_READ, m_param);
+
time_left = wait_for_completion_timeout(&gi2c->done, XFER_TIMEOUT);
if (!time_left)
geni_i2c_abort_xfer(gi2c);
@@ -408,7 +409,6 @@ static int geni_i2c_tx_one_msg(struct geni_i2c_dev *gi2c, struct i2c_msg *msg,
geni_se_select_mode(se, GENI_SE_FIFO);
writel_relaxed(len, se->base + SE_I2C_TX_TRANS_LEN);
- geni_se_setup_m_cmd(se, I2C_WRITE, m_param);
if (dma_buf && geni_se_tx_dma_prep(se, dma_buf, len, &tx_dma)) {
geni_se_select_mode(se, GENI_SE_FIFO);
@@ -416,6 +416,8 @@ static int geni_i2c_tx_one_msg(struct geni_i2c_dev *gi2c, struct i2c_msg *msg,
dma_buf = NULL;
}
+ geni_se_setup_m_cmd(se, I2C_WRITE, m_param);
+
if (!dma_buf) /* Get FIFO IRQ */
writel_relaxed(1, se->base + SE_GENI_TX_WATERMARK_REG);
--
2.28.0.rc0.142.g3c755180ce-goog
Quoting Douglas Anderson (2020-07-22 15:00:21)
> When I have KASAN enabled on my kernel and I start stressing the
> touchscreen my system tends to hang. The touchscreen is one of the
> only things that does a lot of big i2c transfers and ends up hitting
> the DMA paths in the geni i2c driver. It appears that KASAN adds
> enough delay in my system to tickle a race condition in the DMA setup
> code.
>
> When the system hangs, I found that it was running the geni_i2c_irq()
> over and over again. It had these:
>
> m_stat = 0x04000080
> rx_st = 0x30000011
> dm_tx_st = 0x00000000
> dm_rx_st = 0x00000000
> dma = 0x00000001
>
> Notably we're in DMA mode but are getting M_RX_IRQ_EN and
> M_RX_FIFO_WATERMARK_EN over and over again.
>
> Putting some traces in geni_i2c_rx_one_msg() showed that when we
> failed we were getting to the start of geni_i2c_rx_one_msg() but were
> never executing geni_se_rx_dma_prep().
>
> I believe that the problem here is that we are starting the geni
> command before we run geni_se_rx_dma_prep(). If a transfer makes it
> far enough before we do that then we get into the state I have
> observed. Let's change the order, which seems to work fine.
>
> Although problems were seen on the RX path, code inspection suggests
> that the TX should be changed too. Change it as well.
>
> Fixes: 37692de5d523 ("i2c: i2c-qcom-geni: Add bus driver for the Qualcomm GENI I2C controller")
> Signed-off-by: Douglas Anderson <[email protected]>
> Tested-by: Sai Prakash Ranjan <[email protected]>
> Reviewed-by: Akash Asthana <[email protected]>
> ---
Reviewed-by: Stephen Boyd <[email protected]>
On 7/23/2020 6:20 AM, Stephen Boyd wrote:
> Quoting Douglas Anderson (2020-07-22 15:00:21)
>> When I have KASAN enabled on my kernel and I start stressing the
>> touchscreen my system tends to hang. The touchscreen is one of the
>> only things that does a lot of big i2c transfers and ends up hitting
>> the DMA paths in the geni i2c driver. It appears that KASAN adds
>> enough delay in my system to tickle a race condition in the DMA setup
>> code.
>>
>> When the system hangs, I found that it was running the geni_i2c_irq()
>> over and over again. It had these:
>>
>> m_stat = 0x04000080
>> rx_st = 0x30000011
>> dm_tx_st = 0x00000000
>> dm_rx_st = 0x00000000
>> dma = 0x00000001
>>
>> Notably we're in DMA mode but are getting M_RX_IRQ_EN and
>> M_RX_FIFO_WATERMARK_EN over and over again.
>>
>> Putting some traces in geni_i2c_rx_one_msg() showed that when we
>> failed we were getting to the start of geni_i2c_rx_one_msg() but were
>> never executing geni_se_rx_dma_prep().
>>
>> I believe that the problem here is that we are starting the geni
>> command before we run geni_se_rx_dma_prep(). If a transfer makes it
>> far enough before we do that then we get into the state I have
>> observed. Let's change the order, which seems to work fine.
>>
>> Although problems were seen on the RX path, code inspection suggests
>> that the TX should be changed too. Change it as well.
>>
>> Fixes: 37692de5d523 ("i2c: i2c-qcom-geni: Add bus driver for the Qualcomm GENI I2C controller")
>> Signed-off-by: Douglas Anderson <[email protected]>
>> Tested-by: Sai Prakash Ranjan <[email protected]>
>> Reviewed-by: Akash Asthana <[email protected]>
Reviewed-by: Mukesh Kumar Savaliya <[email protected]>
>> ---
> Reviewed-by: Stephen Boyd <[email protected]>
On Wed, Jul 22, 2020 at 03:00:21PM -0700, Douglas Anderson wrote:
> When I have KASAN enabled on my kernel and I start stressing the
> touchscreen my system tends to hang. The touchscreen is one of the
> only things that does a lot of big i2c transfers and ends up hitting
> the DMA paths in the geni i2c driver. It appears that KASAN adds
> enough delay in my system to tickle a race condition in the DMA setup
> code.
>
> When the system hangs, I found that it was running the geni_i2c_irq()
> over and over again. It had these:
>
> m_stat = 0x04000080
> rx_st = 0x30000011
> dm_tx_st = 0x00000000
> dm_rx_st = 0x00000000
> dma = 0x00000001
>
> Notably we're in DMA mode but are getting M_RX_IRQ_EN and
> M_RX_FIFO_WATERMARK_EN over and over again.
>
> Putting some traces in geni_i2c_rx_one_msg() showed that when we
> failed we were getting to the start of geni_i2c_rx_one_msg() but were
> never executing geni_se_rx_dma_prep().
>
> I believe that the problem here is that we are starting the geni
> command before we run geni_se_rx_dma_prep(). If a transfer makes it
> far enough before we do that then we get into the state I have
> observed. Let's change the order, which seems to work fine.
>
> Although problems were seen on the RX path, code inspection suggests
> that the TX should be changed too. Change it as well.
>
> Fixes: 37692de5d523 ("i2c: i2c-qcom-geni: Add bus driver for the Qualcomm GENI I2C controller")
> Signed-off-by: Douglas Anderson <[email protected]>
> Tested-by: Sai Prakash Ranjan <[email protected]>
> Reviewed-by: Akash Asthana <[email protected]>
Applied to for-current, thanks!
On Thu, Jul 23, 2020 at 09:56:34PM +0200, Wolfram Sang wrote:
> On Wed, Jul 22, 2020 at 03:00:21PM -0700, Douglas Anderson wrote:
> > When I have KASAN enabled on my kernel and I start stressing the
> > touchscreen my system tends to hang. The touchscreen is one of the
> > only things that does a lot of big i2c transfers and ends up hitting
> > the DMA paths in the geni i2c driver. It appears that KASAN adds
> > enough delay in my system to tickle a race condition in the DMA setup
> > code.
> >
> > When the system hangs, I found that it was running the geni_i2c_irq()
> > over and over again. It had these:
> >
> > m_stat = 0x04000080
> > rx_st = 0x30000011
> > dm_tx_st = 0x00000000
> > dm_rx_st = 0x00000000
> > dma = 0x00000001
> >
> > Notably we're in DMA mode but are getting M_RX_IRQ_EN and
> > M_RX_FIFO_WATERMARK_EN over and over again.
> >
> > Putting some traces in geni_i2c_rx_one_msg() showed that when we
> > failed we were getting to the start of geni_i2c_rx_one_msg() but were
> > never executing geni_se_rx_dma_prep().
> >
> > I believe that the problem here is that we are starting the geni
> > command before we run geni_se_rx_dma_prep(). If a transfer makes it
> > far enough before we do that then we get into the state I have
> > observed. Let's change the order, which seems to work fine.
> >
> > Although problems were seen on the RX path, code inspection suggests
> > that the TX should be changed too. Change it as well.
> >
> > Fixes: 37692de5d523 ("i2c: i2c-qcom-geni: Add bus driver for the Qualcomm GENI I2C controller")
> > Signed-off-by: Douglas Anderson <[email protected]>
> > Tested-by: Sai Prakash Ranjan <[email protected]>
> > Reviewed-by: Akash Asthana <[email protected]>
>
> Applied to for-current, thanks!
Glad we got this sorted. I just wondered that Alok wasn't part of the
discussion. Is he still interested in maintaining the driver? Also
because there is an unprocessed patch left for this driver:
http://patchwork.ozlabs.org/project/linux-i2c/patch/[email protected]/
On 7/26/2020 6:17 PM, Wolfram Sang wrote:
> On Thu, Jul 23, 2020 at 09:56:34PM +0200, Wolfram Sang wrote:
>> On Wed, Jul 22, 2020 at 03:00:21PM -0700, Douglas Anderson wrote:
>>> When I have KASAN enabled on my kernel and I start stressing the
>>> touchscreen my system tends to hang. The touchscreen is one of the
>>> only things that does a lot of big i2c transfers and ends up hitting
>>> the DMA paths in the geni i2c driver. It appears that KASAN adds
>>> enough delay in my system to tickle a race condition in the DMA setup
>>> code.
>>>
>>> When the system hangs, I found that it was running the geni_i2c_irq()
>>> over and over again. It had these:
>>>
>>> m_stat = 0x04000080
>>> rx_st = 0x30000011
>>> dm_tx_st = 0x00000000
>>> dm_rx_st = 0x00000000
>>> dma = 0x00000001
>>>
>>> Notably we're in DMA mode but are getting M_RX_IRQ_EN and
>>> M_RX_FIFO_WATERMARK_EN over and over again.
>>>
>>> Putting some traces in geni_i2c_rx_one_msg() showed that when we
>>> failed we were getting to the start of geni_i2c_rx_one_msg() but were
>>> never executing geni_se_rx_dma_prep().
>>>
>>> I believe that the problem here is that we are starting the geni
>>> command before we run geni_se_rx_dma_prep(). If a transfer makes it
>>> far enough before we do that then we get into the state I have
>>> observed. Let's change the order, which seems to work fine.
>>>
>>> Although problems were seen on the RX path, code inspection suggests
>>> that the TX should be changed too. Change it as well.
>>>
>>> Fixes: 37692de5d523 ("i2c: i2c-qcom-geni: Add bus driver for the Qualcomm GENI I2C controller")
>>> Signed-off-by: Douglas Anderson <[email protected]>
>>> Tested-by: Sai Prakash Ranjan <[email protected]>
>>> Reviewed-by: Akash Asthana <[email protected]>
>> Applied to for-current, thanks!
> Glad we got this sorted. I just wondered that Alok wasn't part of the
> discussion. Is he still interested in maintaining the driver? Also
> because there is an unprocessed patch left for this driver:
>
> http://patchwork.ozlabs.org/project/linux-i2c/patch/[email protected]/
Alok has moved out of GENI team, he no longer supports GENI I2C drivers.
I have posted a patch to update maintainers list. Patch:
https://patchwork.kernel.org/patch/11686541/ [MAINTAINERS: Update Geni
I2C maintainers list]
Also, Girish Mahadevan, Sagar Dharia and Karthikeyan Ramasubramanian? no
longer supports GENI drivers.
Regards,
Akash
>
--
The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum,\na Linux Foundation Collaborative Project