2019-10-22 08:59:25

by Zhi Chen

[permalink] [raw]
Subject: [PATCH RFC] Revert "ath10k: fix DMA related firmware crashes on multiple devices"

This reverts commit 76d164f582150fd0259ec0fcbc485470bcd8033e.
PCIe hung issue was observed on multiple platforms. The issue was reproduced
when DUT was configured as AP and associated with 50+ STAs.

With PCIe protocol analyzer, we can see DMA Read crossing 4KB boundary when
issue happened. It broke PCIe spec and caused PCIe stuck.

Tested: IPQ8064 + QCA9984 with firmware 10.4-3.10-00047
QCS404 + QCA9984 with firmware 10.4-3.9.0.2--00044
Synaptics AS370 + QCA9888 with firmware 10.4-3.9.0.2--00040

Signed-off-by: Zhi Chen <[email protected]>
---
drivers/net/wireless/ath/ath10k/hw.h | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/drivers/net/wireless/ath/ath10k/hw.h b/drivers/net/wireless/ath/ath10k/hw.h
index 35a3623..17f620b 100644
--- a/drivers/net/wireless/ath/ath10k/hw.h
+++ b/drivers/net/wireless/ath/ath10k/hw.h
@@ -753,7 +753,7 @@ ath10k_is_rssi_enable(struct ath10k_hw_params *hw,
#define TARGET_10X_MAX_FRAG_ENTRIES 0

/* 10.2 parameters */
-#define TARGET_10_2_DMA_BURST_SIZE 0
+#define TARGET_10_2_DMA_BURST_SIZE 1

/* Target specific defines for WMI-TLV firmware */
#define TARGET_TLV_NUM_VDEVS 4
@@ -813,7 +813,7 @@ ath10k_is_rssi_enable(struct ath10k_hw_params *hw,

#define TARGET_10_4_TX_DBG_LOG_SIZE 1024
#define TARGET_10_4_NUM_WDS_ENTRIES 32
-#define TARGET_10_4_DMA_BURST_SIZE 0
+#define TARGET_10_4_DMA_BURST_SIZE 1
#define TARGET_10_4_MAC_AGGR_DELIM 0
#define TARGET_10_4_RX_SKIP_DEFRAG_TIMEOUT_DUP_DETECTION_CHECK 1
#define TARGET_10_4_VOW_CONFIG 0
--
2.7.4


2019-10-22 10:10:35

by Tom Psyborg

[permalink] [raw]
Subject: Re: [PATCH RFC] Revert "ath10k: fix DMA related firmware crashes on multiple devices"

On 22/10/2019, Zhi Chen <[email protected]> wrote:
> This reverts commit 76d164f582150fd0259ec0fcbc485470bcd8033e.
> PCIe hung issue was observed on multiple platforms. The issue was
> reproduced
> when DUT was configured as AP and associated with 50+ STAs.
>
> With PCIe protocol analyzer, we can see DMA Read crossing 4KB boundary when
> issue happened. It broke PCIe spec and caused PCIe stuck.
>
> Tested: IPQ8064 + QCA9984 with firmware 10.4-3.10-00047
> QCS404 + QCA9984 with firmware 10.4-3.9.0.2--00044
> Synaptics AS370 + QCA9888 with firmware 10.4-3.9.0.2--00040
>
> Signed-off-by: Zhi Chen <[email protected]>
> ---
> drivers/net/wireless/ath/ath10k/hw.h | 4 ++--
> 1 file changed, 2 insertions(+), 2 deletions(-)
>
> diff --git a/drivers/net/wireless/ath/ath10k/hw.h
> b/drivers/net/wireless/ath/ath10k/hw.h
> index 35a3623..17f620b 100644
> --- a/drivers/net/wireless/ath/ath10k/hw.h
> +++ b/drivers/net/wireless/ath/ath10k/hw.h
> @@ -753,7 +753,7 @@ ath10k_is_rssi_enable(struct ath10k_hw_params *hw,
> #define TARGET_10X_MAX_FRAG_ENTRIES 0
>
> /* 10.2 parameters */
> -#define TARGET_10_2_DMA_BURST_SIZE 0
> +#define TARGET_10_2_DMA_BURST_SIZE 1
>
> /* Target specific defines for WMI-TLV firmware */
> #define TARGET_TLV_NUM_VDEVS 4
> @@ -813,7 +813,7 @@ ath10k_is_rssi_enable(struct ath10k_hw_params *hw,
>
> #define TARGET_10_4_TX_DBG_LOG_SIZE 1024
> #define TARGET_10_4_NUM_WDS_ENTRIES 32
> -#define TARGET_10_4_DMA_BURST_SIZE 0
> +#define TARGET_10_4_DMA_BURST_SIZE 1
> #define TARGET_10_4_MAC_AGGR_DELIM 0
> #define TARGET_10_4_RX_SKIP_DEFRAG_TIMEOUT_DUP_DETECTION_CHECK 1
> #define TARGET_10_4_VOW_CONFIG 0
> --
> 2.7.4
>
>
> _______________________________________________
> ath10k mailing list
> [email protected]
> http://lists.infradead.org/mailman/listinfo/ath10k
>

What about main and 10x firmware branch?

2019-10-22 17:35:46

by Peter Oh

[permalink] [raw]
Subject: Re: [PATCH RFC] Revert "ath10k: fix DMA related firmware crashes on multiple devices"


On 10/22/19 1:57 AM, Zhi Chen wrote:
> This reverts commit 76d164f582150fd0259ec0fcbc485470bcd8033e.
> PCIe hung issue was observed on multiple platforms. The issue was reproduced
> when DUT was configured as AP and associated with 50+ STAs.
>
> With PCIe protocol analyzer, we can see DMA Read crossing 4KB boundary when
> issue happened. It broke PCIe spec and caused PCIe stuck.

How can you say value 0 (I believe it's 64 bytes) DMA burst size causes
the symptom and 1 fixes it?

Peter

2019-10-22 18:26:37

by Adrian Chadd

[permalink] [raw]
Subject: Re: [PATCH RFC] Revert "ath10k: fix DMA related firmware crashes on multiple devices"

On Tue, 22 Oct 2019 at 10:17, Peter Oh <[email protected]> wrote:
>
>
> On 10/22/19 1:57 AM, Zhi Chen wrote:
> > This reverts commit 76d164f582150fd0259ec0fcbc485470bcd8033e.
> > PCIe hung issue was observed on multiple platforms. The issue was reproduced
> > when DUT was configured as AP and associated with 50+ STAs.
> >
> > With PCIe protocol analyzer, we can see DMA Read crossing 4KB boundary when
> > issue happened. It broke PCIe spec and caused PCIe stuck.
>
> How can you say value 0 (I believe it's 64 bytes) DMA burst size causes
> the symptom and 1 fixes it?


+1 to this question.

Also, shouldn't the DMA engine be doing what the firmware says? Is the
firmware/copy engine actually somehow bursting / prefetching across a
4K page boundary?

Surely this is something that can be fixed in software/firmware by
correctly configuring up buffer size/offsets?



-adrian

2019-10-30 02:45:37

by Zhi Chen

[permalink] [raw]
Subject: Re: [PATCH RFC] Revert "ath10k: fix DMA related firmware crashes on multiple devices"

On 2019-10-22 18:07, Tom Psyborg wrote:

> What about main and 10x firmware branch?

There is no code changes in firmware. It's a configuration change of
host memory access.

Zhi

2019-10-30 06:06:21

by Peter Oh

[permalink] [raw]
Subject: Re: [PATCH RFC] Revert "ath10k: fix DMA related firmware crashes on multiple devices"


On 10/22/19 11:24 AM, Adrian Chadd wrote:
> On Tue, 22 Oct 2019 at 10:17, Peter Oh <[email protected]> wrote:
>>
>> On 10/22/19 1:57 AM, Zhi Chen wrote:
>>> This reverts commit 76d164f582150fd0259ec0fcbc485470bcd8033e.
>>> PCIe hung issue was observed on multiple platforms. The issue was reproduced
>>> when DUT was configured as AP and associated with 50+ STAs.
>>>
>>> With PCIe protocol analyzer, we can see DMA Read crossing 4KB boundary when
>>> issue happened. It broke PCIe spec and caused PCIe stuck.
Can you elaborate how 64 bytes DMA size breaks PCIe spec?
>> How can you say value 0 (I believe it's 64 bytes) DMA burst size causes
>> the symptom and 1 fixes it?
>
> +1 to this question.
>
> Also, shouldn't the DMA engine be doing what the firmware says? Is the
> firmware/copy engine actually somehow bursting / prefetching across a
> 4K page boundary?
>
> Surely this is something that can be fixed in software/firmware by
> correctly configuring up buffer size/offsets?
>
>
>
> -adrian

2019-10-30 06:18:49

by Zhi Chen

[permalink] [raw]
Subject: Re: [PATCH RFC] Revert "ath10k: fix DMA related firmware crashes on multiple devices"

On 2019-10-23 01:16, Peter Oh wrote:
>
> How can you say value 0 (I believe it's 64 bytes) DMA burst size
> causes the symptom and 1 fixes it?
>
> Peter

Confirmed from HW team that the configuration controls AXI burst size of
the RD/WR access to the HOST MEM.
0- No split , RAW read/write transfer size from MAC is put out on bus as
burst length.
1- Split at 256 byte boundary
2,3- Reserved

That's why we see issue with value 0.

Zhi

2019-10-30 06:29:23

by Zhi Chen

[permalink] [raw]
Subject: Re: [PATCH RFC] Revert "ath10k: fix DMA related firmware crashes on multiple devices"

On 2019-10-23 02:24, Adrian Chadd wrote:
> On Tue, 22 Oct 2019 at 10:17, Peter Oh <[email protected]> wrote:
>>
>>
>> On 10/22/19 1:57 AM, Zhi Chen wrote:
>> > This reverts commit 76d164f582150fd0259ec0fcbc485470bcd8033e.
>> > PCIe hung issue was observed on multiple platforms. The issue was reproduced
>> > when DUT was configured as AP and associated with 50+ STAs.
>> >
>> > With PCIe protocol analyzer, we can see DMA Read crossing 4KB boundary when
>> > issue happened. It broke PCIe spec and caused PCIe stuck.
>>
>> How can you say value 0 (I believe it's 64 bytes) DMA burst size
>> causes
>> the symptom and 1 fixes it?
>
>
> +1 to this question.
>
> Also, shouldn't the DMA engine be doing what the firmware says? Is the
> firmware/copy engine actually somehow bursting / prefetching across a
> 4K page boundary?
>
> Surely this is something that can be fixed in software/firmware by
> correctly configuring up buffer size/offsets?
>
>
>
> -Adrian

DMA engine is working as expected as the configuration. It's copy engine
which actually
splits the RD/WR requests and accesses host memory.
And yes it's platform related configuration. We have never hit this
issue on x86 platform.


Zhi

2019-10-30 23:02:45

by Peter Oh

[permalink] [raw]
Subject: Re: [PATCH RFC] Revert "ath10k: fix DMA related firmware crashes on multiple devices"


On 10/29/19 11:16 PM, [email protected] wrote:
> On 2019-10-23 01:16, Peter Oh wrote:
>>
>> How can you say value 0 (I believe it's 64 bytes) DMA burst size
>> causes the symptom and 1 fixes it?
>>
>> Peter
>
> Confirmed from HW team that the configuration controls AXI burst size
> of the RD/WR access to the HOST MEM.
> 0-    No split , RAW read/write transfer size from MAC is put out on
> bus as burst length.
> 1-    Split at 256 byte boundary
> 2,3- Reserved
>
I would ACK to this change. Although QCA4019 host mem max AXI read/write
registers have different descriptions from above, they say 1 (256-byte)
is default and the values, 0, 2, and 3 are reserved. So changing it back
to 1 is reasonable to me.


Thanks,

Peter

2019-11-03 15:36:54

by Tom Psyborg

[permalink] [raw]
Subject: Re: [PATCH RFC] Revert "ath10k: fix DMA related firmware crashes on multiple devices"

On 30/10/2019, [email protected] <[email protected]> wrote:
> On 2019-10-23 01:16, Peter Oh wrote:
>>
>> How can you say value 0 (I believe it's 64 bytes) DMA burst size
>> causes the symptom and 1 fixes it?
>>
>> Peter
>
> Confirmed from HW team that the configuration controls AXI burst size of
> the RD/WR access to the HOST MEM.
> 0- No split , RAW read/write transfer size from MAC is put out on bus as
> burst length.
> 1- Split at 256 byte boundary
> 2,3- Reserved
>
> That's why we see issue with value 0.
>
> Zhi
>
> _______________________________________________
> ath10k mailing list
> [email protected]
> http://lists.infradead.org/mailman/listinfo/ath10k
>

is this true for both wave1 and wave2 ? at least per this commit
message: ath10k: Fix DMA burst size
it's suppose to be:

0 - 128B max (not sure if this means 128B static, or any size between 0 and 128)
1 - 256B

2019-11-04 18:43:01

by Peter Oh

[permalink] [raw]
Subject: Re: [PATCH RFC] Revert "ath10k: fix DMA related firmware crashes on multiple devices"


On 11/3/19 7:36 AM, Tom Psyborg wrote:
> On 30/10/2019, [email protected] <[email protected]> wrote:
>> On 2019-10-23 01:16, Peter Oh wrote:
>>> How can you say value 0 (I believe it's 64 bytes) DMA burst size
>>> causes the symptom and 1 fixes it?
>>>
>>> Peter
>> Confirmed from HW team that the configuration controls AXI burst size of
>> the RD/WR access to the HOST MEM.
>> 0- No split , RAW read/write transfer size from MAC is put out on bus as
>> burst length.
>> 1- Split at 256 byte boundary
>> 2,3- Reserved
>>
>> That's why we see issue with value 0.
>>
>> Zhi
>>
>> _______________________________________________
>> ath10k mailing list
>> [email protected]
>> http://lists.infradead.org/mailman/listinfo/ath10k
>>
> is this true for both wave1 and wave2 ? at least per this commit
> message: ath10k: Fix DMA burst size
> it's suppose to be:
>
> 0 - 128B max (not sure if this means 128B static, or any size between 0 and 128)
> 1 - 256B
I didn't check other chipsets, but QCA4019 registers say AXI read/write
burst size is 64 bytes for 0.

2019-11-06 10:02:51

by Tom Psyborg

[permalink] [raw]
Subject: Re: [PATCH RFC] Revert "ath10k: fix DMA related firmware crashes on multiple devices"

On 30/10/2019, [email protected] <[email protected]> wrote:
> On 2019-10-22 18:07, Tom Psyborg wrote:
>
>> What about main and 10x firmware branch?
>
> There is no code changes in firmware. It's a configuration change of
> host memory access.
>
> Zhi
>

Please change it for main and 10x branch too. Dumping hw1.0 regs with
QCA driver shows DMA burst value set to 1

2019-11-07 05:37:25

by Zhi Chen

[permalink] [raw]
Subject: Re: [PATCH RFC] Revert "ath10k: fix DMA related firmware crashes on multiple devices"

On 2019-11-06 18:01, Tom Psyborg wrote:
> On 30/10/2019, [email protected] <[email protected]> wrote:
>> On 2019-10-22 18:07, Tom Psyborg wrote:
>>
>>> What about main and 10x firmware branch?
>>
>> There is no code changes in firmware. It's a configuration change of
>> host memory access.
>>
>> Zhi
>>
>
> Please change it for main and 10x branch too. Dumping hw1.0 regs with
> QCA driver shows DMA burst value set to 1

Looks some old chips have different definitions of DMA burst size. We
are discussing internally to change it for 10.4 only because we only
tested 10.4. On the other hand we are waiting for HW team to explain
the differences.

Zhi