igc_configure_rx_ring() function will be called as part of XDP program
setup. If Rx hardware timestamp is enabled prio to XDP program setup,
this timestamp enablement will be overwritten when buffer size is
written into SRRCTL register.
Thus, this commit read the register value before write to SRRCTL
register. This commit is tested by using xdp_hw_metadata bpf selftest
tool. The tool enables Rx hardware timestamp and then attach XDP program
to igc driver. It will display hardware timestamp of UDP packet with
port number 9092. Below are detail of test steps and results.
Command on DUT:
sudo ./xdp_hw_metadata <interface name>
Command on Link Partner:
echo -n skb | nc -u -q1 <destination IPv4 addr> 9092
Result before this patch:
skb hwtstamp is not found!
Result after this patch:
found skb hwtstamp = 1677800973.642836757
Optionally, read PHC to confirm the values obtained are almost the same:
Command:
sudo ./testptp -d /dev/ptp0 -g
Result:
clock time: 1677800973.913598978 or Fri Mar 3 07:49:33 2023
Fixes: fc9df2a0b520 ("igc: Enable RX via AF_XDP zero-copy")
Cc: <[email protected]> # 5.14+
Signed-off-by: Song Yoong Siang <[email protected]>
Reviewed-by: Jacob Keller <[email protected]>
Reviewed-by: Jesper Dangaard Brouer <[email protected]>
---
v2 -> v3: Refactor SRRCTL definitions to more human readable definitions
v1 -> v2: Fix indention
---
drivers/net/ethernet/intel/igc/igc_base.h | 11 ++++++++---
drivers/net/ethernet/intel/igc/igc_main.c | 7 +++++--
2 files changed, 13 insertions(+), 5 deletions(-)
diff --git a/drivers/net/ethernet/intel/igc/igc_base.h b/drivers/net/ethernet/intel/igc/igc_base.h
index 7a992befca24..9f3827eda157 100644
--- a/drivers/net/ethernet/intel/igc/igc_base.h
+++ b/drivers/net/ethernet/intel/igc/igc_base.h
@@ -87,8 +87,13 @@ union igc_adv_rx_desc {
#define IGC_RXDCTL_SWFLUSH 0x04000000 /* Receive Software Flush */
/* SRRCTL bit definitions */
-#define IGC_SRRCTL_BSIZEPKT_SHIFT 10 /* Shift _right_ */
-#define IGC_SRRCTL_BSIZEHDRSIZE_SHIFT 2 /* Shift _left_ */
-#define IGC_SRRCTL_DESCTYPE_ADV_ONEBUF 0x02000000
+#define IGC_SRRCTL_BSIZEPKT_MASK GENMASK(6, 0)
+#define IGC_SRRCTL_BSIZEPKT(x) FIELD_PREP(IGC_SRRCTL_BSIZEPKT_MASK, \
+ (x) / 1024) /* in 1 KB resolution */
+#define IGC_SRRCTL_BSIZEHDR_MASK GENMASK(13, 8)
+#define IGC_SRRCTL_BSIZEHDR(x) FIELD_PREP(IGC_SRRCTL_BSIZEHDR_MASK, \
+ (x) / 64) /* in 64 bytes resolution */
+#define IGC_SRRCTL_DESCTYPE_MASK GENMASK(27, 25)
+#define IGC_SRRCTL_DESCTYPE_ADV_ONEBUF FIELD_PREP(IGC_SRRCTL_DESCTYPE_MASK, 1)
#endif /* _IGC_BASE_H */
diff --git a/drivers/net/ethernet/intel/igc/igc_main.c b/drivers/net/ethernet/intel/igc/igc_main.c
index 25fc6c65209b..a2d823e64609 100644
--- a/drivers/net/ethernet/intel/igc/igc_main.c
+++ b/drivers/net/ethernet/intel/igc/igc_main.c
@@ -641,8 +641,11 @@ static void igc_configure_rx_ring(struct igc_adapter *adapter,
else
buf_size = IGC_RXBUFFER_2048;
- srrctl = IGC_RX_HDR_LEN << IGC_SRRCTL_BSIZEHDRSIZE_SHIFT;
- srrctl |= buf_size >> IGC_SRRCTL_BSIZEPKT_SHIFT;
+ srrctl = rd32(IGC_SRRCTL(reg_idx));
+ srrctl &= ~(IGC_SRRCTL_BSIZEPKT_MASK | IGC_SRRCTL_BSIZEHDR_MASK |
+ IGC_SRRCTL_DESCTYPE_MASK);
+ srrctl |= IGC_SRRCTL_BSIZEHDR(IGC_RX_HDR_LEN);
+ srrctl |= IGC_SRRCTL_BSIZEPKT(buf_size);
srrctl |= IGC_SRRCTL_DESCTYPE_ADV_ONEBUF;
wr32(IGC_SRRCTL(reg_idx), srrctl);
--
2.34.1
On 14/04/2023 17.49, Song Yoong Siang wrote:
> igc_configure_rx_ring() function will be called as part of XDP program
> setup. If Rx hardware timestamp is enabled prio to XDP program setup,
> this timestamp enablement will be overwritten when buffer size is
> written into SRRCTL register.
>
> Thus, this commit read the register value before write to SRRCTL
> register. This commit is tested by using xdp_hw_metadata bpf selftest
> tool. The tool enables Rx hardware timestamp and then attach XDP program
> to igc driver. It will display hardware timestamp of UDP packet with
> port number 9092. Below are detail of test steps and results.
>
> Command on DUT:
> sudo ./xdp_hw_metadata <interface name>
>
> Command on Link Partner:
> echo -n skb | nc -u -q1 <destination IPv4 addr> 9092
>
> Result before this patch:
> skb hwtstamp is not found!
>
> Result after this patch:
> found skb hwtstamp = 1677800973.642836757
>
> Optionally, read PHC to confirm the values obtained are almost the same:
> Command:
> sudo ./testptp -d /dev/ptp0 -g
> Result:
> clock time: 1677800973.913598978 or Fri Mar 3 07:49:33 2023
>
> Fixes: fc9df2a0b520 ("igc: Enable RX via AF_XDP zero-copy")
> Cc: <[email protected]> # 5.14+
> Signed-off-by: Song Yoong Siang <[email protected]>
> Reviewed-by: Jacob Keller <[email protected]>
> Reviewed-by: Jesper Dangaard Brouer <[email protected]>
> ---
LGTM, thank for the adjustments :-)
Acked-by: Jesper Dangaard Brouer <[email protected]>
> v2 -> v3: Refactor SRRCTL definitions to more human readable definitions
> v1 -> v2: Fix indention
> ---
> drivers/net/ethernet/intel/igc/igc_base.h | 11 ++++++++---
> drivers/net/ethernet/intel/igc/igc_main.c | 7 +++++--
> 2 files changed, 13 insertions(+), 5 deletions(-)
>
> diff --git a/drivers/net/ethernet/intel/igc/igc_base.h b/drivers/net/ethernet/intel/igc/igc_base.h
> index 7a992befca24..9f3827eda157 100644
> --- a/drivers/net/ethernet/intel/igc/igc_base.h
> +++ b/drivers/net/ethernet/intel/igc/igc_base.h
> @@ -87,8 +87,13 @@ union igc_adv_rx_desc {
> #define IGC_RXDCTL_SWFLUSH 0x04000000 /* Receive Software Flush */
>
> /* SRRCTL bit definitions */
> -#define IGC_SRRCTL_BSIZEPKT_SHIFT 10 /* Shift _right_ */
> -#define IGC_SRRCTL_BSIZEHDRSIZE_SHIFT 2 /* Shift _left_ */
> -#define IGC_SRRCTL_DESCTYPE_ADV_ONEBUF 0x02000000
> +#define IGC_SRRCTL_BSIZEPKT_MASK GENMASK(6, 0)
> +#define IGC_SRRCTL_BSIZEPKT(x) FIELD_PREP(IGC_SRRCTL_BSIZEPKT_MASK, \
> + (x) / 1024) /* in 1 KB resolution */
> +#define IGC_SRRCTL_BSIZEHDR_MASK GENMASK(13, 8)
> +#define IGC_SRRCTL_BSIZEHDR(x) FIELD_PREP(IGC_SRRCTL_BSIZEHDR_MASK, \
> + (x) / 64) /* in 64 bytes resolution */
> +#define IGC_SRRCTL_DESCTYPE_MASK GENMASK(27, 25)
> +#define IGC_SRRCTL_DESCTYPE_ADV_ONEBUF FIELD_PREP(IGC_SRRCTL_DESCTYPE_MASK, 1)
>
> #endif /* _IGC_BASE_H */
> diff --git a/drivers/net/ethernet/intel/igc/igc_main.c b/drivers/net/ethernet/intel/igc/igc_main.c
> index 25fc6c65209b..a2d823e64609 100644
> --- a/drivers/net/ethernet/intel/igc/igc_main.c
> +++ b/drivers/net/ethernet/intel/igc/igc_main.c
> @@ -641,8 +641,11 @@ static void igc_configure_rx_ring(struct igc_adapter *adapter,
> else
> buf_size = IGC_RXBUFFER_2048;
>
> - srrctl = IGC_RX_HDR_LEN << IGC_SRRCTL_BSIZEHDRSIZE_SHIFT;
> - srrctl |= buf_size >> IGC_SRRCTL_BSIZEPKT_SHIFT;
> + srrctl = rd32(IGC_SRRCTL(reg_idx));
> + srrctl &= ~(IGC_SRRCTL_BSIZEPKT_MASK | IGC_SRRCTL_BSIZEHDR_MASK |
> + IGC_SRRCTL_DESCTYPE_MASK);
> + srrctl |= IGC_SRRCTL_BSIZEHDR(IGC_RX_HDR_LEN);
> + srrctl |= IGC_SRRCTL_BSIZEPKT(buf_size);
> srrctl |= IGC_SRRCTL_DESCTYPE_ADV_ONEBUF;
>
> wr32(IGC_SRRCTL(reg_idx), srrctl);
On 14.04.23 17:49, Song Yoong Siang wrote:
> igc_configure_rx_ring() function will be called as part of XDP program
> setup. If Rx hardware timestamp is enabled prio to XDP program setup,
> this timestamp enablement will be overwritten when buffer size is
> written into SRRCTL register.
>
Hi all,
I'm actually searching for the root cause of a similar problem (RX
timestamp lost) that I can reproduce here, but the setup is slightly
different. The setup:
- igc driver
- i225/226 network card
- When starting to transmit frames using XDP with zero copy enabled
another application (ptp4l) complains about missing RX timestamps
- Loading the XDP program has already been removed (not needed for
TX only)
- ptp4l is using the traditional socket API
- The RX timestamps seem to come back once we stop sending frames
using XDP
The "zero copy support" enabled part is important. If ZC support is not
requested everything works fine.
Any ideas?
Best regards,
Florian
On Saturday, April 15, 2023 5:20 PM, Florian Bezdeka <[email protected]> wrote:
>On 14.04.23 17:49, Song Yoong Siang wrote:
>> igc_configure_rx_ring() function will be called as part of XDP program
>> setup. If Rx hardware timestamp is enabled prio to XDP program setup,
>> this timestamp enablement will be overwritten when buffer size is
>> written into SRRCTL register.
>>
>
>Hi all,
>
>I'm actually searching for the root cause of a similar problem (RX timestamp lost)
>that I can reproduce here, but the setup is slightly different. The setup:
>
>- igc driver
>- i225/226 network card
>- When starting to transmit frames using XDP with zero copy enabled
> another application (ptp4l) complains about missing RX timestamps
>- Loading the XDP program has already been removed (not needed for
> TX only)
>- ptp4l is using the traditional socket API
>- The RX timestamps seem to come back once we stop sending frames
> using XDP
>
>The "zero copy support" enabled part is important. If ZC support is not requested
>everything works fine.
>
>Any ideas?
>
>Best regards,
>Florian
>
Hi Florian,
You means this patch does not help on your issue?
Need to understand more on the setup and behavior to tell.
Are ptp4l and XDP ZC Tx apps running on same queue or separate queue?
I suggest you to run " sudo hwstamp_ctl -i eth0 -r 1 " multiple times in the middle
to check the behavior.
Thanks & Regards
Siang
Hi Florian,
> -----Original Message-----
> From: Intel-wired-lan <[email protected]> On Behalf Of
> Song, Yoong Siang
> Sent: Sunday, 16 April, 2023 10:19 AM
> To: Bezdeka, Florian <[email protected]>; Brandeburg, Jesse
> <[email protected]>; Nguyen, Anthony L
> <[email protected]>; David S . Miller <[email protected]>;
> Eric Dumazet <[email protected]>; Jakub Kicinski <[email protected]>;
> Paolo Abeni <[email protected]>; Alexei Starovoitov <[email protected]>;
> Daniel Borkmann <[email protected]>; Jesper Dangaard Brouer
> <[email protected]>; John Fastabend <[email protected]>;
> Fijalkowski, Maciej <[email protected]>; Vedang Patel
> <[email protected]>; Joseph, Jithu <[email protected]>; Andre
> Guedes <[email protected]>; Brouer, Jesper <[email protected]>;
> Stanislav Fomichev <[email protected]>; Keller, Jacob E
> <[email protected]>; David Laight <[email protected]>
> Cc: [email protected]; [email protected]; linux-
> [email protected]; [email protected]; intel-wired-
> [email protected]; [email protected]
> Subject: Re: [Intel-wired-lan] [PATCH net v3 1/1] igc: read before write to
> SRRCTL register
>
> On Saturday, April 15, 2023 5:20 PM, Florian Bezdeka
> <[email protected]> wrote:
> >On 14.04.23 17:49, Song Yoong Siang wrote:
> >> igc_configure_rx_ring() function will be called as part of XDP
> >> program setup. If Rx hardware timestamp is enabled prio to XDP
> >> program setup, this timestamp enablement will be overwritten when
> >> buffer size is written into SRRCTL register.
> >>
> >
> >Hi all,
> >
> >I'm actually searching for the root cause of a similar problem (RX
> >timestamp lost) that I can reproduce here, but the setup is slightly
> different. The setup:
> >
> >- igc driver
> >- i225/226 network card
> >- When starting to transmit frames using XDP with zero copy enabled
> > another application (ptp4l) complains about missing RX timestamps
> >- Loading the XDP program has already been removed (not needed for
> > TX only)
> >- ptp4l is using the traditional socket API
> >- The RX timestamps seem to come back once we stop sending frames
> > using XDP
> >
> >The "zero copy support" enabled part is important. If ZC support is not
> >requested everything works fine.
> >
> >Any ideas?
Are you observing similar issue like below?
ptp4l: timed out while polling for tx timestamp
ptp4l: increasing tx_timestamp_timeout may correct this issue
If yes, only TXSTAMPO register is used for both PTP and non-PTP packets in
the current driver code. There is a possibility that the time stamp
for a PTP packet will be lost when there is a lot of traffic when multiple
applications request for hardware transmission timestamps.
Few months back, I submitted a patch series to enable the DMA
Timestamp for non-ptp packet which can resolve the above issue.
https://lore.kernel.org/netdev/[email protected]/T/
Will continuing back the activity soon.
But you might want to try with series submitted by Vinicius as well.
This patch series add support for four sets of timestamping register.
https://patchwork.ozlabs.org/project/intel-wired-lan/patch/[email protected]/
https://patchwork.ozlabs.org/project/intel-wired-lan/patch/[email protected]/
https://patchwork.ozlabs.org/project/intel-wired-lan/patch/[email protected]/
Would it be possible for you to try any of the above patch series
and see if that fixes your problem?
Thanks,
Husaini
> >
> >Best regards,
> >Florian
> >
>
> Hi Florian,
>
> You means this patch does not help on your issue?
> Need to understand more on the setup and behavior to tell.
> Are ptp4l and XDP ZC Tx apps running on same queue or separate queue?
> I suggest you to run " sudo hwstamp_ctl -i eth0 -r 1 " multiple times in the
> middle to check the behavior.
>
> Thanks & Regards
> Siang
> _______________________________________________
> Intel-wired-lan mailing list
> [email protected]
> https://lists.osuosl.org/mailman/listinfo/intel-wired-lan
On 14/04/2023 22.05, Jesper Dangaard Brouer wrote:
>
> On 14/04/2023 17.49, Song Yoong Siang wrote:
>> igc_configure_rx_ring() function will be called as part of XDP program
>> setup. If Rx hardware timestamp is enabled prio to XDP program setup,
>> this timestamp enablement will be overwritten when buffer size is
>> written into SRRCTL register.
>>
>> Thus, this commit read the register value before write to SRRCTL
>> register. This commit is tested by using xdp_hw_metadata bpf selftest
>> tool. The tool enables Rx hardware timestamp and then attach XDP program
>> to igc driver. It will display hardware timestamp of UDP packet with
>> port number 9092. Below are detail of test steps and results.
>>
[...]
>>
>> Fixes: fc9df2a0b520 ("igc: Enable RX via AF_XDP zero-copy")
>> Cc: <[email protected]> # 5.14+
>> Signed-off-by: Song Yoong Siang <[email protected]>
>> Reviewed-by: Jacob Keller <[email protected]>
>> Reviewed-by: Jesper Dangaard Brouer <[email protected]>
>> ---
>
> LGTM, thank for the adjustments :-)
>
> Acked-by: Jesper Dangaard Brouer <[email protected]>
>
Tested-by: Jesper Dangaard Brouer <[email protected]>
I can confirm that this patch fix the issue I experienced with igc.
This patch clearly fixes a bug in igc when writing the SRRCTL register.
(as bit 30 in register is "Timestamp Received Packet" which got cleared
before).
Florian might have found another bug around RX timestamps, but this
patch should be safe and sane to apply as is.
>> v2 -> v3: Refactor SRRCTL definitions to more human readable definitions
>> v1 -> v2: Fix indention
>> ---
>> drivers/net/ethernet/intel/igc/igc_base.h | 11 ++++++++---
>> drivers/net/ethernet/intel/igc/igc_main.c | 7 +++++--
>> 2 files changed, 13 insertions(+), 5 deletions(-)
On Mon, 17 Apr 2023 02:53:13 +0000 Zulkifli, Muhammad Husaini wrote:
> Are you observing similar issue like below?
> ptp4l: timed out while polling for tx timestamp
> ptp4l: increasing tx_timestamp_timeout may correct this issue
>
> If yes, only TXSTAMPO register is used for both PTP and non-PTP packets in
> the current driver code. There is a possibility that the time stamp
> for a PTP packet will be lost when there is a lot of traffic when multiple
> applications request for hardware transmission timestamps.
> Few months back, I submitted a patch series to enable the DMA
> Timestamp for non-ptp packet which can resolve the above issue.
> https://lore.kernel.org/netdev/[email protected]/T/
> Will continuing back the activity soon.
FWIW the work on selecting the source of the timestamp is progressing
slowly:
https://lore.kernel.org/all/[email protected]/
On Mon, 2023-04-17 at 16:24 +0200, Jesper Dangaard Brouer wrote:
> On 14/04/2023 22.05, Jesper Dangaard Brouer wrote:
> >
> > On 14/04/2023 17.49, Song Yoong Siang wrote:
> > > igc_configure_rx_ring() function will be called as part of XDP program
> > > setup. If Rx hardware timestamp is enabled prio to XDP program setup,
> > > this timestamp enablement will be overwritten when buffer size is
> > > written into SRRCTL register.
> > >
> > > Thus, this commit read the register value before write to SRRCTL
> > > register. This commit is tested by using xdp_hw_metadata bpf selftest
> > > tool. The tool enables Rx hardware timestamp and then attach XDP program
> > > to igc driver. It will display hardware timestamp of UDP packet with
> > > port number 9092. Below are detail of test steps and results.
> > >
> [...]
> > >
> > > Fixes: fc9df2a0b520 ("igc: Enable RX via AF_XDP zero-copy")
> > > Cc: <[email protected]> # 5.14+
> > > Signed-off-by: Song Yoong Siang <[email protected]>
> > > Reviewed-by: Jacob Keller <[email protected]>
> > > Reviewed-by: Jesper Dangaard Brouer <[email protected]>
> > > ---
> >
> > LGTM, thank for the adjustments :-)
> >
> > Acked-by: Jesper Dangaard Brouer <[email protected]>
> >
>
> Tested-by: Jesper Dangaard Brouer <[email protected]>
>
> I can confirm that this patch fix the issue I experienced with igc.
>
> This patch clearly fixes a bug in igc when writing the SRRCTL register.
> (as bit 30 in register is "Timestamp Received Packet" which got cleared
> before).
>
> Florian might have found another bug around RX timestamps, but this
> patch should be safe and sane to apply as is.
After a closer look I'm quite sure now that this patch should fix my
issue as well. The register will be overwritten when setting up a
XSK_POOL as well:
igc_bpf
igc_xdp_setup_pool
igc_enable_rx_ring
igc_configure_rx_ring
wr32(IGC_SRRCTL)
I already removed the BPF loading (which is the use case that the patch
description mentions) from my setup to limit the search scope. If you
like you could extend the patch description, but I'm fine with it.
Thanks a lot for all the support / ideas! Highly appreciated!
Florian
>
> > > v2 -> v3: Refactor SRRCTL definitions to more human readable definitions
> > > v1 -> v2: Fix indention
> > > ---
> > > drivers/net/ethernet/intel/igc/igc_base.h | 11 ++++++++---
> > > drivers/net/ethernet/intel/igc/igc_main.c | 7 +++++--
> > > 2 files changed, 13 insertions(+), 5 deletions(-)
>
On 4/14/2023 18:49, Song Yoong Siang wrote:
> igc_configure_rx_ring() function will be called as part of XDP program
> setup. If Rx hardware timestamp is enabled prio to XDP program setup,
> this timestamp enablement will be overwritten when buffer size is
> written into SRRCTL register.
>
> Thus, this commit read the register value before write to SRRCTL
> register. This commit is tested by using xdp_hw_metadata bpf selftest
> tool. The tool enables Rx hardware timestamp and then attach XDP program
> to igc driver. It will display hardware timestamp of UDP packet with
> port number 9092. Below are detail of test steps and results.
>
> Command on DUT:
> sudo ./xdp_hw_metadata <interface name>
>
> Command on Link Partner:
> echo -n skb | nc -u -q1 <destination IPv4 addr> 9092
>
> Result before this patch:
> skb hwtstamp is not found!
>
> Result after this patch:
> found skb hwtstamp = 1677800973.642836757
>
> Optionally, read PHC to confirm the values obtained are almost the same:
> Command:
> sudo ./testptp -d /dev/ptp0 -g
> Result:
> clock time: 1677800973.913598978 or Fri Mar 3 07:49:33 2023
>
> Fixes: fc9df2a0b520 ("igc: Enable RX via AF_XDP zero-copy")
> Cc: <[email protected]> # 5.14+
> Signed-off-by: Song Yoong Siang <[email protected]>
> Reviewed-by: Jacob Keller <[email protected]>
> Reviewed-by: Jesper Dangaard Brouer <[email protected]>
> ---
> v2 -> v3: Refactor SRRCTL definitions to more human readable definitions
> v1 -> v2: Fix indention
> ---
> drivers/net/ethernet/intel/igc/igc_base.h | 11 ++++++++---
> drivers/net/ethernet/intel/igc/igc_main.c | 7 +++++--
> 2 files changed, 13 insertions(+), 5 deletions(-)
Tested-by: Naama Meir <[email protected]>