On Thu, May 16, 2019 at 02:34:08PM -0400, Lennart Sorensen wrote:
> Here is what I see:
>
> i40e: Intel(R) Ethernet Connection XL710 Network Driver - version 2.1.7-k
> i40e: Copyright (c) 2013 - 2014 Intel Corporation.
> i40e 0000:3d:00.0: fw 3.10.52896 api 1.6 nvm 4.00 0x80001577 1.1767.0
> i40e 0000:3d:00.0: The driver for the device detected a newer version of the NVM image than expected. Please install the most recent version of the network driver.
> i40e 0000:3d:00.0: MAC address: a4:bf:01:4e:0c:87
> i40e 0000:3d:00.0: flow_type: 63 input_mask:0x0000000000004000
> i40e 0000:3d:00.0: flow_type: 46 input_mask:0x0007fff800000000
> i40e 0000:3d:00.0: flow_type: 45 input_mask:0x0007fff800000000
> i40e 0000:3d:00.0: flow_type: 44 input_mask:0x0007ffff80000000
> i40e 0000:3d:00.0: flow_type: 43 input_mask:0x0007fffe00000000
> i40e 0000:3d:00.0: flow_type: 42 input_mask:0x0007fffe00000000
> i40e 0000:3d:00.0: flow_type: 41 input_mask:0x0007fffe00000000
> i40e 0000:3d:00.0: flow_type: 40 input_mask:0x0007fffe00000000
> i40e 0000:3d:00.0: flow_type: 39 input_mask:0x0007fffe00000000
> i40e 0000:3d:00.0: flow_type: 36 input_mask:0x0006060000000000
> i40e 0000:3d:00.0: flow_type: 35 input_mask:0x0006060000000000
> i40e 0000:3d:00.0: flow_type: 34 input_mask:0x0006060780000000
> i40e 0000:3d:00.0: flow_type: 33 input_mask:0x0006060600000000
> i40e 0000:3d:00.0: flow_type: 32 input_mask:0x0006060600000000
> i40e 0000:3d:00.0: flow_type: 31 input_mask:0x0006060600000000
> i40e 0000:3d:00.0: flow_type: 30 input_mask:0x0006060600000000
> i40e 0000:3d:00.0: flow_type: 29 input_mask:0x0006060600000000
> i40e 0000:3d:00.0: Features: PF-id[0] VSIs: 34 QP: 12 TXQ: 13 RSS VxLAN Geneve VEPA
> i40e 0000:3d:00.1: fw 3.10.52896 api 1.6 nvm 4.00 0x80001577 1.1767.0
> i40e 0000:3d:00.1: The driver for the device detected a newer version of the NVM image than expected. Please install the most recent version of the network driver.
> i40e 0000:3d:00.1: MAC address: a4:bf:01:4e:0c:88
> i40e 0000:3d:00.1: flow_type: 63 input_mask:0x0000000000004000
> i40e 0000:3d:00.1: flow_type: 46 input_mask:0x0007fff800000000
> i40e 0000:3d:00.1: flow_type: 45 input_mask:0x0007fff800000000
> i40e 0000:3d:00.1: flow_type: 44 input_mask:0x0007ffff80000000
> i40e 0000:3d:00.1: flow_type: 43 input_mask:0x0007fffe00000000
> i40e 0000:3d:00.1: flow_type: 42 input_mask:0x0007fffe00000000
> i40e 0000:3d:00.1: flow_type: 41 input_mask:0x0007fffe00000000
> i40e 0000:3d:00.1: flow_type: 40 input_mask:0x0007fffe00000000
> i40e 0000:3d:00.1: flow_type: 39 input_mask:0x0007fffe00000000
> i40e 0000:3d:00.1: flow_type: 36 input_mask:0x0006060000000000
> i40e 0000:3d:00.1: flow_type: 35 input_mask:0x0006060000000000
> i40e 0000:3d:00.1: flow_type: 34 input_mask:0x0006060780000000
> i40e 0000:3d:00.1: flow_type: 33 input_mask:0x0006060600000000
> i40e 0000:3d:00.1: flow_type: 32 input_mask:0x0006060600000000
> i40e 0000:3d:00.1: flow_type: 31 input_mask:0x0006060600000000
> i40e 0000:3d:00.1: flow_type: 30 input_mask:0x0006060600000000
> i40e 0000:3d:00.1: flow_type: 29 input_mask:0x0006060600000000
> i40e 0000:3d:00.1: Features: PF-id[1] VSIs: 34 QP: 12 TXQ: 13 RSS VxLAN Geneve VEPA
> i40e 0000:3d:00.1 eth2: NIC Link is Up, 1000 Mbps Full Duplex, Flow Control: None
> i40e_ioctl: power down: eth1
> i40e_ioctl: power down: eth2
Those last two lines is something I added, so ignore those.
--
Len Sorensen
On Thu, May 16, 2019 at 11:37 AM Lennart Sorensen
<[email protected]> wrote:
>
> On Thu, May 16, 2019 at 02:34:08PM -0400, Lennart Sorensen wrote:
> > Here is what I see:
> >
> > i40e: Intel(R) Ethernet Connection XL710 Network Driver - version 2.1.7-k
> > i40e: Copyright (c) 2013 - 2014 Intel Corporation.
> > i40e 0000:3d:00.0: fw 3.10.52896 api 1.6 nvm 4.00 0x80001577 1.1767.0
> > i40e 0000:3d:00.0: The driver for the device detected a newer version of the NVM image than expected. Please install the most recent version of the network driver.
> > i40e 0000:3d:00.0: MAC address: a4:bf:01:4e:0c:87
> > i40e 0000:3d:00.0: flow_type: 63 input_mask:0x0000000000004000
> > i40e 0000:3d:00.0: flow_type: 46 input_mask:0x0007fff800000000
> > i40e 0000:3d:00.0: flow_type: 45 input_mask:0x0007fff800000000
> > i40e 0000:3d:00.0: flow_type: 44 input_mask:0x0007ffff80000000
> > i40e 0000:3d:00.0: flow_type: 43 input_mask:0x0007fffe00000000
> > i40e 0000:3d:00.0: flow_type: 42 input_mask:0x0007fffe00000000
> > i40e 0000:3d:00.0: flow_type: 41 input_mask:0x0007fffe00000000
> > i40e 0000:3d:00.0: flow_type: 40 input_mask:0x0007fffe00000000
> > i40e 0000:3d:00.0: flow_type: 39 input_mask:0x0007fffe00000000
> > i40e 0000:3d:00.0: flow_type: 36 input_mask:0x0006060000000000
> > i40e 0000:3d:00.0: flow_type: 35 input_mask:0x0006060000000000
> > i40e 0000:3d:00.0: flow_type: 34 input_mask:0x0006060780000000
> > i40e 0000:3d:00.0: flow_type: 33 input_mask:0x0006060600000000
> > i40e 0000:3d:00.0: flow_type: 32 input_mask:0x0006060600000000
> > i40e 0000:3d:00.0: flow_type: 31 input_mask:0x0006060600000000
> > i40e 0000:3d:00.0: flow_type: 30 input_mask:0x0006060600000000
> > i40e 0000:3d:00.0: flow_type: 29 input_mask:0x0006060600000000
> > i40e 0000:3d:00.0: Features: PF-id[0] VSIs: 34 QP: 12 TXQ: 13 RSS VxLAN Geneve VEPA
> > i40e 0000:3d:00.1: fw 3.10.52896 api 1.6 nvm 4.00 0x80001577 1.1767.0
> > i40e 0000:3d:00.1: The driver for the device detected a newer version of the NVM image than expected. Please install the most recent version of the network driver.
> > i40e 0000:3d:00.1: MAC address: a4:bf:01:4e:0c:88
> > i40e 0000:3d:00.1: flow_type: 63 input_mask:0x0000000000004000
> > i40e 0000:3d:00.1: flow_type: 46 input_mask:0x0007fff800000000
> > i40e 0000:3d:00.1: flow_type: 45 input_mask:0x0007fff800000000
> > i40e 0000:3d:00.1: flow_type: 44 input_mask:0x0007ffff80000000
> > i40e 0000:3d:00.1: flow_type: 43 input_mask:0x0007fffe00000000
> > i40e 0000:3d:00.1: flow_type: 42 input_mask:0x0007fffe00000000
> > i40e 0000:3d:00.1: flow_type: 41 input_mask:0x0007fffe00000000
> > i40e 0000:3d:00.1: flow_type: 40 input_mask:0x0007fffe00000000
> > i40e 0000:3d:00.1: flow_type: 39 input_mask:0x0007fffe00000000
> > i40e 0000:3d:00.1: flow_type: 36 input_mask:0x0006060000000000
> > i40e 0000:3d:00.1: flow_type: 35 input_mask:0x0006060000000000
> > i40e 0000:3d:00.1: flow_type: 34 input_mask:0x0006060780000000
> > i40e 0000:3d:00.1: flow_type: 33 input_mask:0x0006060600000000
> > i40e 0000:3d:00.1: flow_type: 32 input_mask:0x0006060600000000
> > i40e 0000:3d:00.1: flow_type: 31 input_mask:0x0006060600000000
> > i40e 0000:3d:00.1: flow_type: 30 input_mask:0x0006060600000000
> > i40e 0000:3d:00.1: flow_type: 29 input_mask:0x0006060600000000
> > i40e 0000:3d:00.1: Features: PF-id[1] VSIs: 34 QP: 12 TXQ: 13 RSS VxLAN Geneve VEPA
> > i40e 0000:3d:00.1 eth2: NIC Link is Up, 1000 Mbps Full Duplex, Flow Control: None
> > i40e_ioctl: power down: eth1
> > i40e_ioctl: power down: eth2
>
> Those last two lines is something I added, so ignore those.
No problem.
So just looking at the data provided I am going to guess that IPv6 w/
UDP likely works without any issues and it is just going to be IPv4
that is the problem. When you compare the UDP setup from mine versus
yours it looks like for some reason somebody swapped around the input
bits for the L3 src and destination fields. I'm basing that on the
input set masks in the i40e_txrx.h header:
/* INPUT SET MASK for RSS, flow director, and flexible payload */
#define I40E_L3_SRC_SHIFT 47
#define I40E_L3_SRC_MASK (0x3ULL << I40E_L3_SRC_SHIFT)
#define I40E_L3_V6_SRC_SHIFT 43
#define I40E_L3_V6_SRC_MASK (0xFFULL << I40E_L3_V6_SRC_SHIFT)
#define I40E_L3_DST_SHIFT 35
#define I40E_L3_DST_MASK (0x3ULL << I40E_L3_DST_SHIFT)
#define I40E_L3_V6_DST_SHIFT 35
#define I40E_L3_V6_DST_MASK (0xFFULL << I40E_L3_V6_DST_SHIFT)
#define I40E_L4_SRC_SHIFT 34
#define I40E_L4_SRC_MASK (0x1ULL << I40E_L4_SRC_SHIFT)
#define I40E_L4_DST_SHIFT 33
#define I40E_L4_DST_MASK (0x1ULL << I40E_L4_DST_SHIFT)
#define I40E_VERIFY_TAG_SHIFT 31
#define I40E_VERIFY_TAG_MASK (0x3ULL << I40E_VERIFY_TAG_SHIFT)
The easiest way to verify would be to rewrite the registers for
flow_type 29, 30, and 31 to match the value that I had shown earlier
from my dump:
[ 294.687087] i40e 0000:81:00.1: flow_type: 31 input_mask:0x0001801e00000000
I will take a look at putting together a patch that can be tested to
verify if this is actually the issue tomorrow.
Thanks.
- Alex
On Thu, May 16, 2019 at 4:32 PM Alexander Duyck
<[email protected]> wrote:
>
> On Thu, May 16, 2019 at 11:37 AM Lennart Sorensen
> <[email protected]> wrote:
> >
> > On Thu, May 16, 2019 at 02:34:08PM -0400, Lennart Sorensen wrote:
> > > Here is what I see:
> > >
> > > i40e: Intel(R) Ethernet Connection XL710 Network Driver - version 2.1.7-k
> > > i40e: Copyright (c) 2013 - 2014 Intel Corporation.
> > > i40e 0000:3d:00.0: fw 3.10.52896 api 1.6 nvm 4.00 0x80001577 1.1767.0
> > > i40e 0000:3d:00.0: The driver for the device detected a newer version of the NVM image than expected. Please install the most recent version of the network driver.
> > > i40e 0000:3d:00.0: MAC address: a4:bf:01:4e:0c:87
> > > i40e 0000:3d:00.0: flow_type: 63 input_mask:0x0000000000004000
> > > i40e 0000:3d:00.0: flow_type: 46 input_mask:0x0007fff800000000
> > > i40e 0000:3d:00.0: flow_type: 45 input_mask:0x0007fff800000000
> > > i40e 0000:3d:00.0: flow_type: 44 input_mask:0x0007ffff80000000
> > > i40e 0000:3d:00.0: flow_type: 43 input_mask:0x0007fffe00000000
> > > i40e 0000:3d:00.0: flow_type: 42 input_mask:0x0007fffe00000000
> > > i40e 0000:3d:00.0: flow_type: 41 input_mask:0x0007fffe00000000
> > > i40e 0000:3d:00.0: flow_type: 40 input_mask:0x0007fffe00000000
> > > i40e 0000:3d:00.0: flow_type: 39 input_mask:0x0007fffe00000000
> > > i40e 0000:3d:00.0: flow_type: 36 input_mask:0x0006060000000000
> > > i40e 0000:3d:00.0: flow_type: 35 input_mask:0x0006060000000000
> > > i40e 0000:3d:00.0: flow_type: 34 input_mask:0x0006060780000000
> > > i40e 0000:3d:00.0: flow_type: 33 input_mask:0x0006060600000000
> > > i40e 0000:3d:00.0: flow_type: 32 input_mask:0x0006060600000000
> > > i40e 0000:3d:00.0: flow_type: 31 input_mask:0x0006060600000000
> > > i40e 0000:3d:00.0: flow_type: 30 input_mask:0x0006060600000000
> > > i40e 0000:3d:00.0: flow_type: 29 input_mask:0x0006060600000000
> > > i40e 0000:3d:00.0: Features: PF-id[0] VSIs: 34 QP: 12 TXQ: 13 RSS VxLAN Geneve VEPA
> > > i40e 0000:3d:00.1: fw 3.10.52896 api 1.6 nvm 4.00 0x80001577 1.1767.0
> > > i40e 0000:3d:00.1: The driver for the device detected a newer version of the NVM image than expected. Please install the most recent version of the network driver.
> > > i40e 0000:3d:00.1: MAC address: a4:bf:01:4e:0c:88
> > > i40e 0000:3d:00.1: flow_type: 63 input_mask:0x0000000000004000
> > > i40e 0000:3d:00.1: flow_type: 46 input_mask:0x0007fff800000000
> > > i40e 0000:3d:00.1: flow_type: 45 input_mask:0x0007fff800000000
> > > i40e 0000:3d:00.1: flow_type: 44 input_mask:0x0007ffff80000000
> > > i40e 0000:3d:00.1: flow_type: 43 input_mask:0x0007fffe00000000
> > > i40e 0000:3d:00.1: flow_type: 42 input_mask:0x0007fffe00000000
> > > i40e 0000:3d:00.1: flow_type: 41 input_mask:0x0007fffe00000000
> > > i40e 0000:3d:00.1: flow_type: 40 input_mask:0x0007fffe00000000
> > > i40e 0000:3d:00.1: flow_type: 39 input_mask:0x0007fffe00000000
> > > i40e 0000:3d:00.1: flow_type: 36 input_mask:0x0006060000000000
> > > i40e 0000:3d:00.1: flow_type: 35 input_mask:0x0006060000000000
> > > i40e 0000:3d:00.1: flow_type: 34 input_mask:0x0006060780000000
> > > i40e 0000:3d:00.1: flow_type: 33 input_mask:0x0006060600000000
> > > i40e 0000:3d:00.1: flow_type: 32 input_mask:0x0006060600000000
> > > i40e 0000:3d:00.1: flow_type: 31 input_mask:0x0006060600000000
> > > i40e 0000:3d:00.1: flow_type: 30 input_mask:0x0006060600000000
> > > i40e 0000:3d:00.1: flow_type: 29 input_mask:0x0006060600000000
> > > i40e 0000:3d:00.1: Features: PF-id[1] VSIs: 34 QP: 12 TXQ: 13 RSS VxLAN Geneve VEPA
> > > i40e 0000:3d:00.1 eth2: NIC Link is Up, 1000 Mbps Full Duplex, Flow Control: None
> > > i40e_ioctl: power down: eth1
> > > i40e_ioctl: power down: eth2
> >
> > Those last two lines is something I added, so ignore those.
>
> No problem.
>
> So just looking at the data provided I am going to guess that IPv6 w/
> UDP likely works without any issues and it is just going to be IPv4
> that is the problem. When you compare the UDP setup from mine versus
> yours it looks like for some reason somebody swapped around the input
> bits for the L3 src and destination fields. I'm basing that on the
> input set masks in the i40e_txrx.h header:
> /* INPUT SET MASK for RSS, flow director, and flexible payload */
> #define I40E_L3_SRC_SHIFT 47
> #define I40E_L3_SRC_MASK (0x3ULL << I40E_L3_SRC_SHIFT)
> #define I40E_L3_V6_SRC_SHIFT 43
> #define I40E_L3_V6_SRC_MASK (0xFFULL << I40E_L3_V6_SRC_SHIFT)
> #define I40E_L3_DST_SHIFT 35
> #define I40E_L3_DST_MASK (0x3ULL << I40E_L3_DST_SHIFT)
> #define I40E_L3_V6_DST_SHIFT 35
> #define I40E_L3_V6_DST_MASK (0xFFULL << I40E_L3_V6_DST_SHIFT)
> #define I40E_L4_SRC_SHIFT 34
> #define I40E_L4_SRC_MASK (0x1ULL << I40E_L4_SRC_SHIFT)
> #define I40E_L4_DST_SHIFT 33
> #define I40E_L4_DST_MASK (0x1ULL << I40E_L4_DST_SHIFT)
> #define I40E_VERIFY_TAG_SHIFT 31
> #define I40E_VERIFY_TAG_MASK (0x3ULL << I40E_VERIFY_TAG_SHIFT)
>
> The easiest way to verify would be to rewrite the registers for
> flow_type 29, 30, and 31 to match the value that I had shown earlier
> from my dump:
> [ 294.687087] i40e 0000:81:00.1: flow_type: 31 input_mask:0x0001801e00000000
>
> I will take a look at putting together a patch that can be tested to
> verify if this is actually the issue tomorrow.
>
> Thanks.
>
> - Alex
So the patch below/attached should resolve the issues you are seeing
with your system in terms of UDPv4 RSS. What you should see with this
patch is the first function to come up will display some "update input
mask" messages, and then the remaining functions shouldn't make any
noise about it since the registers being updated are global to the
device.
If you can test this and see if it resolves the UDPv4 RSS issues I
would appreciate it.
Thanks.
- Alex
diff --git a/drivers/net/ethernet/intel/i40e/i40e_main.c
b/drivers/net/ethernet/intel/i40e/i40e_main.c
index 65c2b9d2652b..c0a7f66babd9 100644
--- a/drivers/net/ethernet/intel/i40e/i40e_main.c
+++ b/drivers/net/ethernet/intel/i40e/i40e_main.c
@@ -10998,6 +10998,58 @@ static int i40e_pf_config_rss(struct i40e_pf *pf)
((u64)i40e_read_rx_ctl(hw, I40E_PFQF_HENA(1)) << 32);
hena |= i40e_pf_get_default_rss_hena(pf);
+ for (ret = 64; ret--;) {
+ u64 hash_inset_orig, hash_inset_update;
+
+ if (!(hena & (1ull << ret)))
+ continue;
+
+ /* Read initial input set value for flow type */
+ hash_inset_orig = i40e_read_rx_ctl(hw,
I40E_GLQF_HASH_INSET(1, ret));
+ hash_inset_orig <<= 32;
+ hash_inset_orig |= i40e_read_rx_ctl(hw,
I40E_GLQF_HASH_INSET(0, ret));
+
+ /* Copy value so we can compare later */
+ hash_inset_update = hash_inset_orig;
+
+ /* We should be looking at either the entire IPv6 or IPv4
+ * mask being set. If only part of the IPv6 mask is set, but
+ * the IPv4 mask is not then we have a garbage mask value
+ * and need to reset it.
+ */
+ switch (hash_inset_orig & I40E_L3_V6_SRC_MASK) {
+ case I40E_L3_V6_SRC_MASK:
+ case I40E_L3_SRC_MASK:
+ case 0:
+ break;
+ default:
+ hash_inset_update &= ~I40E_L3_V6_SRC_MASK;
+ hash_inset_update |= I40E_L3_SRC_MASK;
+ }
+
+ switch (hash_inset_orig & I40E_L3_V6_DST_MASK) {
+ case I40E_L3_V6_DST_MASK:
+ case I40E_L3_DST_MASK:
+ case 0:
+ break;
+ default:
+ hash_inset_update &= ~I40E_L3_V6_DST_MASK;
+ hash_inset_update |= I40E_L3_DST_MASK;
+ }
+
+ if (hash_inset_update != hash_inset_orig) {
+ dev_warn(&pf->pdev->dev,
+ "flow type: %d update input mask
from:0x%016llx, to:0x%016llx\n",
+ ret,
+ hash_inset_orig, hash_inset_update);
+ i40e_write_rx_ctl(hw, I40E_GLQF_HASH_INSET(0, ret),
+ (u32)hash_inset_update);
+ hash_inset_update >>= 32;
+ i40e_write_rx_ctl(hw, I40E_GLQF_HASH_INSET(1, ret),
+ (u32)hash_inset_update);
+ }
+ }
+
i40e_write_rx_ctl(hw, I40E_PFQF_HENA(0), (u32)hena);
i40e_write_rx_ctl(hw, I40E_PFQF_HENA(1), (u32)(hena >> 32));
On Fri, May 17, 2019 at 09:42:19AM -0700, Alexander Duyck wrote:
> So the patch below/attached should resolve the issues you are seeing
> with your system in terms of UDPv4 RSS. What you should see with this
> patch is the first function to come up will display some "update input
> mask" messages, and then the remaining functions shouldn't make any
> noise about it since the registers being updated are global to the
> device.
>
> If you can test this and see if it resolves the UDPv4 RSS issues I
> would appreciate it.
>
> Thanks.
>
> - Alex
>
> diff --git a/drivers/net/ethernet/intel/i40e/i40e_main.c
> b/drivers/net/ethernet/intel/i40e/i40e_main.c
> index 65c2b9d2652b..c0a7f66babd9 100644
> --- a/drivers/net/ethernet/intel/i40e/i40e_main.c
> +++ b/drivers/net/ethernet/intel/i40e/i40e_main.c
> @@ -10998,6 +10998,58 @@ static int i40e_pf_config_rss(struct i40e_pf *pf)
> ((u64)i40e_read_rx_ctl(hw, I40E_PFQF_HENA(1)) << 32);
> hena |= i40e_pf_get_default_rss_hena(pf);
>
> + for (ret = 64; ret--;) {
> + u64 hash_inset_orig, hash_inset_update;
> +
> + if (!(hena & (1ull << ret)))
> + continue;
> +
> + /* Read initial input set value for flow type */
> + hash_inset_orig = i40e_read_rx_ctl(hw,
> I40E_GLQF_HASH_INSET(1, ret));
> + hash_inset_orig <<= 32;
> + hash_inset_orig |= i40e_read_rx_ctl(hw,
> I40E_GLQF_HASH_INSET(0, ret));
> +
> + /* Copy value so we can compare later */
> + hash_inset_update = hash_inset_orig;
> +
> + /* We should be looking at either the entire IPv6 or IPv4
> + * mask being set. If only part of the IPv6 mask is set, but
> + * the IPv4 mask is not then we have a garbage mask value
> + * and need to reset it.
> + */
> + switch (hash_inset_orig & I40E_L3_V6_SRC_MASK) {
> + case I40E_L3_V6_SRC_MASK:
> + case I40E_L3_SRC_MASK:
> + case 0:
> + break;
> + default:
> + hash_inset_update &= ~I40E_L3_V6_SRC_MASK;
> + hash_inset_update |= I40E_L3_SRC_MASK;
> + }
> +
> + switch (hash_inset_orig & I40E_L3_V6_DST_MASK) {
> + case I40E_L3_V6_DST_MASK:
> + case I40E_L3_DST_MASK:
> + case 0:
> + break;
> + default:
> + hash_inset_update &= ~I40E_L3_V6_DST_MASK;
> + hash_inset_update |= I40E_L3_DST_MASK;
> + }
> +
> + if (hash_inset_update != hash_inset_orig) {
> + dev_warn(&pf->pdev->dev,
> + "flow type: %d update input mask
> from:0x%016llx, to:0x%016llx\n",
> + ret,
> + hash_inset_orig, hash_inset_update);
> + i40e_write_rx_ctl(hw, I40E_GLQF_HASH_INSET(0, ret),
> + (u32)hash_inset_update);
> + hash_inset_update >>= 32;
> + i40e_write_rx_ctl(hw, I40E_GLQF_HASH_INSET(1, ret),
> + (u32)hash_inset_update);
> + }
> + }
> +
> i40e_write_rx_ctl(hw, I40E_PFQF_HENA(0), (u32)hena);
> i40e_write_rx_ctl(hw, I40E_PFQF_HENA(1), (u32)(hena >> 32));
> i40e: Debug hash inputs
>
> From: Alexander Duyck <[email protected]>
>
>
> ---
> drivers/net/ethernet/intel/i40e/i40e_main.c | 52 +++++++++++++++++++++++++++
> 1 file changed, 52 insertions(+)
>
> diff --git a/drivers/net/ethernet/intel/i40e/i40e_main.c b/drivers/net/ethernet/intel/i40e/i40e_main.c
> index 65c2b9d2652b..c0a7f66babd9 100644
> --- a/drivers/net/ethernet/intel/i40e/i40e_main.c
> +++ b/drivers/net/ethernet/intel/i40e/i40e_main.c
> @@ -10998,6 +10998,58 @@ static int i40e_pf_config_rss(struct i40e_pf *pf)
> ((u64)i40e_read_rx_ctl(hw, I40E_PFQF_HENA(1)) << 32);
> hena |= i40e_pf_get_default_rss_hena(pf);
>
> + for (ret = 64; ret--;) {
> + u64 hash_inset_orig, hash_inset_update;
> +
> + if (!(hena & (1ull << ret)))
> + continue;
> +
> + /* Read initial input set value for flow type */
> + hash_inset_orig = i40e_read_rx_ctl(hw, I40E_GLQF_HASH_INSET(1, ret));
> + hash_inset_orig <<= 32;
> + hash_inset_orig |= i40e_read_rx_ctl(hw, I40E_GLQF_HASH_INSET(0, ret));
> +
> + /* Copy value so we can compare later */
> + hash_inset_update = hash_inset_orig;
> +
> + /* We should be looking at either the entire IPv6 or IPv4
> + * mask being set. If only part of the IPv6 mask is set, but
> + * the IPv4 mask is not then we have a garbage mask value
> + * and need to reset it.
> + */
> + switch (hash_inset_orig & I40E_L3_V6_SRC_MASK) {
> + case I40E_L3_V6_SRC_MASK:
> + case I40E_L3_SRC_MASK:
> + case 0:
> + break;
> + default:
> + hash_inset_update &= ~I40E_L3_V6_SRC_MASK;
> + hash_inset_update |= I40E_L3_SRC_MASK;
> + }
> +
> + switch (hash_inset_orig & I40E_L3_V6_DST_MASK) {
> + case I40E_L3_V6_DST_MASK:
> + case I40E_L3_DST_MASK:
> + case 0:
> + break;
> + default:
> + hash_inset_update &= ~I40E_L3_V6_DST_MASK;
> + hash_inset_update |= I40E_L3_DST_MASK;
> + }
> +
> + if (hash_inset_update != hash_inset_orig) {
> + dev_warn(&pf->pdev->dev,
> + "flow type: %d update input mask from:0x%016llx, to:0x%016llx\n",
> + ret,
> + hash_inset_orig, hash_inset_update);
> + i40e_write_rx_ctl(hw, I40E_GLQF_HASH_INSET(0, ret),
> + (u32)hash_inset_update);
> + hash_inset_update >>= 32;
> + i40e_write_rx_ctl(hw, I40E_GLQF_HASH_INSET(1, ret),
> + (u32)hash_inset_update);
> + }
> + }
> +
> i40e_write_rx_ctl(hw, I40E_PFQF_HENA(0), (u32)hena);
> i40e_write_rx_ctl(hw, I40E_PFQF_HENA(1), (u32)(hena >> 32));
>
OK I applied that and see this:
i40e: Intel(R) Ethernet Connection XL710 Network Driver - version 2.1.7-k
i40e: Copyright (c) 2013 - 2014 Intel Corporation.
i40e 0000:3d:00.0: fw 3.10.52896 api 1.6 nvm 4.00 0x80001577 1.1767.0
i40e 0000:3d:00.0: The driver for the device detected a newer version of the NVM image than expected. Please install the most recent version of the network driver.
i40e 0000:3d:00.0: MAC address: a4:bf:01:4e:0c:87
i40e 0000:3d:00.0: flow type: 36 update input mask from:0x0006060000000000, to:0x0001801800000000
i40e 0000:3d:00.0: flow type: 35 update input mask from:0x0006060000000000, to:0x0001801800000000
i40e 0000:3d:00.0: flow type: 34 update input mask from:0x0006060780000000, to:0x0001801f80000000
i40e 0000:3d:00.0: flow type: 33 update input mask from:0x0006060600000000, to:0x0001801e00000000
i40e 0000:3d:00.0: flow type: 32 update input mask from:0x0006060600000000, to:0x0001801e00000000
i40e 0000:3d:00.0: flow type: 31 update input mask from:0x0006060600000000, to:0x0001801e00000000
i40e 0000:3d:00.0: flow type: 30 update input mask from:0x0006060600000000, to:0x0001801e00000000
i40e 0000:3d:00.0: flow type: 29 update input mask from:0x0006060600000000, to:0x0001801e00000000
i40e 0000:3d:00.0: Features: PF-id[0] VSIs: 34 QP: 12 TXQ: 13 RSS VxLAN Geneve VEPA
i40e 0000:3d:00.1: fw 3.10.52896 api 1.6 nvm 4.00 0x80001577 1.1767.0
i40e 0000:3d:00.1: The driver for the device detected a newer version of the NVM image than expected. Please install the most recent version of the network driver.
i40e 0000:3d:00.1: MAC address: a4:bf:01:4e:0c:88
i40e 0000:3d:00.1: Features: PF-id[1] VSIs: 34 QP: 12 TXQ: 13 RSS VxLAN Geneve VEPA
i40e 0000:3d:00.1 eth2: NIC Link is Up, 1000 Mbps Full Duplex, Flow Control: None
Unfortunately (much to my disappointment, I hoped it would work) I see
no change in behaviour.
--
Len Sorensen
On Fri, May 17, 2019 at 10:23 AM Lennart Sorensen
<[email protected]> wrote:
> OK I applied that and see this:
>
> i40e: Intel(R) Ethernet Connection XL710 Network Driver - version 2.1.7-k
> i40e: Copyright (c) 2013 - 2014 Intel Corporation.
> i40e 0000:3d:00.0: fw 3.10.52896 api 1.6 nvm 4.00 0x80001577 1.1767.0
> i40e 0000:3d:00.0: The driver for the device detected a newer version of the NVM image than expected. Please install the most recent version of the network driver.
> i40e 0000:3d:00.0: MAC address: a4:bf:01:4e:0c:87
> i40e 0000:3d:00.0: flow type: 36 update input mask from:0x0006060000000000, to:0x0001801800000000
> i40e 0000:3d:00.0: flow type: 35 update input mask from:0x0006060000000000, to:0x0001801800000000
> i40e 0000:3d:00.0: flow type: 34 update input mask from:0x0006060780000000, to:0x0001801f80000000
> i40e 0000:3d:00.0: flow type: 33 update input mask from:0x0006060600000000, to:0x0001801e00000000
> i40e 0000:3d:00.0: flow type: 32 update input mask from:0x0006060600000000, to:0x0001801e00000000
> i40e 0000:3d:00.0: flow type: 31 update input mask from:0x0006060600000000, to:0x0001801e00000000
> i40e 0000:3d:00.0: flow type: 30 update input mask from:0x0006060600000000, to:0x0001801e00000000
> i40e 0000:3d:00.0: flow type: 29 update input mask from:0x0006060600000000, to:0x0001801e00000000
> i40e 0000:3d:00.0: Features: PF-id[0] VSIs: 34 QP: 12 TXQ: 13 RSS VxLAN Geneve VEPA
> i40e 0000:3d:00.1: fw 3.10.52896 api 1.6 nvm 4.00 0x80001577 1.1767.0
> i40e 0000:3d:00.1: The driver for the device detected a newer version of the NVM image than expected. Please install the most recent version of the network driver.
> i40e 0000:3d:00.1: MAC address: a4:bf:01:4e:0c:88
> i40e 0000:3d:00.1: Features: PF-id[1] VSIs: 34 QP: 12 TXQ: 13 RSS VxLAN Geneve VEPA
> i40e 0000:3d:00.1 eth2: NIC Link is Up, 1000 Mbps Full Duplex, Flow Control: None
>
> Unfortunately (much to my disappointment, I hoped it would work) I see
> no change in behaviour.
>
> --
> Len Sorensen
I was hoping it would work too. It seemed like it should have been the
answer since it definitely didn't seem right. Now it has me wondering
about some of the other code in the driver.
By any chance have you run anything like DPDK on any of the X722
interfaces on this system recently? I ask because it occurs to me that
if you had and it loaded something like a custom parsing profile it
could cause issues similar to this.
A debugging step you might try would be to revert back to my earlier
patch that only displayed the input mask instead of changing it. Once
you have done that you could look at doing a full power cycle on the
system by either physically disconnecting the power, or using the
power switch on the power supply itself if one is available. It is
necessary to disconnect the motherboard/NIC from power in order to
fully clear the global state stored in the device as it is retained
when the system is in standby.
What I want to verify is if the input mask that we have ran into is
the natural power-on input mask of if that is something that was
overridden by something else. The mask change I made should be reset
if the system loses power, and then it will either default back to the
value with the 6's if that is it's natural state, or it will match
what I had if it was not.
Other than that I really can't think up too much else. I suppose there
is the possibility of the NVM either setting up a DCB setting or
HREGION register causing an override that is limiting the queues to 1.
However, the likelihood of that should be really low.
- Alex
On Fri, May 17, 2019 at 03:20:02PM -0700, Alexander Duyck wrote:
> I was hoping it would work too. It seemed like it should have been the
> answer since it definitely didn't seem right. Now it has me wondering
> about some of the other code in the driver.
>
> By any chance have you run anything like DPDK on any of the X722
> interfaces on this system recently? I ask because it occurs to me that
> if you had and it loaded something like a custom parsing profile it
> could cause issues similar to this.
I have never used DPDK on anything. I was hoping never to do so. :)
This system has so far booted Debian (with a 4.19 kernel) and our own OS
(which has a 4.9 kernel).
> A debugging step you might try would be to revert back to my earlier
> patch that only displayed the input mask instead of changing it. Once
> you have done that you could look at doing a full power cycle on the
> system by either physically disconnecting the power, or using the
> power switch on the power supply itself if one is available. It is
> necessary to disconnect the motherboard/NIC from power in order to
> fully clear the global state stored in the device as it is retained
> when the system is in standby.
>
> What I want to verify is if the input mask that we have ran into is
> the natural power-on input mask of if that is something that was
> overridden by something else. The mask change I made should be reset
> if the system loses power, and then it will either default back to the
> value with the 6's if that is it's natural state, or it will match
> what I had if it was not.
>
> Other than that I really can't think up too much else. I suppose there
> is the possibility of the NVM either setting up a DCB setting or
> HREGION register causing an override that is limiting the queues to 1.
> However, the likelihood of that should be really low.
Here is the register dump after a full power off:
40e: Intel(R) Ethernet Connection XL710 Network Driver - version 2.1.7-k
i40e: Copyright (c) 2013 - 2014 Intel Corporation.
i40e 0000:3d:00.0: fw 3.10.52896 api 1.6 nvm 4.00 0x80001577 1.1767.0
i40e 0000:3d:00.0: The driver for the device detected a newer version of the NVM image than expected. Please install the most recent version of the network driver.
i40e 0000:3d:00.0: MAC address: a4:bf:01:4e:0c:87
i40e 0000:3d:00.0: flow_type: 63 input_mask:0x0000000000004000
i40e 0000:3d:00.0: flow_type: 46 input_mask:0x0007fff800000000
i40e 0000:3d:00.0: flow_type: 45 input_mask:0x0007fff800000000
i40e 0000:3d:00.0: flow_type: 44 input_mask:0x0007ffff80000000
i40e 0000:3d:00.0: flow_type: 43 input_mask:0x0007fffe00000000
i40e 0000:3d:00.0: flow_type: 42 input_mask:0x0007fffe00000000
i40e 0000:3d:00.0: flow_type: 41 input_mask:0x0007fffe00000000
i40e 0000:3d:00.0: flow_type: 40 input_mask:0x0007fffe00000000
i40e 0000:3d:00.0: flow_type: 39 input_mask:0x0007fffe00000000
i40e 0000:3d:00.0: flow_type: 36 input_mask:0x0006060000000000
i40e 0000:3d:00.0: flow_type: 35 input_mask:0x0006060000000000
i40e 0000:3d:00.0: flow_type: 34 input_mask:0x0006060780000000
i40e 0000:3d:00.0: flow_type: 33 input_mask:0x0006060600000000
i40e 0000:3d:00.0: flow_type: 32 input_mask:0x0006060600000000
i40e 0000:3d:00.0: flow_type: 31 input_mask:0x0006060600000000
i40e 0000:3d:00.0: flow_type: 30 input_mask:0x0006060600000000
i40e 0000:3d:00.0: flow_type: 29 input_mask:0x0006060600000000
i40e 0000:3d:00.0: Features: PF-id[0] VSIs: 34 QP: 12 TXQ: 13 RSS VxLAN Geneve VEPA
i40e 0000:3d:00.1: fw 3.10.52896 api 1.6 nvm 4.00 0x80001577 1.1767.0
i40e 0000:3d:00.1: The driver for the device detected a newer version of the NVM image than expected. Please install the most recent version of the network driver.
i40e 0000:3d:00.1: MAC address: a4:bf:01:4e:0c:88
i40e 0000:3d:00.1: flow_type: 63 input_mask:0x0000000000004000
i40e 0000:3d:00.1: flow_type: 46 input_mask:0x0007fff800000000
i40e 0000:3d:00.1: flow_type: 45 input_mask:0x0007fff800000000
i40e 0000:3d:00.1: flow_type: 44 input_mask:0x0007ffff80000000
i40e 0000:3d:00.1: flow_type: 43 input_mask:0x0007fffe00000000
i40e 0000:3d:00.1: flow_type: 42 input_mask:0x0007fffe00000000
i40e 0000:3d:00.1: flow_type: 41 input_mask:0x0007fffe00000000
i40e 0000:3d:00.1: flow_type: 40 input_mask:0x0007fffe00000000
i40e 0000:3d:00.1: flow_type: 39 input_mask:0x0007fffe00000000
i40e 0000:3d:00.1: flow_type: 36 input_mask:0x0006060000000000
i40e 0000:3d:00.1: flow_type: 35 input_mask:0x0006060000000000
i40e 0000:3d:00.1: flow_type: 34 input_mask:0x0006060780000000
i40e 0000:3d:00.1: flow_type: 33 input_mask:0x0006060600000000
i40e 0000:3d:00.1: flow_type: 32 input_mask:0x0006060600000000
i40e 0000:3d:00.1: flow_type: 31 input_mask:0x0006060600000000
i40e 0000:3d:00.1: flow_type: 30 input_mask:0x0006060600000000
i40e 0000:3d:00.1: flow_type: 29 input_mask:0x0006060600000000
i40e 0000:3d:00.1: Features: PF-id[1] VSIs: 34 QP: 12 TXQ: 13 RSS VxLAN Geneve VEPA
i40e 0000:3d:00.1 eth2: NIC Link is Up, 1000 Mbps Full Duplex, Flow Control: None
Pretty sure that is identical to before.
If I dump the registers after doing the update I see this (just did a
reboot this time, not a power cycle):
i40e: Intel(R) Ethernet Connection XL710 Network Driver - version 2.1.7-k
i40e: Copyright (c) 2013 - 2014 Intel Corporation.
i40e 0000:3d:00.0: fw 3.10.52896 api 1.6 nvm 4.00 0x80001577 1.1767.0
i40e 0000:3d:00.0: The driver for the device detected a newer version of the NVM image than expected. Please install the most recent version of the network driver.
i40e 0000:3d:00.0: MAC address: a4:bf:01:4e:0c:87
i40e 0000:3d:00.0: flow_type: 63 input_mask:0x0000000000004000
i40e 0000:3d:00.0: flow_type: 46 input_mask:0x0007fff800000000
i40e 0000:3d:00.0: flow_type: 45 input_mask:0x0007fff800000000
i40e 0000:3d:00.0: flow_type: 44 input_mask:0x0007ffff80000000
i40e 0000:3d:00.0: flow_type: 43 input_mask:0x0007fffe00000000
i40e 0000:3d:00.0: flow_type: 42 input_mask:0x0007fffe00000000
i40e 0000:3d:00.0: flow_type: 41 input_mask:0x0007fffe00000000
i40e 0000:3d:00.0: flow_type: 40 input_mask:0x0007fffe00000000
i40e 0000:3d:00.0: flow_type: 39 input_mask:0x0007fffe00000000
i40e 0000:3d:00.0: flow_type: 36 input_mask:0x0006060000000000
i40e 0000:3d:00.0: flow_type: 35 input_mask:0x0006060000000000
i40e 0000:3d:00.0: flow_type: 34 input_mask:0x0006060780000000
i40e 0000:3d:00.0: flow_type: 33 input_mask:0x0006060600000000
i40e 0000:3d:00.0: flow_type: 32 input_mask:0x0006060600000000
i40e 0000:3d:00.0: flow_type: 31 input_mask:0x0006060600000000
i40e 0000:3d:00.0: flow_type: 30 input_mask:0x0006060600000000
i40e 0000:3d:00.0: flow_type: 29 input_mask:0x0006060600000000
i40e 0000:3d:00.0: flow type: 36 update input mask from:0x0006060000000000, to:0x0001801800000000
i40e 0000:3d:00.0: flow type: 35 update input mask from:0x0006060000000000, to:0x0001801800000000
i40e 0000:3d:00.0: flow type: 34 update input mask from:0x0006060780000000, to:0x0001801f80000000
i40e 0000:3d:00.0: flow type: 33 update input mask from:0x0006060600000000, to:0x0001801e00000000
i40e 0000:3d:00.0: flow type: 32 update input mask from:0x0006060600000000, to:0x0001801e00000000
i40e 0000:3d:00.0: flow type: 31 update input mask from:0x0006060600000000, to:0x0001801e00000000
i40e 0000:3d:00.0: flow type: 30 update input mask from:0x0006060600000000, to:0x0001801e00000000
i40e 0000:3d:00.0: flow type: 29 update input mask from:0x0006060600000000, to:0x0001801e00000000
i40e 0000:3d:00.0: flow_type after update: 63 input_mask:0x0000000000004000
i40e 0000:3d:00.0: flow_type after update: 46 input_mask:0x0007fff800000000
i40e 0000:3d:00.0: flow_type after update: 45 input_mask:0x0007fff800000000
i40e 0000:3d:00.0: flow_type after update: 44 input_mask:0x0007ffff80000000
i40e 0000:3d:00.0: flow_type after update: 43 input_mask:0x0007fffe00000000
i40e 0000:3d:00.0: flow_type after update: 42 input_mask:0x0007fffe00000000
i40e 0000:3d:00.0: flow_type after update: 41 input_mask:0x0007fffe00000000
i40e 0000:3d:00.0: flow_type after update: 40 input_mask:0x0007fffe00000000
i40e 0000:3d:00.0: flow_type after update: 39 input_mask:0x0007fffe00000000
i40e 0000:3d:00.0: flow_type after update: 36 input_mask:0x0001801800000000
i40e 0000:3d:00.0: flow_type after update: 35 input_mask:0x0001801800000000
i40e 0000:3d:00.0: flow_type after update: 34 input_mask:0x0001801f80000000
i40e 0000:3d:00.0: flow_type after update: 33 input_mask:0x0001801e00000000
i40e 0000:3d:00.0: flow_type after update: 32 input_mask:0x0001801e00000000
i40e 0000:3d:00.0: flow_type after update: 31 input_mask:0x0001801e00000000
i40e 0000:3d:00.0: flow_type after update: 30 input_mask:0x0001801e00000000
i40e 0000:3d:00.0: flow_type after update: 29 input_mask:0x0001801e00000000
i40e 0000:3d:00.0: Features: PF-id[0] VSIs: 34 QP: 12 TXQ: 13 RSS VxLAN Geneve VEPA
So at least it appears the update did apply.
--
Len Sorensen
On Tue, May 21, 2019 at 8:15 AM Lennart Sorensen
<[email protected]> wrote:
>
> On Fri, May 17, 2019 at 03:20:02PM -0700, Alexander Duyck wrote:
> > I was hoping it would work too. It seemed like it should have been the
> > answer since it definitely didn't seem right. Now it has me wondering
> > about some of the other code in the driver.
> >
> > By any chance have you run anything like DPDK on any of the X722
> > interfaces on this system recently? I ask because it occurs to me that
> > if you had and it loaded something like a custom parsing profile it
> > could cause issues similar to this.
>
> I have never used DPDK on anything. I was hoping never to do so. :)
>
> This system has so far booted Debian (with a 4.19 kernel) and our own OS
> (which has a 4.9 kernel).
>
> > A debugging step you might try would be to revert back to my earlier
> > patch that only displayed the input mask instead of changing it. Once
> > you have done that you could look at doing a full power cycle on the
> > system by either physically disconnecting the power, or using the
> > power switch on the power supply itself if one is available. It is
> > necessary to disconnect the motherboard/NIC from power in order to
> > fully clear the global state stored in the device as it is retained
> > when the system is in standby.
> >
> > What I want to verify is if the input mask that we have ran into is
> > the natural power-on input mask of if that is something that was
> > overridden by something else. The mask change I made should be reset
> > if the system loses power, and then it will either default back to the
> > value with the 6's if that is it's natural state, or it will match
> > what I had if it was not.
> >
> > Other than that I really can't think up too much else. I suppose there
> > is the possibility of the NVM either setting up a DCB setting or
> > HREGION register causing an override that is limiting the queues to 1.
> > However, the likelihood of that should be really low.
>
> Here is the register dump after a full power off:
>
> 40e: Intel(R) Ethernet Connection XL710 Network Driver - version 2.1.7-k
> i40e: Copyright (c) 2013 - 2014 Intel Corporation.
> i40e 0000:3d:00.0: fw 3.10.52896 api 1.6 nvm 4.00 0x80001577 1.1767.0
> i40e 0000:3d:00.0: The driver for the device detected a newer version of the NVM image than expected. Please install the most recent version of the network driver.
> i40e 0000:3d:00.0: MAC address: a4:bf:01:4e:0c:87
> i40e 0000:3d:00.0: flow_type: 63 input_mask:0x0000000000004000
> i40e 0000:3d:00.0: flow_type: 46 input_mask:0x0007fff800000000
> i40e 0000:3d:00.0: flow_type: 45 input_mask:0x0007fff800000000
> i40e 0000:3d:00.0: flow_type: 44 input_mask:0x0007ffff80000000
> i40e 0000:3d:00.0: flow_type: 43 input_mask:0x0007fffe00000000
> i40e 0000:3d:00.0: flow_type: 42 input_mask:0x0007fffe00000000
> i40e 0000:3d:00.0: flow_type: 41 input_mask:0x0007fffe00000000
> i40e 0000:3d:00.0: flow_type: 40 input_mask:0x0007fffe00000000
> i40e 0000:3d:00.0: flow_type: 39 input_mask:0x0007fffe00000000
> i40e 0000:3d:00.0: flow_type: 36 input_mask:0x0006060000000000
> i40e 0000:3d:00.0: flow_type: 35 input_mask:0x0006060000000000
> i40e 0000:3d:00.0: flow_type: 34 input_mask:0x0006060780000000
> i40e 0000:3d:00.0: flow_type: 33 input_mask:0x0006060600000000
> i40e 0000:3d:00.0: flow_type: 32 input_mask:0x0006060600000000
> i40e 0000:3d:00.0: flow_type: 31 input_mask:0x0006060600000000
> i40e 0000:3d:00.0: flow_type: 30 input_mask:0x0006060600000000
> i40e 0000:3d:00.0: flow_type: 29 input_mask:0x0006060600000000
> i40e 0000:3d:00.0: Features: PF-id[0] VSIs: 34 QP: 12 TXQ: 13 RSS VxLAN Geneve VEPA
> i40e 0000:3d:00.1: fw 3.10.52896 api 1.6 nvm 4.00 0x80001577 1.1767.0
> i40e 0000:3d:00.1: The driver for the device detected a newer version of the NVM image than expected. Please install the most recent version of the network driver.
> i40e 0000:3d:00.1: MAC address: a4:bf:01:4e:0c:88
> i40e 0000:3d:00.1: flow_type: 63 input_mask:0x0000000000004000
> i40e 0000:3d:00.1: flow_type: 46 input_mask:0x0007fff800000000
> i40e 0000:3d:00.1: flow_type: 45 input_mask:0x0007fff800000000
> i40e 0000:3d:00.1: flow_type: 44 input_mask:0x0007ffff80000000
> i40e 0000:3d:00.1: flow_type: 43 input_mask:0x0007fffe00000000
> i40e 0000:3d:00.1: flow_type: 42 input_mask:0x0007fffe00000000
> i40e 0000:3d:00.1: flow_type: 41 input_mask:0x0007fffe00000000
> i40e 0000:3d:00.1: flow_type: 40 input_mask:0x0007fffe00000000
> i40e 0000:3d:00.1: flow_type: 39 input_mask:0x0007fffe00000000
> i40e 0000:3d:00.1: flow_type: 36 input_mask:0x0006060000000000
> i40e 0000:3d:00.1: flow_type: 35 input_mask:0x0006060000000000
> i40e 0000:3d:00.1: flow_type: 34 input_mask:0x0006060780000000
> i40e 0000:3d:00.1: flow_type: 33 input_mask:0x0006060600000000
> i40e 0000:3d:00.1: flow_type: 32 input_mask:0x0006060600000000
> i40e 0000:3d:00.1: flow_type: 31 input_mask:0x0006060600000000
> i40e 0000:3d:00.1: flow_type: 30 input_mask:0x0006060600000000
> i40e 0000:3d:00.1: flow_type: 29 input_mask:0x0006060600000000
> i40e 0000:3d:00.1: Features: PF-id[1] VSIs: 34 QP: 12 TXQ: 13 RSS VxLAN Geneve VEPA
> i40e 0000:3d:00.1 eth2: NIC Link is Up, 1000 Mbps Full Duplex, Flow Control: None
>
> Pretty sure that is identical to before.
>
> If I dump the registers after doing the update I see this (just did a
> reboot this time, not a power cycle):
>
> i40e: Intel(R) Ethernet Connection XL710 Network Driver - version 2.1.7-k
> i40e: Copyright (c) 2013 - 2014 Intel Corporation.
> i40e 0000:3d:00.0: fw 3.10.52896 api 1.6 nvm 4.00 0x80001577 1.1767.0
> i40e 0000:3d:00.0: The driver for the device detected a newer version of the NVM image than expected. Please install the most recent version of the network driver.
> i40e 0000:3d:00.0: MAC address: a4:bf:01:4e:0c:87
> i40e 0000:3d:00.0: flow_type: 63 input_mask:0x0000000000004000
> i40e 0000:3d:00.0: flow_type: 46 input_mask:0x0007fff800000000
> i40e 0000:3d:00.0: flow_type: 45 input_mask:0x0007fff800000000
> i40e 0000:3d:00.0: flow_type: 44 input_mask:0x0007ffff80000000
> i40e 0000:3d:00.0: flow_type: 43 input_mask:0x0007fffe00000000
> i40e 0000:3d:00.0: flow_type: 42 input_mask:0x0007fffe00000000
> i40e 0000:3d:00.0: flow_type: 41 input_mask:0x0007fffe00000000
> i40e 0000:3d:00.0: flow_type: 40 input_mask:0x0007fffe00000000
> i40e 0000:3d:00.0: flow_type: 39 input_mask:0x0007fffe00000000
> i40e 0000:3d:00.0: flow_type: 36 input_mask:0x0006060000000000
> i40e 0000:3d:00.0: flow_type: 35 input_mask:0x0006060000000000
> i40e 0000:3d:00.0: flow_type: 34 input_mask:0x0006060780000000
> i40e 0000:3d:00.0: flow_type: 33 input_mask:0x0006060600000000
> i40e 0000:3d:00.0: flow_type: 32 input_mask:0x0006060600000000
> i40e 0000:3d:00.0: flow_type: 31 input_mask:0x0006060600000000
> i40e 0000:3d:00.0: flow_type: 30 input_mask:0x0006060600000000
> i40e 0000:3d:00.0: flow_type: 29 input_mask:0x0006060600000000
> i40e 0000:3d:00.0: flow type: 36 update input mask from:0x0006060000000000, to:0x0001801800000000
> i40e 0000:3d:00.0: flow type: 35 update input mask from:0x0006060000000000, to:0x0001801800000000
> i40e 0000:3d:00.0: flow type: 34 update input mask from:0x0006060780000000, to:0x0001801f80000000
> i40e 0000:3d:00.0: flow type: 33 update input mask from:0x0006060600000000, to:0x0001801e00000000
> i40e 0000:3d:00.0: flow type: 32 update input mask from:0x0006060600000000, to:0x0001801e00000000
> i40e 0000:3d:00.0: flow type: 31 update input mask from:0x0006060600000000, to:0x0001801e00000000
> i40e 0000:3d:00.0: flow type: 30 update input mask from:0x0006060600000000, to:0x0001801e00000000
> i40e 0000:3d:00.0: flow type: 29 update input mask from:0x0006060600000000, to:0x0001801e00000000
> i40e 0000:3d:00.0: flow_type after update: 63 input_mask:0x0000000000004000
> i40e 0000:3d:00.0: flow_type after update: 46 input_mask:0x0007fff800000000
> i40e 0000:3d:00.0: flow_type after update: 45 input_mask:0x0007fff800000000
> i40e 0000:3d:00.0: flow_type after update: 44 input_mask:0x0007ffff80000000
> i40e 0000:3d:00.0: flow_type after update: 43 input_mask:0x0007fffe00000000
> i40e 0000:3d:00.0: flow_type after update: 42 input_mask:0x0007fffe00000000
> i40e 0000:3d:00.0: flow_type after update: 41 input_mask:0x0007fffe00000000
> i40e 0000:3d:00.0: flow_type after update: 40 input_mask:0x0007fffe00000000
> i40e 0000:3d:00.0: flow_type after update: 39 input_mask:0x0007fffe00000000
> i40e 0000:3d:00.0: flow_type after update: 36 input_mask:0x0001801800000000
> i40e 0000:3d:00.0: flow_type after update: 35 input_mask:0x0001801800000000
> i40e 0000:3d:00.0: flow_type after update: 34 input_mask:0x0001801f80000000
> i40e 0000:3d:00.0: flow_type after update: 33 input_mask:0x0001801e00000000
> i40e 0000:3d:00.0: flow_type after update: 32 input_mask:0x0001801e00000000
> i40e 0000:3d:00.0: flow_type after update: 31 input_mask:0x0001801e00000000
> i40e 0000:3d:00.0: flow_type after update: 30 input_mask:0x0001801e00000000
> i40e 0000:3d:00.0: flow_type after update: 29 input_mask:0x0001801e00000000
> i40e 0000:3d:00.0: Features: PF-id[0] VSIs: 34 QP: 12 TXQ: 13 RSS VxLAN Geneve VEPA
>
> So at least it appears the update did apply.
>
> --
> Len Sorensen
I think we need to narrow this down a bit more. Let's try forcing the
lookup table all to one value and see if traffic is still going to
queue 0.
Specifically what we need to is run the following command to try and
force all RSS traffic to queue 8, you can verify the result with
"ethtool -x":
ethtool -X <iface> weight 0 0 0 0 0 0 0 0 1
If that works and the IPSec traffic goes to queue 8 then we are likely
looking at some sort of input issue, either in the parsing or the
population of things like the input mask that we can then debug
further.
If traffic still goes to queue 0 then that tells us the output of the
RSS hash and lookup table are being ignored, this would imply either
some other filter is rerouting the traffic or is directing us to limit
the queue index to 0 bits.
Thanks.
- Alex
On Tue, May 21, 2019 at 09:51:33AM -0700, Alexander Duyck wrote:
> I think we need to narrow this down a bit more. Let's try forcing the
> lookup table all to one value and see if traffic is still going to
> queue 0.
>
> Specifically what we need to is run the following command to try and
> force all RSS traffic to queue 8, you can verify the result with
> "ethtool -x":
> ethtool -X <iface> weight 0 0 0 0 0 0 0 0 1
>
> If that works and the IPSec traffic goes to queue 8 then we are likely
> looking at some sort of input issue, either in the parsing or the
> population of things like the input mask that we can then debug
> further.
>
> If traffic still goes to queue 0 then that tells us the output of the
> RSS hash and lookup table are being ignored, this would imply either
> some other filter is rerouting the traffic or is directing us to limit
> the queue index to 0 bits.
# ethtool -x eth2
RX flow hash indirection table for eth2 with 12 RX ring(s):
0: 7 7 7 7 7 7 7 7
8: 7 7 7 7 7 7 7 7
16: 7 7 7 7 7 7 7 7
24: 7 7 7 7 7 7 7 7
32: 7 7 7 7 7 7 7 7
...
472: 7 7 7 7 7 7 7 7
480: 7 7 7 7 7 7 7 7
488: 7 7 7 7 7 7 7 7
496: 7 7 7 7 7 7 7 7
504: 7 7 7 7 7 7 7 7
RSS hash key:
0b:1f:ae:ed:60:04:7d:e5:8a:2b:43:3f:1d:ee:6c:99:89:29:94:b0:25:db:c7:4b:fa:da:4d:3f:e8:cc:bc:00:ad:32:01:d6:1c:30:3f:f8:79:3e:f4:48:04:1f:51:d2:5a:39:f0:90
root@ECA:~# ethtool --show-priv-flags eth2
Private flags for eth2:
MFP : off
LinkPolling : off
flow-director-atr: off
veb-stats : off
hw-atr-eviction : on
legacy-rx : off
All ipsec packets are still hitting queue 0.
Seems it is completely ignoring RSS for these packets. That is
impressively weird.
--
Len Sorensen
On Tue, May 21, 2019 at 10:55 AM Lennart Sorensen
<[email protected]> wrote:
>
> On Tue, May 21, 2019 at 09:51:33AM -0700, Alexander Duyck wrote:
> > I think we need to narrow this down a bit more. Let's try forcing the
> > lookup table all to one value and see if traffic is still going to
> > queue 0.
> >
> > Specifically what we need to is run the following command to try and
> > force all RSS traffic to queue 8, you can verify the result with
> > "ethtool -x":
> > ethtool -X <iface> weight 0 0 0 0 0 0 0 0 1
> >
> > If that works and the IPSec traffic goes to queue 8 then we are likely
> > looking at some sort of input issue, either in the parsing or the
> > population of things like the input mask that we can then debug
> > further.
> >
> > If traffic still goes to queue 0 then that tells us the output of the
> > RSS hash and lookup table are being ignored, this would imply either
> > some other filter is rerouting the traffic or is directing us to limit
> > the queue index to 0 bits.
>
> # ethtool -x eth2
> RX flow hash indirection table for eth2 with 12 RX ring(s):
> 0: 7 7 7 7 7 7 7 7
> 8: 7 7 7 7 7 7 7 7
> 16: 7 7 7 7 7 7 7 7
> 24: 7 7 7 7 7 7 7 7
> 32: 7 7 7 7 7 7 7 7
> ...
> 472: 7 7 7 7 7 7 7 7
> 480: 7 7 7 7 7 7 7 7
> 488: 7 7 7 7 7 7 7 7
> 496: 7 7 7 7 7 7 7 7
> 504: 7 7 7 7 7 7 7 7
> RSS hash key:
> 0b:1f:ae:ed:60:04:7d:e5:8a:2b:43:3f:1d:ee:6c:99:89:29:94:b0:25:db:c7:4b:fa:da:4d:3f:e8:cc:bc:00:ad:32:01:d6:1c:30:3f:f8:79:3e:f4:48:04:1f:51:d2:5a:39:f0:90
> root@ECA:~# ethtool --show-priv-flags eth2
> Private flags for eth2:
> MFP : off
> LinkPolling : off
> flow-director-atr: off
> veb-stats : off
> hw-atr-eviction : on
> legacy-rx : off
>
> All ipsec packets are still hitting queue 0.
>
> Seems it is completely ignoring RSS for these packets. That is
> impressively weird.
>
> --
> Len Sorensen
So we are either using 0 bits of the LUT or we are just not performing
hashing because this is somehow being parsed into a type that doesn't
support it.
I have attached 2 more patches we can test. The first enables hashing
on what are called "OAM" packets, The thing is we shouldn't be
identifying these packets as "OAM", Operations Administration &
Management, as normally it would have to be recognized as a tunnel
first and then have a specific flag set in either the GENEVE or
VXLAN-GPE header. The second patch will dump the contents of the
HREGION registers. They should all be 0, however I thought it best to
dump the contents and verify that since I know that these registers
can be used to change the traffic class of a given packet type and if
we are encountering that it might be moving it to an uninitialized TC
which would be using queue offset 0 with 0 bits of the LUT.
These last 2 patches would pretty much eliminate the entire RSS
subsystem. If we don't see HREGION values set and the OAM flags have
no effect I can only assume there is something going on with the
parser in the NIC since it isn't recognizing the packet type.
Thanks.
- Alex
On Tue, May 21, 2019 at 04:22:17PM -0700, Alexander Duyck wrote:
> So we are either using 0 bits of the LUT or we are just not performing
> hashing because this is somehow being parsed into a type that doesn't
> support it.
>
> I have attached 2 more patches we can test. The first enables hashing
> on what are called "OAM" packets, The thing is we shouldn't be
> identifying these packets as "OAM", Operations Administration &
> Management, as normally it would have to be recognized as a tunnel
> first and then have a specific flag set in either the GENEVE or
> VXLAN-GPE header. The second patch will dump the contents of the
> HREGION registers. They should all be 0, however I thought it best to
> dump the contents and verify that since I know that these registers
> can be used to change the traffic class of a given packet type and if
> we are encountering that it might be moving it to an uninitialized TC
> which would be using queue offset 0 with 0 bits of the LUT.
>
> These last 2 patches would pretty much eliminate the entire RSS
> subsystem. If we don't see HREGION values set and the OAM flags have
> no effect I can only assume there is something going on with the
> parser in the NIC since it isn't recognizing the packet type.
>
> Thanks.
>
> - Alex
OK I applied those two patches and get this:
i40e: Intel(R) Ethernet Connection XL710 Network Driver - version 2.1.7-k
i40e: Copyright (c) 2013 - 2014 Intel Corporation.
i40e 0000:3d:00.0: fw 3.10.52896 api 1.6 nvm 4.00 0x80001577 1.1767.0
i40e 0000:3d:00.0: The driver for the device detected a newer version of the NVM image than expected. Please install the most recent version of the network driver.
i40e 0000:3d:00.0: MAC address: a4:bf:01:4e:0c:87
i40e 0000:3d:00.0: PFQF_HREGION[7]: 0x00000000
i40e 0000:3d:00.0: PFQF_HREGION[6]: 0x00000000
i40e 0000:3d:00.0: PFQF_HREGION[5]: 0x00000000
i40e 0000:3d:00.0: PFQF_HREGION[4]: 0x00000000
i40e 0000:3d:00.0: PFQF_HREGION[3]: 0x00000000
i40e 0000:3d:00.0: PFQF_HREGION[2]: 0x00000000
i40e 0000:3d:00.0: PFQF_HREGION[1]: 0x00000000
i40e 0000:3d:00.0: PFQF_HREGION[0]: 0x00000000
i40e 0000:3d:00.0: flow_type: 63 input_mask:0x0000000000004000
i40e 0000:3d:00.0: flow_type: 46 input_mask:0x0007fff800000000
i40e 0000:3d:00.0: flow_type: 45 input_mask:0x0007fff800000000
i40e 0000:3d:00.0: flow_type: 44 input_mask:0x0007ffff80000000
i40e 0000:3d:00.0: flow_type: 43 input_mask:0x0007fffe00000000
i40e 0000:3d:00.0: flow_type: 42 input_mask:0x0007fffe00000000
i40e 0000:3d:00.0: flow_type: 41 input_mask:0x0007fffe00000000
i40e 0000:3d:00.0: flow_type: 40 input_mask:0x0007fffe00000000
i40e 0000:3d:00.0: flow_type: 39 input_mask:0x0007fffe00000000
i40e 0000:3d:00.0: flow_type: 36 input_mask:0x0006060000000000
i40e 0000:3d:00.0: flow_type: 35 input_mask:0x0006060000000000
i40e 0000:3d:00.0: flow_type: 34 input_mask:0x0006060780000000
i40e 0000:3d:00.0: flow_type: 33 input_mask:0x0006060600000000
i40e 0000:3d:00.0: flow_type: 32 input_mask:0x0006060600000000
i40e 0000:3d:00.0: flow_type: 31 input_mask:0x0006060600000000
i40e 0000:3d:00.0: flow_type: 30 input_mask:0x0006060600000000
i40e 0000:3d:00.0: flow_type: 29 input_mask:0x0006060600000000
i40e 0000:3d:00.0: flow_type: 27 input_mask:0x00000000002c0000
i40e 0000:3d:00.0: flow_type: 26 input_mask:0x00000000002c0000
i40e 0000:3d:00.0: flow type: 36 update input mask from:0x0006060000000000, to:0x0001801800000000
i40e 0000:3d:00.0: flow type: 35 update input mask from:0x0006060000000000, to:0x0001801800000000
i40e 0000:3d:00.0: flow type: 34 update input mask from:0x0006060780000000, to:0x0001801f80000000
i40e 0000:3d:00.0: flow type: 33 update input mask from:0x0006060600000000, to:0x0001801e00000000
i40e 0000:3d:00.0: flow type: 32 update input mask from:0x0006060600000000, to:0x0001801e00000000
i40e 0000:3d:00.0: flow type: 31 update input mask from:0x0006060600000000, to:0x0001801e00000000
i40e 0000:3d:00.0: flow type: 30 update input mask from:0x0006060600000000, to:0x0001801e00000000
i40e 0000:3d:00.0: flow type: 29 update input mask from:0x0006060600000000, to:0x0001801e00000000
So seems the regions are all 0.
All ipsec packets still hitting queue 0.
--
Len Sorensen
On Wed, May 22, 2019 at 10:39:56AM -0400, Lennart Sorensen wrote:
> OK I applied those two patches and get this:
>
> i40e: Intel(R) Ethernet Connection XL710 Network Driver - version 2.1.7-k
> i40e: Copyright (c) 2013 - 2014 Intel Corporation.
> i40e 0000:3d:00.0: fw 3.10.52896 api 1.6 nvm 4.00 0x80001577 1.1767.0
> i40e 0000:3d:00.0: The driver for the device detected a newer version of the NVM image than expected. Please install the most recent version of the network driver.
> i40e 0000:3d:00.0: MAC address: a4:bf:01:4e:0c:87
> i40e 0000:3d:00.0: PFQF_HREGION[7]: 0x00000000
> i40e 0000:3d:00.0: PFQF_HREGION[6]: 0x00000000
> i40e 0000:3d:00.0: PFQF_HREGION[5]: 0x00000000
> i40e 0000:3d:00.0: PFQF_HREGION[4]: 0x00000000
> i40e 0000:3d:00.0: PFQF_HREGION[3]: 0x00000000
> i40e 0000:3d:00.0: PFQF_HREGION[2]: 0x00000000
> i40e 0000:3d:00.0: PFQF_HREGION[1]: 0x00000000
> i40e 0000:3d:00.0: PFQF_HREGION[0]: 0x00000000
> i40e 0000:3d:00.0: flow_type: 63 input_mask:0x0000000000004000
> i40e 0000:3d:00.0: flow_type: 46 input_mask:0x0007fff800000000
> i40e 0000:3d:00.0: flow_type: 45 input_mask:0x0007fff800000000
> i40e 0000:3d:00.0: flow_type: 44 input_mask:0x0007ffff80000000
> i40e 0000:3d:00.0: flow_type: 43 input_mask:0x0007fffe00000000
> i40e 0000:3d:00.0: flow_type: 42 input_mask:0x0007fffe00000000
> i40e 0000:3d:00.0: flow_type: 41 input_mask:0x0007fffe00000000
> i40e 0000:3d:00.0: flow_type: 40 input_mask:0x0007fffe00000000
> i40e 0000:3d:00.0: flow_type: 39 input_mask:0x0007fffe00000000
> i40e 0000:3d:00.0: flow_type: 36 input_mask:0x0006060000000000
> i40e 0000:3d:00.0: flow_type: 35 input_mask:0x0006060000000000
> i40e 0000:3d:00.0: flow_type: 34 input_mask:0x0006060780000000
> i40e 0000:3d:00.0: flow_type: 33 input_mask:0x0006060600000000
> i40e 0000:3d:00.0: flow_type: 32 input_mask:0x0006060600000000
> i40e 0000:3d:00.0: flow_type: 31 input_mask:0x0006060600000000
> i40e 0000:3d:00.0: flow_type: 30 input_mask:0x0006060600000000
> i40e 0000:3d:00.0: flow_type: 29 input_mask:0x0006060600000000
> i40e 0000:3d:00.0: flow_type: 27 input_mask:0x00000000002c0000
> i40e 0000:3d:00.0: flow_type: 26 input_mask:0x00000000002c0000
> i40e 0000:3d:00.0: flow type: 36 update input mask from:0x0006060000000000, to:0x0001801800000000
> i40e 0000:3d:00.0: flow type: 35 update input mask from:0x0006060000000000, to:0x0001801800000000
> i40e 0000:3d:00.0: flow type: 34 update input mask from:0x0006060780000000, to:0x0001801f80000000
> i40e 0000:3d:00.0: flow type: 33 update input mask from:0x0006060600000000, to:0x0001801e00000000
> i40e 0000:3d:00.0: flow type: 32 update input mask from:0x0006060600000000, to:0x0001801e00000000
> i40e 0000:3d:00.0: flow type: 31 update input mask from:0x0006060600000000, to:0x0001801e00000000
> i40e 0000:3d:00.0: flow type: 30 update input mask from:0x0006060600000000, to:0x0001801e00000000
> i40e 0000:3d:00.0: flow type: 29 update input mask from:0x0006060600000000, to:0x0001801e00000000
>
> So seems the regions are all 0.
>
> All ipsec packets still hitting queue 0.
So any news or more ideas to try or are we stuck hoping someone can fix
the firmware?
--
Len Sorensen
On Fri, Jun 7, 2019 at 7:39 AM Lennart Sorensen
<[email protected]> wrote:
>
> On Wed, May 22, 2019 at 10:39:56AM -0400, Lennart Sorensen wrote:
> > OK I applied those two patches and get this:
> >
> > i40e: Intel(R) Ethernet Connection XL710 Network Driver - version 2.1.7-k
> > i40e: Copyright (c) 2013 - 2014 Intel Corporation.
> > i40e 0000:3d:00.0: fw 3.10.52896 api 1.6 nvm 4.00 0x80001577 1.1767.0
> > i40e 0000:3d:00.0: The driver for the device detected a newer version of the NVM image than expected. Please install the most recent version of the network driver.
> > i40e 0000:3d:00.0: MAC address: a4:bf:01:4e:0c:87
> > i40e 0000:3d:00.0: PFQF_HREGION[7]: 0x00000000
> > i40e 0000:3d:00.0: PFQF_HREGION[6]: 0x00000000
> > i40e 0000:3d:00.0: PFQF_HREGION[5]: 0x00000000
> > i40e 0000:3d:00.0: PFQF_HREGION[4]: 0x00000000
> > i40e 0000:3d:00.0: PFQF_HREGION[3]: 0x00000000
> > i40e 0000:3d:00.0: PFQF_HREGION[2]: 0x00000000
> > i40e 0000:3d:00.0: PFQF_HREGION[1]: 0x00000000
> > i40e 0000:3d:00.0: PFQF_HREGION[0]: 0x00000000
> > i40e 0000:3d:00.0: flow_type: 63 input_mask:0x0000000000004000
> > i40e 0000:3d:00.0: flow_type: 46 input_mask:0x0007fff800000000
> > i40e 0000:3d:00.0: flow_type: 45 input_mask:0x0007fff800000000
> > i40e 0000:3d:00.0: flow_type: 44 input_mask:0x0007ffff80000000
> > i40e 0000:3d:00.0: flow_type: 43 input_mask:0x0007fffe00000000
> > i40e 0000:3d:00.0: flow_type: 42 input_mask:0x0007fffe00000000
> > i40e 0000:3d:00.0: flow_type: 41 input_mask:0x0007fffe00000000
> > i40e 0000:3d:00.0: flow_type: 40 input_mask:0x0007fffe00000000
> > i40e 0000:3d:00.0: flow_type: 39 input_mask:0x0007fffe00000000
> > i40e 0000:3d:00.0: flow_type: 36 input_mask:0x0006060000000000
> > i40e 0000:3d:00.0: flow_type: 35 input_mask:0x0006060000000000
> > i40e 0000:3d:00.0: flow_type: 34 input_mask:0x0006060780000000
> > i40e 0000:3d:00.0: flow_type: 33 input_mask:0x0006060600000000
> > i40e 0000:3d:00.0: flow_type: 32 input_mask:0x0006060600000000
> > i40e 0000:3d:00.0: flow_type: 31 input_mask:0x0006060600000000
> > i40e 0000:3d:00.0: flow_type: 30 input_mask:0x0006060600000000
> > i40e 0000:3d:00.0: flow_type: 29 input_mask:0x0006060600000000
> > i40e 0000:3d:00.0: flow_type: 27 input_mask:0x00000000002c0000
> > i40e 0000:3d:00.0: flow_type: 26 input_mask:0x00000000002c0000
> > i40e 0000:3d:00.0: flow type: 36 update input mask from:0x0006060000000000, to:0x0001801800000000
> > i40e 0000:3d:00.0: flow type: 35 update input mask from:0x0006060000000000, to:0x0001801800000000
> > i40e 0000:3d:00.0: flow type: 34 update input mask from:0x0006060780000000, to:0x0001801f80000000
> > i40e 0000:3d:00.0: flow type: 33 update input mask from:0x0006060600000000, to:0x0001801e00000000
> > i40e 0000:3d:00.0: flow type: 32 update input mask from:0x0006060600000000, to:0x0001801e00000000
> > i40e 0000:3d:00.0: flow type: 31 update input mask from:0x0006060600000000, to:0x0001801e00000000
> > i40e 0000:3d:00.0: flow type: 30 update input mask from:0x0006060600000000, to:0x0001801e00000000
> > i40e 0000:3d:00.0: flow type: 29 update input mask from:0x0006060600000000, to:0x0001801e00000000
> >
> > So seems the regions are all 0.
> >
> > All ipsec packets still hitting queue 0.
>
> So any news or more ideas to try or are we stuck hoping someone can fix
> the firmware?
I had reached out to some folks over in the networking division hoping
that they can get a reproduction as I don't have the hardware that you
are seeing the issue on so I have no way to reproduce it.
Maybe someone from that group can reply and tell us where they are on that?
Thanks.
- Alex
On Fri, 7 Jun 2019, Alexander Duyck wrote:
> On Fri, Jun 7, 2019 at 7:39 AM Lennart Sorensen
> <[email protected]> wrote:
>>
>> On Wed, May 22, 2019 at 10:39:56AM -0400, Lennart Sorensen wrote:
>>> OK I applied those two patches and get this:
>>>
>>> i40e: Intel(R) Ethernet Connection XL710 Network Driver - version 2.1.7-k
>>> i40e: Copyright (c) 2013 - 2014 Intel Corporation.
>>> i40e 0000:3d:00.0: fw 3.10.52896 api 1.6 nvm 4.00 0x80001577 1.1767.0
>>> i40e 0000:3d:00.0: The driver for the device detected a newer version of the NVM image than expected. Please install the most recent version of the network driver.
>>> i40e 0000:3d:00.0: MAC address: a4:bf:01:4e:0c:87
>>> i40e 0000:3d:00.0: PFQF_HREGION[7]: 0x00000000
>>> i40e 0000:3d:00.0: PFQF_HREGION[6]: 0x00000000
>>> i40e 0000:3d:00.0: PFQF_HREGION[5]: 0x00000000
>>> i40e 0000:3d:00.0: PFQF_HREGION[4]: 0x00000000
>>> i40e 0000:3d:00.0: PFQF_HREGION[3]: 0x00000000
>>> i40e 0000:3d:00.0: PFQF_HREGION[2]: 0x00000000
>>> i40e 0000:3d:00.0: PFQF_HREGION[1]: 0x00000000
>>> i40e 0000:3d:00.0: PFQF_HREGION[0]: 0x00000000
>>> i40e 0000:3d:00.0: flow_type: 63 input_mask:0x0000000000004000
>>> i40e 0000:3d:00.0: flow_type: 46 input_mask:0x0007fff800000000
>>> i40e 0000:3d:00.0: flow_type: 45 input_mask:0x0007fff800000000
>>> i40e 0000:3d:00.0: flow_type: 44 input_mask:0x0007ffff80000000
>>> i40e 0000:3d:00.0: flow_type: 43 input_mask:0x0007fffe00000000
>>> i40e 0000:3d:00.0: flow_type: 42 input_mask:0x0007fffe00000000
>>> i40e 0000:3d:00.0: flow_type: 41 input_mask:0x0007fffe00000000
>>> i40e 0000:3d:00.0: flow_type: 40 input_mask:0x0007fffe00000000
>>> i40e 0000:3d:00.0: flow_type: 39 input_mask:0x0007fffe00000000
>>> i40e 0000:3d:00.0: flow_type: 36 input_mask:0x0006060000000000
>>> i40e 0000:3d:00.0: flow_type: 35 input_mask:0x0006060000000000
>>> i40e 0000:3d:00.0: flow_type: 34 input_mask:0x0006060780000000
>>> i40e 0000:3d:00.0: flow_type: 33 input_mask:0x0006060600000000
>>> i40e 0000:3d:00.0: flow_type: 32 input_mask:0x0006060600000000
>>> i40e 0000:3d:00.0: flow_type: 31 input_mask:0x0006060600000000
>>> i40e 0000:3d:00.0: flow_type: 30 input_mask:0x0006060600000000
>>> i40e 0000:3d:00.0: flow_type: 29 input_mask:0x0006060600000000
>>> i40e 0000:3d:00.0: flow_type: 27 input_mask:0x00000000002c0000
>>> i40e 0000:3d:00.0: flow_type: 26 input_mask:0x00000000002c0000
>>> i40e 0000:3d:00.0: flow type: 36 update input mask from:0x0006060000000000, to:0x0001801800000000
>>> i40e 0000:3d:00.0: flow type: 35 update input mask from:0x0006060000000000, to:0x0001801800000000
>>> i40e 0000:3d:00.0: flow type: 34 update input mask from:0x0006060780000000, to:0x0001801f80000000
>>> i40e 0000:3d:00.0: flow type: 33 update input mask from:0x0006060600000000, to:0x0001801e00000000
>>> i40e 0000:3d:00.0: flow type: 32 update input mask from:0x0006060600000000, to:0x0001801e00000000
>>> i40e 0000:3d:00.0: flow type: 31 update input mask from:0x0006060600000000, to:0x0001801e00000000
>>> i40e 0000:3d:00.0: flow type: 30 update input mask from:0x0006060600000000, to:0x0001801e00000000
>>> i40e 0000:3d:00.0: flow type: 29 update input mask from:0x0006060600000000, to:0x0001801e00000000
>>>
>>> So seems the regions are all 0.
>>>
>>> All ipsec packets still hitting queue 0.
>>
>> So any news or more ideas to try or are we stuck hoping someone can fix
>> the firmware?
>
> I had reached out to some folks over in the networking division hoping
> that they can get a reproduction as I don't have the hardware that you
> are seeing the issue on so I have no way to reproduce it.
>
> Maybe someone from that group can reply and tell us where they are on that?
>
> Thanks.
>
> - Alex
For some reason this isn't showing up in my work email. We had an
internal conference this week and I think people are away. I'll see if I
can chase some people down if they're still here and not on the way
home.
--
Hisashi T Fujinaka - [email protected]
Just a quick update with the response I got and I'll make sure this is in our internal bug database.
Here's what I got back, and it looks like you guys have tried this already:
Have they tried these steps to configure RSS:
RSS Hash Flow
-------------
Allows you to set the hash bytes per flow type and any combination of one or
more options for Receive Side Scaling (RSS) hash byte configuration.
#ethtool -N <dev> rx-flow-hash <type> <option>
Where <type> is:
tcp4 signifying TCP over IPv4
udp4 signifying UDP over IPv4
tcp6 signifying TCP over IPv6
udp6 signifying UDP over IPv6
And <option> is one or more of:
s Hash on the IP source address of the rx packet.
d Hash on the IP destination address of the rx packet.
f Hash on bytes 0 and 1 of the Layer 4 header of the rx packet.
n Hash on bytes 2 and 3 of the Layer 4 header of the rx packet.
Also, looks like the driver needs to be updated to latest version:
>>> 1.1767.0 i40e 0000:3d:00.0: The driver for the device detected a
>>> newer version of the NVM image than expected. Please install the
>>> most recent version of the network driver.
Out of tree: https://sourceforge.net/projects/e1000/files/i40e%20stable/
Todd Fujinaka
Software Application Engineer
Datacenter Engineering Group
Intel Corporation
[email protected]
-----Original Message-----
From: Hisashi T Fujinaka [mailto:[email protected]]
Sent: Friday, June 7, 2019 1:50 PM
To: Alexander Duyck <[email protected]>
Cc: [email protected]; Netdev <[email protected]>; intel-wired-lan <[email protected]>; LKML <[email protected]>; Lennart Sorensen <[email protected]>
Subject: Re: [E1000-devel] [Intel-wired-lan] i40e X722 RSS problem with NAT-Traversal IPsec packets
On Fri, 7 Jun 2019, Alexander Duyck wrote:
> On Fri, Jun 7, 2019 at 7:39 AM Lennart Sorensen
> <[email protected]> wrote:
>>
>> On Wed, May 22, 2019 at 10:39:56AM -0400, Lennart Sorensen wrote:
>>> OK I applied those two patches and get this:
>>>
>>> i40e: Intel(R) Ethernet Connection XL710 Network Driver - version
>>> 2.1.7-k
>>> i40e: Copyright (c) 2013 - 2014 Intel Corporation.
>>> i40e 0000:3d:00.0: fw 3.10.52896 api 1.6 nvm 4.00 0x80001577
>>> 1.1767.0 i40e 0000:3d:00.0: The driver for the device detected a newer version of the NVM image than expected. Please install the most recent version of the network driver.
>>> i40e 0000:3d:00.0: MAC address: a4:bf:01:4e:0c:87 i40e 0000:3d:00.0:
>>> PFQF_HREGION[7]: 0x00000000 i40e 0000:3d:00.0: PFQF_HREGION[6]:
>>> 0x00000000 i40e 0000:3d:00.0: PFQF_HREGION[5]: 0x00000000 i40e
>>> 0000:3d:00.0: PFQF_HREGION[4]: 0x00000000 i40e 0000:3d:00.0:
>>> PFQF_HREGION[3]: 0x00000000 i40e 0000:3d:00.0: PFQF_HREGION[2]:
>>> 0x00000000 i40e 0000:3d:00.0: PFQF_HREGION[1]: 0x00000000 i40e
>>> 0000:3d:00.0: PFQF_HREGION[0]: 0x00000000 i40e 0000:3d:00.0:
>>> flow_type: 63 input_mask:0x0000000000004000 i40e 0000:3d:00.0:
>>> flow_type: 46 input_mask:0x0007fff800000000 i40e 0000:3d:00.0:
>>> flow_type: 45 input_mask:0x0007fff800000000 i40e 0000:3d:00.0:
>>> flow_type: 44 input_mask:0x0007ffff80000000 i40e 0000:3d:00.0:
>>> flow_type: 43 input_mask:0x0007fffe00000000 i40e 0000:3d:00.0:
>>> flow_type: 42 input_mask:0x0007fffe00000000 i40e 0000:3d:00.0:
>>> flow_type: 41 input_mask:0x0007fffe00000000 i40e 0000:3d:00.0:
>>> flow_type: 40 input_mask:0x0007fffe00000000 i40e 0000:3d:00.0:
>>> flow_type: 39 input_mask:0x0007fffe00000000 i40e 0000:3d:00.0:
>>> flow_type: 36 input_mask:0x0006060000000000 i40e 0000:3d:00.0:
>>> flow_type: 35 input_mask:0x0006060000000000 i40e 0000:3d:00.0:
>>> flow_type: 34 input_mask:0x0006060780000000 i40e 0000:3d:00.0:
>>> flow_type: 33 input_mask:0x0006060600000000 i40e 0000:3d:00.0:
>>> flow_type: 32 input_mask:0x0006060600000000 i40e 0000:3d:00.0:
>>> flow_type: 31 input_mask:0x0006060600000000 i40e 0000:3d:00.0:
>>> flow_type: 30 input_mask:0x0006060600000000 i40e 0000:3d:00.0:
>>> flow_type: 29 input_mask:0x0006060600000000 i40e 0000:3d:00.0:
>>> flow_type: 27 input_mask:0x00000000002c0000 i40e 0000:3d:00.0:
>>> flow_type: 26 input_mask:0x00000000002c0000 i40e 0000:3d:00.0: flow
>>> type: 36 update input mask from:0x0006060000000000,
>>> to:0x0001801800000000 i40e 0000:3d:00.0: flow type: 35 update input
>>> mask from:0x0006060000000000, to:0x0001801800000000 i40e
>>> 0000:3d:00.0: flow type: 34 update input mask
>>> from:0x0006060780000000, to:0x0001801f80000000 i40e 0000:3d:00.0:
>>> flow type: 33 update input mask from:0x0006060600000000,
>>> to:0x0001801e00000000 i40e 0000:3d:00.0: flow type: 32 update input
>>> mask from:0x0006060600000000, to:0x0001801e00000000 i40e
>>> 0000:3d:00.0: flow type: 31 update input mask
>>> from:0x0006060600000000, to:0x0001801e00000000 i40e 0000:3d:00.0:
>>> flow type: 30 update input mask from:0x0006060600000000,
>>> to:0x0001801e00000000 i40e 0000:3d:00.0: flow type: 29 update input
>>> mask from:0x0006060600000000, to:0x0001801e00000000
>>>
>>> So seems the regions are all 0.
>>>
>>> All ipsec packets still hitting queue 0.
>>
>> So any news or more ideas to try or are we stuck hoping someone can
>> fix the firmware?
>
> I had reached out to some folks over in the networking division hoping
> that they can get a reproduction as I don't have the hardware that you
> are seeing the issue on so I have no way to reproduce it.
>
> Maybe someone from that group can reply and tell us where they are on that?
>
> Thanks.
>
> - Alex
For some reason this isn't showing up in my work email. We had an internal conference this week and I think people are away. I'll see if I can chase some people down if they're still here and not on the way home.
--
Hisashi T Fujinaka - [email protected]
_______________________________________________
E1000-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/e1000-devel
To learn more about Intel® Ethernet, visit http://communities.intel.com/community/wired
On Fri, Jun 07, 2019 at 10:08:31PM +0000, Fujinaka, Todd wrote:
> Just a quick update with the response I got and I'll make sure this is in our internal bug database.
>
> Here's what I got back, and it looks like you guys have tried this already:
>
> Have they tried these steps to configure RSS:
>
> RSS Hash Flow
> -------------
>
> Allows you to set the hash bytes per flow type and any combination of one or
> more options for Receive Side Scaling (RSS) hash byte configuration.
>
> #ethtool -N <dev> rx-flow-hash <type> <option>
>
> Where <type> is:
> tcp4 signifying TCP over IPv4
> udp4 signifying UDP over IPv4
> tcp6 signifying TCP over IPv6
> udp6 signifying UDP over IPv6
> And <option> is one or more of:
> s Hash on the IP source address of the rx packet.
> d Hash on the IP destination address of the rx packet.
> f Hash on bytes 0 and 1 of the Layer 4 header of the rx packet.
> n Hash on bytes 2 and 3 of the Layer 4 header of the rx packet.
With potentially 10000 ipsec connections, we don't even want to look at
creating manual flow entries. There isn't enough room for that. We just
wanted RSS to do its job the way it does on every other NIC in the past.
After years of using mostly intel NICs that just worked, this one has
been quite the surprise.
> Also, looks like the driver needs to be updated to latest version:
> >>> 1.1767.0 i40e 0000:3d:00.0: The driver for the device detected a
> >>> newer version of the NVM image than expected. Please install the
> >>> most recent version of the network driver.
>
> Out of tree: https://sourceforge.net/projects/e1000/files/i40e%20stable/
Already tried with 4.19 kernel which is essentially identical to the
latest out of tree driver (I diffed them and found no functional
differences at all) and it didn't help. Well it was essentially identical
to the latest out of tree a few weeks ago. It seems there is now a
newer one with some changes although nothing in the list of changes
sound relevant.
We do not want to use the out of tree driver and even trying it out is
a lot of work. We used to use it in the past for some NIC types but
stopped due to the hassle of maintaining the integration. If any problems
exist in the in kernel driver we will patch it, but so far that does not
appear to be the problem. The tests we did so far indicate the firmware
isn't applying an RSS value to certain packet types. Even mapping every
RSS value to queue 7 still saw these packets arrive on queue 0 which
should of course be impossible if the firmware was working. Now if
there is anything in the out of tree driver that you think can explain
this problem, I will look at it and consider trying it, but so far I
see nothing that makes that worth the effort. It just doesn't look like
a driver problem. If someone has access to a S2600WFT board (or some
other C612 based board) it should be simple enough to try replaying
the captured packet and see what RSS queue it hits (with ATR disabled
of course).
The message is because we tried installing an NVM update to see if it
fixed anything, and it did not. We could put the old version back,
but since neither version works I didn't bother yet.
--
Len Sorensen
On Fri, Jun 07, 2019 at 12:32:51PM -0700, Alexander Duyck wrote:
> I had reached out to some folks over in the networking division hoping
> that they can get a reproduction as I don't have the hardware that you
> are seeing the issue on so I have no way to reproduce it.
>
> Maybe someone from that group can reply and tell us where they are on that?
Well I still never heard anything from anyone. Just installed 4.10
firmware in case that security fix (the only change to happen in over
12 months) did something, but no.
So all UDP encapsulated IPsec packets still always have RSS value of 0.
I am tempted to write a test to see if all UDP encapsulated IP packets
that are not of one of the explicitly handled types have this problem
since I have a suspicion they do.
--
Len Sorensen