Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67;
MIME-Version: 1.0
References: <20190514163443.glfjva3ofqcy7lbg@csclub.uwaterloo.ca>
 <CAKgT0UdPDyCBsShQVwwE5C8fBKkMcfS6_S5m3T7JP-So9fzVgA@mail.gmail.com>
 <20190516183407.qswotwyjwtjqfdqm@csclub.uwaterloo.ca> <20190516183705.e4zflbli7oujlbek@csclub.uwaterloo.ca>
 <CAKgT0UfSa-dM2+7xntK9tB7Zw5N8nDd3U1n4OSK0gbWbkNSKJQ@mail.gmail.com>
 <CAKgT0Ucd0s_0F5_nwqXknRngwROyuecUt+4bYzWvp1-2cNSg7g@mail.gmail.com>
 <20190517172317.amopafirjfizlgej@csclub.uwaterloo.ca> <CAKgT0UdM28pSTCsaT=TWqmQwCO44NswS0PqFLAzgs9pmn41VeQ@mail.gmail.com>
 <20190521151537.xga4aiq3gjtiif4j@csclub.uwaterloo.ca> <CAKgT0UfpZ-ve3Hx26gDkb+YTDHvN3=MJ7NZd2NE7ewF5g=kHHw@mail.gmail.com>
 <20190521175456.zlkiiov5hry2l4q2@csclub.uwaterloo.ca>
In-Reply-To: <20190521175456.zlkiiov5hry2l4q2@csclub.uwaterloo.ca>
From:   Alexander Duyck <alexander.duyck@gmail.com>
Date:   Tue, 21 May 2019 16:22:17 -0700
Message-ID: <CAKgT0UcR3q1maBmJz7xj_i+_oux_6FQxua9DOjXQSZzyq6FhkQ@mail.gmail.com>
Subject: Re: [Intel-wired-lan] i40e X722 RSS problem with NAT-Traversal IPsec packets
To:     Lennart Sorensen <lsorense@csclub.uwaterloo.ca>
Cc:     Jeff Kirsher <jeffrey.t.kirsher@intel.com>,
        LKML <linux-kernel@vger.kernel.org>,
        Netdev <netdev@vger.kernel.org>,
        intel-wired-lan <intel-wired-lan@lists.osuosl.org>,
        e1000-devel@lists.sourceforge.net
Content-Type: multipart/mixed; boundary="00000000000059aa6005896e1fad"
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk

--00000000000059aa6005896e1fad
Content-Type: text/plain; charset="UTF-8"

On Tue, May 21, 2019 at 10:55 AM Lennart Sorensen
<lsorense@csclub.uwaterloo.ca> wrote:
>
> On Tue, May 21, 2019 at 09:51:33AM -0700, Alexander Duyck wrote:
> > I think we need to narrow this down a bit more. Let's try forcing the
> > lookup table all to one value and see if traffic is still going to
> > queue 0.
> >
> > Specifically what we need to is run the following command to try and
> > force all RSS traffic to queue 8, you can verify the result with
> > "ethtool -x":
> > ethtool -X <iface> weight 0 0 0 0 0 0 0 0 1
> >
> > If that works and the IPSec traffic goes to queue 8 then we are likely
> > looking at some sort of input issue, either in the parsing or the
> > population of things like the input mask that we can then debug
> > further.
> >
> > If traffic still goes to queue 0 then that tells us the output of the
> > RSS hash and lookup table are being ignored, this would imply either
> > some other filter is rerouting the traffic or is directing us to limit
> > the queue index to 0 bits.
>
> # ethtool -x eth2
> RX flow hash indirection table for eth2 with 12 RX ring(s):
>     0:      7     7     7     7     7     7     7     7
>     8:      7     7     7     7     7     7     7     7
>    16:      7     7     7     7     7     7     7     7
>    24:      7     7     7     7     7     7     7     7
>    32:      7     7     7     7     7     7     7     7
> ...
>   472:      7     7     7     7     7     7     7     7
>   480:      7     7     7     7     7     7     7     7
>   488:      7     7     7     7     7     7     7     7
>   496:      7     7     7     7     7     7     7     7
>   504:      7     7     7     7     7     7     7     7
> RSS hash key:
> 0b:1f:ae:ed:60:04:7d:e5:8a:2b:43:3f:1d:ee:6c:99:89:29:94:b0:25:db:c7:4b:fa:da:4d:3f:e8:cc:bc:00:ad:32:01:d6:1c:30:3f:f8:79:3e:f4:48:04:1f:51:d2:5a:39:f0:90
> root@ECA:~# ethtool --show-priv-flags eth2
> Private flags for eth2:
> MFP              : off
> LinkPolling      : off
> flow-director-atr: off
> veb-stats        : off
> hw-atr-eviction  : on
> legacy-rx        : off
>
> All ipsec packets are still hitting queue 0.
>
> Seems it is completely ignoring RSS for these packets.  That is
> impressively weird.
>
> --
> Len Sorensen

So we are either using 0 bits of the LUT or we are just not performing
hashing because this is somehow being parsed into a type that doesn't
support it.

I have attached 2 more patches we can test. The first enables hashing
on what are called "OAM" packets, The thing is we shouldn't be
identifying these packets as "OAM", Operations Administration &
Management, as normally it would have to be recognized as a tunnel
first and then have a specific flag set in either the GENEVE or
VXLAN-GPE header. The second patch will dump the contents of the
HREGION registers. They should all be 0, however I thought it best to
dump the contents and verify that since I know that these registers
can be used to change the traffic class of a given packet type and if
we are encountering that it might be moving it to an uninitialized TC
which would be using queue offset 0 with 0 bits of the LUT.

These last 2 patches would pretty much eliminate the entire RSS
subsystem. If we don't see HREGION values set and the OAM flags have
no effect I can only assume there is something going on with the
parser in the NIC since it isn't recognizing the packet type.

Thanks.

- Alex

--00000000000059aa6005896e1fad
Content-Type: text/x-patch; charset="US-ASCII"; name="i40e-enable-oam-flag-tunnel.patch"
Content-Disposition: attachment; 
	filename="i40e-enable-oam-flag-tunnel.patch"
Content-Transfer-Encoding: base64
Content-ID: <f_jvydbsln1>
X-Attachment-Id: f_jvydbsln1

aTQwZTogRW5hYmxlIE9BTSBmbGFnIHR1bm5lbCBoYXNoaW5nCgpGcm9tOiBBbGV4YW5kZXIgRHV5
Y2sgPGFsZXhhbmRlci5oLmR1eWNrQGxpbnV4LmludGVsLmNvbT4KCkFkZCBzdXBwb3J0IGZvciBo
YXNoaW5nIHBhY2tldCB0eXBlcyAyNiBhbmQgMjcgb24gWDcyMiBhZGFwdGVycy4gVGhlCmRlZmF1
bHQgaW5wdXQgc2V0IGlzIHN1cHBvc2VkIHRvIGJlIHNvdXJjZSBvdXRlciBVRFAgYW5kIFZOSS4g
Rm9yIG5vdyBhbGwgSQpjYXJlIGFib3V0IGlzIGVuYWJsaW5nIGhhc2hpbmcgb24gdGhpcyB0byBz
ZWUgaWYgd2UgY2FuIGdldCBFU1AgdHJhZmZpYyB0bwpub3QgaGl0IHF1ZXVlIDAgZm9yIGV2ZXJ5
dGhpbmcuCi0tLQogZHJpdmVycy9uZXQvZXRoZXJuZXQvaW50ZWwvaTQwZS9pNDBlX3R4cnguaCB8
ICAgIDMgKystCiAxIGZpbGUgY2hhbmdlZCwgMiBpbnNlcnRpb25zKCspLCAxIGRlbGV0aW9uKC0p
CgpkaWZmIC0tZ2l0IGEvZHJpdmVycy9uZXQvZXRoZXJuZXQvaW50ZWwvaTQwZS9pNDBlX3R4cngu
aCBiL2RyaXZlcnMvbmV0L2V0aGVybmV0L2ludGVsL2k0MGUvaTQwZV90eHJ4LmgKaW5kZXggMTAw
ZTkyZDI5ODJmLi5hZDNlMTZlOGNkN2EgMTAwNjQ0Ci0tLSBhL2RyaXZlcnMvbmV0L2V0aGVybmV0
L2ludGVsL2k0MGUvaTQwZV90eHJ4LmgKKysrIGIvZHJpdmVycy9uZXQvZXRoZXJuZXQvaW50ZWwv
aTQwZS9pNDBlX3R4cnguaApAQCAtOTUsNyArOTUsOCBAQCBlbnVtIGk0MGVfZHluX2lkeF90IHsK
IAlCSVRfVUxMKEk0MEVfRklMVEVSX1BDVFlQRV9OT05GX01VTFRJQ0FTVF9JUFY0X1VEUCkgfCBc
CiAJQklUX1VMTChJNDBFX0ZJTFRFUl9QQ1RZUEVfTk9ORl9JUFY2X1RDUF9TWU5fTk9fQUNLKSB8
IFwKIAlCSVRfVUxMKEk0MEVfRklMVEVSX1BDVFlQRV9OT05GX1VOSUNBU1RfSVBWNl9VRFApIHwg
XAotCUJJVF9VTEwoSTQwRV9GSUxURVJfUENUWVBFX05PTkZfTVVMVElDQVNUX0lQVjZfVURQKSkK
KwlCSVRfVUxMKEk0MEVfRklMVEVSX1BDVFlQRV9OT05GX01VTFRJQ0FTVF9JUFY2X1VEUCkgfCBc
CisJQklUX1VMTCgyNikgfCBCSVRfVUxMKDI3KSkgLyogQWRkZWQgYml0cyBmb3IgdHVubmVsIE9B
TSAqLwogCiAjZGVmaW5lIGk0MGVfcGZfZ2V0X2RlZmF1bHRfcnNzX2hlbmEocGYpIFwKIAkoKChw
ZiktPmh3X2ZlYXR1cmVzICYgSTQwRV9IV19NVUxUSVBMRV9UQ1BfVURQX1JTU19QQ1RZUEUpID8g
XAo=
--00000000000059aa6005896e1fad
Content-Type: text/x-patch; charset="US-ASCII"; name="i40e-dump-extra-hregion.patch"
Content-Disposition: attachment; filename="i40e-dump-extra-hregion.patch"
Content-Transfer-Encoding: base64
Content-ID: <f_jvydbslb0>
X-Attachment-Id: f_jvydbslb0

aTQwZTogRHVtcCBIUkVHSU9OIGVudHJpZXMKCkZyb206IEFsZXhhbmRlciBEdXljayA8YWxleGFu
ZGVyLmguZHV5Y2tAbGludXguaW50ZWwuY29tPgoKCi0tLQogZHJpdmVycy9uZXQvZXRoZXJuZXQv
aW50ZWwvaTQwZS9pNDBlX21haW4uYyB8ICAgIDggKysrKysrKysKIDEgZmlsZSBjaGFuZ2VkLCA4
IGluc2VydGlvbnMoKykKCmRpZmYgLS1naXQgYS9kcml2ZXJzL25ldC9ldGhlcm5ldC9pbnRlbC9p
NDBlL2k0MGVfbWFpbi5jIGIvZHJpdmVycy9uZXQvZXRoZXJuZXQvaW50ZWwvaTQwZS9pNDBlX21h
aW4uYwppbmRleCAzMjA1NjJiMzk2ODYuLjM3MGY2NmRmNGU0ZiAxMDA2NDQKLS0tIGEvZHJpdmVy
cy9uZXQvZXRoZXJuZXQvaW50ZWwvaTQwZS9pNDBlX21haW4uYworKysgYi9kcml2ZXJzL25ldC9l
dGhlcm5ldC9pbnRlbC9pNDBlL2k0MGVfbWFpbi5jCkBAIC0xMTA5NCw2ICsxMTA5NCwxNCBAQCBz
dGF0aWMgaW50IGk0MGVfcGZfY29uZmlnX3JzcyhzdHJ1Y3QgaTQwZV9wZiAqcGYpCiAJdTY0IGhl
bmE7CiAJaW50IHJldDsKIAorCS8qIFRoZXNlIHNob3VsZCBhbGwgYmUgMCwgZHVtcCB0aGVtIHRv
IHZlcmlmeSB0aGV5IGFyZSAqLworCWZvciAocmV0ID0gODsgcmV0LS07KSB7CisJCXJlZ192YWwg
PSBpNDBlX3JlYWRfcnhfY3RsKGh3LCBJNDBFX1BGUUZfSFJFR0lPTihyZXQpKTsKKworCQlkZXZf
aW5mbygmcGYtPnBkZXYtPmRldiwKKwkJCSAiUEZRRl9IUkVHSU9OWyVkXTogMHglMDh4XG4iLCBy
ZXQsIHJlZ192YWwpOworCX0KKwogCS8qIEJ5IGRlZmF1bHQgd2UgZW5hYmxlIFRDUC9VRFAgd2l0
aCBJUHY0L0lQdjYgcHR5cGVzICovCiAJaGVuYSA9ICh1NjQpaTQwZV9yZWFkX3J4X2N0bChodywg
STQwRV9QRlFGX0hFTkEoMCkpIHwKIAkJKCh1NjQpaTQwZV9yZWFkX3J4X2N0bChodywgSTQwRV9Q
RlFGX0hFTkEoMSkpIDw8IDMyKTsK
--00000000000059aa6005896e1fad--