Received-SPF: pass (google.com: domain of linux-kernel+bounces-18402-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.199.223 as permitted sender) client-ip=147.75.199.223;
From: Ale Crismani <ale.crismani@automattic.com>
Content-Type: text/plain;
	charset=us-ascii
Content-Transfer-Encoding: quoted-printable
Precedence: bulk
Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.700.6\))
Subject: Performance regression in ip_set_swap on 6.1.69
Message-Id: <C0829B10-EAA6-4809-874E-E1E9C05A8D84@automattic.com>
Date: Fri, 5 Jan 2024 23:45:27 +0100
To: linux-kernel@vger.kernel.org

Dear all,

When upgrading some of our Debian hosts that compose a Kubernetes =
cluster we found a regression in ip_set_swap on 6.1.69. Calls to =
ip_set_swap now take roughly 15ms, while they used to take just tens of =
microseconds before.

The issue is very visible for use, since we use kube-router as our =
Kubernetes networking interface, and it uses ipset swap all the time to =
populate sets that enforce firewall policies between containers.

We tracked the issue down with strace, and then took stats with bpftrace =
running:
---
kfunc:ip_set:ip_set_swap {
        @start[tid] =3D nsecs;
}

kretfunc:ip_set:ip_set_swap {
        if (@start[tid]) {
                @srlat =3D hist((nsecs - @start[tid])/1000);
                delete(@start[tid]);
        }
}

interval:s:20 {
        printf("ip_set_swap() latency, milliseconds:\n");
---

On 6.1.69 results look like:
=20
ip_set_swap() latency, milliseconds:
[8K, 16K)           1848 =
|@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
[16K, 32K)         1017 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@
[32K, 64K)         38     |@

while on 6.1.67:
ip_set_swap() latency, milliseconds:

[0]                  166   |@
[1]                  378   |@@
[2, 4)              762   |@@@@@
[4, 8)              1624 |@@@@@@@@@@@
[8, 16)            3493 |@@@@@@@@@@@@@@@@@@@@@@@@
[16, 32)          7308 =
|@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
[32, 64)          6412 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
[64, 128)              1 |

We tried compiling commits between 6.1.67 and 6.1.69 and it seems the =
performance regression was introduced by 875ee3a, ip_set_swap is fast on =
602505 that precedes it, and slow on it.

First time I post here, hope the format is appropriate, and thanks for =
any help with this! Also, if possible, I'd appreciate if any reply could =
CC me, as I am not subscribed.

Alessandro Crismani=