This test fails routinely in our prod testing environment, and I can
reproduce it locally as well.
The test allocates dcache inside a cgroup, then drops the memory limit
and checks that usage drops correspondingly. The reason it fails is
because dentries are freed with an RCU delay - a debugging sleep shows
that usage drops as expected shortly after.
Insert a 1s sleep after dropping the limit. This should be good
enough, assuming that machines running those tests are otherwise not
very busy.
Signed-off-by: Johannes Weiner <[email protected]>
---
tools/testing/selftests/cgroup/test_kmem.c | 4 ++++
1 file changed, 4 insertions(+)
diff --git a/tools/testing/selftests/cgroup/test_kmem.c b/tools/testing/selftests/cgroup/test_kmem.c
index 258ddc565deb..1b2cec9d18a4 100644
--- a/tools/testing/selftests/cgroup/test_kmem.c
+++ b/tools/testing/selftests/cgroup/test_kmem.c
@@ -70,6 +70,10 @@ static int test_kmem_basic(const char *root)
goto cleanup;
cg_write(cg, "memory.high", "1M");
+
+ /* wait for RCU freeing */
+ sleep(1);
+
slab1 = cg_read_key_long(cg, "memory.stat", "slab ");
if (slab1 <= 0)
goto cleanup;
--
2.41.0
On Tue, Aug 01, 2023 at 09:56:32AM -0400, Johannes Weiner wrote:
> This test fails routinely in our prod testing environment, and I can
> reproduce it locally as well.
>
> The test allocates dcache inside a cgroup, then drops the memory limit
> and checks that usage drops correspondingly. The reason it fails is
> because dentries are freed with an RCU delay - a debugging sleep shows
> that usage drops as expected shortly after.
>
> Insert a 1s sleep after dropping the limit. This should be good
> enough, assuming that machines running those tests are otherwise not
> very busy.
>
> Signed-off-by: Johannes Weiner <[email protected]>
I am putting together something more formal, but this will certainly
improve things, as Johannes says, assuming the system goes mostly
idle during that one-second wait. So:
Acked-by: Paul E. McKenney <[email protected]>
Yes, there are corner cases, such as the system having millions of
RCU callbacks queued and being unable to invoke them all during that
one-second interval. But that is a corner case, and that is exactly
why I will be putting together something more formal. ;-)
Thanx, Paul
> ---
> tools/testing/selftests/cgroup/test_kmem.c | 4 ++++
> 1 file changed, 4 insertions(+)
>
> diff --git a/tools/testing/selftests/cgroup/test_kmem.c b/tools/testing/selftests/cgroup/test_kmem.c
> index 258ddc565deb..1b2cec9d18a4 100644
> --- a/tools/testing/selftests/cgroup/test_kmem.c
> +++ b/tools/testing/selftests/cgroup/test_kmem.c
> @@ -70,6 +70,10 @@ static int test_kmem_basic(const char *root)
> goto cleanup;
>
> cg_write(cg, "memory.high", "1M");
> +
> + /* wait for RCU freeing */
> + sleep(1);
> +
> slab1 = cg_read_key_long(cg, "memory.stat", "slab ");
> if (slab1 <= 0)
> goto cleanup;
> --
> 2.41.0
>
On Tue, Aug 01, 2023 at 09:39:28AM -0700, Paul E. McKenney wrote:
> On Tue, Aug 01, 2023 at 09:56:32AM -0400, Johannes Weiner wrote:
> > This test fails routinely in our prod testing environment, and I can
> > reproduce it locally as well.
> >
> > The test allocates dcache inside a cgroup, then drops the memory limit
> > and checks that usage drops correspondingly. The reason it fails is
> > because dentries are freed with an RCU delay - a debugging sleep shows
> > that usage drops as expected shortly after.
> >
> > Insert a 1s sleep after dropping the limit. This should be good
> > enough, assuming that machines running those tests are otherwise not
> > very busy.
> >
> > Signed-off-by: Johannes Weiner <[email protected]>
>
> I am putting together something more formal, but this will certainly
> improve things, as Johannes says, assuming the system goes mostly
> idle during that one-second wait. So:
>
> Acked-by: Paul E. McKenney <[email protected]>
>
> Yes, there are corner cases, such as the system having millions of
> RCU callbacks queued and being unable to invoke them all during that
> one-second interval. But that is a corner case, and that is exactly
> why I will be putting together something more formal. ;-)
>
> Thanx, Paul
>
> > ---
> > tools/testing/selftests/cgroup/test_kmem.c | 4 ++++
> > 1 file changed, 4 insertions(+)
> >
> > diff --git a/tools/testing/selftests/cgroup/test_kmem.c b/tools/testing/selftests/cgroup/test_kmem.c
> > index 258ddc565deb..1b2cec9d18a4 100644
> > --- a/tools/testing/selftests/cgroup/test_kmem.c
> > +++ b/tools/testing/selftests/cgroup/test_kmem.c
> > @@ -70,6 +70,10 @@ static int test_kmem_basic(const char *root)
> > goto cleanup;
> >
> > cg_write(cg, "memory.high", "1M");
> > +
> > + /* wait for RCU freeing */
> > + sleep(1);
> > +
> > slab1 = cg_read_key_long(cg, "memory.stat", "slab ");
> > if (slab1 <= 0)
> > goto cleanup;
> > --
> > 2.41.0
> >
The same issue exists in the test case test_kmem_memcg_deletion. I
wouldn't mind posting the patch, but it seems you want to propose
something more formal. Let me know your opinion.
Thanks,
Lucas
On Thu, Aug 03, 2023 at 12:13:26PM -0400, Lucas Karpinski wrote:
> On Tue, Aug 01, 2023 at 09:39:28AM -0700, Paul E. McKenney wrote:
> > On Tue, Aug 01, 2023 at 09:56:32AM -0400, Johannes Weiner wrote:
> > > This test fails routinely in our prod testing environment, and I can
> > > reproduce it locally as well.
> > >
> > > The test allocates dcache inside a cgroup, then drops the memory limit
> > > and checks that usage drops correspondingly. The reason it fails is
> > > because dentries are freed with an RCU delay - a debugging sleep shows
> > > that usage drops as expected shortly after.
> > >
> > > Insert a 1s sleep after dropping the limit. This should be good
> > > enough, assuming that machines running those tests are otherwise not
> > > very busy.
> > >
> > > Signed-off-by: Johannes Weiner <[email protected]>
> >
> > I am putting together something more formal, but this will certainly
> > improve things, as Johannes says, assuming the system goes mostly
> > idle during that one-second wait. So:
> >
> > Acked-by: Paul E. McKenney <[email protected]>
> >
> > Yes, there are corner cases, such as the system having millions of
> > RCU callbacks queued and being unable to invoke them all during that
> > one-second interval. But that is a corner case, and that is exactly
> > why I will be putting together something more formal. ;-)
> >
> > Thanx, Paul
> >
> > > ---
> > > tools/testing/selftests/cgroup/test_kmem.c | 4 ++++
> > > 1 file changed, 4 insertions(+)
> > >
> > > diff --git a/tools/testing/selftests/cgroup/test_kmem.c b/tools/testing/selftests/cgroup/test_kmem.c
> > > index 258ddc565deb..1b2cec9d18a4 100644
> > > --- a/tools/testing/selftests/cgroup/test_kmem.c
> > > +++ b/tools/testing/selftests/cgroup/test_kmem.c
> > > @@ -70,6 +70,10 @@ static int test_kmem_basic(const char *root)
> > > goto cleanup;
> > >
> > > cg_write(cg, "memory.high", "1M");
> > > +
> > > + /* wait for RCU freeing */
> > > + sleep(1);
> > > +
> > > slab1 = cg_read_key_long(cg, "memory.stat", "slab ");
> > > if (slab1 <= 0)
> > > goto cleanup;
> > > --
> > > 2.41.0
> > >
>
> The same issue exists in the test case test_kmem_memcg_deletion. I
> wouldn't mind posting the patch, but it seems you want to propose
> something more formal. Let me know your opinion.
I am proposing a /sys/module/rcutree/parameters/do_rcu_barrier
file. Writing a "1" into this file results in an rcu_barrier()
in the kernel, but set up so that there is no more than a single
rcu_barrier() call per second.
So you could do the following:
run-a-test
echo 1 > /sys/module/rcutree/parameters/do_rcu_barrier # As root
# All RCU callbacks from run-a-test have now been invoked
run-another-test
Please note that this handles only RCU, as in call_rcu(), and not
SRCU, Tasks RCU, and so on.
Thanx, Paul