Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
MIME-Version: 1.0
References: <xm26fsjotqda.fsf@google.com> <20220629165542.da7fc8a2a5dbd53cf99572aa@linux-foundation.org>
 <CALvZod5KX7XEHR9h_jFHf5pJcYB+dODEeaLKrQLtSy9EUqgvWw@mail.gmail.com> <20220629192435.df27c0dbb07ef72165e1de5e@linux-foundation.org>
In-Reply-To: <20220629192435.df27c0dbb07ef72165e1de5e@linux-foundation.org>
From:   Shakeel Butt <shakeelb@google.com>
Date:   Thu, 30 Jun 2022 07:59:05 -0700
Message-ID: <CALvZod5hJ8VJ4E9jhqjCKc8au8_b-h_q+g=2pbQVUSBvappE6g@mail.gmail.com>
Subject: Re: [RESEND RFC PATCH] epoll: autoremove wakers even more aggressively
To:     Andrew Morton <akpm@linux-foundation.org>
Cc:     Benjamin Segall <bsegall@google.com>,
        Alexander Viro <viro@zeniv.linux.org.uk>,
        linux-fsdevel <linux-fsdevel@vger.kernel.org>,
        LKML <linux-kernel@vger.kernel.org>,
        Linus Torvalds <torvalds@linux-foundation.org>,
        Eric Dumazet <edumazet@google.com>,
        Roman Penyaev <rpenyaev@suse.de>,
        Jason Baron <jbaron@akamai.com>,
        Khazhismel Kumykov <khazhy@google.com>, Heiher <r@hev.cc>
Content-Type: text/plain; charset="UTF-8"
Precedence: bulk

On Wed, Jun 29, 2022 at 7:24 PM Andrew Morton <akpm@linux-foundation.org> wrote:
>
> On Wed, 29 Jun 2022 18:12:46 -0700 Shakeel Butt <shakeelb@google.com> wrote:
>
> > On Wed, Jun 29, 2022 at 4:55 PM Andrew Morton <akpm@linux-foundation.org> wrote:
> > >
> > > On Wed, 15 Jun 2022 14:24:23 -0700 Benjamin Segall <bsegall@google.com> wrote:
> > >
> > > > If a process is killed or otherwise exits while having active network
> > > > connections and many threads waiting on epoll_wait, the threads will all
> > > > be woken immediately, but not removed from ep->wq. Then when network
> > > > traffic scans ep->wq in wake_up, every wakeup attempt will fail, and
> > > > will not remove the entries from the list.
> > > >
> > > > This means that the cost of the wakeup attempt is far higher than usual,
> > > > does not decrease, and this also competes with the dying threads trying
> > > > to actually make progress and remove themselves from the wq.
> > > >
> > > > Handle this by removing visited epoll wq entries unconditionally, rather
> > > > than only when the wakeup succeeds - the structure of ep_poll means that
> > > > the only potential loss is the timed_out->eavail heuristic, which now
> > > > can race and result in a redundant ep_send_events attempt. (But only
> > > > when incoming data and a timeout actually race, not on every timeout)
> > > >
> > >
> > > Thanks.  I added people from 412895f03cbf96 ("epoll: atomically remove
> > > wait entry on wake up") to cc.  Hopefully someone there can help review
> > > and maybe test this.
> > >
> > >
> >
> > Thanks Andrew. Just wanted to add that we are seeing this issue in
> > production with real workloads and it has caused hard lockups.
> > Particularly network heavy workloads with a lot of threads in
> > epoll_wait() can easily trigger this issue if they get killed
> > (oom-killed in our case).
>
> Hard lockups are undesirable.  Is a cc:stable justified here?

Not for now as I don't know if we can blame a patch which might be the
source of this behavior.