Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67;
MIME-Version: 1.0
References: <000000000000742e9e05a10170bc@google.com> <87a74arown.fsf@nanos.tec.linutronix.de>
 <CAM_iQpV3S0xv5xzSrA5COYa3uyy_TBGpDA9Wcj9Qt_vn1n3jBQ@mail.gmail.com> <87ftdypyec.fsf@nanos.tec.linutronix.de>
In-Reply-To: <87ftdypyec.fsf@nanos.tec.linutronix.de>
From:   Cong Wang <xiyou.wangcong@gmail.com>
Date:   Mon, 23 Mar 2020 16:14:55 -0700
Message-ID: <CAM_iQpVR8Ve3Jy8bb9VB6RcQ=p22ZTyTqjxJxL11RZmO7rkWeg@mail.gmail.com>
Subject: Re: WARNING: ODEBUG bug in tcindex_destroy_work (3)
To:     Thomas Gleixner <tglx@linutronix.de>
Cc:     syzbot <syzbot+46f513c3033d592409d2@syzkaller.appspotmail.com>,
        David Miller <davem@davemloft.net>,
        Jamal Hadi Salim <jhs@mojatatu.com>,
        Jiri Pirko <jiri@resnulli.us>,
        Jakub Kicinski <kuba@kernel.org>,
        LKML <linux-kernel@vger.kernel.org>,
        Linux Kernel Network Developers <netdev@vger.kernel.org>,
        syzkaller-bugs <syzkaller-bugs@googlegroups.com>
Content-Type: text/plain; charset="UTF-8"
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk

On Mon, Mar 23, 2020 at 2:14 PM Thomas Gleixner <tglx@linutronix.de> wrote:
>
> Cong Wang <xiyou.wangcong@gmail.com> writes:
> > On Sat, Mar 21, 2020 at 3:19 AM Thomas Gleixner <tglx@linutronix.de> wrote:
> >> > ------------[ cut here ]------------
> >> > ODEBUG: free active (active state 0) object type: work_struct hint: tcindex_destroy_rexts_work+0x0/0x20 net/sched/cls_tcindex.c:143
> >> ...
> >> >  __debug_check_no_obj_freed lib/debugobjects.c:967 [inline]
> >> >  debug_check_no_obj_freed+0x2e1/0x445 lib/debugobjects.c:998
> >> >  kfree+0xf6/0x2b0 mm/slab.c:3756
> >> >  tcindex_destroy_work+0x2e/0x70 net/sched/cls_tcindex.c:231
> >>
> >> So this is:
> >>
> >>         kfree(p->perfect);
> >>
> >> Looking at the place which queues that work:
> >>
> >> tcindex_destroy()
> >>
> >>    if (p->perfect) {
> >>         if (tcf_exts_get_net(&r->exts))
> >>             tcf_queue_work(&r-rwork, tcindex_destroy_rexts_work);
> >>         else
> >>             __tcindex_destroy_rexts(r)
> >>    }
> >>
> >>    .....
> >>
> >>    tcf_queue_work(&p->rwork, tcindex_destroy_work);
> >>
> >> So obviously if tcindex_destroy_work() runs before
> >> tcindex_destroy_rexts_work() then the above happens.
> >
> > We use an ordered workqueue for tc filters, so these two
> > works are executed in the same order as they are queued.
>
> The workqueue is ordered, but look how the work is queued on the work
> queue:
>
> tcf_queue_work()
>   queue_rcu_work()
>     call_rcu(&rwork->rcu, rcu_work_rcufn);
>
> So after the grace period elapses rcu_work_rcufn() queues it in the
> actual work queue.
>
> Now tcindex_destroy() is invoked via tcf_proto_destroy() which can be
> invoked from preemtible context. Now assume the following:
>
> CPU0
>   tcf_queue_work()
>     tcf_queue_work(&r->rwork, tcindex_destroy_rexts_work);
>
> -> Migration
>
> CPU1
>    tcf_queue_work(&p->rwork, tcindex_destroy_work);
>
> So your RCU callbacks can be placed on different CPUs which obviously
> has no ordering guarantee at all. See also:

Good catch!

I thought about this when I added this ordered workqueue, but it
seems I misinterpret max_active, so despite we have max_active==1,
more than 1 work could still be queued on different CPU's here.

I don't know how to fix this properly, I think essentially RCU work
should be guaranteed the same ordering with regular work. But this
seems impossible unless RCU offers some API to achieve that.

Thanks.