Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67;
MIME-Version: 1.0
References: <20191111140502.17541-1-tonylu@linux.alibaba.com>
In-Reply-To: <20191111140502.17541-1-tonylu@linux.alibaba.com>
From:   Cong Wang <xiyou.wangcong@gmail.com>
Date:   Mon, 11 Nov 2019 13:26:13 -0800
Message-ID: <CAM_iQpUaPsFHrDmd7fLjWZLbbo8j1uD6opuT+zKqPTVuQPKniA@mail.gmail.com>
Subject: Re: [PATCH] net: remove static inline from dev_put/dev_hold
To:     Tony Lu <tonylu@linux.alibaba.com>
Cc:     David Miller <davem@davemloft.net>, shemminger@osdl.org,
        Linux Kernel Network Developers <netdev@vger.kernel.org>,
        LKML <linux-kernel@vger.kernel.org>
Content-Type: text/plain; charset="UTF-8"
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk

On Mon, Nov 11, 2019 at 6:12 AM Tony Lu <tonylu@linux.alibaba.com> wrote:
>
> This patch removes static inline from dev_put/dev_hold in order to help
> trace the pcpu_refcnt leak of net_device.
>
> We have sufferred this kind of issue for several times during
> manipulating NIC between different net namespaces. It prints this
> log in dmesg:
>
>   unregister_netdevice: waiting for eth0 to become free. Usage count = 1

I debugged a nasty dst refcnt leak in TCP a long time ago, so I can
feel your pain.


>
> However, it is hard to find out who called and leaked refcnt in time. It
> only left the crime scene but few evidence. Once leaked, it is not
> safe to fix it up on the running host. We can't trace dev_put/dev_hold
> directly, for the functions are inlined and used wildly amoung modules.
> And this issue is common, there are tens of patches fix net_device
> refcnt leak for various causes.
>
> To trace the refcnt manipulating, this patch removes static inline from
> dev_put/dev_hold. We can use handy tools, such as eBPF with kprobe, to
> find out who holds but forgets to put refcnt. This will not be called
> frequently, so the overhead is limited.

I think tracepoint serves the purpose of tracking function call history,
you can add tracepoint for each of dev_put()/dev_hold(), which could
also inherit the trace filter and trigger too.

The netdev refcnt itself is not changed very frequently, but it is
refcnt'ed by other things like dst too which is changed frequently.
This is why usually when you see the netdev refcnt leak warning,
the problem is probably somewhere else, like dst refcnt leak.

Hope this helps.

Thanks.