Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
Date:   Wed, 12 May 2021 08:39:13 -0700
From:   Jakub Kicinski <kuba@kernel.org>
To:     Rocco yue <rocco.yue@mediatek.com>
Cc:     "David S . Miller" <davem@davemloft.net>,
        Matthias Brugger <matthias.bgg@gmail.com>,
        Andrew Morton <akpm@linux-foundation.org>,
        Masahiro Yamada <masahiroy@kernel.org>,
        Nick Desaulniers <ndesaulniers@google.com>,
        Andy Shevchenko <andriy.shevchenko@linux.intel.com>,
        Peter Zijlstra <peterz@infradead.org>,
        Tetsuo Handa <penguin-kernel@I-love.SAKURA.ne.jp>,
        <linux-kernel@vger.kernel.org>, <netdev@vger.kernel.org>,
        <linux-arm-kernel@lists.infradead.org>,
        <linux-mediatek@lists.infradead.org>, <wsd_upstream@mediatek.com>
Subject: Re: [PATCH][v3] rtnetlink: add rtnl_lock debug log
Message-ID: <20210512083913.733f9803@kicinski-fedora-pc1c0hjn.dhcp.thefacebook.com>
In-Reply-To: <20210511113257.2094-1-rocco.yue@mediatek.com>
References: <20210511113257.2094-1-rocco.yue@mediatek.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=US-ASCII
Content-Transfer-Encoding: 7bit
Precedence: bulk

On Tue, 11 May 2021 19:32:57 +0800 Rocco yue wrote:
> We often encounter system hangs caused by certain process
> holding rtnl_lock for a long time. Even if there is a lock
> detection mechanism in Linux, it is a bit troublesome and
> affects the system performance. We hope to add a lightweight
> debugging mechanism for detecting rtnl_lock.
> 
> Up to now, we have discovered and solved some potential bugs
> through this lightweight rtnl_lock debugging mechanism, which
> is helpful for us.
> 
> When you say Y for RTNL_LOCK_DEBUG, then the kernel will
> detect if any function hold rtnl_lock too long and some key
> information will be printed out to help locate the problem.
> 
> i.e: from the following logs, we can clearly know that the
> pid=2206 RfxSender_4 process holds rtnl_lock for a long time,
> causing the system to hang. And we can also speculate that the
> delay operation may be performed in devinet_ioctl(), resulting
> in rtnl_lock was not released in time.

You can achieve that with a pair of fexit/fentry hooks or kprobes,
and maybe a bit of BPF. No need for config options, and hardcoded
parameters..