Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <53ba30f1-c867-f34e-b262-08df67ef26fb@kernel.org>
Date:   Tue, 20 Sep 2022 09:42:44 +0200
MIME-Version: 1.0
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101
 Thunderbird/102.2.1
Subject: Re: [PATCH] trace/osnoise: Fix possible recursive locking in
 stop_per_cpu_kthreads
To:     Nico Pache <npache@redhat.com>, linux-kernel@vger.kernel.org
Cc:     rostedt@goodmis.org, mingo@redhat.com
References: <20220919144932.3064014-1-npache@redhat.com>
Content-Language: en-US
From:   Daniel Bristot de Oliveira <bristot@kernel.org>
In-Reply-To: <20220919144932.3064014-1-npache@redhat.com>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit
Precedence: bulk

Hi Nico!

On 9/19/22 16:49, Nico Pache wrote:
> There is a recursive lock on the cpu_hotplug_lock.
> 
> In kernel/trace/trace_osnoise.c:<start/stop>_per_cpu_kthreads:
>     - start_per_cpu_kthreads calls cpus_read_lock() and if
> 	start_kthreads returns a error it will call stop_per_cpu_kthreads.
>     - stop_per_cpu_kthreads then calls cpus_read_lock() again causing
>       deadlock.

Yep, I see the problem.

> Fix this by calling cpus_read_unlock() before calling
> stop_per_cpu_kthreads. This behavior can also be seen in commit
> f46b16520a08 ("trace/hwlat: Implement the per-cpu mode").

Correct, we have a similar function that *without* this problem.

> This error was noticed during the LTP ftrace-stress-test:
> 
> WARNING: possible recursive locking detected
> --------------------------------------------
> sh/275006 is trying to acquire lock:
> ffffffffb02f5400 (cpu_hotplug_lock){++++}-{0:0}, at: stop_per_cpu_kthreads
> 
> but task is already holding lock:
> ffffffffb02f5400 (cpu_hotplug_lock){++++}-{0:0}, at: start_per_cpu_kthreads
> 
> other info that might help us debug this:
>  Possible unsafe locking scenario:
> 
>       CPU0
>       ----
>  lock(cpu_hotplug_lock);
>  lock(cpu_hotplug_lock);
> 
>  *** DEADLOCK ***
> 
> May be due to missing lock nesting notation
> 
> 3 locks held by sh/275006:
>  #0: ffff8881023f0470 (sb_writers#24){.+.+}-{0:0}, at: ksys_write
>  #1: ffffffffb084f430 (trace_types_lock){+.+.}-{3:3}, at: rb_simple_write
>  #2: ffffffffb02f5400 (cpu_hotplug_lock){++++}-{0:0}, at: start_per_cpu_kthreads
> 
> Fixes: c8895e271f79 ("trace/osnoise: Support hotplug operations")
> Signed-off-by: Nico Pache <npache@redhat.com>

Acked-by: Daniel Bristot de Oliveira <bristot@kernel.org>

Thanks!
-- Daniel