Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67;
Date:   Mon, 7 Jan 2019 16:28:33 -0500
From:   Steven Rostedt <rostedt@goodmis.org>
To:     Andrea Righi <righi.andrea@gmail.com>
Cc:     Masami Hiramatsu <mhiramat@kernel.org>,
        Ingo Molnar <mingo@redhat.com>, peterz@infradead.org,
        Mathieu Desnoyers <mathieu.desnoyers@efficios.com>,
        linux-kernel <linux-kernel@vger.kernel.org>
Subject: Re: [PATCH 0/2] kprobes: Fix kretprobe incorrect stacking order
 problem
Message-ID: <20190107162833.1e034fc7@gandalf.local.home>
In-Reply-To: <20190107211904.GC5966@xps-13>
References: <154686789378.15479.2886543882215785247.stgit@devbox>
        <20190107183444.GA5966@xps-13>
        <20190107142749.34231bb6@gandalf.local.home>
        <20190107195209.GB5966@xps-13>
        <20190107145918.407b851b@gandalf.local.home>
        <20190107211904.GC5966@xps-13>
MIME-Version: 1.0
Content-Type: text/plain; charset=US-ASCII
Content-Transfer-Encoding: 7bit
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk

On Mon, 7 Jan 2019 22:19:04 +0100
Andrea Righi <righi.andrea@gmail.com> wrote:

> > > If we put a kretprobe to raw_spin_lock_irqsave() it looks like
> > > kretprobe is going to call kretprobe...  
> > 
> > Right, but we should be able to add some recursion protection to stop
> > that. I have similar protection in the ftrace code.  
> 
> If we assume that __raw_spin_lock/unlock*() are always inlined a

I wouldn't assume that.

> possible way to prevent this recursion could be to use directly those
> functions to do locking from the kretprobe trampoline.
> 
> But I'm not sure if that's a safe assumption... if not I'll see if I can
> find a better solution.

All you need to do is have a per_cpu variable, where you just do:

	preempt_disable_notrace();
	if (this_cpu_read(kprobe_recursion))
		goto out;
	this_cpu_inc(kprobe_recursion);
	[...]
	this_cpu_dec(kprobe_recursion);
out:
	preempt_enable_notrace();

And then just ignore any kprobes that trigger while you are processing
the current kprobe.

Something like that. If you want (or if it already happens) replace
preempt_disable() with local_irq_save().

-- Steve

> 
> Thanks,
> 
> From: Andrea Righi <righi.andrea@gmail.com>
> Subject: [PATCH] kprobes: prevent recursion deadlock with kretprobe and
>  spinlocks
> 
> kretprobe_trampoline() is using a spinlock to protect the hash of
> kretprobes. Adding a kretprobe to the spinlock functions may cause
> a recursion deadlock where kretprobe is calling itself:
> 
>  kretprobe_trampoline()
>    -> trampoline_handler()
>      -> kretprobe_hash_lock()
>        -> raw_spin_lock_irqsave()
>          -> _raw_spin_lock_irqsave()  
>  kretprobe_trampoline from _raw_spin_lock_irqsave => DEADLOCK
> 
>  kretprobe_trampoline()
>    -> trampoline_handler()
>      -> recycle_rp_inst()
>        -> raw_spin_lock()
>          -> _raw_spin_lock()  
>  kretprobe_trampoline from _raw_spin_lock => DEADLOCK
> 
> Use the corresponding inlined spinlock functions to prevent this
> recursion.
> 
> Signed-off-by: Andrea Righi <righi.andrea@gmail.com>
> ---
>  kernel/kprobes.c | 8 ++++----
>  1 file changed, 4 insertions(+), 4 deletions(-)
> 
> diff --git a/kernel/kprobes.c b/kernel/kprobes.c
> index f4ddfdd2d07e..b89bef5e3d80 100644
> --- a/kernel/kprobes.c
> +++ b/kernel/kprobes.c
> @@ -1154,9 +1154,9 @@ void recycle_rp_inst(struct kretprobe_instance *ri,
>  	hlist_del(&ri->hlist);
>  	INIT_HLIST_NODE(&ri->hlist);
>  	if (likely(rp)) {
> -		raw_spin_lock(&rp->lock);
> +		__raw_spin_lock(&rp->lock);
>  		hlist_add_head(&ri->hlist, &rp->free_instances);
> -		raw_spin_unlock(&rp->lock);
> +		__raw_spin_unlock(&rp->lock);
>  	} else
>  		/* Unregistering */
>  		hlist_add_head(&ri->hlist, head);
> @@ -1172,7 +1172,7 @@ __acquires(hlist_lock)
>  
>  	*head = &kretprobe_inst_table[hash];
>  	hlist_lock = kretprobe_table_lock_ptr(hash);
> -	raw_spin_lock_irqsave(hlist_lock, *flags);
> +	*flags = __raw_spin_lock_irqsave(hlist_lock);
>  }
>  NOKPROBE_SYMBOL(kretprobe_hash_lock);
>  
> @@ -1193,7 +1193,7 @@ __releases(hlist_lock)
>  	raw_spinlock_t *hlist_lock;
>  
>  	hlist_lock = kretprobe_table_lock_ptr(hash);
> -	raw_spin_unlock_irqrestore(hlist_lock, *flags);
> +	__raw_spin_unlock_irqrestore(hlist_lock, *flags);
>  }
>  NOKPROBE_SYMBOL(kretprobe_hash_unlock);
>