Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67;
Subject: Re: [PATCH 4/7] objtool: Add support for return trampoline call
To:     Peter Zijlstra <peterz@infradead.org>,
        Josh Poimboeuf <jpoimboe@redhat.com>
Cc:     Julien Thierry <jthierry@redhat.com>, x86@kernel.org,
        linux-kernel@vger.kernel.org, tglx@linutronix.de
References: <20200402082220.808-5-alexandre.chartre@oracle.com>
 <c0f265ed-c86b-d3f1-3894-941c25e42d0e@redhat.com>
 <fc224792-bd1c-08ff-072f-e584740521b4@oracle.com>
 <a250f29d-969a-b704-6dd6-c6cc7b84f526@redhat.com>
 <20200402154022.GG20730@hirez.programming.kicks-ass.net>
 <bc3a31dc-9d59-5756-aad3-187533f05654@redhat.com>
 <20200403151757.lhhia7pzqptvlqz5@treble>
 <20200403154620.GS20730@hirez.programming.kicks-ass.net>
 <20200404133218.GL20760@hirez.programming.kicks-ass.net>
 <20200404142232.wpn7estahnabfy3z@treble>
 <20200404155126.GF20730@hirez.programming.kicks-ass.net>
From:   Alexandre Chartre <alexandre.chartre@oracle.com>
Message-ID: <27f433a3-6109-9831-5292-3df4155e10fb@oracle.com>
Date:   Mon, 6 Apr 2020 10:19:56 +0200
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101
 Thunderbird/68.2.0
MIME-Version: 1.0
In-Reply-To: <20200404155126.GF20730@hirez.programming.kicks-ass.net>
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Language: en-US
Content-Transfer-Encoding: 7bit
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk


On 4/4/20 5:51 PM, Peter Zijlstra wrote:
> On Sat, Apr 04, 2020 at 09:22:32AM -0500, Josh Poimboeuf wrote:
>> On Sat, Apr 04, 2020 at 03:32:18PM +0200, Peter Zijlstra wrote:
>>> On Fri, Apr 03, 2020 at 05:46:20PM +0200, Peter Zijlstra wrote:
>>>> On Fri, Apr 03, 2020 at 10:17:57AM -0500, Josh Poimboeuf wrote:
>>>>> Peter, I think my previous idea for UNWIND_HINT_ADJUST stack_add=8 would
>>>>> work here?
>>>>
>>>> Yes, it would.
>>>
>>> Sorry, I have reconsidered. While it will shut up objtool, it will not
>>> 'work'. That is, the ORC data generated will not correctly unwind.
>>>
>>> I'll try and write a longer email tonight.
>>
>> Right, that's what I've been trying to say.  The ORC data will be
>> non-deterministic unless we unroll the loop.  Or did you mean something
>> else?
> 
> The below should result in deterministic code.
> 
> diff --git a/arch/x86/include/asm/nospec-branch.h b/arch/x86/include/asm/nospec-branch.h
> index 07e95dcb40ad..109ee65f4a11 100644
> --- a/arch/x86/include/asm/nospec-branch.h
> +++ b/arch/x86/include/asm/nospec-branch.h
> @@ -59,8 +59,8 @@
>   	jmp	775b;				\
>   774:						\
>   	dec	reg;				\
> -	jnz	771b;				\
> -	add	$(BITS_PER_LONG/8) * nr, sp;
> +	add	$(BITS_PER_LONG/8) * $2, sp;	\
> +	jnz	771b;
>   
>   #ifdef __ASSEMBLY__

Nice. This works fine and allows to remove ANNOTATE_NOSPEC_ALTERNATIVE when
using __FILL_RETURN_BUFFER. However this is probably less performant because
we now have nr/2 add instructions instead of just 1.

Here is a variant where I unroll half of the loop. This way we have 2
add+dec+jnz instruction instructions instead of nr/2 dec+jnz and 1 add
instruction.

#define __FILL_RETURN_BUFFER(reg, nr, sp)	\
	mov	$1, reg;			\
771:						\
	.rept (nr/2);				\
	call	772f;				\
773:	/* speculation trap */			\
	pause;					\
	lfence;					\
	jmp	773b;				\
772:	;					\
	.endr;					\
	add	$(BITS_PER_LONG/8) * (nr/2), sp; \
	dec	reg;				\
	jnz	771b;


Note that we can't unroll the entire loop: it won't work with nr=32 because
the code is then too large to fit into an alternative (the alternative size
is encoded on only one byte so this allows a maximum size of 255, while with
nr=32 the size is around 390).

alex.