Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
From:   Arvind Sankar <nivedita@alum.mit.edu>
Date:   Wed, 2 Sep 2020 12:14:29 -0400
To:     David Laight <David.Laight@ACULAB.COM>
Cc:     'Arvind Sankar' <nivedita@alum.mit.edu>,
        Linus Torvalds <torvalds@linux-foundation.org>,
        Miguel Ojeda <miguel.ojeda.sandonis@gmail.com>,
        Sedat Dilek <sedat.dilek@gmail.com>,
        Segher Boessenkool <segher@kernel.crashing.org>,
        Thomas Gleixner <tglx@linutronix.de>,
        Nick Desaulniers <ndesaulniers@google.com>,
        "Paul E. McKenney" <paulmck@kernel.org>,
        Ingo Molnar <mingo@redhat.com>, Arnd Bergmann <arnd@arndb.de>,
        Borislav Petkov <bp@alien8.de>,
        "maintainer:X86 ARCHITECTURE (32-BIT AND 64-BIT)" <x86@kernel.org>,
        "H. Peter Anvin" <hpa@zytor.com>,
        "Kirill A. Shutemov" <kirill.shutemov@linux.intel.com>,
        Kees Cook <keescook@chromium.org>,
        Peter Zijlstra <peterz@infradead.org>,
        Juergen Gross <jgross@suse.com>,
        Andy Lutomirski <luto@kernel.org>,
        Andrew Cooper <andrew.cooper3@citrix.com>,
        LKML <linux-kernel@vger.kernel.org>,
        clang-built-linux <clang-built-linux@googlegroups.com>,
        Will Deacon <will@kernel.org>,
        "nadav.amit@gmail.com" <nadav.amit@gmail.com>,
        Nathan Chancellor <natechancellor@gmail.com>
Subject: Re: [PATCH v2] x86/asm: Replace __force_order with memory clobber
Message-ID: <20200902161429.GB3297881@rani.riverdale.lan>
References: <20200823212550.3377591-1-nivedita@alum.mit.edu>
 <20200902153346.3296117-1-nivedita@alum.mit.edu>
 <55a35a3d8fba417aabe63ad13d519198@AcuMS.aculab.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
In-Reply-To: <55a35a3d8fba417aabe63ad13d519198@AcuMS.aculab.com>
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk

On Wed, Sep 02, 2020 at 03:58:38PM +0000, David Laight wrote:
> From: Arvind Sankar
> > Sent: 02 September 2020 16:34
> > 
> > The CRn accessor functions use __force_order as a dummy operand to
> > prevent the compiler from reordering the inline asm.
> > 
> > The fact that the asm is volatile should be enough to prevent this
> > already, however older versions of GCC had a bug that could sometimes
> > result in reordering. This was fixed in 8.1, 7.3 and 6.5. Versions prior
> > to these, including 5.x and 4.9.x, may reorder volatile asm.
> > 
> > There are some issues with __force_order as implemented:
> > - It is used only as an input operand for the write functions, and hence
> >   doesn't do anything additional to prevent reordering writes.
> > - It allows memory accesses to be cached/reordered across write
> >   functions, but CRn writes affect the semantics of memory accesses, so
> >   this could be dangerous.
> > - __force_order is not actually defined in the kernel proper, but the
> >   LLVM toolchain can in some cases require a definition: LLVM (as well
> >   as GCC 4.9) requires it for PIE code, which is why the compressed
> >   kernel has a definition, but also the clang integrated assembler may
> >   consider the address of __force_order to be significant, resulting in
> >   a reference that requires a definition.
> > 
> > Fix this by:
> > - Using a memory clobber for the write functions to additionally prevent
> >   caching/reordering memory accesses across CRn writes.
> > - Using a dummy input operand with an arbitrary constant address for the
> >   read functions, instead of a global variable. This will prevent reads
> >   from being reordered across writes, while allowing memory loads to be
> >   cached/reordered across CRn reads, which should be safe.
> 
> How much does using a full memory clobber for the reads cost?
> 
> It would remove any chance that the compiler decides it needs to
> get the address of the 'dummy' location into a register so that
> it can be used as a memory reference in a generated instruction
> (which is probably what was happening for PIE compiles).
> 
> 	David
> 

It doesn't cost much. When I tested it, the only differences were in
startup code and sleep/hibernate/reboot code.

The compiler doesn't load 0x1000 into a register even for PIE code, the
reason it was doing it with a real symbol is to go through the GOT.

Thanks.