Received: by 2002:a25:868d:0:0:0:0:0 with SMTP id z13csp971181ybk; Fri, 15 May 2020 19:41:27 -0700 (PDT) X-Google-Smtp-Source: ABdhPJzogQBNJ4CZ6OeizD3kNtdzCURE4YcnagAIqkZ/XQ+cduIgQh5+r3+de2hlQZZoL4I2JRKQ X-Received: by 2002:a17:906:4753:: with SMTP id j19mr5817783ejs.83.1589596886834; Fri, 15 May 2020 19:41:26 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1589596886; cv=none; d=google.com; s=arc-20160816; b=EtE12xxcO9AJ1dz4+arhMcfC626/ZhaEQPJxzlkqlnOZBH4OwPTVK2cPbPIz8RzONR 87CemVMr5m2UCtdIyUmna2UxhhcJXkJl3X1QCUefWnbd8/hVqSpSw1HBSmRPhzUpG+32 yfXtKQvmwX4Cmcf2oU1fqm1xdewN2MMTSravQs4HaoCtAu5gQVeoaPTfEhLgtIURO+Y3 01GpNMZuQR9Q3oiNxo/trKOkQ0F1n3nvp4g71gE5VVmJyqclxCoyA5cLDLnMuHPqIP+R qipppIW17vDOY+GAlMkVa4VD0r993QpcQaB8g49wgpMjW+tWyhBhKzVT7MgxS0sirRi8 3wpA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:dkim-signature; bh=e+Om/GsyDrpyV1h+nZkU0dBuPGFU3TkzcVyywQUT9C8=; b=N4RB4w6hb5JD5c7vV6vz48+6OyqMo92AChWm2M7rcz2SqhxVa/qcMVxQ9O2kKy9Igb Cu2NpZI0/xf6yEPutx8E+YBRMJxmkhWTARsh7V4LRbyEGuCx34UZXzAhH1sZgEChMhlM 6IIfNGeb0F7WLjNpeNqjq+WC+ujnR4Fkhw5JxSW2gxb6WRm7jrlxXhUUNWhs2juy1iNa Lbh3i3PCw5Hm11l3oB4VUeV4apgC+6vC4RYItY6NVN3XpLXUsBwE73sb0n1vgTQUF66L IsqBS1AzQHQ9Dt3i9U0w+0iyjGpI18Qz3O0ERzo9A/iOQwpyVNOG+Cy6Px/WC3C2vmEW u5yQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=YlSSCiFz; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from vger.kernel.org (vger.kernel.org. [23.128.96.18]) by mx.google.com with ESMTP id l4si2427660ejr.97.2020.05.15.19.41.03; Fri, 15 May 2020 19:41:26 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=YlSSCiFz; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727943AbgEPChi (ORCPT + 99 others); Fri, 15 May 2020 22:37:38 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36670 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-FAIL-OK-FAIL) by vger.kernel.org with ESMTP id S1726247AbgEPChh (ORCPT ); Fri, 15 May 2020 22:37:37 -0400 Received: from mail-il1-x144.google.com (mail-il1-x144.google.com [IPv6:2607:f8b0:4864:20::144]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 73229C061A0C; Fri, 15 May 2020 19:37:37 -0700 (PDT) Received: by mail-il1-x144.google.com with SMTP id 17so4506660ilj.3; Fri, 15 May 2020 19:37:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=e+Om/GsyDrpyV1h+nZkU0dBuPGFU3TkzcVyywQUT9C8=; b=YlSSCiFz7D2P0SHo/hB1dOKbVDBgwqB3bbFXiseSa4VR0BMjp+xU3hiKpTRCWg4Duc rGklV2WRDYVYENVj9vnH8RDWSgOIT1CP4nshdYJLzGA8vZiFvKDToDDux9xoEVtGYSRS Hzz6+zJxYI7P8tgWP5eJte4FmLpUssj/zf56zH4/DHKqC9dnssp3LvPczqvzzgMrIi8k B/PWF5fN6n4FhTIvD+xq7OcclfTByKkkPRozO05tJM2Mty0z7vscexgQNIYk69lFIwF8 6TQYYLwb+KuF0bsj7FviLEwcDz/gonYP1l05OhpE/zjvwGFiA0cWYC6UhdFgibA9pUo6 dwog== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=e+Om/GsyDrpyV1h+nZkU0dBuPGFU3TkzcVyywQUT9C8=; b=ErwlvQ0apG3ol+IS5FfBA1WJjDpN7N6p+KKPFqGVbkgoTMPTHJrPp9+RQKhDw7szQ/ UDu7cIEbM8Ndo94Xu1u+rGBaJNV5RdRefnmyvTz46LYjIVUW/WvxARsYHr51VKVYf2+Q sJPdbVn9BGHNRttT+iabuAaMfWnnBwPIK9daFMjDO3TcTitEDWPwfvGOTX9kHZy2zR60 7/HdzvADeIHMHUi6wLLWOuVaOvbwO84VjkX2WZsOwrH0vm78wGxHUS44m3jExH2W3XYD RiOh0dusXh6TxDjR8kv7VNez0RXGDWOOnueQsT7Yu9yIMETiDmvTHFbdtuqNs8a1p3hk br/A== X-Gm-Message-State: AOAM532sM5pmo68AjkHtlorWdUDZKhhbSFpmBTEMZ5KHYGIVFWp1pDyi FUSUjawQ3RfWLJY0dcAc3RvCGx1SjaTYVAR+GfY= X-Received: by 2002:a92:9f4b:: with SMTP id u72mr4727883ili.273.1589596656775; Fri, 15 May 2020 19:37:36 -0700 (PDT) MIME-Version: 1.0 References: <20200429220732.31602-1-yu-cheng.yu@intel.com> <20200429220732.31602-2-yu-cheng.yu@intel.com> <5cc163ff9058d1b27778e5f0a016c88a3b1a1598.camel@intel.com> <44c055342bda4fb4730703f987ae35195d1d0c38.camel@intel.com> <32235ffc-6e6c-fb3d-80c4-a0478e2d0e0f@intel.com> <6272c481-af90-05c5-7231-3ba44ff9bd02@citrix.com> In-Reply-To: <6272c481-af90-05c5-7231-3ba44ff9bd02@citrix.com> From: "H.J. Lu" Date: Fri, 15 May 2020 19:37:00 -0700 Message-ID: Subject: Re: [PATCH v10 01/26] Documentation/x86: Add CET description To: Andrew Cooper Cc: Dave Hansen , Yu-cheng Yu , "the arch/x86 maintainers" , "H. Peter Anvin" , Thomas Gleixner , Ingo Molnar , LKML , "open list:DOCUMENTATION" , Linux-MM , linux-arch , Linux API , Arnd Bergmann , Andy Lutomirski , Balbir Singh , Borislav Petkov , Cyrill Gorcunov , Dave Hansen , Eugene Syromiatnikov , Florian Weimer , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , "Ravi V. Shankar" , Vedvyas Shanbhogue , Dave Martin , Weijiang Yang Content-Type: text/plain; charset="UTF-8" Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Fri, May 15, 2020 at 5:13 PM Andrew Cooper wrote: > > On 15/05/2020 23:43, Dave Hansen wrote: > > On 5/15/20 2:33 PM, Yu-cheng Yu wrote: > >> On Fri, 2020-05-15 at 11:39 -0700, Dave Hansen wrote: > >>> On 5/12/20 4:20 PM, Yu-cheng Yu wrote: > >>> Can a binary compiled with CET run without CET? > >> Yes, but a few details: > >> > >> - The shadow stack is transparent to the application. A CET application does > >> not have anything different from a non-CET application. However, if a CET > >> application uses any CET instructions (e.g. INCSSP), it must first check if CET > >> is turned on. > >> - If an application is compiled for IBT, the compiler inserts ENDBRs at branch > >> targets. These are nops if IBT is not on. > > I appreciate the detailed response, but it wasn't quite what I was > > asking. Let's ignore IBT for now and just talk about shadow stacks. > > > > An app compiled with the new ELF flags and running on a CET-enabled > > kernel and CPU will start off with shadow stacks allocated and enabled, > > right? It can turn its shadow stack off per-thread with the new prctl. > > But, otherwise, it's stuck, the only way to turn shadow stacks off at > > startup would be editing the binary. > > > > Basically, if there ends up being a bug in an app that violates the > > shadow stack rules, the app is broken, period. The only recourse is to > > have the kernel disable CET and reboot. > > > > Is that right? > > If I may interject with the experience of having got supervisor shadow > stacks working for Xen. > > Turning shadow stacks off is quite easy - clear MSR_U_CET.SHSTK_EN and > the shadow stack will stay in whatever state it was in, and you can > largely forget about it. (Of course, in a sandbox scenario, it would be > prudent to prevent the ability to disable shadow stacks.) > > Turning shadow stacks on is much more tricky. You cannot enable it in > any function you intend to return from, as the divergence between the > stack and shadow stack will constitute a control flow violation. > > > When it comes to binaries, you can reasonably arrange for clone() to > start a thread on a new stack/shstk, as you can prepare both stacks > suitably before execution starts. > > You cannot reasonably implement a system call for "turn shadow stacks on > for me", because you'll crash on the ret out of the VDSO from the system > call. It would be possible to conceive of an exec()-like system call > which is "discard my current stack, turn on shstk, and start me on this > new stack/shstk". > > In principle, with a pair of system calls to atomically manage the ststk > settings and stack switching, it might possible to construct a > `run_with_shstk_enabled(func, stack, shstk)` API which executes in the > current threads context and doesn't explode. > > Fork() is a problem when shadow stacks are disabled in the parent. The > moment shadow stacks are disabled, the regular stack diverges from the > shadow stack. A CET-enabled app which turns off shstk and then fork()'s > must have the child inherit the shstk-off property. If the child were > to start with shstk enabled, it would explode almost immediately due to > the parent's stack divergence which it inherited. > > > Finally seeing as the question was asked but not answered, it is > actually quite easy to figure out whether shadow stacks are enabled in > the current thread. > > mov $1, %eax > rdsspd %eax This is for 32-bit mode. I use /* Check if shadow stack is in use. */ xorl %esi, %esi rdsspq %rsi testq %rsi, %rsi /* Normal return if shadow stack isn't in use. */ je L(no_shstk) > cmp $1, %eax > je no_shstk > ... > no_shsk: > > rdssp is allocated from the hint nop encoding space, and the minimum > alignment of the shadow stack pointer is 4. On older parts, or with > shstk disabled (either at the system level, or for the thread), the $1 > will be preserved in %eax, while if CET is active, it will be clobbered > with something that has the bottom two bits clear. > > It turns out this is a lifesaver for codepaths (e.g. the NMI handler) > which need to use other CET instructions which aren't from the hint nop > space, and run before the BSP can set everything up. > > ~Andrew -- H.J.