Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.37 as permitted sender) client-ip=23.128.96.37;
Message-ID: <c72f3ef9-ee82-84c8-c802-c9593a1dd5d1@intel.com>
Date:   Fri, 15 Sep 2023 14:30:11 +0800
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:102.0) Gecko/20100101
 Thunderbird/102.15.0
Subject: Re: [PATCH v6 03/25] x86/fpu/xstate: Add CET supervisor mode state
 support
Content-Language: en-US
To:     "Edgecombe, Rick P" <rick.p.edgecombe@intel.com>,
        "kvm@vger.kernel.org" <kvm@vger.kernel.org>,
        "pbonzini@redhat.com" <pbonzini@redhat.com>,
        "Christopherson,, Sean" <seanjc@google.com>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>
CC:     "peterz@infradead.org" <peterz@infradead.org>,
        "Hansen, Dave" <dave.hansen@intel.com>,
        "Gao, Chao" <chao.gao@intel.com>,
        "john.allen@amd.com" <john.allen@amd.com>
References: <20230914063325.85503-1-weijiang.yang@intel.com>
 <20230914063325.85503-4-weijiang.yang@intel.com>
 <35393ed30962125fd6ce1f91f71595d1111413ec.camel@intel.com>
From:   "Yang, Weijiang" <weijiang.yang@intel.com>
In-Reply-To: <35393ed30962125fd6ce1f91f71595d1111413ec.camel@intel.com>
Content-Type: text/plain; charset="UTF-8"; format=flowed
Content-Transfer-Encoding: 8bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?TDlhUXNOd2tJUW5VRUVINWQvZHJGSGNWOURZVVljWWIrazFrVm5DSGZnTzdy?=
 =?utf-8?B?UzJIQWNFWVB2VXdMd2tISDVlQ2V5czBnbkxIbjJjZTJlbWlUWHRXak1NZ24z?=
 =?utf-8?B?WlErQmZpc0xob1pjcjN4czhqTmFIcHlrL3NyTVdWRzZwWTNoRUhLKzJZQklJ?=
 =?utf-8?B?cWV1ckhqRXN4RUpPbE0rUEtHWlJYSnNWL3UzVGVOK1pxY0MvYkFmaExrS3dW?=
 =?utf-8?B?aXU0Q0NmK3pKMzgvdVkvdFgrYlk0UmowMGlnTW5nTEluRmN6Qkw3WlFNQkRJ?=
 =?utf-8?B?OXR2Q1ZncWhqZVErZ01yWDRkR0k5YmxJWjhoVG5reTJpNnRHNVJ0UGd6RGxH?=
 =?utf-8?B?MWFWZFNVaHA4TGs5VlVmYSs3b3lsYUpCMlpJdXhOWHJRZjNXMXF2WllIQnB3?=
 =?utf-8?B?Z2VZNTVaaXdORE5tOVNDQm5lWTZPNXh3Ym94RlQxYlJKbnN1WUdnQXpLOTZx?=
 =?utf-8?B?bXZPYmMyMTZmemxFRVJodE9BNnlZb2ovaEFMVnZMdTJMMmtXL0xuQ3ZjZVk2?=
 =?utf-8?B?aWp6NWZLQWRsVXEwL2hHbnJjN3ZIYVl5MHBSQUpUTHZZOFVpTjQ5Zm5ZYmpY?=
 =?utf-8?B?RlFRQ2Q4QWtMZllaL3NYR3JKWDVGd3FCZCthTUtwZkZGN3BseXV4dTU0dytm?=
 =?utf-8?B?eUVjYkhJM1JLeERIbUNOdktuN1l4OSs0cTJlZHJnRlBNN1FqOFh0MVdLWGZK?=
 =?utf-8?B?MjJqT1drMWNOMXZnOHlGVklIQmdMSFlYSUZTdS9PSW1LVkIwWXhEL0c4ejVq?=
 =?utf-8?B?QWt2ZjVDSnMzOGhZRTBuVVJTanBXSFRlMUd5YzVLbXptQ3JhNDJBWmhZV0ti?=
 =?utf-8?B?OEU4WE9kUy9tZXNZQ2hkaVQzLzM4aDdISGdUWTF2ejhsM080UXhraVRCdXhR?=
 =?utf-8?B?WjMwMVlrUUsrYmRscTlGS3FOT2s1UmZTb0J2dGZSVHYwZjlkRHpsd0hvQzdO?=
 =?utf-8?B?dThwZFF4MGNkN3c4RWpiaUR1ay9ubzB1bU9qTjA3ZUNJOXkrVGkrS0NRWkdQ?=
 =?utf-8?B?NHZkWDl3ZWdFVUF0bmtJc1lqTnF6QnR2VGNxbTBMT2s3dXVXVTFEQ2RNWWFz?=
 =?utf-8?B?Qml3SUdKWWpzbUhoUHJoUWlBaHNJVTFZUTYzTE1JeXVPVjZENzlBdjhuZUlr?=
 =?utf-8?B?UE16Y3dldEJiK3ZEYlkzWDJodTVHQ25sSTFzMmdwZlhwdGNOZEE0dWdsc0Fn?=
 =?utf-8?B?eHM0SG5yc1M3U05ZZkZRS09RY0J1L1lCQzUxNkNnQUFUUnVQR0lwSHNXTmZi?=
 =?utf-8?B?S25IeUVwajJmemFaeHhjYm4ycWp3M2pNakIvVDRHUUFNa0VLbWNYVC8wR1FU?=
 =?utf-8?B?b1pjOTFTcFUxTEFWeFh0dk1yVmRZRDFCTjZDMXhGMnZmZG9qNVEyaDI1Zk85?=
 =?utf-8?B?SmxabE1MNjFuNzYvY0V3L2dSdzhUSE1YdzhVTFU3ZXFMTnJXUHVrdzhsUnp2?=
 =?utf-8?B?d2NqUTRSUi82ejR0MHViQXFSdnA1b004M0JTMDBVeC93ZlQwcmZlY0hZaXZI?=
 =?utf-8?B?UXd6ZUNmS1Y0c2xMTmwxOWxmUHZmWVo3dUpvWVVQdUV4MitaYWpLVmlEUWxW?=
 =?utf-8?B?UklLempSUCtKRnlMTWhtdDN6UytwZG96cmRWKytpYlFUUTk0VTV1ZU8wMGtn?=
 =?utf-8?B?UkJDYTdCMER6TzQzUkovVVJOQnhQNTVycisvK2VsSzZxcm13V0ZuV21rUkU1?=
 =?utf-8?B?WDROQTFMdCtTOElRRHhvMDZXL0NHd3g5UHRTaVpQZUhPckFqa0xMTnJtbmJS?=
 =?utf-8?B?c2tDL1duQ25qbnNUcXg3aHlTZ25XVmptZ3JZbjBQLzErdlpnNEFZNkNBV1Bm?=
 =?utf-8?B?WUtXbkNacFc2MU55SHkveXdXYUliT3EwZmtVVFdYU3dQYUhHclBuSE5uUWVV?=
 =?utf-8?B?RDdPZUdDejV0dWlxUEJVVW1GUVZiTjdWek9MOEpyT0xNYkNYTHljelhIa1Nq?=
 =?utf-8?B?b2ZLTFhrTUV2UFYwYkxsd2xmNHo5WXJtYnBLSUplSHBjZmorK3p6WDNPeUZl?=
 =?utf-8?B?TWJQamtEWkt5cTNqb2VnVm4rcDNFUlRCWlh5TWlZNlBERmYzKzZoVFhMa3d1?=
 =?utf-8?B?N2txd0I5VXhUTm1ieXFHZ1FCTW8ydlN0U1hnT21ZMnlFcHdTY0FMUXBkQ1lG?=
 =?utf-8?B?VnVSd1RtZFRmN2MxWGlZRHcrY3lsR2pnSzZacjVrdDBET2dBajhmN0xPK0Nh?=
 =?utf-8?B?cHc9PQ==?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 633d2faa-ee46-49a1-62a7-08dbb5b53d25
X-MS-Exchange-CrossTenant-AuthSource: PH0PR11MB4965.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 15 Sep 2023 06:30:22.4082
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: wkrpyl4emC/As/IAUev8TbeeScSvfvwNGLKPxVG6SFSLMp7bw6vLzhtBDnmvG+w4wXsbQIsl+ojv65Mm2DfERw==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY8PR11MB6939
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-3.6 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,NICE_REPLY_A,
        RCVD_IN_DNSWL_BLOCKED,RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL,
        SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org
X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (snail.vger.email [0.0.0.0]); Thu, 14 Sep 2023 23:31:12 -0700 (PDT)

On 9/15/2023 8:06 AM, Edgecombe, Rick P wrote:
> On Thu, 2023-09-14 at 02:33 -0400, Yang Weijiang wrote:
>> Add supervisor mode state support within FPU xstate management
>> framework.
>> Although supervisor shadow stack is not enabled/used today in
>> kernel,KVM
>           ^ Nit: needs a space
>> requires the support because when KVM advertises shadow stack feature
>> to
>> guest, architechturally it claims the support for both user and
>           ^ Spelling: "architecturally"

Thank you!!

>> supervisor
>> modes for Linux and non-Linux guest OSes.
>>
>> With the xstate support, guest supervisor mode shadow stack state can
>> be
>> properly saved/restored when 1) guest/host FPU context is swapped
>> 2) vCPU
>> thread is sched out/in.
> (2) is a little bit confusing, because the lazy FPU stuff won't always
> save/restore while scheduling.

It's true for normal thread, but for vCPU thread, it's a bit different, on the path to
vm-entry, after host/guest fpu states swapped, preemption is not disabled and
vCPU thread could be sched out/in, in this case,  guest FPU states will  be saved/
restored because TIF_NEED_FPU_LOAD is always cleared after swap.

> But trying to explain the details in
> this commit log is probably unnecessary. Maybe something like?
>
>     2) At the proper times while other tasks are scheduled

I just want to justify that enabling of supervisor xstate is necessary for guest.
Maybe I need to reword a bit :-)

> I think also a key part of this is that XFEATURE_CET_KERNEL is not
> *all* of the "guest supervisor mode shadow stack state", at least with
> respect to the MSRs. It might be worth calling that out a little more
> loudly.

OK, I will call it out that supervisor mode shadow stack state also includes IA32_S_CET msr.

>> The alternative is to enable it in KVM domain, but KVM maintainers
>> NAKed
>> the solution. The external discussion can be found at [*], it ended
>> up
>> with adding the support in kernel instead of KVM domain.
>>
>> Note, in KVM case, guest CET supervisor state i.e.,
>> IA32_PL{0,1,2}_MSRs,
>> are preserved after VM-Exit until host/guest fpstates are swapped,
>> but
>> since host supervisor shadow stack is disabled, the preserved MSRs
>> won't
>> hurt host.
> It might beg the question of if this solution will need to be redone by
> some future Linux supervisor shadow stack effort. I *think* the answer
> is no.

AFAICT KVM needs to be modified if host shadow stack is implemented, at least
guest/host CET supervisor MSRs should be swapped at the earliest time after
vm-exit so that host won't misbehavior on *guest*  MSR contents.

> Most of the xsave managed features are restored before returning to
> userspace because they would have userspace effect. But
> XFEATURE_CET_KERNEL is different. It only effects the kernel. But the
> IA32_PL{0,1,2}_MSRs are used when transitioning to those rings. So for
> Linux they would get used when transitioning back from userspace. In
> order for it to be used when control transfers back *from* userspace,
> it needs to be restored before returning *to* userspace. So despite
> being needed only for the kernel, and having no effect on userspace, it
> might need to be swapped/restored at the same time as the rest of the
> FPU state that only affects userspace.

You're right, for enabling of supervisor mode shadow stack, we need to take
it carefully whenever ring/stack is switching. But we still have time to figure out
the points.

Thanks a lot for bring up such kind of thinking!

> Probably supervisor shadow stack for Linux needs much more analysis,
> but trying to leave some breadcrumbs on the thinking from internal
> reviews. I don't know if it might be good to include some of this
> reasoning in the commit log. It's a bit hand wavy.

IMO, we have put much assumption on the fact that CET supervisor shadow stack is not
enabled in kernel and this patch itself is straightforward and simple, it's just a small
brick for enabling supervisor shadow stack, we would revisit whether something is an
issue based on how SSS is implemented in kernel. So let's not add such kind of reasoning :-)

Thank you for the enlightenment!
>> [*]:
>> https://lore.kernel.org/all/806e26c2-8d21-9cc9-a0b7-7787dd231729@intel.com/
>>
>> Signed-off-by: Yang Weijiang <weijiang.yang@intel.com>
> Otherwise, the code looked good to me.