Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
Subject: Re: [RFC PATCH 11/13] x86/uintr: Introduce uintr_wait() syscall
To:     Andy Lutomirski <luto@kernel.org>,
        Thomas Gleixner <tglx@linutronix.de>,
        the arch/x86 maintainers <x86@kernel.org>
CC:     Tony Luck <tony.luck@intel.com>,
        Dave Hansen <dave.hansen@intel.com>,
        Ingo Molnar <mingo@redhat.com>, Borislav Petkov <bp@alien8.de>,
        "H. Peter Anvin" <hpa@zytor.com>, Jens Axboe <axboe@kernel.dk>,
        Christian Brauner <christian@brauner.io>,
        "Peter Zijlstra (Intel)" <peterz@infradead.org>,
        Shuah Khan <shuah@kernel.org>, Arnd Bergmann <arnd@arndb.de>,
        Jonathan Corbet <corbet@lwn.net>,
        Raj Ashok <ashok.raj@intel.com>,
        Jacob Pan <jacob.jun.pan@linux.intel.com>,
        Gayatri Kammela <gayatri.kammela@intel.com>,
        Zeng Guang <guang.zeng@intel.com>,
        "Williams, Dan J" <dan.j.williams@intel.com>,
        Randy E Witt <randy.e.witt@intel.com>,
        "Shankar, Ravi V" <ravi.v.shankar@intel.com>,
        Ramesh Thomas <ramesh.thomas@intel.com>,
        Linux API <linux-api@vger.kernel.org>,
        <linux-arch@vger.kernel.org>,
        "Linux Kernel Mailing List" <linux-kernel@vger.kernel.org>,
        <linux-kselftest@vger.kernel.org>
References: <20210913200132.3396598-1-sohil.mehta@intel.com>
 <20210913200132.3396598-12-sohil.mehta@intel.com>
 <f5a971e4-6b0d-477f-992c-89110a2ceb03@www.fastmail.com>
 <c6e83d0e-6551-4e16-0822-0abbc4d656c4@intel.com>
 <fd54f257-fa02-4ec3-a81b-b5e60f24bf94@www.fastmail.com> <877dex7tgj.ffs@tglx>
 <b537a890-4b9f-462e-8c17-5c7aa9b60138@www.fastmail.com> <87tui162am.ffs@tglx>
 <25ba1e1f-c05b-4b67-b547-6b5dbc958a2f@www.fastmail.com> <87pmsp5aqx.ffs@tglx>
 <0364c572-4bc2-4538-8d65-485dbfa81f0d@www.fastmail.com>
From:   Sohil Mehta <sohil.mehta@intel.com>
Message-ID: <d9515c72-d8c5-eba4-56d1-1779fb751908@intel.com>
Date:   Fri, 1 Oct 2021 11:04:22 -0700
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0) Gecko/20100101
 Firefox/78.0 Thunderbird/78.14.0
In-Reply-To: <0364c572-4bc2-4538-8d65-485dbfa81f0d@www.fastmail.com>
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 7bit
Content-Language: en-US
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?R2k4OUpRWUVFS2dpTC9iQ25yV29sWG9UVjZsTkVVd1dvVDE4Snc5VXZ0L0V5?=
 =?utf-8?B?ZGNwY0tUWGx1Y1pBcXI4VmdHWm95QWxrdDFuUytEOUV3MUFaajRTR2dDQVFM?=
 =?utf-8?B?a01BRHlVWkZOb2NtN1I4S1ZSOVo1VUFaajQ4bkp6UHQ0azNnWXMwWms2c1pw?=
 =?utf-8?B?TkNUTEhzYnRFdzFEdElUZjhYclVNZHJHZzUvV29vN3pvRjFoQlpUb3lRclU1?=
 =?utf-8?B?SXhPU3NHQldCSm13eXhVZjhVNE5sVWtLajdjb05UdUpOd2Vva0NuenUzaE5R?=
 =?utf-8?B?ZVdZUEFyYVZWNGQ2ZlAvNkJMN1hmaDdIOWlDVUpVcmdFMVpFVEkzc1NhRklV?=
 =?utf-8?B?eG1tbHFRN1lCQmVRZzNWcHBzVzBqVHFtS2hNV1pmbVZLLzJ6S1JrZFU0Zk1W?=
 =?utf-8?B?bFl4WjZrTWViZ2cyUU0wWWxLZXFOL0NjVlFGT1lWR2VjbkN6UVh3RUlNNDJV?=
 =?utf-8?B?ckFiOTBxaUFhRmxBMnVMRHE3azcrNHAweWhtUXM0RWc4VldpeGIzTG9xanZV?=
 =?utf-8?B?dSt2dkppcmVTUnJ0ejlwSW5jMWtHQm9sKzc1bDY4T3B0a3lOUlVTTkNYQXBP?=
 =?utf-8?B?V05wS0EwcmtURnV4cWVyQjMxbW9FZmRDc0JncjhOWm1aL0d2b0MvRkVtSWg0?=
 =?utf-8?B?bnN4eGlmU1lPZU55VjhWWDh3eHJ0clVjT3VPazdoMitSTjV0SzlXc211elBL?=
 =?utf-8?B?M2xkR1pzNFpIVzNlMlFwZTVEOHk3MFNjYm05SzdDUEhLdGtiSTRUNzFYSUVt?=
 =?utf-8?B?VDZhNVZUOTNJUjFOV09jQWpjWEJuMDZTbmEzMERPdFJmMWxQcmRmR083QUlC?=
 =?utf-8?B?cGllaGRJWGFqa1pUVHJWS3NRRUgraDhUbVkyUUY1YVc5SWhBUFZncS9saU9T?=
 =?utf-8?B?K1BGQkw0UGMycjhmTGJNa2wwV0lXTU5TcTAybWllK3RaQlAyNDFoNWNuNDBB?=
 =?utf-8?B?WStySjQ3QVZ1dU5vczVuQXhIR0pidGgzQk9ZbmlnckVJd0xWR1dzcjVhc3pJ?=
 =?utf-8?B?TFFielkzSk5IMmp2cTJoSjBEWnplZzBLejIvUTNXdkhkcjlqVTgxNWdIRzBo?=
 =?utf-8?B?RGJLaDlOVkRnRDhlQlBWOWZ0VTRtVTZqNDY5K3dudFJIZUpDZGRQeW40SHZx?=
 =?utf-8?B?MkRpc2JZZ0xJQk42b3ZOT0VXdHc3M2lSZE1aSlY3Vk5mZklpdVhuOEZiMUg5?=
 =?utf-8?B?VkY3d1ZyeWhZQTU5QUtqcGR2dUg2SWZRd2U2d2p6VFJoTGdvMUd2V2ZSQkdB?=
 =?utf-8?B?ZHZ2ZVJlN0QrS2kzUlFXWGtzaWdFY1RFa1RGbWl4RHI1bm91cEdTTWNvak40?=
 =?utf-8?B?RC8zWi81a2RoTW8vSklxM2FvU2U2MW5qRGNiK1lHVjZWMlVaOVBmUEF6ckNM?=
 =?utf-8?B?cENPUGEvQ0h6ZlR3ZVZ0eWdodlRZTlplSkJnTXYzTXMrNVIzRUdzVGpzWGRG?=
 =?utf-8?B?NGlWb0ZnM1BlMGtqTEtJTXlhR1FHUGdiWW5oMmdhWmxWMi9yaytPck13bTA2?=
 =?utf-8?B?dmwwcERRSXhXZlFwV09QZW9CVXFnS2J6dG0wNFhUcld4VFo2NFlXSWcyRHds?=
 =?utf-8?B?VEFvWGF0eUwvL0dNdHlpVktYS283Ry9RSktYcGNCeHR5LzZLcUxiT3U2eVFo?=
 =?utf-8?B?a2cxU2ZtQWNiTWxWb3d6Z1ViT3RtMnNWM05mUmRtbTdrbU9xRGE5UFBVM0I4?=
 =?utf-8?B?UnBxMWYyWGIvYlcxMlR2MGFYN1VwZ0l1bnRyTm1nd0ZNdVU5U0g5ZzkvL0Rn?=
 =?utf-8?Q?1kmBps8xfapdgrgAHBGAsB6L7Fatg9H19QsvgKi?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 216b2c30-a73e-4c9d-da5f-08d98505e6c5
X-MS-Exchange-CrossTenant-AuthSource: BYAPR11MB3320.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 01 Oct 2021 18:04:24.6610
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: 9S99FOFHTUwVtnJ2PmtWvBpT+S5Axj2MhGEL+8nc1q7Ev7wEjy8RiuKpxB5pNLxjEXw8eCZLRXBLPjvZ6hpj4Q==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BYAPR11MB2936
X-OriginatorOrg: intel.com
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On 10/1/2021 8:13 AM, Andy Lutomirski wrote:
>
> I'm just reading TFM, which is some kind of dystopian fantasy.
>
> 11.8.2.4 XRSTORS
>
> Before restoring the user-interrupt state component, XRSTORS verifies that UINV is 0. If it is not, XRSTORS
> causes a general-protection fault (#GP) before loading any part of the user-interrupt state component. (UINV
> is IA32_UINTR_MISC[39:32]; XRSTORS does not check the contents of the remainder of that MSR.)
>
> So if UINV is set in the memory image and you XRSTORS five times in a row, the first one will work assuming UINV was zero.  The second one will #GP.  And:
>
> 11.8.2.3 XSAVES
> After saving the user-interrupt state component, XSAVES clears UINV. (UINV is IA32_UINTR_MISC[39:32];
> XSAVES does not modify the remainder of that MSR.)
>
> So if we're running a UPID-enabled user task and we switch to a kernel thread, we do XSAVES and UINV is cleared.  Then we switch back to the same task and don't do XRSTORS (or otherwise write IA32_UINTR_MISC) and UINV is still clear.

Andy,

I am still catching up with the rest of the discussion but I wanted to 
provide some input here.

Have you had a chance to look at the discussion on this topic in patch 5?
https://lore.kernel.org/lkml/87bl4fcxz8.ffs@tglx/
The pseudo code Thomas provided and my comments on the same cover the 
above situation.

The UINV bits in the IA32_UINTR_MISC act as an on/off switch for 
detecting user interrupts (i.e. moving them from UPID.PIR to UIRR). When 
XSAVES saves UIRR into memory we want the switch to atomically turn off 
to stop detecting additional interrupts. When we restore the state back 
the hardware wants to be sure the switch is off before writing to UIRR. 
If not, the UIRR state could potentially be overwritten.

That's how I understand the XSAVES/XRSTORS behavior. I can confirm with 
the hardware architects if you want more details here.

Regarding the #GP trick proposal, I am planning to get some feedback 
from the hardware folks to see if any potential issues could arise.

I am on a pre-planned break next week. I apologize (in advance) for the 
delay in responding.

Thanks,
Sohil