Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
Subject: Re: [RFC PATCH 11/13] x86/uintr: Introduce uintr_wait() syscall
To:     Andy Lutomirski <luto@kernel.org>,
        the arch/x86 maintainers <x86@kernel.org>
CC:     Tony Luck <tony.luck@intel.com>,
        Dave Hansen <dave.hansen@intel.com>,
        Thomas Gleixner <tglx@linutronix.de>,
        Ingo Molnar <mingo@redhat.com>, Borislav Petkov <bp@alien8.de>,
        "H. Peter Anvin" <hpa@zytor.com>, Jens Axboe <axboe@kernel.dk>,
        Christian Brauner <christian@brauner.io>,
        "Peter Zijlstra (Intel)" <peterz@infradead.org>,
        Shuah Khan <shuah@kernel.org>, Arnd Bergmann <arnd@arndb.de>,
        Jonathan Corbet <corbet@lwn.net>,
        Raj Ashok <ashok.raj@intel.com>,
        Jacob Pan <jacob.jun.pan@linux.intel.com>,
        "Gayatri Kammela" <gayatri.kammela@intel.com>,
        Zeng Guang <guang.zeng@intel.com>,
        "Williams, Dan J" <dan.j.williams@intel.com>,
        Randy E Witt <randy.e.witt@intel.com>,
        "Shankar, Ravi V" <ravi.v.shankar@intel.com>,
        Ramesh Thomas <ramesh.thomas@intel.com>,
        Linux API <linux-api@vger.kernel.org>,
        <linux-arch@vger.kernel.org>,
        "Linux Kernel Mailing List" <linux-kernel@vger.kernel.org>,
        <linux-kselftest@vger.kernel.org>
References: <20210913200132.3396598-1-sohil.mehta@intel.com>
 <20210913200132.3396598-12-sohil.mehta@intel.com>
 <f5a971e4-6b0d-477f-992c-89110a2ceb03@www.fastmail.com>
From:   Sohil Mehta <sohil.mehta@intel.com>
Message-ID: <c6e83d0e-6551-4e16-0822-0abbc4d656c4@intel.com>
Date:   Tue, 28 Sep 2021 21:56:08 -0700
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0) Gecko/20100101
 Firefox/78.0 Thunderbird/78.14.0
In-Reply-To: <f5a971e4-6b0d-477f-992c-89110a2ceb03@www.fastmail.com>
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 8bit
Content-Language: en-US
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?bkZOUmhGNjMxWWJkT0pwZEdib292akx1MDVNNUltSXMzMUMxSTF3VHFJd3hx?=
 =?utf-8?B?VEJMNXpYMWMyZThTOG44NXpINTM0VE1Hb2dvTHJia0g2UFhDNk1wNko2YkV5?=
 =?utf-8?B?MGR2WXVOc2pTeFFSOG5HQ1BhemM4SnE5T0xGQUNFcFJxR3YrcTdGcVdCazFB?=
 =?utf-8?B?djZFazV4UGIzbWU5MVNYbnJRQVBxNzAvRWpwc2hQeW1iSkdZem9SS0NiOURQ?=
 =?utf-8?B?SXR1MktacnJHVUdhc0RJazRHdmpkeitZeGtON1EyOEpjU0IzVWU1cTJLMTNi?=
 =?utf-8?B?akx2Q25UYkhhWm9STXk0KzRvcng0SVZZZUQ3bU1GZzhtUmJ5YkIzN0JlWU5j?=
 =?utf-8?B?cnl1aVhPeWVzekU5NDJwU1hPTzJEQURVcVE1ZlNMeHk4VFNYWkVIQkNhT04w?=
 =?utf-8?B?Vkx5UURZb1hsdVlYaXNDbWx4c1pLcGxSTGVIYml1UWdJSTU1RENkMHBoMTRJ?=
 =?utf-8?B?NXFQUEpMZHI0TmF2cWNVOTFldm5pMTU0K2RpaVRacXk1U1V4TXpqUDhHQll5?=
 =?utf-8?B?ZXVmc3NrVm91bGVJOHVYZDh3eWxpWEllSzd5cG1PczNsdzArQnYvTHJ1Q2ha?=
 =?utf-8?B?NENId1NRcVZGSGUrSXdzRUg3bmRBVFhPVFVJTWNyMElYNE1rcDBSUEF4bmx0?=
 =?utf-8?B?Zld0MjZPRW83QXJqSUNOdSsrUXBJYk5qZlN4bnhSK2theVNwWkdaYnM3UVpT?=
 =?utf-8?B?Y0Z3dGppMHdkU3VrOWVOUlk4cjVYRkkxcUs2Yk5QdGgrVFY3N3lwUjd1cTVy?=
 =?utf-8?B?Q2creUlQVkp2S2Q1OXhOUHdiMjVpR1NrVXVzbDRjcTNCSUxoeFUwYmpkeE9a?=
 =?utf-8?B?N2cyRlBoQ2kvRnpLc0lVL1hzZy9POVZKdHpTYm15NVpNU2VHTDBIMkU1N2pK?=
 =?utf-8?B?cEc2bnNjSExvV3pPUTBwTyt1VXhwbUxZVUNxL01Fa3hkdVdNSnU4bjB2Ynlm?=
 =?utf-8?B?T25Ia0NWWHJ4d2ZvamNOdVVMb3E0UmVBVkRyRnJuZ09YcWhudnJlTnQvT2N1?=
 =?utf-8?B?eVI5eGliY2JEa2h4ZHZkREZmNjUzUm5aOTJ0SnROd0F5allTRGUvNXE2dS9n?=
 =?utf-8?B?V1BJYzNPRlJMVndBVlRtNUhrd285NUh5WHlXcys3cjlySmhpNjlBenZtaWk2?=
 =?utf-8?B?L0xGWHNIbHFmSjNNY1FrUHFxYm1KTFBBSGxPamtRKzZ1NFVxb1RPVkcxWE5w?=
 =?utf-8?B?bzQ3dktHT3hWcEc4TGZ5OWplQlh0SlZRR0g4ZmNxUW9TNHg4V0pXbE5jQ25u?=
 =?utf-8?B?TTYzd2dHTG04R3k5TXNsZld3TktYdkpQWTQ4ZTNHRHFSMmVYSUVmNUxpNDNp?=
 =?utf-8?B?NlE3YjcxTTAvN0NNcktkYmJMTWZSdkEvVVJCNVA5eWFCR2VzY0V3c05rVEts?=
 =?utf-8?B?MWtvUyt6c0xVcE4vcmQ1R2ZkS3BRNEtwQXkyTHBuZjlWMEY3NjdmenJVSTlD?=
 =?utf-8?B?NURuQks3S3p1Rmc4Y0NDK2JuQ0V0NG5EVDNFcElDRFVsdGN2NTNEZDM3alAx?=
 =?utf-8?B?R3VaWDdZZ0Z4ZElWanpkdGpaRFlpdXlZd2M5c0RoWXRKQXJtQ0ZyamVQekFI?=
 =?utf-8?B?OEhTRkhKbGcvaHR0UzFLYUJ0azBCSVV5ZHY5SXArMEhwYnpWdDIyaFFQWU9F?=
 =?utf-8?B?TUFTeEd2ZW0rdWlRKzA5c2pBdFBkSjUwdmZXMjBGOFZ0YUJ3ZTdkN1RTVXQ1?=
 =?utf-8?B?NC9BTnBRb1BQZ1g3UUV3c0F0NlVFOEErYS85aC9mZ0pkSjg0R0VxYzBpVmFx?=
 =?utf-8?Q?ezgQY2hI2rydY46aXa/wMlZRsSGh4RCu9X6R4Sa?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 248e4fdd-4ce7-46b4-be2e-08d9830574d2
X-MS-Exchange-CrossTenant-AuthSource: BYAPR11MB3320.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 29 Sep 2021 04:56:11.0387
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: pQ7RRFh6mxA8iStN3DA3AtLd2Oh/EG2ymdJhQpSJbZUzBWyu3zadWtWqbrrszHhP6EMwx+tb2gaH/qOQJAvrbA==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ0PR11MB4797
X-OriginatorOrg: intel.com
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On 9/28/2021 8:30 PM, Andy Lutomirski wrote:
> On Mon, Sep 13, 2021, at 1:01 PM, Sohil Mehta wrote:
>> Add a new system call to allow applications to block in the kernel and
>> wait for user interrupts.
>>
> ...
>
>> When the application makes this syscall the notification vector is
>> switched to a new kernel vector. Any new SENDUIPI will invoke the kernel
>> interrupt which is then used to wake up the process.
> Any new SENDUIPI that happens to hit the target CPU's ucode at a time when the kernel vector is enabled will deliver the interrupt.  Any new SENDUIPI that happens to hit the target CPU's ucode at a time when a different UIPI-using task is running will *not* deliver the interrupt, unless I'm missing some magic.  Which means that wakeups will be missed, which I think makes this whole idea a nonstarter.
>
> Am I missing something?


The current kernel implementation reserves 2 notification vectors (NV) 
for the 2 states of a thread (running vs blocked).

NV-1 – used only for tasks that are running. (results in a user 
interrupt or a spurious kernel interrupt)

NV-2 – used only for a tasks that are blocked in the kernel. (always 
results in a kernel interrupt)

The UPID.UINV bits are switched between NV-1 and NV-2 based on the state 
of the task.

However, NV-1 is also programmed in the running task's MISC_MSR UINV 
bits. This is what tells the ucode that the notification vector received 
is for the user instead of the kernel.

NV-2 is never programmed in the MISC_MSR of a task. When NV-2 arrives on 
any cpu there is never a possibility of it being detected as a User 
Interrupt. It will always be delivered to the kernel.

Does this help clarify the above?


I just realized, we need to be careful when the notification vectors are 
switched in the UPID. Any pending vectors detected after the switch 
should abort the blocking call. The current code is wrong in a lot of 
places where it touches the UPID.

Thanks,
Sohil