Message-ID: <3918b4e8-d612-8394-dad3-78d13218b4cb@intel.com>
Date:   Thu, 18 Nov 2021 13:44:55 -0800
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:91.0) Gecko/20100101
 Firefox/91.0 Thunderbird/91.3.1
Subject: Re: [RFC PATCH 00/13] x86 User Interrupts support
Content-Language: en-US
To:     Prakash Sangappa <prakash.sangappa@oracle.com>
CC:     "x86@kernel.org" <x86@kernel.org>, Tony Luck <tony.luck@intel.com>,
        "Dave Hansen" <dave.hansen@intel.com>,
        Thomas Gleixner <tglx@linutronix.de>,
        "Ingo Molnar" <mingo@redhat.com>, Borislav Petkov <bp@alien8.de>,
        "H . Peter Anvin" <hpa@zytor.com>,
        Andy Lutomirski <luto@kernel.org>,
        Jens Axboe <axboe@kernel.dk>,
        Christian Brauner <christian@brauner.io>,
        Peter Zijlstra <peterz@infradead.org>,
        Shuah Khan <shuah@kernel.org>, Arnd Bergmann <arnd@arndb.de>,
        Jonathan Corbet <corbet@lwn.net>,
        Ashok Raj <ashok.raj@intel.com>,
        Jacob Pan <jacob.jun.pan@linux.intel.com>,
        "Gayatri Kammela" <gayatri.kammela@intel.com>,
        Zeng Guang <guang.zeng@intel.com>,
        "Dan Williams" <dan.j.williams@intel.com>,
        Randy E Witt <randy.e.witt@intel.com>,
        Ravi V Shankar <ravi.v.shankar@intel.com>,
        Ramesh Thomas <ramesh.thomas@intel.com>,
        Linux API <linux-api@vger.kernel.org>,
        "linux-arch@vger.kernel.org" <linux-arch@vger.kernel.org>,
        LKML <linux-kernel@vger.kernel.org>,
        "linux-kselftest@vger.kernel.org" <linux-kselftest@vger.kernel.org>
References: <20210913200132.3396598-1-sohil.mehta@intel.com>
 <D4A544CE-15E8-4C9A-9245-8B0146E272F3@oracle.com>
From:   Sohil Mehta <sohil.mehta@intel.com>
In-Reply-To: <D4A544CE-15E8-4C9A-9245-8B0146E272F3@oracle.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?RmxhbW91MG1BOGxZZ0JiZkNub0lmcDhyMlZYUm1raUU0MjFsOXYwdit2SHVi?=
 =?utf-8?B?aUxzcWk5QlRYMjdvWktKL3o2NnF1blR4M0ZBY1A5bVNVM0M3ZmVIRDAyWXMw?=
 =?utf-8?B?azVkQy81cE1obFJrb2ZML0MxY1pja2JaeG95dXVZL0hnYjJPYnhMQkw5bW85?=
 =?utf-8?B?eVZGR1Foa3NJbElTOUZJMEJJL2hLalRTZ013UXJ3Z0ZMMk8ySzJ4OERkdG5K?=
 =?utf-8?B?eHJmOGRvckIzUGVsVjM0RGpYeWg3MmxoT2RqbGxwTkg3R0pRcjcxTjJDTE5o?=
 =?utf-8?B?RzlKMnphYU5PeW9RRjdBNnBXcnl1dWlDcUtoV3R3Mm5PSll2YndSUEtONjBo?=
 =?utf-8?B?a1VLUVNJQ3p6QUNHaEpjQ1BUS1NDTTR2dkVGWVEyQmpoMjVaZlo2a1VwQXJr?=
 =?utf-8?B?eTZYNC9TNlhydjlOblRNeDE4WjBZTll0RVJaaDA2bDNRYU52K1RJNGg4V28z?=
 =?utf-8?B?Q2hsYTJtOHd3YUNnU3BMdk5sOVdXVENNeEdDaEFCeHI0ZUwwb1RjN1dIQXl3?=
 =?utf-8?B?MFJtMlFnalhxN0lqelhlbmJ3ZHJTK01hQndjMTB6VWYzVlJmVHFMWGZEeEpt?=
 =?utf-8?B?NkQyNC9qV3pOMXl5TDV3NHN5L3hlMHVXNk91dHJVRGlhaTJSSDBDaWYyQXcr?=
 =?utf-8?B?M2lmbWhqQjZpLzdrVzliYmNxKzdMT1Q0WnB6V2JyWC9aenc3ajVpdjRZd0N3?=
 =?utf-8?B?ZDFUbTNKamtUazVFRE9YUkNwb0s4V3pDcFM4UDkvdVk3L05xaXlHVTg4UTVC?=
 =?utf-8?B?ZFA5N093cDBqT0xqa3hQbmVleDE4RmxHTGVXZ2VYZTJpajNnLzcrc3FlZ1lo?=
 =?utf-8?B?cFB3TXcxL3Ixb1NydGpxdkFPbTFWeUdWOStIb3oyeVJaVC9qVVRWZTZMV290?=
 =?utf-8?B?UUxDYUdialdkMTBJNFRlMVcwVnJrMzB4K3BRQXZvMmJUUnRWLzRNZGFidEhS?=
 =?utf-8?B?WmRFd3dlbGpMWldod2RXOWJYNjZPRzY3WExuek0xWHJkdmI3QjBTVDFycnUx?=
 =?utf-8?B?QkhMNXpDcStTMnpLdm1hbkIvOTlKNXQ2cGk1TjgyL2duT2VTRGpYajJSWWFP?=
 =?utf-8?B?Vkp3ZHhKUFE1dE9OR2Q5cnRSZVFPOUdYc1g2WWs5NGlnNVVtLzFvSHpvZkdk?=
 =?utf-8?B?c1NlU2dPelI3d255dERyV0VNRUFucmw1K0JPb0VPSFBoMWRYSGxTbHRSa0FW?=
 =?utf-8?B?UDZicDJwS1A3M1V0c201WUJBazYveFFLNVQ0V0Y3R0V1UkMzNDhpQ0dEQmxT?=
 =?utf-8?B?MkoyU2toa2tXcmY0bzRkZFhia0UvcHdPSWgyYlhLNVU5WkhVb1lRbGJRSEtH?=
 =?utf-8?B?T21qK0NEbSthb2JpN1RIZmduNVBjWTYrdzJMU05oMnFJVmJmSTZUK0IzQ3F0?=
 =?utf-8?B?NHFEVXg2cVBQazgyZnF6dDFtRjdVaHdXajE4dnAxOUdrSzFlNEZGdHRnMTVK?=
 =?utf-8?B?VUc3TDN3WWV2N2JGWVVGTkIxelZCNG9LZU5iTXdOV1ZGRVFqZ2VyUG1ud3BR?=
 =?utf-8?B?Rm9Vd2VKTXpGbGdNODE4clZhcTJhOGdwTnVRcUZEQ0Fsc3Jld0dPMkpHVnJK?=
 =?utf-8?B?bm9xR3lYaDQwTzlETmNXVkk4MVdWSU9FS25zWXQ5b3puTHNmK1lZWnRKeTZG?=
 =?utf-8?B?aklHV1FRcGhYTzBibFhGdUZFMWFBdE1KMm5sK2c4ZEVDTkJTbTdGTVlYT0hF?=
 =?utf-8?B?NHg1UE1YbWFWRExLa3llOGxHMWxvejNHaVB5Q20zamYvQkFhVGgwM29iWmU4?=
 =?utf-8?B?UHo4amxBcUNRTEJWb1hEZTArcUJTTElOanRwWHhpNDgzSlhYZkk0ZFVPTHFv?=
 =?utf-8?B?OXRWTWNnOC9zcCtyOUFkU0UxWk1PNHkrd2U2NFhPTHV5aU9Kd29mWDZoSVNp?=
 =?utf-8?B?YjRJZHVmWjR5ZzI3aklVWUlvdVJ3S0NaaWlVMkZUcGh6dklQZUg1Z1RERHMv?=
 =?utf-8?B?bWJBRnJHN0ZrV3hNZDhFK2hldXVyNjFNeWZmZVM3bzY1cU9tZXhjbEo1R1o4?=
 =?utf-8?B?OGdST2J6UER5K0NUdXMwY3RrWW9rUjVIeXFKVkJqaFpMc1QxTk9qVU5CZm13?=
 =?utf-8?B?UmZzUnZUMENjYlBJUnlwVU5xRldiWDVXL1VkSDh4RjJqNU9EYTR5eU92MHdQ?=
 =?utf-8?B?TnBMdmZ2M2lLR284R2ttMHR1anhjMUFzbGY2ZEthNjV6NFBZRlFOdnpkNFpH?=
 =?utf-8?Q?p9nXFu6qY+ka3NRYM16N9YQ=3D?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 4dbad686-aab4-4503-6233-08d9aadcaa54
X-MS-Exchange-CrossTenant-AuthSource: BYAPR11MB3320.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 18 Nov 2021 21:44:58.0047
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: cjEmyCj/bKf81ljpQVpqTyBxAmsM5f0un9/yqzWydj2hmEacoqeuPEvlH0QpPhi6K5VVCZaQ+qGEy8GVftMvAw==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BYAPR11MB3045
X-OriginatorOrg: intel.com
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On 11/15/2021 7:49 PM, Prakash Sangappa wrote:
> 

> Here are some use cases received from our Databases(Oracle) group.

Thank you Prakash for providing the potential use cases. This would 
really help with the design and validation of the UINTR APIs.

> 
> Aim is to use user interrupts as one mechanism, for fast IPC and to signal
> target thread blocked in the kernel in a system call.
> i.e replace use of signals with user interrupts.
> 

Mimicking this signal behavior would likely add some complexity to the 
implementation. Since there is interest, we'll work on prototyping this 
to evaluate tradeoffs and present them here.

> Following enhancements with respect to sharing UITT table will be beneficial.
> 
> Oracle DB creates large number of multithreaded processes. A thread in a
> process may need to communicate(using user interrupts) with another
> thread in any other process. Current proposal of receiver sending an FD
> per vector to each of the sender will be an overhead. Also every sender
> process/thread allocating a sender table for storing same receiver UPIDs
> will be duplication resulting in wasted memory.
> > In addition to the current FD based registration approach, having a way
> for a group of DB processes to share a sender(UITT) table and  allowing
> each of the receiver threads to directly register itself in the shared UITT
> table,  will be efficient. For this the receiver need not create an fd. The
> receiver’s UPID index in UITT got from the registration will  be shared
> with all senders via shared memory(IPC).
> 

Sharing the UITT between tasks of the same process would be relatively 
easier compared to the sharing the UITT across processes. We would need 
a scalable mechanism to authenticate the sharing of this kernel resource 
across the process boundary.

I am working on a proposal for this. I'll send it out once I have 
something concrete.

> DB maintains a process table of all the DB processes/threads in the shared
> memory. The receiver can register itself in the shared UITT table and store
> its UPID index in the process table. Sender will lookup target process from
> the process table to get the UITT index and send the user interrupt.
> 

Thanks,
Sohil