Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
Subject: Re: [RFC PATCH 00/13] x86 User Interrupts support
To:     Stefan Hajnoczi <stefanha@redhat.com>
CC:     <x86@kernel.org>, Tony Luck <tony.luck@intel.com>,
        Dave Hansen <dave.hansen@intel.com>,
        Thomas Gleixner <tglx@linutronix.de>,
        Ingo Molnar <mingo@redhat.com>, Borislav Petkov <bp@alien8.de>,
        "H . Peter Anvin" <hpa@zytor.com>,
        Andy Lutomirski <luto@kernel.org>,
        Jens Axboe <axboe@kernel.dk>,
        Christian Brauner <christian@brauner.io>,
        Peter Zijlstra <peterz@infradead.org>,
        Shuah Khan <shuah@kernel.org>, Arnd Bergmann <arnd@arndb.de>,
        Jonathan Corbet <corbet@lwn.net>,
        Ashok Raj <ashok.raj@intel.com>,
        Jacob Pan <jacob.jun.pan@linux.intel.com>,
        "Gayatri Kammela" <gayatri.kammela@intel.com>,
        Zeng Guang <guang.zeng@intel.com>,
        "Dan Williams" <dan.j.williams@intel.com>,
        Randy E Witt <randy.e.witt@intel.com>,
        Ravi V Shankar <ravi.v.shankar@intel.com>,
        Ramesh Thomas <ramesh.thomas@intel.com>,
        <linux-api@vger.kernel.org>, <linux-arch@vger.kernel.org>,
        <linux-kernel@vger.kernel.org>, <linux-kselftest@vger.kernel.org>
References: <20210913200132.3396598-1-sohil.mehta@intel.com>
 <YVXlQIt/oWQlIupu@stefanha-x1.localdomain>
From:   Sohil Mehta <sohil.mehta@intel.com>
Message-ID: <d0e9d6cd-16ea-173d-36ba-24ab814553b1@intel.com>
Date:   Thu, 30 Sep 2021 17:40:24 -0700
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0) Gecko/20100101
 Firefox/78.0 Thunderbird/78.14.0
In-Reply-To: <YVXlQIt/oWQlIupu@stefanha-x1.localdomain>
Content-Type: text/plain; charset=windows-1252; format=flowed
Content-Transfer-Encoding: 7bit
Content-Language: en-US
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?Windows-1252?Q?HI0rd6/EGpvb7asDPWAPwYaFkxRXRlpvMaRo1gSxY9RBDFXw5EmN7Uzs?=
 =?Windows-1252?Q?uP617BRfSCCaktcb0YA6x4OSHDARWOTX20hambzcfA95+h8zDXIzsF0v?=
 =?Windows-1252?Q?Zi3yteKDICcmvEb04hz1uVxtERcNVEn5IKbeyx1NmPb8PHa6FUEF3/C1?=
 =?Windows-1252?Q?yQaeUngIQ12krMcZh35p9L5A6wLvEDnDHVrqpWI5yWzZGziCPsNrvj2G?=
 =?Windows-1252?Q?oxo68cePPbSDReYNk3mEzkpfhLNOfzw2V2cnKgPNmDue6vNpTsTSOYBT?=
 =?Windows-1252?Q?JWG/uEnamjgLU6F9o2MpsLmQdz1Ou6UR96JniJlT8YKJAS16QKHGUkXd?=
 =?Windows-1252?Q?Tjf8s8X/vWvEJEj9vbMpHf/I52KDhlJoZr0rz2lIX2XBwVFDx1h2zKq0?=
 =?Windows-1252?Q?XCIMPPvnLfp5atj6igFg1oIkRQ3stjEJ666PUk4uDzb/EQSyqBxl6SzN?=
 =?Windows-1252?Q?BhCw8iqOU9qVOWia6+ZvPQ8tzYw/oG/XdR8UZnwH6HzZfrDgYH7r30iV?=
 =?Windows-1252?Q?QfjMKsy0GdJb2NkbrRe74zUNTO/JrCKHvIKRSery8Sge+ecxpihz4ubT?=
 =?Windows-1252?Q?Td5pieh0T2YabUEIyyuz0/fiSjBhb00Lok76dDculQCrVp+xSy8rj+R2?=
 =?Windows-1252?Q?2nuHLnGD0AJyAPzWv7PrQCLAhElzIrHCGojRGWMZl+A7zIu1YBPONvOF?=
 =?Windows-1252?Q?tmmoyU+D3tLk4pfjL77GMqofl/xtrNd9yhMZQRS9K996KYjyifZwp7I9?=
 =?Windows-1252?Q?RK+IWL1ftBABeQNJ/EZVy+AW3pbLg+Iye461iJNDOnXz7bupwRLJxxVU?=
 =?Windows-1252?Q?+yI3bS6xUpWRbuBTs9sIXMpJaSr6JUGWevPbSroOnNkIJsoWzt17EB2Q?=
 =?Windows-1252?Q?TMWyEiGaUe5iMswnxniqaSMBQZe9VOs+11DWeTcUQ59BUo0aSIzBPVmT?=
 =?Windows-1252?Q?TgWrzXncLFSdaJSwAlkUVFqC7R9M2pbkQGOvnZ+/SGZxwjLtF7GF8rIK?=
 =?Windows-1252?Q?u/xqdeQFcllyNYQScgTlhDhdE6jyVqHN8xlRDn0TZgK3zpNUro59VV05?=
 =?Windows-1252?Q?QiW4Cj5EntatLPIArk8ntk9gio97DqseeLU2BsMsJfpYr38I0HpWBuc0?=
 =?Windows-1252?Q?Jy3WeacPo7AwybHb3hdduqzBjc6YFUy6hlGTgjQkEGU4C/qJyyeZaQPq?=
 =?Windows-1252?Q?yeDsggapk21abCS5I7/24/X7VIrtm8b56zRveJOX2fphAZsu572ia9xF?=
 =?Windows-1252?Q?v5tGnYZVx2mY3GSEbsWRdN6mwCG/M5u+wyOhaN6wxj6ExM4M6tWCC1nf?=
 =?Windows-1252?Q?YykAzlFcuqUR941+w1L2jEzDTprfHPHaETz6xzh56A/QrqdrJ/XyQ3/2?=
 =?Windows-1252?Q?ghC8Qvis9orWMlAytxjH2Z7bDwkfQZ7OHYfQ0HXdUZY8BD+BAduJas5b?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 5011ac40-95c7-48d9-d06b-08d984740f93
X-MS-Exchange-CrossTenant-AuthSource: BYAPR11MB3320.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 01 Oct 2021 00:40:26.4902
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: OnjXpLwYwp0fmiglsSVEgESwZkoQjQW6x7UTLHL+GljOw5+SaOkzzNycqImj9UxXpbyYpKiBGf5C6RZJxa7bAw==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BYAPR11MB2760
X-OriginatorOrg: intel.com
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On 9/30/2021 9:26 AM, Stefan Hajnoczi wrote:
> On Mon, Sep 13, 2021 at 01:01:19PM -0700, Sohil Mehta wrote:
>> +------------+-------------------------+
>> | IPC type   |   Relative Latency      |
>> |            |(normalized to User IPI) |
>> +------------+-------------------------+
>> | User IPI   |                     1.0 |
>> | Signal     |                    14.8 |
>> | Eventfd    |                     9.7 |
> Is this the bi-directional eventfd benchmark?
> https://github.com/intel/uintr-ipc-bench/blob/linux-rfc-v1/source/eventfd/eventfd-bi.c

Yes. I have left it unmodified from the original source. But, I should 
have looked at it more closely.

> Two things stand out:
>
> 1. The server and client threads are racing on the same eventfd.
>     Eventfds aren't bi-directional! The eventfd_wait() function has code
>     to write the value back, which is a waste of CPU cycles and hinders
>     progress. I've never seen eventfd used this way in real applications.
>     Can you use two separate eventfds?

Sure. I can do that.


> 2. The fd is in blocking mode and the task may be descheduled, so we're
>     measuring eventfd read/write latency plus scheduler/context-switch
>     latency. A fairer comparison against user interrupts would be to busy
>     wait on a non-blocking fd so the scheduler/context-switch latency is
>     mostly avoided. After all, the uintrfd-bi.c benchmark does this in
>     uintrfd_wait():
>
>       // Keep spinning until the interrupt is received
>       while (!uintr_received[token]);

That makes sense. I'll give this a try and send out the updated results.

Thanks,
Sohil