Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <9ab99257-844b-de15-cd20-7cda61295186@suse.com>
Date:   Tue, 8 Nov 2022 17:26:48 +0100
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:102.0) Gecko/20100101
 Thunderbird/102.4.1
Subject: Re: [PATCH v4] x86/xen: Add support for
 HVMOP_set_evtchn_upcall_vector
Content-Language: en-US
From:   Jan Beulich <jbeulich@suse.com>
To:     Juergen Gross <jgross@suse.com>,
        Stefano Stabellini <sstabellini@kernel.org>
Cc:     Boris Ostrovsky <boris.ostrovsky@oracle.com>,
        Thomas Gleixner <tglx@linutronix.de>,
        Ingo Molnar <mingo@redhat.com>, Borislav Petkov <bp@alien8.de>,
        Dave Hansen <dave.hansen@linux.intel.com>, x86@kernel.org,
        "H. Peter Anvin" <hpa@zytor.com>,
        Oleksandr Tyshchenko <oleksandr_tyshchenko@epam.com>,
        Maximilian Heyne <mheyne@amazon.de>,
        xen-devel@lists.xenproject.org, LKML <linux-kernel@vger.kernel.org>
References: <20220729070416.23306-1-jane.malalane@citrix.com>
 <1918c63f-0065-b468-3691-de6e314f4374@suse.com>
 <e7341a89-208c-8845-fbab-cb0326cc0883@suse.com>
In-Reply-To: <e7341a89-208c-8845-fbab-cb0326cc0883@suse.com>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?eEl3YksydDM4bk1ZckRUL3Y1b2NjWlo4ckRnZERIWmJjUTRGc2k1WkZuRjdF?=
 =?utf-8?B?ZjAwVDgrTEMrUXNqSVpCa2RZeXY5UVpVMFNlaDF5M3c0UlI2OGpIcTNxaUJF?=
 =?utf-8?B?anpCRGprbnhubTNxNVpSZFF5ejIyTjJReWd2aUlONjFtMnkvUUVaNzBUNFZx?=
 =?utf-8?B?SitYSVFVRnRoNXJrR09qaUl1VHU3Q2FyRlRPY0s5Ykcza0w3TUJoeGZNTWU2?=
 =?utf-8?B?RzNGL1d5VFVJbFhoUjVHRjJYTytmMktoVWdzSDJLQnBLMDdvcVhkWWk2ZXJh?=
 =?utf-8?B?MEZ2WmdrUURtOUR0c1NDWDU3UW9kTExWbWpNSFhZNWU3SjBOWU5RYTA5d29X?=
 =?utf-8?B?Sm56OGovRmFZVFY1eUwybHBibjF0OUpKVlhuWGsvUnRUWjRLNlVoV3dlU2FJ?=
 =?utf-8?B?dURkNkk4a0ZFUGk0WEEwZ25haEQrMjBQOGg2VXRxL3ZoWGhEY2Z3Rlkyb1ds?=
 =?utf-8?B?QjB4bmVRU3BMYzhHMnE2cWZUT1ZKSC9WUjNZT0xmamROZ3hHNUlPdlhndjdF?=
 =?utf-8?B?Q2FJYjR4Rktjdm9IU0I3aFEySHpiTmVqaEFqT1BsTXJnUEd5QnBrbnlqc045?=
 =?utf-8?B?RG5KLzZ1aUVKcFBBZWVzUldtK2YzR3lIWVNXVmNjN3VSc3drT25QdEtNWXU3?=
 =?utf-8?B?Vng3MEZVdWZHbUtYVm5ZNXdFTDBhelNlMVlEdlpQQ2FWOG12Vm9ZT2pwWWdO?=
 =?utf-8?B?MjZNT0ZiMUs3Yko2TFAxWU9zWDJhSjNMUDc1QnR0bDhmdnZTb0k0a3RvbVor?=
 =?utf-8?B?YWgzcGlLWUp4MFRCcTRzUFF4ZE1NVFVxT3pEOFVScHNlM3ZSTTB0bEw1REhT?=
 =?utf-8?B?dlJIMnR2YmZoQnoxWUpJYVNzbUI4Um80c3lHUjB5L2p6YXYzRUgxK1RWSTVz?=
 =?utf-8?B?Nmc1YUt0ZU9qYmovMnpXbG4vL2E1VU5nK2p1WHZhZlh5czhESDhRNDhUUXY0?=
 =?utf-8?B?T2pjNDhoMS9pU2lwQTdVMktMdWVMTHpJSXlFTlVIUEhmQUhHUCtud3JjdEhn?=
 =?utf-8?B?bWRKZFlmL0IyWFU5MDliYm1aeERvbXkxSm1nS05VSm51clV2aEtaNlN6ck5q?=
 =?utf-8?B?ZU9wTGUvM3dxdEtlOVd0NE5kclpwZVdVbXJUejhwZ2FyZFh3UkRjS2ltNTA3?=
 =?utf-8?B?NkNNMkdpdGREZkJ4VWE5anpreXpMSmZEemRDZVYxc2pkdEVoUld5RWJqRzE1?=
 =?utf-8?B?d1VYU0NicWwzZzM5N0VudHhQQ0lNQWFPcGs5Zng3a1hVU0tZcDREZzRZV0xx?=
 =?utf-8?B?Ui9XV25mKy92Q0pwK1dTZHorUkk0L3Z1RXVRa2M5NEM5YlFlNWRHQWhvRUxV?=
 =?utf-8?B?TWlITkkvWGo4WUVLdUhyYnRleE5OUG5wNlBraHA3cDhwaStKd3d3OURMSk9J?=
 =?utf-8?B?TEhBWGQ0VjI1SFJkL0pYVFdmL3VVODBETGlKT0VQSy8wUXFKWVpBVUx2S2xs?=
 =?utf-8?B?T2Izb0VGN3RjcGdvR05aQ0ZtZlp6Y1h1TkF4eWZCMjkzdEVnZ3YrOXZWL1NF?=
 =?utf-8?B?S3FlY0pRWEtwVU1XeGFkUmwwejVIUnJEU0wrSjVxU0h4UFhabENrcmx4dC9U?=
 =?utf-8?B?Z3Bad2xYRURwMldWQk5oeTlEK0lzNDNKTmlBNlVwRmZFZ0t4TU9Ka3MvSkcy?=
 =?utf-8?B?c0dlVEdUSDgzOFNaTll3UHpTZlppQnhSUkR4L2QyVXROWnFtUFFVSGd1Mkdj?=
 =?utf-8?B?MTFxNlp6Q1FrYUJHTFgvYzkwLzYvaGpQZVEvOHNFUlB0OVB4WmxmZkhlcGhl?=
 =?utf-8?B?T0NvQXgvVzBxU0orQ2R6ckZ0L0hmaXFLM2pSVlZoZXNGcVd2eUtUdlJYZVBz?=
 =?utf-8?B?d01ZZ0VhTXJRb2ZhRWtiTEhSZmpranMrbmVUMkx2RzdzVU93UjJ5Z0RKM1RH?=
 =?utf-8?B?ekZQZitLRCtKQUZaSy9xUXNReSsydERML0NrWk1Vb1JzKzZvNnJLZi93UVha?=
 =?utf-8?B?a0lkdUNGY3M0L0tBY294Rzk4Vnk4OVlva3RPbGhqR2dJM0FOQ0o4WjNKSzFY?=
 =?utf-8?B?bUZtdFJYUy8vdmNoVVJaVGlRbEZka2NKakpXQmFESWo0VzJyMXRxR3YrVWdi?=
 =?utf-8?B?OGdBRlBZSFRnMzlDTjZkekZMN2x1NFdzeWVTQ1BlSm5ySEhJcVlTVHdUVlVL?=
 =?utf-8?Q?PQdmzKOIHbyMgMEDbvkwNLHO3?=
X-OriginatorOrg: suse.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 30945652-f8ac-4649-07b3-08dac1a60934
X-MS-Exchange-CrossTenant-AuthSource: VE1PR04MB6560.eurprd04.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 08 Nov 2022 16:26:49.0889
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: f7a17af6-1c5c-4a36-aa8b-f5be247aa4ba
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: Vyuiw8/TNV/DaGFR1538DPSaSCgxpaUw6cte0x2qQIzUZGFgnXRQtMox6zxMCZFErHcmA5Tn2B6H4LFGVSVreQ==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DBBPR04MB8042
X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED,
        DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,NICE_REPLY_A,RCVD_IN_DNSWL_NONE,
        RCVD_IN_MSPIKE_H2,SPF_HELO_NONE,SPF_PASS autolearn=ham
        autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On 03.11.2022 16:41, Jan Beulich wrote:
> On 03.11.2022 14:38, Jan Beulich wrote:
>> On 29.07.2022 09:04, Jane Malalane wrote:
>>> @@ -125,6 +130,9 @@ DEFINE_IDTENTRY_SYSVEC(sysvec_xen_hvm_callback)
>>>  {
>>>  	struct pt_regs *old_regs = set_irq_regs(regs);
>>>  
>>> +	if (xen_percpu_upcall)
>>> +		ack_APIC_irq();
>>> +
>>>  	inc_irq_stat(irq_hv_callback_count);
>>>  
>>>  	xen_hvm_evtchn_do_upcall();
>>> @@ -168,6 +176,15 @@ static int xen_cpu_up_prepare_hvm(unsigned int cpu)
>>>  	if (!xen_have_vector_callback)
>>>  		return 0;
>>>  
>>> +	if (xen_percpu_upcall) {
>>> +		rc = xen_set_upcall_vector(cpu);
>>
>> From all I can tell at least for APs this happens before setup_local_apic().
>> With there being APIC interaction in this operation mode, as seen e.g. in
>> the earlier hunk above, I think this is logically wrong. And it leads to
>> apic_pending_intr_clear() issuing its warning: The vector registration, as
>> an intentional side effect, marks the vector as pending. Unless IRQs were
>> enabled at any point between the registration and the check, there's
>> simply no way for the corresponding IRR bit to be dealt with (by
>> propagating to ISR when the interrupt is delivered, and then being cleared
>> from ISR by EOI).
> 
> With Roger's help I now have a pointer to osstest also exposing the issue:
> 
> http://logs.test-lab.xenproject.org/osstest/logs/174592/test-amd64-amd64-xl-pvhv2-intel/huxelrebe0---var-log-xen-console-guest-debian.guest.osstest.log.gz

I've noticed only now that my mail to Jane bounced, and I'm now told
she's no longer in her role at Citrix. Since I don't expect to have time
to investigate an appropriate solution here, may I ask whether one of
the two of you could look into this, being the maintainers of this code?

Thanks, Jan