Received: by 2002:a05:6a10:5bc5:0:0:0:0 with SMTP id os5csp3686208pxb; Wed, 13 Oct 2021 10:49:23 -0700 (PDT) X-Google-Smtp-Source: ABdhPJyuLrhPARyKTaPNzTcg4zW+XMIRql3c/9Pi+kWYPcokScqS3ojSgDvdxh/V6oKjbXA/EiT2 X-Received: by 2002:a05:6402:1d52:: with SMTP id dz18mr1109970edb.49.1634147363621; Wed, 13 Oct 2021 10:49:23 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1634147363; cv=none; d=google.com; s=arc-20160816; b=pZMGJO1laESb1xvcCQxxmWicrq5htcsQiJCJzuwpNMNWFTWSr6LQizfMm355arzMba OPR46kGYyII5zKLsKxCegEtuG8kzZja+fVbzOr8wgmvyToRf3A9EctxX6roPuVfSZkbR TTzRSY+NNdxyVas+XjO0V9RJz+sX6akiiiGwfWRJwxaYwrKxjRRh8SE392OJAMXmkbSc 4nOCJkImvd2XVeSiqOXNZuEOcAa/q0QU9oZE3R1xQzgIgT+DpcL/1V81N1+NFIcSZsZJ Vh3Xl1p1LT8TwcOABaLuX+qFy5D3ymLNh/5zLb4nyppGjX7gOxe3d9mO3NoSYaXSsij7 jfVw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:content-language :in-reply-to:mime-version:user-agent:date:message-id:from:references :cc:to:subject; bh=OHhiY01fPn94B4nPXjRAP7rMP5ARb37ZCKiTSaoR/lk=; b=fFqIySyVYayELGdugIgFbKcNQ2CjrIM19ssLy/ZrN+rNZhF7usx1i+qFt0E9TmoReM OWlxskMH87ZlSxZ0z2TyLqEcYBIW/XPg4O1jwKNoUgO4+hqu6ZVrNHzX1Pd9o4Gtic4U hEV13WSIMlvq77zUL2kEq4nTSLXv5ufxcScPtkOlyOmHj0KLfNIdoun5+AYvDFBXtjGs ENA7RWUwFB76NTcim8f772H14/sv7oUu9cFX3/0lprXpwnemxzrNbvgdRShdZ+YP+ZvE URaxwT7P5UxScz+yNK/WBqCcWsJA5nODTb7EkuAWl4Cx07j72fRvfaRFKhOo40iDYSoA Fmqw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=arm.com Return-Path: Received: from vger.kernel.org (vger.kernel.org. [23.128.96.18]) by mx.google.com with ESMTP id r8si136454edo.627.2021.10.13.10.48.59; Wed, 13 Oct 2021 10:49:23 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18; Authentication-Results: mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=arm.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237959AbhJMRq4 (ORCPT + 99 others); Wed, 13 Oct 2021 13:46:56 -0400 Received: from foss.arm.com ([217.140.110.172]:43090 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229714AbhJMRqz (ORCPT ); Wed, 13 Oct 2021 13:46:55 -0400 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id E85541063; Wed, 13 Oct 2021 10:44:51 -0700 (PDT) Received: from [10.1.196.28] (eglon.cambridge.arm.com [10.1.196.28]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id E710E3F694; Wed, 13 Oct 2021 10:44:50 -0700 (PDT) Subject: Re: [PATCH V2] ACPI / APEI: restore interrupt before panic in sdei flow To: Liguang Zhang Cc: linux-acpi@vger.kernel.org, linux-kernel@vger.kernel.org, Tony Luck , linux-arm-kernel@lists.infradead.org, Borislav Petkov , Len Brown , "Rafael J. Wysocki" References: <20211012142910.9688-1-zhangliguang@linux.alibaba.com> From: James Morse Message-ID: <5951ad5b-d755-0150-0f2a-c567eb454dac@arm.com> Date: Wed, 13 Oct 2021 18:44:49 +0100 User-Agent: Mozilla/5.0 (X11; Linux aarch64; rv:78.0) Gecko/20100101 Thunderbird/78.12.0 MIME-Version: 1.0 In-Reply-To: <20211012142910.9688-1-zhangliguang@linux.alibaba.com> Content-Type: text/plain; charset=utf-8 Content-Language: en-GB Content-Transfer-Encoding: 7bit Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Hello! On 12/10/2021 15:29, Liguang Zhang wrote: > When hest acpi table configure Hardware Error Notification type as > Software Delegated Exception(0x0B) for RAS event, OS RAS interacts with > ATF by SDEI mechanism. On the firmware first system, OS was notified by > ATF sdei call. > > The calling flow like as below when fatal RAS error happens: > > ATF notify OS flow: > sdei_dispatch_event() > ehf_activate_priority() > call sdei callback // callback registered by OS > ehf_deactivate_priority() > > OS sdei callback: > sdei_asm_handler() > __sdei_handler() > _sdei_handler() > sdei_event_handler() > ghes_sdei_critical_callback() > ghes_in_nmi_queue_one_entry() > /* if RAS error is fatal */ > __ghes_panic() > panic() > > If fatal RAS error occured, panic was called in sdei_asm_handle() > without ehf_deactivate_priority executed, which lead interrupt masked. So far the story is: Firmware generated and SDEI event (a kind of software NMI) because of a firmware interrupt, but it hasn't completely handled the interrupt. > If interrupt masked, system would be halted in kdump flow like this: > > arm-smmu-v3 arm-smmu-v3.3.auto: allocated 65536 entries for cmdq > arm-smmu-v3 arm-smmu-v3.3.auto: allocated 32768 entries for evtq > arm-smmu-v3 arm-smmu-v3.3.auto: allocated 65536 entries for priq > arm-smmu-v3 arm-smmu-v3.3.auto: SMMU currently enabled! Resetting... How and why do firmware interrupts affect the IOMMU? It sounds like you are sharing something with firmware that you shouldn't. > After debug, we found accurate halted position is: > arm_smmu_device_probe() > arm_smmu_device_reset() > arm_smmu_device_disable() > arm_smmu_write_reg_sync() > readl_relaxed_poll_timeout() > readx_poll_timeout() > read_poll_timeout() > usleep_range() // hrtimer is never waked. > > So interrupt should be restored before panic otherwise kdump will trigger > error. Why can't firmware finish with the interrupt before injecting the SDEI event? If you need it to not happen a second time while the handler runs, you can always disable it. The text in the spec about the interaction of complete and physical interrupts is for bound interrupts. Linux doesn't support these. It isn't possible for linux to know whether firmware tied any other kind of event to a physical interrupt or not. > In the process of sdei, a SDEI_EVENT_COMPLETE_AND_RESUME call > should be called before panic for a completed run of ehf_deactivate_priority(). SDEI_EVENT_COMPLETE_AND_RESUME is a complete, it tells firmware to restore the execution state from before the event. You get almost get away with x17-x30 being corrupted as panic() won't return - but the stack trace produced will be corrupt. If the original exception was from user-space, SP_EL0 will have been restored to be the user value. The kernel uses this for 'current'. The way this is supposed to work is the die-ing kernel calls SDEI_PE_MASK while it does the kdump reboot. Once the kdump kernel has started, the SDEI_PRIVATE_RESET and SDEI_SHARED_RESET calls should fix anything left over in firmware. Could you debug why firmware interrupts being active prevent the SMMU from being reset. As far as I can tell, those should be totally independent. Thanks, James