Received: by 10.192.165.148 with SMTP id m20csp4900339imm; Tue, 1 May 2018 05:59:49 -0700 (PDT) X-Google-Smtp-Source: AB8JxZojrcdreW4JMa4QLq30sPSnj0FiEv1yKy78DgoQ0Jg/eIQp5E9Sen3M7JEccl/QmWJ+ze27 X-Received: by 2002:a17:902:6b8b:: with SMTP id p11-v6mr15845752plk.212.1525179588955; Tue, 01 May 2018 05:59:48 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1525179588; cv=none; d=google.com; s=arc-20160816; b=tbOv6oQdXj9PoPGpxuwPwbu50ZjMlKHtRoUJvfMubj3l1/u8Y7SXTWCnShZNWpjgHh jcyiIcNpPl/nYdstHqwQPcBiGIGc0iJASg9xLxhF+toS3wJH+1SXEV5oJ7HFHXzFSXmN IIAPoyv8OJGCcBWIdakKxf/vsdQKCaciSFzHVuVDehoRqHSfmp6XSWuQkAZ3YIQT3qlO MKDAakacQ/s60KEFmPqkG4RXq7q006d4oQj934KmHL0cvy/m7wwb48YMB5qf2HzZ1pkK o4Lp21ehY1fqeHEITgcsdLh9hydjfpGITUkgxIrTQVPajnz0md2Z/rwGKcqFfUJiE5gA RDLw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:content-transfer-encoding :content-language:in-reply-to:mime-version:user-agent:date :message-id:organization:from:references:cc:to:subject :arc-authentication-results; bh=EskXttwsdMCR83GKwyA5Pyua5EXIuNszrGlequKwvcU=; b=jR+cFPODP31/A0rIfWM3KtwWy7Abk/iRUV6Vl/9hUw7It3oyyyPVIO4Wdwr/pWocPK ZbWaovad5A6U2oSHJBwSPXwoNwUgCOdzrES4Fo/UhlrxxUxppwwPmyZG6gby6NgNgciB M8hLRszV75ms4hQlPxoF8gml+FqNwljr6nEu24rWiuhK6hQOCo9ZFoAI2/4GyWgmamd/ lKiNT2E89L3XyNgCk9yv1EEofHe+q7kn2mx2Ys78WDxFR/gSkh1b88g65zQCj2WasV// RX1iQFqKFsSai23Kf9kx8HTUMlIBNWxA0gjaoEDd4EwjqkwHPRbRlfSW3yvV3HjYPZtx cqUA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id w22-v6si9069823plq.115.2018.05.01.05.59.34; Tue, 01 May 2018 05:59:48 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1755453AbeEAM7Z (ORCPT + 99 others); Tue, 1 May 2018 08:59:25 -0400 Received: from usa-sjc-mx-foss1.foss.arm.com ([217.140.101.70]:46830 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1755015AbeEAM7Y (ORCPT ); Tue, 1 May 2018 08:59:24 -0400 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id CCC29F; Tue, 1 May 2018 05:59:23 -0700 (PDT) Received: from [10.1.206.75] (usa-sjc-imap-foss1.foss.arm.com [10.72.51.249]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id EBFC73F5A0; Tue, 1 May 2018 05:59:21 -0700 (PDT) Subject: Re: pciehp 0000:00:1c.0:pcie004: Timeout on hotplug command 0x1038 (issued 65284 msec ago) To: Sinan Kaya , Bjorn Helgaas Cc: Paul Menzel , Dave Young , linux-pci@vger.kernel.org, kexec@lists.infradead.org, linux-kernel@vger.kernel.org, Lukas Wunner , Eric Biederman , Bjorn Helgaas , Vivek Goyal References: <8770820b-85a0-172b-7230-3a44524e6c9f@molgen.mpg.de> <20180427192207.GG8199@bhelgaas-glaptop.roam.corp.google.com> <20180427211255.GI8199@bhelgaas-glaptop.roam.corp.google.com> <20180428005620.GB1675@dhcp-128-65.nay.redhat.com> <20180428011845.GC1675@dhcp-128-65.nay.redhat.com> <3ebc908fb196168bf0373875ffc5679e@codeaurora.org> <20180430211740.GG95643@bhelgaas-glaptop.roam.corp.google.com> <7285da70-2c3e-c3b7-62e1-fdbb55a77729@codeaurora.org> <3549ffe8-7605-d72c-5c09-1436a4288c7d@codeaurora.org> From: Marc Zyngier Organization: ARM Ltd Message-ID: Date: Tue, 1 May 2018 13:59:20 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.7.0 MIME-Version: 1.0 In-Reply-To: <3549ffe8-7605-d72c-5c09-1436a4288c7d@codeaurora.org> Content-Type: text/plain; charset=utf-8 Content-Language: en-GB Content-Transfer-Encoding: 7bit Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 01/05/18 13:38, Sinan Kaya wrote: > +Marc, > > On 4/30/2018 5:27 PM, Sinan Kaya wrote: >> On 4/30/2018 5:17 PM, Bjorn Helgaas wrote: >>>> What should we do about this? >>>> >>>> Since there is an actual HW errata involved, should we quirk this >>>> root port and not wait as if remove/shutdown doesn't exist? >>> I was hoping to avoid a quirk because AFAIK all Intel parts have this >>> issue so it will be an ongoing maintenance issue. I tried to avoid >>> the timeout delays, e.g., with 40b960831cfa ("PCI: pciehp: Compute >>> timeout from hotplug command start time"). >>> >>> But we still see the alarming messages, so we should probably add a >>> quirk to get rid of those. >>> >>> But I haven't given up on the idea of getting rid of the >>> pciehp_remove() path. I'm not convinced yet that we actually need to >>> do anything to shut this device down. I don't like the assumption >>> that kexec requires this. The kexec is fundamentally just a branch, >>> and anything we do before the branch (i.e., in the old kernel), we >>> should also be able to do after the branch (i.e., in the kexec-ed >>> kernel). >>> >> >> In my experience with kexec, MSI type edge interrupts are harmless. >> You might just see a few unhandled interrupt messages during boot >> if something is pending from the first kernel. Unfortunately, that's not always the case. A number of GICv3/v4 implementations (a very common interrupt controller on ARM servers) cannot be disabled, which means they will keep writing to their pending tables long after kexec will have started the new kernel. And since we don't track memory allocation across kexec, you end-up with significant chances of observing single bit corruption as interrupts carry on being delivered. Oh, and you won't actually be able to take MSIs because you can't even reprogram the damn thing. Yes, this can be considered a HW bug. >> It is the level interrupts that are more concerning. It remains pending >> until the interrupt source is cleared. CPU never returns from the >> interrupt handler to actually continue booting the second kernel. > > This makes me wonder why kexec doesn't disable all interrupt sources by > itself instead of relying on the drivers shutdown routine. Some drivers > don't even have a shutdown callback. Kexec could have done both as another > example. Something like. > > 1. Call shutdown for all drivers if available. > 2. Disable all interrupt sources in the interrupt controller > 3. Start the new kernel. See above. Although you can shut off the end-point and to some extent mask interrupts before jumping into the payload, it is not always possible to go back to a reasonable state where you can take actually MSIs. Thanks, M. -- Jazz is not dead. It just smells funny...