Received: by 2002:a05:6a10:16a7:0:0:0:0 with SMTP id gp39csp3803172pxb; Tue, 17 Nov 2020 04:06:39 -0800 (PST) X-Google-Smtp-Source: ABdhPJwGhEIEaAPLSHTUJ53+LqgOH/HSI50hvAd4ScbDP/k0felROrJkKIaETUJltvT5+LfHnNIy X-Received: by 2002:a05:6402:b6e:: with SMTP id cb14mr20773472edb.308.1605614799206; Tue, 17 Nov 2020 04:06:39 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1605614799; cv=none; d=google.com; s=arc-20160816; b=EU8Yxmu+GpD+PWfc6yQySB8YNMlWZgfDQD4SaYcpF3jE4ULfZ3Lhkd+3goY2RjxD7H Wv/l3fVJ2S4bT7WpjFgMz+o8FUejZ+Y/y6Kt8OvMotpdqztfAqoTAlPc+jzbdoxT2VXH ewyyhWporkOQ7TsGM+ayCzDXMrH+T53SrYZNsSvvH4OcykTouSt+jbbDQ7LZgXVlP7fi u0Fzcd92pVJ+4SU91E6fLOa02opwC4Neck8ZMUa4ToNCsURMoG3d4JWwU5XP7d0LC7Cl qEYGZMGv0VUlUyKdhVGP/wq+Ddy4JZZZKGfe4fvhsucTtzKuNAzSv3OMfy8gd2DtuKmX tO4g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:subject:message-id:date:from:in-reply-to :references:mime-version; bh=SVhaLNmBtuqnmi4tGMJBvp0RPN7vCBWkGFN0wZbfqIU=; b=RdrJc/A6b1ZE7rIAw+rVwzSXJuAQuOQVSl4hnihfUbQ72nWsBs7iSCGpPPE8XGD+5s iHTAd4lqlaXckR0OTcV6rVopx7uLqpN5LTYBcTP9s5kw7TSW+soG4uImYq25IM5rFkhV nKkFUmvScutfA33b3/jtxeKwml9mZs7UvpPYBw0bCjmrX/PcYhsoOHd1hXqBohs1+0xD n+cQvEHwYuixgc+KnaTN2FL6rTFo5Cm6260SDKzFPzrpbaz+Lfjoi/5Cihpu+9ur++HC sA82DDdpHoA4KyBHng5FlXZwRCNZiAxu4VCTDG/JqCym3iPIOXq86HrC9T2ZV59lri5Z +vRQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=canonical.com Return-Path: Received: from vger.kernel.org (vger.kernel.org. [23.128.96.18]) by mx.google.com with ESMTP id w26si4249901ejn.528.2020.11.17.04.06.14; Tue, 17 Nov 2020 04:06:39 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18; Authentication-Results: mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=canonical.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727885AbgKQMEs (ORCPT + 99 others); Tue, 17 Nov 2020 07:04:48 -0500 Received: from youngberry.canonical.com ([91.189.89.112]:45449 "EHLO youngberry.canonical.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726807AbgKQMEr (ORCPT ); Tue, 17 Nov 2020 07:04:47 -0500 Received: from mail-ed1-f70.google.com ([209.85.208.70]) by youngberry.canonical.com with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.86_2) (envelope-from ) id 1kezj2-0003C6-Jk for linux-kernel@vger.kernel.org; Tue, 17 Nov 2020 12:04:44 +0000 Received: by mail-ed1-f70.google.com with SMTP id l12so9530576edw.11 for ; Tue, 17 Nov 2020 04:04:44 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=SVhaLNmBtuqnmi4tGMJBvp0RPN7vCBWkGFN0wZbfqIU=; b=mbYn/S7r2k4X1/z6EIdCvnSnMPijzE7XKNlRRqhbY1r6+92kZaA/GPHdaz5ti072r0 cn1SVkRQxvV5CnlBO2jcR2YvjfUn7iYyMwR2nTkbQ3tJhkshsjFk7BVNTBbg8Ir1sWqL kz664PliYoi9mJz3Mb4J8J2R9n5oJGIik2QkoKKc2wSRFYm7MvNF87UbzJeWClj8s/p0 KRYVh5wWgfMlGvvrV2F/YANEDlu7XPS0O3DRvHZOHhX+3Q6Lqsw1LdBmNhZlC/Q8f90M i3QIMWqz6FRgnASD3N+DxHsgXdvhX25Fj2JSWdFi8/4NpPbzaDcwjZo1WnrNXJ91Am5/ jJZQ== X-Gm-Message-State: AOAM533bnidDZTm+zwCQrDkKcZRyj7zp+vwkH61uqarHkhASmXhEnbjt A5B34/zhZHzXwPMlDQIjP6EfYLtWcQPNkh0ujV3Czhp8azZUniM+4x/Qq9foBGXd6EemNiW6v9W eQDCSNoVNXiieRGfTQlx5rCGbWDnPw+FRMhzV2lyYmTkBv8+txNeh6QUFdw== X-Received: by 2002:a17:906:fcdb:: with SMTP id qx27mr19442471ejb.470.1605614684124; Tue, 17 Nov 2020 04:04:44 -0800 (PST) X-Received: by 2002:a17:906:fcdb:: with SMTP id qx27mr19442435ejb.470.1605614683864; Tue, 17 Nov 2020 04:04:43 -0800 (PST) MIME-Version: 1.0 References: <20201117001907.GA1342260@bjorn-Precision-5520> <87h7poeqqn.fsf@x220.int.ebiederm.org> In-Reply-To: <87h7poeqqn.fsf@x220.int.ebiederm.org> From: Guilherme Piccoli Date: Tue, 17 Nov 2020 09:04:07 -0300 Message-ID: Subject: Re: [PATCH 1/3] x86/quirks: Scan all busses for early PCI quirks To: "Eric W. Biederman" , Bjorn Helgaas , Thomas Gleixner Cc: lukas@wunner.de, linux-pci@vger.kernel.org, Pingfan Liu , andi@firstfloor.org, "H. Peter Anvin" , Baoquan He , x86@kernel.org, Sinan Kaya , Ingo Molnar , Jay Vosburgh , Dave Young , Gavin Guo , Borislav Petkov , Bjorn Helgaas , Guowen Shan , "Rafael J. Wysocki" , "Guilherme G. Piccoli" , kexec mailing list , LKML , Dan Streetman , Vivek Goyal Content-Type: text/plain; charset="UTF-8" Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Mon, Nov 16, 2020 at 10:07 PM Eric W. Biederman wrote: > [...] > > I think we need to disable MSIs in the crashing kernel before the > > kexec. It adds a little more code in the crash_kexec() path, but it > > seems like a worthwhile tradeoff. > > Disabling MSIs in the b0rken kernel is not possible. > > Walking the device tree or even a significant subset of it hugely > decreases the chances that we will run into something that is incorrect > in the known broken kernel. I expect the code to do that would double > or triple the amount of code that must be executed in the known broken > kernel. The last time something like that happened (switching from xchg > to ordinary locks) we had cases that stopped working. Walking all of > the pci devices in the system is much more invasive. > I think we could try to decouple this problem in 2, if that makes sense. Bjorn/others can jump in and correct me if I'm wrong. So, the problem is to walk a PCI topology tree, identify every device and disable MSI(/INTx maybe) in them, and the big deal with doing that in the broken kernel before the kexec is that this task is complex due to the tree walk, mainly. But what if we keep a table (a simple 2-D array) with the address and data to be written to disable the MSIs, and before the kexec we could have a parameter enabling a function that just goes through this array and performs the writes? The table itself would be constructed by the PCI core (and updated in the hotplug path), when device discovery is happening. This table would live in a protected area in memory, with no write/execute access, this way if the kernel itself tries to corrupt that, we get a fault (yeah, I know DMAs can corrupt anywhere, but IOMMU could protect against that). If the parameter "kdump_clear_msi" is passed in the cmdline of the regular kernel, for example, then the function walks this simple table and performs the devices' writes before the kexec... If that's not possible or a bad idea for any reason, I still think the early_quirks() idea hereby proposed is not something we should discard, because it'll help a lot of users even with its limitations (in our case it worked very well). Also, taking here the opportunity to clarify my understanding about the limitations of that approach: Bjorn, in our reproducer machine we had 3 parents in the PCI tree (as per lspci -t), 0000:00, 0000:ff and 0000:80 - are those all under "segment 0" as per your verbiage? In our case the troublemaker was under 0000:80, and the early_quirks() shut the device up successfully. Thanks, Guilherme