Received: by 2002:a05:7412:40d:b0:e2:908c:2ebd with SMTP id 13csp1045799rdf; Wed, 22 Nov 2023 04:23:20 -0800 (PST) X-Google-Smtp-Source: AGHT+IHR/lz6dQ9qPJChs2Q5UcDnWGCtg0ajKlToco0nKgrDZ7bVqjU7TlKvJDCfT+73Mgi/cPRs X-Received: by 2002:a17:902:e5c2:b0:1cc:43af:f568 with SMTP id u2-20020a170902e5c200b001cc43aff568mr2063466plf.6.1700655799686; Wed, 22 Nov 2023 04:23:19 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1700655799; cv=none; d=google.com; s=arc-20160816; b=WvSqL2z29da7YmdwLlKSKGbXioC3fi951B6ko75JDuPj+cInzSjFSyVdehwlQBokwo GP3aT2aMy9Iu/xHwJTxi6BPK7Eoyd/E4x9ro+iyxXrEdEAvf6X6bD0w1p23wBNjKpzl0 SDTZgatLihHygn8D7474dRpDE3BnSlvnqPs5B/VXRB44feRul6ZBCUtHJQUqsA4seI1s VY/mmEILG/Q9FeJan5Wk/AuJ8MY9DojAmf2jh9mBpj2UCO0BsWzxRIoWjgsOGaFCmkK4 EEm//wItChn7SDHjN2JY0iF3P+JLY9FPLLHz5ZuLyakhLJh1rACy4TmOE0iXC3O7gKdS DMpQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :message-id:date:references:in-reply-to:subject:cc:to:from :dkim-signature; bh=PYMteO/8nXAEz+nEcUViWhq7sS5PnhcbZf0mu+EALEg=; fh=f0t/BMND6bhW7L6QZ0SxZJK9ICuGXpfJg+JyVXZ4eew=; b=rbzADeI409flhRRrNopSGfIuiCt0uhgTz+Lk9BGKkvSaLDcfxL6utWkN6z3tHWSvE/ nQc9dVSD9MmaunxKKXEzAlV0cwxdJkzQEtC+1vjNkBjO0NErHENnTnncNL2Wehz6gzfT eClOOVaaEwerbp9v7dcsMaJg0jN2oDLoT9ZEBs+vP7hXAp2d0SIJsJfHjtLMY+jynMDO 5Qc76TLbTPX+dLDtNqB/15EIA0hnEwAwmV0od/stywUKlvAbyXTecR+EOLkifT0+Vdva 5KE2g8i4ni8PVo4UYM3GRQ7XSfjFZgtmgy6E4fIi2UUNBrdyD/ojQ33mxD86KsjTxdTR q3sQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=PsHuEKgs; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.33 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Return-Path: Received: from lipwig.vger.email (lipwig.vger.email. [23.128.96.33]) by mx.google.com with ESMTPS id a16-20020a170902ecd000b001cf7e9809e2si80008plh.389.2023.11.22.04.23.19 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 22 Nov 2023 04:23:19 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.33 as permitted sender) client-ip=23.128.96.33; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=PsHuEKgs; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.33 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by lipwig.vger.email (Postfix) with ESMTP id 14EA48077507; Wed, 22 Nov 2023 04:23:17 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at lipwig.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1343719AbjKVMXG (ORCPT + 99 others); Wed, 22 Nov 2023 07:23:06 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39574 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1343927AbjKVMXE (ORCPT ); Wed, 22 Nov 2023 07:23:04 -0500 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 78E0298 for ; Wed, 22 Nov 2023 04:23:00 -0800 (PST) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 9B255C433C7; Wed, 22 Nov 2023 12:22:59 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1700655780; bh=6S2477i1eBrWi6MdXtHpYcNgUIbufoSnif77dzfTWyk=; h=From:To:Cc:Subject:In-Reply-To:References:Date:From; b=PsHuEKgss+IP6NdSdts7W9nwVVimGzqCAt0Amtf3T/GnYWgSfi+oMymO7ViH5LR8Z yQhqhFtmbM6R7YxLbcL+IQ+zQLh7ciJmRk4kVSDcPjN+Q/3vb+VCQNE66hMM+ZU1BY TFdkG4Ad5Kes9f+Cvoh0MDv5jm8y/LKgsDAQVpwnOzkrWQnCHn/7OYKgH6eQEQ+9OR fwsxmvroxShojZhQs7VhPRXgB+/jNUD5wsaIe8eO3sA6FYyb2v+wJ/UH5x+JkwC9qi HQY+8yiVVttoX9CP8C4C4UfGYBu50shQqtxZivV3NM4XvxneF9qRWRBm3DEzFA9ds4 7IWWFHa3te4EQ== From: =?utf-8?B?QmrDtnJuIFTDtnBlbA==?= To: Sunil V L , Bjorn Helgaas Cc: linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, linux-acpi@vger.kernel.org, linux-pci@vger.kernel.org, linux-serial@vger.kernel.org, Catalin Marinas , Will Deacon , Paul Walmsley , Palmer Dabbelt , Albert Ou , "Rafael J . Wysocki" , Len Brown , Bjorn Helgaas , Anup Patel , Thomas Gleixner , Greg Kroah-Hartman , Jiri Slaby , Conor Dooley , Andrew Jones , Atish Kumar Patra , Haibo Xu , Marc Zyngier Subject: Re: [RFC PATCH v2 06/21] RISC-V: Kconfig: Select deferred GSI probe for ACPI systems In-Reply-To: References: <20231106221606.GA264641@bhelgaas> Date: Wed, 22 Nov 2023 13:22:56 +0100 Message-ID: <87a5r6rn8f.fsf@all.your.base.are.belong.to.us> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Status: No, score=-1.3 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lipwig.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (lipwig.vger.email [0.0.0.0]); Wed, 22 Nov 2023 04:23:17 -0800 (PST) Hi Sunil! I'm trying to decipher this thread, so apologies in advance for the stupid questions! :-P Sunil V L writes: > Hi Bjorn, > > On Mon, Nov 06, 2023 at 04:16:06PM -0600, Bjorn Helgaas wrote: >> On Fri, Oct 27, 2023 at 06:25:03PM +0530, Sunil V L wrote: >> > On Thu, Oct 26, 2023 at 12:04:08PM -0500, Bjorn Helgaas wrote: >> > > On Thu, Oct 26, 2023 at 01:53:29AM +0530, Sunil V L wrote: >> > > > On RISC-V platforms, apart from root interrupt controllers (which >> > > > provide local interrupts and IPI), other interrupt controllers in = the >> > > > hierarchy are probed late. Enable this select this CONFIG option f= or >> > > > RISC-V platforms so that device drivers which connect to deferred >> > > > interrupt controllers can take appropriate action. >> > >=20 >> > > Quite a bit of this series seems related to the question of interrupt >> > > controllers being probed "late". >> > >=20 >> > > I don't see anything specific about *how* late this might be, but fr= om >> > > the use of -EPROBE_DEFER in individual drivers (8250_pnp explicitly, >> > > and acpi_register_gsi() and pnp_irq() and acpi_pci_irq_enable(), whi= ch >> > > are called from driver .probe() paths) it seems like interrupt >> > > controllers might be detected even after devices that use them. >> > >=20 >> > > That seems like a fairly invasive change to the driver probe flow. >> > > If we really need to do that, I think it might merit a little more >> > > background as justification since we haven't had to do it for any >> > > other arch yet. >> >=20 >> > In RISC-V, the APLIC can be a converter from wired (GSI) to MSI interr= upts. >> > Hence, especially in this mode, it has to be a platform device to use >> > device MSI domain. Also, according to Marc Zyngier there is no reason = to >> > probe interrupt controllers early apart from root controller. So, the >> > device drivers which use wired interrupts need to be probed after APLI= C. >> >=20 >> > The PNP devices and PCI INTx GSI links use either >> > acpi_dev_resource_interrupt() (PNP) or acpi_register_gsi() directly >> > (PCI). The approach taken here is to follow the example of >> > acpi_irq_get() which is enhanced to return EPROBE_DEFER and several >> > platform device drivers which use platform_get_irq() seem to be handli= ng >> > this already. >>=20 >> This series (patch 04/21 "ACPI: irq: Add support for deferred probe in >> acpi_register_gsi()" [1]) makes acpi_register_gsi() return >> -EPROBE_DEFER, which percolates up through pci_enable_device(). >>=20 >> Maybe that's ok, but this affects *all* PCI drivers, and it's a new >> case that did not occur before. Many drivers emit warning or error >> messages for any pci_enable_device() failure, which you probably don't >> want in this case, since -EPROBE_DEFER is not really a "failure"; >> IIUC, it just means "probe again later." >> > Yeah, I think all the drivers which need to be supported on RISC-V > ACPI based systems will have to support deferred probe with this scheme. > >> > Using ResourceSource dependency (mbigen uses) in the namespace as part= of >> > Extended Interrupt Descriptor will not ensure the order since PNP/INTx >> > GSI devices don't work with that. >>=20 >> Are these PNP/INTx GSI devices described in ACPI? In the namespace? >> Or in a static table? >>=20 > Yes, these are standard devices in the namespace. For ex: PNP0501(16550) > or PNP0C0F (PCI interrupt link devices) are in the namespace. > >> > Is there any other better way to create dependency between IO devices >> > and the interrupt controllers when interrupt controller itself is a >> > platform device? While using core_initcall() for interrupt controllers >> > seem to work which forces the interrupt controller to be probed first, >> > Marc is not in favor of that approach since it is fragile. >>=20 >> I guess PCI interrupts from the PCI host bridges (PNP0A03 devices) >> feed into the APLIC? And APLIC is described via MADT? Based on this >> series, it looks like this: >>=20 >> acpi_init >> + acpi_riscv_init >> + riscv_acpi_aplic_platform_init >> + acpi_table_parse_madt(ACPI_MADT_TYPE_APLIC, aplic_parse_madt, = 0) >> acpi_scan_init >> acpi_pci_root_init >> acpi_pci_link_init >> acpi_bus_scan # add PCI host bridges, etc >>=20 >> If that's the sequence, it looks like aplic_parse_madt() should be >> called before the PCI host bridges are added. >>=20 >> Or maybe this isn't how the APLICs are enumerated? >>=20 > That's partly correct. APLIC platform devices are created prior to PCI > host bridges added. But the actual APLIC driver which creates the > irqdomain will be probed as a regular platform driver for the APLIC > device. The platform driver probe will happen using DD framework and > devices don't have any dependency on APLIC which can cause device probe > prior to APLIC driver probe. > > DT supports fw_devlink framework which makes it easier for IRQ devices > to use regular platform drivers and produces-consumers are probed in the > order without requiring drivers to do deferred probe. But I don't see > that supported for ACPI framework. Also, the way PNP devices get added > there is an assumption that interrupt controller is already setup fully. AFAIU, the -EPROBE_DEFER changes are needed for GSIs (and the way the IMSIC/APLIC irqchip series is structured), right? There's a couple of separate pieces in play here: 1. IMSIC-IPI (MADT init) 2. IMSIC-MSI (MADT init, imsic_platform_acpi_probe() patch 14) 3. APLIC-wired (platform) 4. APLIC-MSI-bridge (platform) APLIC-MSI-bridge is pretty much a RISC-V mbigen. Some devices do not have ResourceSource parsing implemented yet. The PNP devices that cannot use ResourceSource (you mention PNP0501 (16550) and PNP0C0F (PCI interrupt link devices), do we really need to care about them for the RISC-V platforms using ACPI? If that would change, the kernel drivers can be adjusted (d44fa3d46079 ("ACPI: Add support for ResourceSource/IRQ domain mapping"))? I guess my question is we need to care about GSIs w/o explicit ResourceSource, so that APLIC-MSI-bridge can be used. GED works nicely with ResourceSource, and covers a lot of the GSI use-cases, no? And if we do care, then *both* 3 and 4 would need at MADT scan point/init, and not be a platform device (late init). From my, probably naive perspective, it's a bit weird *not* to create the irq domains at MADT scan time. > With this new use case in RISC-V, here are the alternatives I am aware of. > > 1) Use core_initcall() in the APLIC drivers which makes APLIC driver to > be probed prior to PNP or PCI INTx devices. But this was ruled out in > the context of DT from Marc. > > 2) Like the approach tried in this series, add support for deferred > probe in drivers. This will be invasive change requiring many drivers to > change like you pointed. Again is this only for GSIs? Patch 14 moves the IMSIC-MSI init to MADT for PCIe devices (which is different from DT), so it's not for PCIe devices. I wonder if it's a lot of churn for something that will not be used for RISC-V ACPI systems... A quick look at what Arm's GICv3 does, all irq domains are created at MADT init. Bj=C3=B6rn (no, not PCI-Bjorn ;-))