Received: by 2002:a6b:500f:0:0:0:0:0 with SMTP id e15csp4681750iob; Sun, 8 May 2022 21:49:21 -0700 (PDT) X-Google-Smtp-Source: ABdhPJx27e+G2XWJt6ycEtDx8UHLo69st51Xc7kf9wtWjR653S3ehg2DVXWFRPVeWqPjH1MAjYNk X-Received: by 2002:a63:e146:0:b0:39d:1b00:e475 with SMTP id h6-20020a63e146000000b0039d1b00e475mr11810158pgk.537.1652071760862; Sun, 08 May 2022 21:49:20 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1652071760; cv=none; d=google.com; s=arc-20160816; b=ySX50GUptPaFzcV4rHpqFEFCbNnAKRNa8rveFNKLF16hq7Aph7Qhk6DTj0TkqHFyOu Zt7bZ+I8nnzxqJaJ1C67XLSoNROOSmECpAP+ESIZiGDULDElHlpjqTIMga3hDhQu+nk8 w+uX4auqg3OMFVaCGwwR+yGvsDtxLJurU/Pf1yGCE1S8IBNQJkLY5RkLqBpQ06ikK0Fv kKUv9E+Er2oMF9n0IZe2wcrpWnwFW8kBY38sz8lV7yPEYhBbJuCUmTN6uP54t9WXHxMJ tF76+HEaP2VSPk0Ok7zPvc/B1dQ7vDRuCi7/Xy8E28xlfMSCdT6oPh/aiXkkNttDtCJP +EaQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from; bh=bUqRLasqP8HmRQw7AcqDRNdgza/JQd7KJrJaLEJp7Wg=; b=de+21zauX30LWFi9o/Gv1eLLuTCUR2TnG2nA5VIu9Do3FEs874+W6QZslnhWKvG0pY 7N4i0UR6ZeQ+nj18xhN5i+ma8VP/B6sXWTotS3B5n6G40+lpF6TcNUnxtxWFAU99qVbP 97DeUrUfU89KgVYUEw4K8TZnm7tNAt+hzgryQIZa4Q3IMTZii/GH820YLYkAL750jP0N CDHzOqgxIj4PVl1g3E5djxMVaL6tFpW0cf+eaHv3+JlIIHulmABE4siNzIAcrbZ6uvno 1NLT89m3N+ANkSiVrE7w9dkRltAhVA/IaFxzSCwaRYPllI5tZ7pQYwcBqgSiS3ZRnwAZ n+VQ== ARC-Authentication-Results: i=1; mx.google.com; spf=softfail (google.com: domain of transitioning linux-kernel-owner@vger.kernel.org does not designate 23.128.96.19 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from lindbergh.monkeyblade.net (lindbergh.monkeyblade.net. [23.128.96.19]) by mx.google.com with ESMTPS id t8-20020a632248000000b003c68f44a226si7161567pgm.653.2022.05.08.21.49.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 08 May 2022 21:49:20 -0700 (PDT) Received-SPF: softfail (google.com: domain of transitioning linux-kernel-owner@vger.kernel.org does not designate 23.128.96.19 as permitted sender) client-ip=23.128.96.19; Authentication-Results: mx.google.com; spf=softfail (google.com: domain of transitioning linux-kernel-owner@vger.kernel.org does not designate 23.128.96.19 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id 1A8EA12A71F; Sun, 8 May 2022 21:49:03 -0700 (PDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1376890AbiEDS3h (ORCPT + 99 others); Wed, 4 May 2022 14:29:37 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48352 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1376761AbiEDS3N (ORCPT ); Wed, 4 May 2022 14:29:13 -0400 Received: from cloudserver094114.home.pl (cloudserver094114.home.pl [79.96.170.134]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A76BE4757F; Wed, 4 May 2022 11:00:37 -0700 (PDT) Received: from localhost (127.0.0.1) (HELO v370.home.net.pl) by /usr/run/smtp (/usr/run/postfix/private/idea_relay_lmtp) via UNIX with SMTP (IdeaSmtpServer 5.0.0) id 51a7cf1e4c0649f0; Wed, 4 May 2022 20:00:35 +0200 Received: from kreacher.localnet (unknown [213.134.161.219]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by v370.home.net.pl (Postfix) with ESMTPSA id A61BA66C2D7; Wed, 4 May 2022 20:00:34 +0200 (CEST) From: "Rafael J. Wysocki" To: Nathan Chancellor Cc: "Rafael J. Wysocki" , Linux PCI , Linux PM , LKML , Bjorn Helgaas , Mika Westerberg Subject: Re: [PATCH v3 4/9] PCI/PM: Rework changing power states of PCI devices Date: Wed, 04 May 2022 20:00:33 +0200 Message-ID: <2650302.mvXUDI8C0e@kreacher> In-Reply-To: References: <4419002.LvFx2qVVIh@kreacher> MIME-Version: 1.0 Content-Transfer-Encoding: 7Bit Content-Type: text/plain; charset="UTF-8" X-CLIENT-IP: 213.134.161.219 X-CLIENT-HOSTNAME: 213.134.161.219 X-VADE-SPAMSTATE: clean X-VADE-SPAMCAUSE: gggruggvucftvghtrhhoucdtuddrgedvfedrvdelgdduudelucetufdoteggodetrfdotffvucfrrhhofhhilhgvmecujffqoffgrffnpdggtffipffknecuuegrihhlohhuthemucduhedtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmdenucfjughrpefhvfevufffkfgjfhgggfgtsehtufertddttdejnecuhfhrohhmpedftfgrfhgrvghlucflrdcuhgihshhotghkihdfuceorhhjfiesrhhjfiihshhotghkihdrnhgvtheqnecuggftrfgrthhtvghrnhepledtieekkeekveeikeetgffgteeuteefjeevjeegudelvdduheeiuedvieehieevnecuffhomhgrihhnpehgihhthhhusgdrtghomhenucfkphepvddufedrudefgedrudeiuddrvdduleenucevlhhushhtvghrufhiiigvpedtnecurfgrrhgrmhepihhnvghtpedvudefrddufeegrdduiedurddvudelpdhhvghlohepkhhrvggrtghhvghrrdhlohgtrghlnhgvthdpmhgrihhlfhhrohhmpedftfgrfhgrvghlucflrdcuhgihshhotghkihdfuceorhhjfiesrhhjfiihshhotghkihdrnhgvtheqpdhnsggprhgtphhtthhopeejpdhrtghpthhtohepnhgrthhhrghnsehkvghrnhgvlhdrohhrghdprhgtphhtthhopehrrghfrggvlheskhgvrhhnvghlrdhorhhgpdhrtghpthhtoheplhhinhhugidqphgtihesvhhgvghrrdhkvghrnhgvlhdrohhrghdprhgtphhtthhopehlihhnuhigqdhpmhesvhhgvghrrdhkvghrnhgvlhdrohhrghdp rhgtphhtthhopehlihhnuhigqdhkvghrnhgvlhesvhhgvghrrdhkvghrnhgvlhdrohhrghdprhgtphhtthhopehhvghlghgrrghssehkvghrnhgvlhdrohhrghdprhgtphhtthhopehmihhkrgdrfigvshhtvghrsggvrhhgsehlihhnuhigrdhinhhtvghlrdgtohhm X-DCC--Metrics: v370.home.net.pl 1024; Body=7 Fuz1=7 Fuz2=7 X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,RDNS_NONE, SPF_HELO_NONE,T_SCC_BODY_TEXT_LINE autolearn=no autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Wednesday, May 4, 2022 6:36:00 PM CEST Nathan Chancellor wrote: > On Wed, May 04, 2022 at 02:59:17PM +0200, Rafael J. Wysocki wrote: > > On Tue, May 3, 2022 at 7:59 PM Nathan Chancellor wrote: > > > > > > Hi Rafael, > > > > > > On Thu, Apr 14, 2022 at 03:11:21PM +0200, Rafael J. Wysocki wrote: > > > > From: Rafael J. Wysocki > > > > > > > > There are some issues related to changing power states of PCI > > > > devices, mostly related to carrying out unnecessary actions in some > > > > places, and the code is generally hard to follow. > > > > > > > > 1. pci_power_up() has two callers, pci_set_power_state() and > > > > pci_pm_default_resume_early(). The latter updates the current > > > > power state of the device right after calling pci_power_up() > > > > and it restores the entire config space of the device right > > > > after that, so pci_power_up() itself need not read the > > > > PCI_PM_CTRL register or restore the BARs after programming the > > > > device into D0 in that case. > > > > > > > > 2. It is generally hard to get a clear view of the pci_power_up() > > > > code flow, especially in some corner cases, due to all of the > > > > involved PCI_PM_CTRL register reads and writes occurring in > > > > pci_platform_power_transition() and in pci_raw_set_power_state(), > > > > some of which are redundant. > > > > > > > > 3. The transitions from low-power states to D0 and the other way > > > > around are unnecessarily tangled in pci_raw_set_power_state() > > > > which causes it to use a redundant local variable and makes it > > > > rather hard to follow. > > > > > > > > To address the above shortcomings, make the following changes: > > > > > > > > a. Remove the code handling transitions into D0 > > > > from pci_raw_set_power_state() and rename it as > > > > pci_set_low_power_state(). > > > > > > > > b. Add the code handling transitions into D0 directly > > > > to pci_power_up() and to a new wrapper function > > > > pci_set_full_power_state() calling it internally that is > > > > only used in pci_set_power_state(). > > > > > > > > c. Make pci_power_up() avoid redundant PCI_PM_CTRL register reads > > > > and make it work in the same way for transitions from any > > > > low-power states (transitions from D1 and D2 are handled > > > > slightly differently before the change). > > > > > > > > d. Put the restoration of the BARs and the PCI_PM_CTRL > > > > register read confirming the power state change into > > > > pci_set_full_power_state() to avoid doing that in > > > > pci_pm_default_resume_early() unnecessarily. > > > > > > > > Signed-off-by: Rafael J. Wysocki > > > > Reviewed-by: Mika Westerberg > > > > > > This change as commit 5bffe4c611f5 ("PCI/PM: Rework changing power > > > states of PCI devices") causes my AMD-based system to fail to fully > > > boot. As far as I can tell, this might be NVMe related, which might make > > > getting a full log difficult, as journalctl won't have anywhere to save > > > it. I see: > > > > > > nvme nvme0: I/O 8 QID 0 timeout, completion polled > > > > > > then shortly afterwards: > > > > > > nvme nvme0: I/O 24 QID 0 timeout, completion polled > > > nvme nvme0: missing or invalid SUBNQN field > > > > > > then I am dropped into an emergency shell. > > > > Thanks for the report! > > > > > This is a log from the previous commit, which may give some hints about > > > the configuration of this particular system. > > > > > > https://gist.github.com/nathanchance/8a56f0939410cb187896e904c72e41e7/raw/b47b2620bdd32d43c7a3b209fcfd9e3d4668f058/good-boot.log > > > > > > If there is any additional debugging information I can provide or > > > patches I can try, please let me know! > > > > Please see what happens if the "if (dev->current_state == PCI_D0)" > > check and the following "return 0" statement in pci_power_up() are > > commented out. > > If I understand you correctly, this? Unfortunately, that does not help. Thanks for testing. Please check if the patch below makes any difference. --- drivers/pci/pci.c | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) Index: linux-pm/drivers/pci/pci.c =================================================================== --- linux-pm.orig/drivers/pci/pci.c +++ linux-pm/drivers/pci/pci.c @@ -1245,7 +1245,7 @@ int pci_power_up(struct pci_dev *dev) /* There's nothing more to do if current_state is D0 at this point. */ if (dev->current_state == PCI_D0) - return 0; + goto done; /* * Program the device into PCI_D0 by forcing the entire word to 0 (this @@ -1260,6 +1260,11 @@ int pci_power_up(struct pci_dev *dev) udelay(PCI_PM_D2_DELAY); dev->current_state = PCI_D0; + +done: + if (dev->bus->self) + pcie_aspm_pm_state_change(dev->bus->self); + return 1; fail: @@ -1339,9 +1344,6 @@ static int pci_set_full_power_state(stru pci_restore_bars(dev); } - if (dev->bus->self) - pcie_aspm_pm_state_change(dev->bus->self); - return 0; }