Received: by 2002:ab2:6857:0:b0:1ef:ffd0:ce49 with SMTP id l23csp1434158lqp; Fri, 22 Mar 2024 15:05:11 -0700 (PDT) X-Forwarded-Encrypted: i=3; AJvYcCWnRdUvOiiMlzhaOIjEJAvsHAIOhXGpLZvB5vbxvpFot5EELdTzQPFO0FMZnmhpgMAKaVP+KC6eQ2f8YbLji3AGwu9NJitkF1qGaPVWyw== X-Google-Smtp-Source: AGHT+IHMysGptP6YIxJiyZtv1xFZSCSEWgZVHganTKyzP4/4zit7wmiS0HMRz8AUW/Mlq3N9nArE X-Received: by 2002:a17:906:f348:b0:a47:39d1:5b3f with SMTP id hg8-20020a170906f34800b00a4739d15b3fmr745413ejb.61.1711145111665; Fri, 22 Mar 2024 15:05:11 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1711145111; cv=pass; d=google.com; s=arc-20160816; b=elmkfvJ6/SfJY/gwX9cYjWB86jGjNfTVA45anHV7GW1xe2I3iAc51u1bg33V+f6kF9 WOhdl0kUiGEg2VievekLnGxWkISejWve8m608eiV2iKa257r7ywqnCR1GsLPkrfP0O+2 n87sm+/5GZVmvwCPvUsKrr/eleIg970y8nPby0z5reE6Czq1r16yiHXYvXcAYxzqcXCT CNKiQhP8kJrIxyPEMYOuzIwJYf7VUHDvVhVRygCLZgqJ81XrC23L2arYGvbcwVkBlW+g meYDWwf/hSVhsxHtlcJNApG8bi4BZV+mWCL15bP+Dq2N6E/ehofzzgx7tdZJJxokPay7 bQ8A== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=in-reply-to:content-disposition:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:message-id:subject:cc:to:from :date:dkim-signature; bh=1gKO+WczKzpEwViDlBjiu7iS5XKitHcsO4UcYdA3uh4=; fh=pGZ5SAoLmMKfRFon89sVgUIrlJj8dNp2JQ+1nD5QD54=; b=pjVFLFWF0Gv4xHyn9gq/yb7zMytUzsuYwdY7F1eU0tpzHCbb5dWO4ccyr90HLbDmNd 6Siq6G6zA9w/Ly3VxrOIenA0ZgK286nAmC+2TsLAxph1CYJigiN/DKCWxt4CivPDbc+B okJiwKDo4m44G/mvq6W5Z3/al+PZCEBEt554UTk7FMyFsb9V2ECeIRfStTxUXgUBqU+o uxNIzmtXKCwfPXOde6L4WQ7jrK7+gt2WWVubbh6du/JpXXvb1zeg4q437m+NbKRT/ZxL PjHESfhWmg6Egdw4AcknoJ2LYFM0YUKjioLFoyzctP/O/J6+cJbPKZeNhTyiVM5Sap58 p2xA==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=fcX3UWz6; arc=pass (i=1 dkim=pass dkdomain=kernel.org); spf=pass (google.com: domain of linux-kernel+bounces-112055-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) smtp.mailfrom="linux-kernel+bounces-112055-linux.lists.archive=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Return-Path: Received: from am.mirrors.kernel.org (am.mirrors.kernel.org. [2604:1380:4601:e00::3]) by mx.google.com with ESMTPS id a8-20020a1709064a4800b00a46df08220dsi224482ejv.895.2024.03.22.15.05.11 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 22 Mar 2024 15:05:11 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel+bounces-112055-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) client-ip=2604:1380:4601:e00::3; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=fcX3UWz6; arc=pass (i=1 dkim=pass dkdomain=kernel.org); spf=pass (google.com: domain of linux-kernel+bounces-112055-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) smtp.mailfrom="linux-kernel+bounces-112055-linux.lists.archive=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by am.mirrors.kernel.org (Postfix) with ESMTPS id 60EBF1F22434 for ; Fri, 22 Mar 2024 22:05:11 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 3985A8174C; Fri, 22 Mar 2024 22:05:00 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="fcX3UWz6" Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 48F148005F; Fri, 22 Mar 2024 22:04:58 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711145099; cv=none; b=BfyEoDmKddQncpWGppW2JyVqnaqO5Lkh0jaMx1dYaHC4Cere+nW+2b9Ev+RyfHzrhb8CLzlTZwWIZvBM/7ZANfnEIFRRi+QRTMlz8ULaFxe/RCsuoFrEDHiYHSRSpbDoS4T0ZMVRREnxXuIuG9GfGdc1fLOvWT56In9LD6J1LSM= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711145099; c=relaxed/simple; bh=RB4dv/BfPAPMuLZqjIrxuoGF9WQVhOll7NLtUmXcVcQ=; h=Date:From:To:Cc:Subject:Message-ID:MIME-Version:Content-Type: Content-Disposition:In-Reply-To; b=UMass6A8kNM9xn7MvZ/QUztkK3KaCwVnb6mkYVJ9Zf8U3Nntwev6ndj5Ji/7n+48fHTHS1RZiqWRBYhIhwZIUlrtt6Kaqlv2W9svxt/HP1Sa+xdZDiQ79mMpVnk+M+nA+nUGhktIFmrG4Ei2l6KXCqy/6STlvn481mVSQ5T9lUs= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=fcX3UWz6; arc=none smtp.client-ip=10.30.226.201 Received: by smtp.kernel.org (Postfix) with ESMTPSA id 84096C433F1; Fri, 22 Mar 2024 22:04:58 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1711145098; bh=RB4dv/BfPAPMuLZqjIrxuoGF9WQVhOll7NLtUmXcVcQ=; h=Date:From:To:Cc:Subject:In-Reply-To:From; b=fcX3UWz6eptpiLRAnqJrHhJZqbBhdWWJVHjwbkMKx2RSyA4OKINOHfU9Wy3f8s/Ls Kaz6yfSfWAE/3eG5AwxlEtXd1/JlLdm5l2vFwTjHqLKZul/SOioqCjS2oh+mSxA9uK Q+jc3q+HHpectJe+Vvcl7HZUBQmTg9ln/S29j83SXRb45eyEZvB26UBqbK1dccoNAG kMpfT7jW4FHgOfglLJH55dhsahBnX0Dl/Ml5KJRxET0SzDH9AIZkrSPiMqYEBfoHrX NfREHoSJraofJI7hdFYnD5YoZ8wY6zb0a2V3CK8y26Rhsy9Ruw2kXU1ueDh2R8dV4C Qonfg6V4u7igg== Date: Fri, 22 Mar 2024 17:04:56 -0500 From: Bjorn Helgaas To: Manivannan Sadhasivam Cc: Krishna Chaitanya Chundru , "Rafael J. Wysocki" , Jingoo Han , Gustavo Pimentel , Lorenzo Pieralisi , Krzysztof =?utf-8?Q?Wilczy=C5=84ski?= , Rob Herring , Bjorn Helgaas , linux-pci@vger.kernel.org, linux-kernel@vger.kernel.org, quic_vbadigan@quicinc.com, quic_ramkri@quicinc.com, quic_nitegupt@quicinc.com, quic_skananth@quicinc.com, quic_parass@quicinc.com Subject: Re: [PATCH v2] PCI: dwc: Enable runtime pm of the host bridge Message-ID: <20240322220456.GA1379507@bhelgaas> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20240319111148.GF52500@thinkpad> On Tue, Mar 19, 2024 at 04:41:48PM +0530, Manivannan Sadhasivam wrote: > On Fri, Mar 08, 2024 at 11:12:48AM -0600, Bjorn Helgaas wrote: > > On Fri, Mar 08, 2024 at 08:38:52AM +0530, Krishna Chaitanya Chundru wrote: > > > On 3/8/2024 3:25 AM, Bjorn Helgaas wrote: > > > > [+to Rafael, sorry, another runtime PM question, beginning of thread: > > > > https://lore.kernel.org/r/20240305-runtime_pm_enable-v2-1-a849b74091d1@quicinc.com] > > > > > > > > On Thu, Mar 07, 2024 at 07:28:54AM +0530, Krishna Chaitanya Chundru wrote: > > > > > On 3/6/2024 1:27 AM, Bjorn Helgaas wrote: > > > > > > On Tue, Mar 05, 2024 at 03:19:01PM +0530, Krishna chaitanya chundru wrote: > > > > > > > The Controller driver is the parent device of the PCIe host bridge, > > > > > > > PCI-PCI bridge and PCIe endpoint as shown below. > > > > > > > > > > > > > > PCIe controller(Top level parent & parent of host bridge) > > > > > > > | > > > > > > > v > > > > > > > PCIe Host bridge(Parent of PCI-PCI bridge) > > > > > > > | > > > > > > > v > > > > > > > PCI-PCI bridge(Parent of endpoint driver) > > > > > > > | > > > > > > > v > > > > > > > PCIe endpoint driver > > > > > > > > > > > > > > Since runtime PM is disabled for host bridge, the state of the child > > > > > > > devices under the host bridge is not taken into account by PM framework > > > > > > > for the top level parent, PCIe controller. So PM framework, allows > > > > > > > the controller driver to enter runtime PM irrespective of the state > > > > > > > of the devices under the host bridge. > > > > > > > > > > > > IIUC this says that we runtime suspend the controller even though > > > > > > runtime PM is disabled for the host bridge? I have a hard time > > > > > > parsing this; can you cite a function that does this or some relevant > > > > > > documentation about how this part of runtime PM works? > > > > > > > > > > > Generally controller should go to runtime suspend when endpoint client > > > > > drivers and pci-pci host bridge drivers goes to runtime suspend as the > > > > > controller driver is the parent, but we are observing controller driver > > > > > goes to runtime suspend even when client drivers and PCI-PCI bridge are > > > > > in active state. > > > > > > > > It surprises me that a device could be suspended while children are > > > > active. A PCI-PCI bridge must be in D0 for any devices below it to be > > > > active. The controller is a platform device, not a PCI device, but I > > > > am similarly surprised that we would suspend it when children are > > > > active, which makes me think we didn't set the hierarchy up correctly. > > > > > > > > It doesn't seem like we should need to enable runtime PM for a parent > > > > to keep it from being suspended when children are active. > > > > > > Here we are not enabling runtime PM of the controller device, we are > > > enabling runtime PM for the bridge device which is maintained by the > > > PCIe framework. The bridge device is the parent of the PCI-PCI > > > bridge and child of the controller device. As the bridge device's > > > runtime PM is not enabled the PM framework is ignoring the child's > > > runtime status. > > > > OK, it's the host bridge, not the controller. > > > > I'm still surprised that the PM framework will runtime suspend a > > device when child devices are active. > > There is a catch here. Even though the child devices are funtionally > active, PM framework will only consider their runtime_pm state, > which is initially set to 'disabled' for all devices. It is upto the > device drivers to enable it when required. > > Here is the initial runtime PM status of each device post boot: > > Controller device -> disabled initially but enabled by pcie-qcom.c > Host bridge -> disabled initially > PCIe bridge -> disabled initially but conditionally enabled by portdrv.c > PCIe devices -> disabled initially but enabled by respective drivers like WLAN > > Now, when the controller device goes to runtime suspend, PM > framework will check the runtime PM state of the child device (host > bridge) and will find it to be disabled. So it will allow the parent > (controller device) to go to runtime suspend. Only if the child > device's state was 'active' it will prevent the parent to get > suspended. > > But you may wonder if this is ideal? IMO NO. But we cannot blame the > PM framework here. The responsibility is within the device drivers > to handle the PM state based on the usecase. Ideally, the host > bridge driver should've handled runtime PM state during the probe > time. Otherwise, PM framework wouldn't know when would be the best > time to suspend the devices. My expectation is that adding new functionality should only require changes in drivers that want to take advantage of it. For example, if we add runtime PM support in the controller driver, the result should be functionally correct even if we don't update drivers for downstream devices. If that's not the way it works, I suggest that would be a problem in the PM framework. The host bridge might be a special case because we don't have a separate "host bridge" driver; that code is kind of integrated with the controller drivers. So maybe it's OK to do controller + host bridge runtime PM support at the same time, as long as any time we add runtime PM to a controller, we sure it's also set up for the host bridge. Bjorn