Received-SPF: pass (google.com: domain of linux-kernel+bounces-139256-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) client-ip=147.75.80.249;
Date: Wed, 10 Apr 2024 14:28:40 -0500
From: Bjorn Helgaas <helgaas@kernel.org>
To: Vidya Sagar <vidyas@nvidia.com>
Cc: "Rafael J. Wysocki" <rafael@kernel.org>,
	Krzysztof =?utf-8?Q?Wilczy=C5=84ski?= <kw@linux.com>,
	Lorenzo Pieralisi <lpieralisi@kernel.org>,
	"linux-pci@vger.kernel.org" <linux-pci@vger.kernel.org>,
	"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
	Manikanta Maddireddy <mmaddireddy@nvidia.com>,
	Shanker Donthineni <sdonthineni@nvidia.com>,
	Krishna Thota <kthota@nvidia.com>, Will Deacon <will@kernel.org>,
	Joerg Roedel <joro@8bytes.org>
Subject: Re: [Query] ACS enablement in the DT based boot flow
Message-ID: <20240410192840.GA2147526@bhelgaas>
Precedence: bulk
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <PH8PR12MB667446D4A4CAD6E0A2F488B5B83F2@PH8PR12MB6674.namprd12.prod.outlook.com>

[+cc Will, Joerg]

On Mon, Apr 01, 2024 at 10:40:15AM +0000, Vidya Sagar wrote:
> Hi folks,
> ACS (Access Control Services) is configured for a PCI device through
> pci_enable_acs().  The first thing pci_enable_acs() checks for is
> whether the global flag 'pci_acs_enable' is set or not.  The global
> flag 'pci_acs_enable' is set by the function pci_request_acs().
> 
> pci_enable_acs() function is called whenever a new PCI device is
> added to the system
> 
>  pci_enable_acs+0x4c/0x2a4
>  pci_acs_init+0x38/0x60
>  pci_device_add+0x1a0/0x670
>  pci_scan_single_device+0xc4/0x100
>  pci_scan_slot+0x6c/0x1e0
>  pci_scan_child_bus_extend+0x48/0x2e0
>  pci_scan_root_bus_bridge+0x64/0xf0
>  pci_host_probe+0x18/0xd0
> 
> In the case of a system that boots using device-tree blob,
> pci_request_acs() is called when the device driver binds with the
> respective device
> 
> of_iommu_configure+0xf4/0x230
> of_dma_configure_id+0x110/0x340
> pci_dma_configure+0x54/0x120
> really_probe+0x80/0x3e0
> __driver_probe_device+0x88/0x1c0
> driver_probe_device+0x3c/0x140
> __device_attach_driver+0xe8/0x1e0
> bus_for_each_drv+0x78/0xf0
> __device_attach+0x104/0x1e0
> device_attach+0x14/0x30
> pci_bus_add_device+0x50/0xd0
> pci_bus_add_devices+0x38/0x90
> pci_host_probe+0x40/0xd0
> 
> Since the device addition always happens first followed by the
> driver binding, this flow effectively makes sure that ACS never gets
> enabled.
> 
> Ideally, I would expect the pci_request_acs() get called (probably
> by the OF framework itself) before calling pci_enable_acs().
> 
> This happens in the ACPI flow where pci_request_acs() is called
> during IORT node initialization (i.e. iort_init_platform_devices()
> function).
> 
> Is this understanding correct? If yes, would it make sense to call
> pci_request_acs() during OF initialization (similar to IORT
> initialization in ACPI flow)?

Your understanding looks correct to me.  My call graph notes, FWIW:

  mem_init
    pci_iommu_alloc                   # x86 only
      amd_iommu_detect                # init_state = IOMMU_START_STATE
        iommu_go_to_state(IOMMU_IVRS_DETECTED)
          state_next
            switch (init_state)
            case IOMMU_START_STATE:
              detect_ivrs
                pci_request_acs
                  pci_acs_enable = 1  # <--
      detect_intel_iommu
        pci_request_acs
          pci_acs_enable = 1          # <--

  pci_scan_single_device              # PCI enumeration
    ...
      pci_init_capabilities
        pci_acs_init
          pci_enable_acs
            if (pci_acs_enable)       # <--
              pci_std_enable_acs

  __driver_probe_device
    really_probe
      pci_dma_configure               # pci_bus_type.dma_configure
        if (OF)
          of_dma_configure
            of_dma_configure_id
              of_iommu_configure
                pci_request_acs       # <-- 6bf6c24720d3
                iommu_probe_device
        else if (ACPI)
          acpi_dma_configure
            acpi_dma_configure_id
              acpi_iommu_configure_id
                iommu_probe_device

The pci_request_acs() in of_iommu_configure(), which happens too late
to affect pci_enable_acs(), was added by 6bf6c24720d3 ("iommu/of:
Request ACS from the PCI core when configuring IOMMU linkage"), so I
cc'd Will and Joerg.  I don't know if that *used* to work and got
broken somehow, or if it never worked as intended.

Bjorn