Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <48744abf-bd9b-927d-9ca3-a3e4bdc07fbb@linux.intel.com>
Date:   Sat, 23 Apr 2022 17:41:05 +0800
MIME-Version: 1.0
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101
 Thunderbird/91.7.0
Subject: Re: [PATCH 02/13] iommu: Move bus setup to IOMMU device registration
Content-Language: en-US
From:   Lu Baolu <baolu.lu@linux.intel.com>
To:     Robin Murphy <robin.murphy@arm.com>, joro@8bytes.org,
        will@kernel.org
Cc:     jean-philippe@linaro.org, zhang.lyra@gmail.com,
        linux-kernel@vger.kernel.org, iommu@lists.linux-foundation.org,
        thierry.reding@gmail.com, gerald.schaefer@linux.ibm.com,
        linux-arm-kernel@lists.infradead.org
References: <cover.1649935679.git.robin.murphy@arm.com>
 <e607a32be8e84c56d65160902f4bd3fb434ee9d3.1649935679.git.robin.murphy@arm.com>
 <d838c7f8-2d31-2327-9b4d-c05b06a516bb@linux.intel.com>
 <1d9a2146-9e48-ba97-d049-5d2332ebbd3a@arm.com>
 <d928d256-b1bc-2d45-a9c3-382edfe67b80@linux.intel.com>
 <7f693a3a-bbab-237a-af9a-4395e501a5b2@arm.com>
 <36764562-ad54-b192-4084-842fee8fca78@linux.intel.com>
 <8cd04084-54d0-9e10-f16f-99541e42d7eb@arm.com>
 <f72349ab-19d9-ed9f-28a7-6654fd5556cd@linux.intel.com>
 <60d6a35e-5d12-b8c2-b0d2-7155965a10e5@linux.intel.com>
In-Reply-To: <60d6a35e-5d12-b8c2-b0d2-7155965a10e5@linux.intel.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit
Precedence: bulk

On 2022/4/23 17:00, Lu Baolu wrote:
> On 2022/4/23 16:51, Lu Baolu wrote:
>> On 2022/4/23 16:37, Robin Murphy wrote:
>>> On 2022-04-23 09:01, Lu Baolu wrote:
>>>> Hi Robin,
>>>>
>>>> On 2022/4/19 15:20, Robin Murphy wrote:
>>>>> On 2022-04-19 00:37, Lu Baolu wrote:
>>>>>> On 2022/4/19 6:09, Robin Murphy wrote:
>>>>>>> On 2022-04-16 01:04, Lu Baolu wrote:
>>>>>>>> On 2022/4/14 20:42, Robin Murphy wrote:
>>>>>>>>> @@ -1883,27 +1900,12 @@ static int iommu_bus_init(struct 
>>>>>>>>> bus_type *bus)
>>>>>>>>>    */
>>>>>>>>>   int bus_set_iommu(struct bus_type *bus, const struct 
>>>>>>>>> iommu_ops *ops)
>>>>>>>>>   {
>>>>>>>>> -    int err;
>>>>>>>>> -
>>>>>>>>> -    if (ops == NULL) {
>>>>>>>>> -        bus->iommu_ops = NULL;
>>>>>>>>> -        return 0;
>>>>>>>>> -    }
>>>>>>>>> -
>>>>>>>>> -    if (bus->iommu_ops != NULL)
>>>>>>>>> +    if (bus->iommu_ops && ops && bus->iommu_ops != ops)
>>>>>>>>>           return -EBUSY;
>>>>>>>>>       bus->iommu_ops = ops;
>>>>>>>>
>>>>>>>> Do we still need to keep above lines in bus_set_iommu()?
>>>>>>>
>>>>>>> It preserves the existing behaviour until each callsite and its 
>>>>>>> associated error handling are removed later on, which seems like 
>>>>>>> as good a thing to do as any. Since I'm already relaxing 
>>>>>>> iommu_device_register() to a warn-but-continue behaviour while it 
>>>>>>> keeps the bus ops on life-support internally, I figured not 
>>>>>>> changing too much at once would make it easier to bisect any 
>>>>>>> potential issues arising from this first step.
>>>>>>
>>>>>> Fair enough. Thank you for the explanation.
>>>>>>
>>>>>> Do you have a public tree that I could pull these patches and try 
>>>>>> them
>>>>>> on an Intel hardware? Or perhaps you have done this? I like the whole
>>>>>> idea of this series, but it's better to try it with a real hardware.
>>>>>
>>>>> I haven't bothered with separate branches since there's so many 
>>>>> different pieces in-flight, but my complete (unstable) development 
>>>>> branch can be found here:
>>>>>
>>>>> https://gitlab.arm.com/linux-arm/linux-rm/-/commits/iommu/bus
>>>>>
>>>>> For now I'd recommend winding the head back to "iommu: Clean up 
>>>>> bus_set_iommu()" for testing - some of the patches above that have 
>>>>> already been posted and picked up by their respective subsystems, 
>>>>> but others are incomplete and barely compile-tested. I'll probably 
>>>>> rearrange it later this week to better reflect what's happened so far.
>>>>
>>>> I wound the head back to "iommu: Clean up bus_set_iommu" and tested it
>>>> on an Intel machine. It got stuck during boot. This test was on a 
>>>> remote
>>>> machine and I have no means to access it physically. So I can't get any
>>>> kernel debugging messages. (I have to work from home these days. :-()
>>>>
>>>> I guess it's due to the fact that intel_iommu_probe_device() callback
>>>> only works for the pci devices. The issue occurs when probing a device
>>>> other than a PCI one.
>>>
>>> Yeah, I was wondering if that would be significant, since it's the 
>>> only driver that never registered itself for platform_bus_type so 
>>> won't have actually seen those calls before. I'm inclined to bodge 
>>> that as below for now, as long as it then works OK in terms of the 
>>> rest of the changes.
>>>
>>> Thanks,
>>> Robin.
>>>
>>> ----->8-----
>>> diff --git a/drivers/iommu/intel/iommu.c b/drivers/iommu/intel/iommu.c
>>> index 9fa1b98186a3..6e359f92ec00 100644
>>> --- a/drivers/iommu/intel/iommu.c
>>> +++ b/drivers/iommu/intel/iommu.c
>>> @@ -4565,6 +4565,10 @@ static struct iommu_device 
>>> *intel_iommu_probe_device(struct device *dev)
>>>       unsigned long flags;
>>>       u8 bus, devfn;
>>>
>>> +    /* ANDD platform device support needs fixing */
>>> +    if (!pdev)
>>> +        return ERR_PTR(-ENODEV);
>>> +
>>>       iommu = device_to_iommu(dev, &bus, &devfn);
>>>       if (!iommu)
>>>           return ERR_PTR(-ENODEV);
>>
>> I haven't seen any real ANDD platform devices, hence this works for me.
> 
> Or more precisely, return NULL when @dev goes through device_to_iommu()?
> 
> diff --git a/drivers/iommu/intel/iommu.c b/drivers/iommu/intel/iommu.c
> index df5c62ecf942..0d447739e076 100644
> --- a/drivers/iommu/intel/iommu.c
> +++ b/drivers/iommu/intel/iommu.c
> @@ -797,8 +797,11 @@ struct intel_iommu *device_to_iommu(struct device 
> *dev, u8 *bus, u8 *devfn)
>                  pf_pdev = pci_physfn(pdev);
>                  dev = &pf_pdev->dev;
>                  segment = pci_domain_nr(pdev->bus);
> -       } else if (has_acpi_companion(dev))
> +       } else if (has_acpi_companion(dev)) {
>                  dev = &ACPI_COMPANION(dev)->dev;
> +       } else {
> +               return NULL;
> +       }
> 
>          rcu_read_lock();
>          for_each_iommu(iommu, drhd) {

Robin, please ignore this. "has_acpi_companion(dev)" isn't equal to an
ANDD device. Please use yours. Sorry for the noise.

Best regards,
baolu