Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <7090732d-291f-de14-21d2-afb4dc5e0d8c@intel.com>
Date:   Mon, 19 Sep 2022 16:01:46 +0200
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101
 Thunderbird/91.11.0
Subject: Re: [topic/core-for-CI] Revert "iommu/dma: Fix race condition during
 iova_domain initialization"
Content-Language: en-US
To:     Lucas De Marchi <lucas.demarchi@intel.com>
CC:     Robin Murphy <robin.murphy@arm.com>,
        <intel-gfx@lists.freedesktop.org>,
        Gwan-gyeong Mun <gwan-gyeong.mun@intel.com>,
        Yunfei Wang <yf.wang@mediatek.com>,
        Ning Li <ning.li@mediatek.com>,
        Miles Chen <miles.chen@mediatek.com>,
        Joerg Roedel <jroedel@suse.de>, <iommu@lists.linux.dev>,
        <linux-kernel@vger.kernel.org>
References: <20220914124045.903883-1-karolina.drobnik@intel.com>
 <20220914150121.y6ucj4mav65mt7we@ldmartin-desk2.lan>
 <c539a4fb-795b-0b33-2543-6a3e94164676@arm.com>
 <fbca082a-7fb1-bdb5-af93-69061e597b34@intel.com>
 <20220916203238.g364hpyrcuoo4h5m@ldmartin-desk2.lan>
From:   Karolina Drobnik <karolina.drobnik@intel.com>
Organization: Intel Technology Poland sp. z o.o. - ul. Slowackiego 173, 80-298
 Gdansk - KRS 101882 - NIP 957-07-52-316
In-Reply-To: <20220916203238.g364hpyrcuoo4h5m@ldmartin-desk2.lan>
Content-Type: text/plain; charset="UTF-8"; format=flowed
Content-Transfer-Encoding: 8bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?ZlFIZDE1dXhXMXJwWmhFSmRhMzhNZDBTbG5ZYkp2aXVYbGN1NnpFV25BNFk4?=
 =?utf-8?B?QnBxN2xZRnp5VGU2eWd3c1JJU2VaR0Fwekc3eWtmOFo2S3o3K3pieTJ2YW5Y?=
 =?utf-8?B?anh0M0hBVkFkMVFIK290aWNmODd5Wm5Od05EdUJtQ2xzdlAwYzQwbEVXaGdh?=
 =?utf-8?B?VHdoY1ptSjdSL0o1TUdZZXNQNlBTZGt3NlZSS2lWSG9TaVN2Yk5Xa0VlOVJE?=
 =?utf-8?B?YkZ0RVNTZkh5Q202VHo3SE9BWHRZc3JJNnZnNXNqNEkvMmZoeXZYT0JOZ09u?=
 =?utf-8?B?QkFBNGVYMWpmN1hUN0JpZ3VJRkFnSWJQQ3kxWnNzMXROVHM4dVNibFhzc3Nr?=
 =?utf-8?B?UHplb09YSERGZlFyL3JYd3dEWnZyRGt1amxXa3RZZzZtaVZoVGdVTC85Y0xC?=
 =?utf-8?B?cXZDd3pxMDZidm9IbUxjYkEvV1E3Y3ZsajBkSzFObGZxWXN3Z0xSQk9mOU1z?=
 =?utf-8?B?cEFLbWlNWHNxRDNDSHNGeVpQTXFBVk4yMklBWTVZYjNpZk5iUDRXRTB0ckc2?=
 =?utf-8?B?bU9JRnVWWm9DNzlKYzQyQkZDRjlyVkxjSnpMOE5paEc0ZGNzZG5NWldDb1Fx?=
 =?utf-8?B?amRpbkg3Q3A3OVFjaUdGczMwRDBJR29HVThQM01uMGloV3ViNmdNMUVDdHEv?=
 =?utf-8?B?YVVReEhFM1lTOFdTdGttR2l6OUdvZ2t0dCtjdkp3dXBmeHBxVmNXMVFDcU9X?=
 =?utf-8?B?bWNkcFBLbVBPU3JodWsxS3BKN1R2c0toUTB4NU5xTWRUWmswTlpxUlZ2cHVC?=
 =?utf-8?B?Z21HVWdGVndlTzBmQkQ1aHBGY3NZUjAvUHp4QVBtcFJtV0ovMXIvQWdLdWJU?=
 =?utf-8?B?Wi9OMzdhemIzdzRUMmMvZXViTHJQVk16UHplMkY3dzFBY1Bac1BFdGRWaEJY?=
 =?utf-8?B?THdPd1poQlBuc3plMlFjVWQwRy9HbUhCczZ2YklVc3c1RFZnL1pPbE5iKysx?=
 =?utf-8?B?Zm9ETldEd3puN2k4KzhZMmNnMVg1ekhtM0hpdzg3by9PaFhkWW9WOU9aczJJ?=
 =?utf-8?B?RGxLOTdPdGZvTC9PcStHay8xY25kRjlTNW1kakF5dTZsR2xjaWJENmZ3endW?=
 =?utf-8?B?a2VxVHc0TCt2Y2QydlFhem5NWU05Q2EyK0RjbzRCT3ByRGFaTW81OXlNRXdv?=
 =?utf-8?B?SHZNam1qV2ZZUGdFQXgwNWR4Z1VNQjZWTHdZSVNlaGRUUVlValhKbUZ0cmFt?=
 =?utf-8?B?M3dSc3pYMkVhMjhOa25nbjN6MmxadjUwaTQzZlorZ2JBOFBWeWt5NnJhR2ti?=
 =?utf-8?B?THJmSHZFT3Y1SWF6UHAzYkVwOE16T3VyOXlLcTNINVVieE9hZUpKUVdsbWdZ?=
 =?utf-8?B?c3Z2dWR0QkJCSFNheTl4UXMxMDhwTVh6SFBZZk45aVU2ZjNDY1NyY3M1RzNH?=
 =?utf-8?B?emluMUNmUWtqdG5TTTVqWjhaZUJjVkFQdTcxckh6YnVxUkJJTXM2OXI3U2M5?=
 =?utf-8?B?dUs1S1NLeDFkWWhOUUFLTW1ObDQwcE5ZcUlMMzZubmU5cFppU2pUYUVIWEVl?=
 =?utf-8?B?VE1Wbnc5V1duU2xZRlVyMnQ5cWQ1RlJDUFllSlQzdXdOMGtxYWI2bllZcFJt?=
 =?utf-8?B?MWxvaUlZdEhJQjYzV3Z5Y2xmbEdxQ1dDRUkyQkVOZWdwL2ZwbzkrK1hRMDlP?=
 =?utf-8?B?RHRBRVhSbSt0cUdNMlNEZkl4bTNMT3ZwMEVqVVpRamVpeEhxTkpJQlRJRTlr?=
 =?utf-8?B?S1h3eE5UMmdXRmdrY1NYaHorZzdwOGtXaDREVnVPUUxNZEw0YUhDemtVSVRu?=
 =?utf-8?B?b3YwTlZyTlhnNWxSVVRLRU5aUW1JTjI1R3pPYU9hSGpXNEhoQ3JtNk1DRnh5?=
 =?utf-8?B?YVQyYitkVFNmTHNJZ0VFNTBBblIzVVdFdVJEY2syNmt4RzJrZG5kRlErMGl0?=
 =?utf-8?B?QWFnQzg4Z2xHbTFkSmJ1VDV0K05ZaXZ3amhXcXRsOWdldld3V2pUV0lFOTZP?=
 =?utf-8?B?WSt0UC9wamo5ZVd5cmRQWi9BRW94cTRsYmNhWkhsd0hVdGoxWTlMYU9VQmhq?=
 =?utf-8?B?SlU0NTBuZm9QSzE3U2gxdlhjVXN4M1o4UlBlNXU5ekN0aG52czNSWjNtU3Rm?=
 =?utf-8?B?RGRDREY1VWg1cHhHVmFIOGlVYzBOcFc1OUdMbjdqYm9EYWNxSjQ1ZE9oZVVy?=
 =?utf-8?B?TE9VMjRNZDl2THZTbm15akFSdFVoQmdWL2ExMWs3M0pDTGFHTWhwN3NBK2hN?=
 =?utf-8?B?K0E9PQ==?=
X-MS-Exchange-CrossTenant-Network-Message-Id: a28bba6f-92e2-457f-13b0-08da9a477f7c
X-MS-Exchange-CrossTenant-AuthSource: BY5PR11MB4243.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 19 Sep 2022 14:01:49.9946
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: 60O0I8BZy+mlFTWzgJh0hAJpLJVLI17iuXDWwgx/7VIIdvVLqLrFGxbpTS5KhWYp+gvYZCoGLJlzX5l9OMLpJXroqpOydAoFjLlPJY2k6Oo=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: CO1PR11MB5042
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-5.4 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,NICE_REPLY_A,
        RCVD_IN_DNSWL_MED,SPF_HELO_PASS,SPF_NONE autolearn=ham
        autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On 16.09.2022 22:32, Lucas De Marchi wrote:
> On Fri, Sep 16, 2022 at 02:24:00PM +0200, Karolina Drobnik wrote:
>> On 14.09.2022 17:54, Robin Murphy wrote:
>>> On 2022-09-14 16:01, Lucas De Marchi wrote:
>>>> On Wed, Sep 14, 2022 at 02:40:45PM +0200, Karolina Drobnik wrote:
>>>>> This reverts commit ac9a5d522bb80be50ea84965699e1c8257d745ce.
>>>>>
>>>>> This change introduces a regression on Alder Lake that
>>>>> completely blocks testing. To enable CI and avoid possible
>>>>> circular locking warning, revert the patch.
>>>>
>>>> We are already on rc5. Are iommu authors involved aware of this 
>>>> issue? We could do this in our "for CI only" branch, but it's 
>>>> equally important that this is fixed for 6.0
>>>>
>>>> Cc'ing them.
>>>
>>> The lockdep report doesn't make much sense to me - the deadlock cycle
>>> it's reporting doesn't even involve the mutex added by that commit,
>>> and otherwise the lock ordering between the IOMMU bus notifier(s) and
>>> cpu_hotplug_lock has existed for ages. Has lockdep somehow got
>>> multiple different and unrelated bus notifiers mixed up, maybe?
>>>
>>> FWIW nobody else has reported anything, and that mutex addresses a 
>>> real-world concurrency issue, so I'm not convinced a revert is 
>>> appropriate without at least a much clearer justification.
>>
>> I'll share more background on this regression. We've noticed that no
>> tests were run for Alder Lake platforms. This may happens when, for
>> example, there is a kernel taint or lockdep warning.
>>
>> Links:
>> https://intel-gfx-ci.01.org/tree/drm-tip/bat-adlm-1.html
>> https://intel-gfx-ci.01.org/tree/drm-tip/bat-adlp-6.html
>>
>> The CI logs (which can be found for example here[1], boot0 file)
>> revealed a lockdep warning. One of the recent changes in the area was
>> commit ac9a5d522bb8 ("iommu/dma: Fix race condition during iova_domain
>> initialization"), and I sent a revert patch to test it on CI[2]. This
>> proved to be effective, as the tests started running on Alder Lake
>> platform:
>> https://intel-gfx-ci.01.org/tree/drm-tip/Trybot_108474v1/index.html?hosts=adlp 
>>
>>
>> To be clear, that revert is just a way of unblocking CI testing, the
>> problem requires a specific fix.
>>
>> Lucas, would it be possible to merge this revert to the topic branch to
>> unblock Alder Lake until this issue is fixed? I'm afraid that some
>> regressions could slip through the cracks if we don't do it soon enough.
> 
> Yeah. Let's have CI running with the revertt so we can see if on next runs
> it will really show it was a regression or if it's something else. I
> think it will help us understand why it's failing.

Thanks for the merge. It looks like all adls are doing better now
(revert went in CI_DRM_12147):
https://intel-gfx-ci.01.org/tree/drm-tip/bat-adlp-6.html
https://intel-gfx-ci.01.org/tree/drm-tip/bat-adln-1.html
https://intel-gfx-ci.01.org/tree/drm-tip/bat-adlm-1.html (CI_DRM_12149 
seems to show a different problem)

All the best,
Karolina
> 
> Lucas De Marchi
> 
> 
>>
>> Thanks,
>> Karolina
>>
>> ----
>> [1] -
>> https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_12145/bat-adlm-1/igt@runner@aborted.html 
>>
>> [2] - https://patchwork.freedesktop.org/series/108474/
>>
>>> Robin.
>>>
>>>> thanks Lucas De Marchi
>>>>
>>>>>
>>>>> kernel log:
>>>>>
>>>>> ====================================================== WARNING: 
>>>>> possible circular locking dependency detected 
>>>>> 6.0.0-rc5-CI_DRM_12132-g6c93e979e542+ #1 Not tainted 
>>>>> ------------------------------------------------------ cpuhp/0/15
>>>>> is trying to acquire lock: ffff8881013df278 
>>>>> (&(&priv->bus_notifier)->rwsem){++++}-{3:3}, at: 
>>>>> blocking_notifier_call_chain+0x20/0x50 but task is already holding 
>>>>> lock: ffffffff826490c0 (cpuhp_state-up){+.+.}-{0:0}, at:
>>>>> cpuhp_thread_fun+0x48/0x1f0 which lock already depends on the
>>>>> new loc the existing dependency chain (in reverse order) is: ->
>>>>> #3 (cpuhp_state-up){+.+.}-{0:0}: lock_acquire+0xd3/0x310 
>>>>> cpuhp_thread_fun+0xa6/0x1f0 smpboot_thread_fn+0x1b5/0x260 
>>>>> kthread+0xed/0x120 ret_from_fork+0x1f/0x30 -> #2 
>>>>> (cpu_hotplug_lock){++++}-{0:0}: lock_acquire+0xd3/0x310 
>>>>> __cpuhp_state_add_instance+0x43/0x1c0 
>>>>> iova_domain_init_rcaches+0x199/0x1c0 
>>>>> iommu_setup_dma_ops+0x130/0x440 bus_iommu_probe+0x26a/0x2d0 
>>>>> bus_set_iommu+0x82/0xd0 intel_iommu_init+0xe33/0x1039 
>>>>> pci_iommu_init+0x9/0x31 do_one_initcall+0x53/0x2f0 
>>>>> kernel_init_freeable+0x18f/0x1e1 kernel_init+0x11/0x120 
>>>>> ret_from_fork+0x1f/0x30 -> #1 
>>>>> (&domain->iova_cookie->mutex){+.+.}-{3:3}: lock_acquire+0xd3/0x310 
>>>>> __mutex_lock+0x97/0xf10 iommu_setup_dma_ops+0xd7/0x440 
>>>>> iommu_probe_device+0xa4/0x180 iommu_bus_notifier+0x2d/0x40 
>>>>> notifier_call_chain+0x31/0x90 
>>>>> blocking_notifier_call_chain+0x3a/0x50 device_add+0x3c1/0x900 
>>>>> pci_device_add+0x255/0x580 pci_scan_single_device+0xa6/0xd0 
>>>>> pci_scan_slot+0x7a/0x1b0 pci_scan_child_bus_extend+0x35/0x2a0 
>>>>> vmd_probe+0x5cd/0x970 pci_device_probe+0x95/0x110 
>>>>> really_probe+0xd6/0x350 __driver_probe_device+0x73/0x170 
>>>>> driver_probe_device+0x1a/0x90 __driver_attach+0xbc/0x190 
>>>>> bus_for_each_dev+0x72/0xc0 bus_add_driver+0x1bb/0x210 
>>>>> driver_register+0x66/0xc0 do_one_initcall+0x53/0x2f0 
>>>>> kernel_init_freeable+0x18f/0x1e1 kernel_init+0x11/0x120 
>>>>> ret_from_fork+0x1f/0x30 -> #0 
>>>>> (&(&priv->bus_notifier)->rwsem){++++}-{3:3}: 
>>>>> validate_chain+0xb3f/0x2000 __lock_acquire+0x5a4/0xb70 
>>>>> lock_acquire+0xd3/0x310 down_read+0x39/0x140 
>>>>> blocking_notifier_call_chain+0x20/0x50 device_add+0x3c1/0x900 
>>>>> platform_device_add+0x108/0x240 coretemp_cpu_online+0xe1/0x15e 
>>>>> [coretemp] cpuhp_invoke_callback+0x181/0x8a0 
>>>>> cpuhp_thread_fun+0x188/0x1f0 smpboot_thread_fn+0x1b5/0x260 
>>>>> kthread+0xed/0x120 ret_from_fork+0x1f/0x30 other info that might
>>>>> help us debug thi Chain exists of &(&priv->bus_notifier)->rwsem --> 
>>>>> cpu_hotplug_lock --> cpuhp_state- Possible unsafe locking scenari 
>>>>> CPU0 CPU1 ----                    ---- lock(cpuhp_state-up); 
>>>>> lock(cpu_hotplug_lock); lock(cpuhp_state-up); 
>>>>> lock(&(&priv->bus_notifier)->rwsem); *** DEADLOCK * 2 locks held by 
>>>>> cpuhp/0/15: #0: ffffffff82648f10 (cpu_hotplug_lock){++++}-{0:0}, 
>>>>> at: cpuhp_thread_fun+0x48/0x1f0 #1: ffffffff826490c0 
>>>>> (cpuhp_state-up){+.+.}-{0:0}, at: cpuhp_thread_fun+0x48/0x1f0 stack 
>>>>> backtrace: CPU: 0 PID: 15 Comm:
>>>>> cpuhp/0 Not tainted 6.0.0-rc5-CI_DRM_12132-g6c93e979e542+ #1
>>>>> Hardware name: Intel Corporation Alder Lake Client 
>>>>> Platform/AlderLake-P DDR4 RVP, BIOS 
>>>>> ADLPFWI1.R00.3135.A00.2203251419 03/25/2022 Call Trace: <TASK> 
>>>>> dump_stack_lvl+0x56/0x7f check_noncircular+0x132/0x150 
>>>>> validate_chain+0xb3f/0x2000 __lock_acquire+0x5a4/0xb70 
>>>>> lock_acquire+0xd3/0x310 ? blocking_notifier_call_chain+0x20/0x50 
>>>>> down_read+0x39/0x140 ? blocking_notifier_call_chain+0x20/0x50 
>>>>> blocking_notifier_call_chain+0x20/0x50 device_add+0x3c1/0x900 ? 
>>>>> dev_set_name+0x4e/0x70 platform_device_add+0x108/0x240 
>>>>> coretemp_cpu_online+0xe1/0x15e [coretemp] ? 
>>>>> create_core_data+0x550/0x550 [coretemp] 
>>>>> cpuhp_invoke_callback+0x181/0x8a0 cpuhp_thread_fun+0x188/0x1f0 ?
>>>>> smpboot_thread_fn+0x1e/0x260 smpboot_thread_fn+0x1b5/0x260 ? 
>>>>> sort_range+0x20/0x20 kthread+0xed/0x120 ? 
>>>>> kthread_complete_and_exit+0x20/0x20 ret_from_fork+0x1f/0x30 </TASK>
>>>>>
>>>>> Closes: https://gitlab.freedesktop.org/drm/intel/-/issues/6641
>>>>>
>>>>> Signed-off-by: Karolina Drobnik <karolina.drobnik@intel.com> Cc:
>>>>> Lucas De Marchi <lucas.demarchi@intel.com> --- 
>>>>> drivers/iommu/dma-iommu.c | 17 ++++------------- 1 file changed,
>>>>> 4 insertions(+), 13 deletions(-)
>>>>>
>>>>> diff --git a/drivers/iommu/dma-iommu.c b/drivers/iommu/dma-iommu.c 
>>>>> index 17dd683b2fce..9616b473e4c7 100644 --- 
>>>>> a/drivers/iommu/dma-iommu.c +++ b/drivers/iommu/dma-iommu.c @@ 
>>>>> -65,7 +65,6 @@ struct iommu_dma_cookie {
>>>>>
>>>>> /* Domain for flush queue callback; NULL if flush queue not in use 
>>>>> */ struct iommu_domain        *fq_domain; -    struct mutex mutex; };
>>>>>
>>>>> static DEFINE_STATIC_KEY_FALSE(iommu_deferred_attach_enabled); @@
>>>>> -312,7 +311,6 @@ int iommu_get_dma_cookie(struct iommu_domain 
>>>>> *domain) if (!domain->iova_cookie) return -ENOMEM;
>>>>>
>>>>> -    mutex_init(&domain->iova_cookie->mutex); return 0; }
>>>>>
>>>>> @@ -563,33 +561,26 @@ static int iommu_dma_init_domain(struct 
>>>>> iommu_domain *domain, dma_addr_t base, }
>>>>>
>>>>> /* start_pfn is always nonzero for an already-initialised domain
>>>>> */ -    mutex_lock(&cookie->mutex); if (iovad->start_pfn) { if (1UL 
>>>>> << order != iovad->granule || base_pfn != iovad->start_pfn)
>>>>> { pr_warn("Incompatible range for DMA domain\n"); - ret = -EFAULT; 
>>>>> -            goto done_unlock; +            return -EFAULT; }
>>>>>
>>>>> -        ret = 0; -        goto done_unlock; +        return 0; }
>>>>>
>>>>> init_iova_domain(iovad, 1UL << order, base_pfn); ret = 
>>>>> iova_domain_init_rcaches(iovad); if (ret) -        goto 
>>>>> done_unlock; +        return ret;
>>>>>
>>>>> /* If the FQ fails we can simply fall back to strict mode */ if 
>>>>> (domain->type == IOMMU_DOMAIN_DMA_FQ && iommu_dma_init_fq(domain)) 
>>>>> domain->type = IOMMU_DOMAIN_DMA;
>>>>>
>>>>> -    ret = iova_reserve_iommu_regions(dev, domain); - -done_unlock: 
>>>>> -    mutex_unlock(&cookie->mutex); -    return
>>>>> ret; +    return iova_reserve_iommu_regions(dev, domain); }
>>>>>
>>>>> /** -- 2.25.1
>>>>>