Received-SPF: pass (google.com: domain of linux-kernel+bounces-303-linux.lists.archive=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) client-ip=139.178.88.99;
Message-ID: <8c8a1e49-5447-4d61-b3ba-782873cc0f4a@linux.intel.com>
Date: Fri, 15 Dec 2023 09:03:37 +0800
Precedence: bulk
MIME-Version: 1.0
User-Agent: Mozilla Thunderbird
Subject: Re: [PATCH 2/2] iommu/vt-d: don's issue devTLB flush request when
 device is disconnected
From: Ethan Zhao <haifeng.zhao@linux.intel.com>
To: Baolu Lu <baolu.lu@linux.intel.com>, bhelgaas@google.com,
 dwmw2@infradead.org, will@kernel.org, robin.murphy@arm.com
Cc: linux-pci@vger.kernel.org, iommu@lists.linux.dev,
 linux-kernel@vger.kernel.org, Haorong Ye <yehaorong@bytedance.com>
References: <20231213034637.2603013-1-haifeng.zhao@linux.intel.com>
 <20231213034637.2603013-3-haifeng.zhao@linux.intel.com>
 <96051115-c928-4f3c-bd65-4f3f8e83ca9c@linux.intel.com>
 <e6fddbb0-fa8f-43d4-8a5e-d177e9465912@linux.intel.com>
In-Reply-To: <e6fddbb0-fa8f-43d4-8a5e-d177e9465912@linux.intel.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit


On 12/14/2023 10:26 AM, Ethan Zhao wrote:
>
> On 12/13/2023 7:59 PM, Baolu Lu wrote:
>> On 2023/12/13 11:46, Ethan Zhao wrote:
>>> For those endpoint devices connect to system via hotplug capable ports,
>>> users could request a warm reset to the device by flapping device's 
>>> link
>>> through setting the slot's link control register, as pciehpt_ist() 
>>> DLLSC
>>> interrupt sequence response, pciehp will unload the device driver and
>>> then power it off.
>>
>> Is it possible for pciehp to disable ATS on the device before unloading
>> the driver? Or should the device follow some specific steps to warm
>> reset the device?
>>
> In this case, link down first, then pciehp_ist() got DLLSC interrupt 
> to know
>
> that, I don't think it makes sense to disable device ATS here, but it 
> could
>
> flag the device is ATS disabled, well,  "disconnected" is enough to let
>
> vt-d like software knows the device state.
>
>
For hot "unplug" cases:

1. safe_removal

   Users request unplug the device via sysfs or press the attention button,

   Then pciehp will response to unconfig device/unload device driver, power

   if off, and devcie is ready to remove. in this case, devTLB invalidate

   request is sent before device link to be brought down or device power

   to be turned off. so it doesn't trigger the hard lockup.

2. supprise_removal

  Users remove the devece directly or bring the device link down/turn off

  device power first by setting pci config space, link-down/not-present/

  power-off are all handled by pciehp the same way "supprise_removal", in

  such case, pciehp_ist() will flag the device as "disconnected" first, then

  unconfig the devcie, unload driver, iommu release device(issing devTLB 
flush)

  delete device. so we checking the device state could work such cases.

But I am still think about if there are other windows.


Thanks,

Ethan


>> What happens if IOMMU issues device TLB invalidation after link down but
>> before pci_dev_is_disconnected() returns true?
>
> Seems it wouldn't happen with hotplug cases, safe_removal or
>
> supprise_removal.
>
>
>
> Thanks,
>
> Ethan
>
>>
>> Best regards,
>> baolu