Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
Subject: Re: [PATCH v2] nvme-multipath: Early exit if no path is available
To:     Hannes Reinecke <hare@suse.de>, Sagi Grimberg <sagi@grimberg.me>,
        "Daniel Wagner" <dwagner@suse.de>
CC:     <linux-nvme@lists.infradead.org>, <linux-kernel@vger.kernel.org>,
        "Jens Axboe" <axboe@fb.com>, Keith Busch <kbusch@kernel.org>,
        Christoph Hellwig <hch@lst.de>
References: <20210127103033.15318-1-dwagner@suse.de>
 <db9baae0-547c-7ff4-8b2c-0b95f14be67c@huawei.com>
 <20210128075837.u5u56t23fq5gu6ou@beryllium.lan>
 <69575290-200e-b4a1-4269-c71e4c2cc37b@huawei.com>
 <20210128094004.erwnszjqcxlsi2kd@beryllium.lan>
 <ebb1d098-3ded-e592-4419-e905aabe824f@huawei.com>
 <675d3cf7-1ae8-adc5-b6d0-359fe10f6b23@grimberg.me>
 <59cd053e-46cb-0235-141f-4ce919c93f48@huawei.com>
 <65392653-6b03-9195-f686-5fe4b3290bd2@suse.de>
 <81b22bbf-4dd3-6161-e63a-9699690a4e4f@huawei.com>
 <715dd943-0587-be08-2840-e0948cf0bc62@suse.de>
 <eb131d8f-f009-42e7-105d-58b84060f0dd@huawei.com>
 <ac019690-7f02-d28c-ed58-bfc8c1d48879@suse.de>
 <6ceff3cb-c9e9-7e74-92f0-dd745987c943@huawei.com>
 <114751ac-1f7d-ce5e-12c5-7d6303bdb999@suse.de>
 <aebc6c2d-4711-95ba-daf2-1cd17fc6f0e7@huawei.com>
 <47a1b796-9d91-5947-4bac-dd8f397041a3@suse.de>
 <bd37abd5-759d-efe2-fdcd-8b004a41c75a@huawei.com>
 <2752ed93-6bd5-1a13-0e05-b91e2dcc24e1@suse.de>
From:   Chao Leng <lengchao@huawei.com>
Message-ID: <6ef6df55-318b-887d-b14f-62cae6419b4a@huawei.com>
Date:   Tue, 2 Feb 2021 09:12:30 +0800
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:68.0) Gecko/20100101
 Thunderbird/68.9.0
MIME-Version: 1.0
In-Reply-To: <2752ed93-6bd5-1a13-0e05-b91e2dcc24e1@suse.de>
Content-Type: text/plain; charset="utf-8"; format=flowed
Content-Language: en-US
Content-Transfer-Encoding: 8bit
Precedence: bulk


On 2021/2/1 18:45, Hannes Reinecke wrote:
> On 2/1/21 10:40 AM, Chao Leng wrote:
>>
>>
>> On 2021/2/1 16:57, Hannes Reinecke wrote:
>>> On 2/1/21 9:47 AM, Chao Leng wrote:
>>>>
>>>>
>>>> On 2021/2/1 15:29, Hannes Reinecke wrote:[ .. ]
>>>>> Urgh. Please, no. That is well impossible to debug.
>>>>> Can you please open-code it to demonstrate where the difference to the current (and my fixed) versions is?
>>>>> I'm still not clear where the problem is once we applied both patches.
>>>> For example assume the list has three path, and all path is not NVME_ANA_OPTIMIZED:
>>>> head->next = ns1;
>>>> ns1->next = ns2;
>>>> ns2->next = head;
>>>> old->next = ns2;
>>>>
>>> And this is where I have issues with.
>>> Where does 'old' come from?
>>> Clearly it was part of the list at one point; so what happened to it?
>> I explained this earlier.
>> In nvme_ns_remove, there is a hole between list_del_rcu and
>> nvme_mpath_clear_current_path. If head->current_path is the "old", and
>> the "old" is removing. The "old" is already removed from the list by
>> list_del_rcu, but head->current_path is not clear to NULL by
>> nvme_mpath_clear_current_path.
>> Find path is race with nvme_ns_remove, use the "old" pass to
>> nvme_round_robin_path to find path.
> 
> Ah. So this should be better:
> 
> @@ -202,10 +202,12 @@ static struct nvme_ns *__nvme_find_path(struct nvme_ns_head *head, int node)
>   static struct nvme_ns *nvme_next_ns(struct nvme_ns_head *head,
>                  struct nvme_ns *ns)
>   {
> -       ns = list_next_or_null_rcu(&head->list, &ns->siblings, struct nvme_ns,
> -                       siblings);
> -       if (ns)
> -               return ns;
> +       if (ns && !test_bit(NVME_NS_REMOVING, &ns->flags)) {
> +               ns = list_next_or_null_rcu(&head->list, &ns->siblings,
> +                                          struct nvme_ns, siblings);
> +               if (ns)
> +                       return ns;
> +       }
>          return list_first_or_null_rcu(&head->list, struct nvme_ns, siblings);
>   }
> 
> The 'NVME_NS_REMOVING' bit is set before list_del_rcu() is called, so it should guard against the issue you mentioned.
Looks useless, it is still infinite loop. You can check the workflow for the scenario I mentioned.
> 
> Cheers,
> 
> Hannes