Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <86065501-ab2e-09b4-71cd-c0b18ede00ed@acm.org>
Date:   Tue, 16 May 2023 08:12:19 -0700
MIME-Version: 1.0
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101
 Thunderbird/102.10.0
Subject: Re: [PATCH 2/2] ufs: don't use the fair tag sharings
Content-Language: en-US
To:     Yu Kuai <yukuai1@huaweicloud.com>,
        Christoph Hellwig <hch@infradead.org>
Cc:     Ed Tsai <ed.tsai@mediatek.com>, axboe@kernel.dk,
        linux-block@vger.kernel.org, linux-scsi@vger.kernel.org,
        linux-kernel@vger.kernel.org, martin.petersen@oracle.com,
        stanley.chu@mediatek.com, peter.wang@mediatek.com,
        chun-hung.wu@mediatek.com, alice.chao@mediatek.com,
        powen.kao@mediatek.com, naomi.chu@mediatek.com,
        wsd_upstream@mediatek.com, "yukuai (C)" <yukuai3@huawei.com>
References: <20230509065230.32552-1-ed.tsai@mediatek.com>
 <20230509065230.32552-3-ed.tsai@mediatek.com>
 <ZF0K7A6G2cYBjSgn@infradead.org>
 <aa9af9ae-62a4-6469-244c-b5d9106bb044@acm.org>
 <ZF5G5ztMng8Xbd1W@infradead.org>
 <2740ee82-e35f-1cbf-f5d0-373f94eb14a5@acm.org>
 <de3f41a0-b13d-d4f6-765a-19b857bce53e@huaweicloud.com>
From:   Bart Van Assche <bvanassche@acm.org>
In-Reply-To: <de3f41a0-b13d-d4f6-765a-19b857bce53e@huaweicloud.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit
Precedence: bulk

On 5/12/23 20:09, Yu Kuai wrote:
> 在 2023/05/13 2:12, Bart Van Assche 写道:
>> The fair tag sharing algorithm has a negative impact on all SCSI 
>> devices with multiple logical units. This is because logical units are 
>> considered active until (request timeout) seconds have elapsed after 
>> the logical unit stopped being used (see also the blk_mq_tag_idle() 
>> call in blk_mq_timeout_work()). UFS users are hit by this because UFS 
>> 3.0 devices have a limited queue depth (32) and because power 
>> management commands are submitted to a logical unit (WLUN). Hence, it 
>> happens often that the block layer "active queue" counter is equal to 
>> 2 while only one logical unit is being used actively (a logical unit 
>> backed by NAND flash). The performance difference between queue depths 
>> 16 and 32 for UFS devices is significant.
> 
> We meet similiar problem before, but I think remove tag fair sharing
> might cause some problems, because get tag is not fair currently, for
> example 2 devices share 32 tag, while device a issue large amount of
> io concurrently, and device b only issue one io, in this case, if fair
> tag sharing is removed, device b can get bad io latency.
> 
> By the way, I tried to propose a way to workaround this by following:
> 
> 1) disable fair tag sharing untill get tag found no tag is avaiable;
> 2) enable fair tag sharing again if the disk donesn't faild to get tag
> for a period of time;
> 
> Can this approch be considered?

I'm afraid that this approach won't help for the UFS driver since it is 
likely that all tags are in use by a single logical unit during an IOPS 
test. Hence, fair sharing would be enabled even when we don't want it to 
be enabled.

I propose that we switch to one of these two approaches:
* Either remove the fair tag sharing code entirely and rely on the 
fairness mechanism provided by the sbitmap code. I'm referring to how 
__sbitmap_queue_wake_up() uses the wake_index member variable.
* Or make the behavior of the fairness algorithm configurable from user 
space. One possible approach is to make the proportion of tags for a 
logical unit / NVMe namespace configurable via sysfs. This will allow to 
reduce the number of tags for the WLUN of UFS devices.

Thanks,

Bart.