As described in the comment of blkcg_activate_policy(),
*Update of each blkg is protected by both queue and blkcg locks so
that holding either lock and testing blkcg_policy_enabled() is
always enough for dereferencing policy data.*
with queue lock held, there is no need to hold blkcg lock in
blkcg_deactivate_policy(). Similar case is in
blkcg_activate_policy(), which has removed holding of blkcg lock in
commit 4c55f4f9ad3001ac1fefdd8d8ca7641d18558e23.
Signed-off-by: Jiang Biao <[email protected]>
Signed-off-by: Wen Yang <[email protected]>
CC: Tejun Heo <[email protected]>
CC: Jens Axboe <[email protected]>
---
block/blk-cgroup.c | 5 -----
1 file changed, 5 deletions(-)
diff --git a/block/blk-cgroup.c b/block/blk-cgroup.c
index c2033a2..2b7f8d0 100644
--- a/block/blk-cgroup.c
+++ b/block/blk-cgroup.c
@@ -1367,17 +1367,12 @@ void blkcg_deactivate_policy(struct request_queue *q,
__clear_bit(pol->plid, q->blkcg_pols);
list_for_each_entry(blkg, &q->blkg_list, q_node) {
- /* grab blkcg lock too while removing @pd from @blkg */
- spin_lock(&blkg->blkcg->lock);
-
if (blkg->pd[pol->plid]) {
if (pol->pd_offline_fn)
pol->pd_offline_fn(blkg->pd[pol->plid]);
pol->pd_free_fn(blkg->pd[pol->plid]);
blkg->pd[pol->plid] = NULL;
}
-
- spin_unlock(&blkg->blkcg->lock);
}
spin_unlock_irq(q->queue_lock);
--
2.7.4
> Il giorno 17 apr 2018, alle ore 09:10, Jiang Biao <[email protected]> ha scritto:
>
> As described in the comment of blkcg_activate_policy(),
> *Update of each blkg is protected by both queue and blkcg locks so
> that holding either lock and testing blkcg_policy_enabled() is
> always enough for dereferencing policy data.*
> with queue lock held, there is no need to hold blkcg lock in
> blkcg_deactivate_policy(). Similar case is in
> blkcg_activate_policy(), which has removed holding of blkcg lock in
> commit 4c55f4f9ad3001ac1fefdd8d8ca7641d18558e23.
>
Hi,
by chance, did you check whether this may cause problems with bfq,
being the latter not protected by the queue lock as cfq?
Thanks,
Paolo
> Signed-off-by: Jiang Biao <[email protected]>
> Signed-off-by: Wen Yang <[email protected]>
> CC: Tejun Heo <[email protected]>
> CC: Jens Axboe <[email protected]>
> ---
> block/blk-cgroup.c | 5 -----
> 1 file changed, 5 deletions(-)
>
> diff --git a/block/blk-cgroup.c b/block/blk-cgroup.c
> index c2033a2..2b7f8d0 100644
> --- a/block/blk-cgroup.c
> +++ b/block/blk-cgroup.c
> @@ -1367,17 +1367,12 @@ void blkcg_deactivate_policy(struct request_queue *q,
> __clear_bit(pol->plid, q->blkcg_pols);
>
> list_for_each_entry(blkg, &q->blkg_list, q_node) {
> - /* grab blkcg lock too while removing @pd from @blkg */
> - spin_lock(&blkg->blkcg->lock);
> -
> if (blkg->pd[pol->plid]) {
> if (pol->pd_offline_fn)
> pol->pd_offline_fn(blkg->pd[pol->plid]);
> pol->pd_free_fn(blkg->pd[pol->plid]);
> blkg->pd[pol->plid] = NULL;
> }
> -
> - spin_unlock(&blkg->blkcg->lock);
> }
>
> spin_unlock_irq(q->queue_lock);
> --
> 2.7.4
>
> Il giorno 18 apr 2018, alle ore 11:18, [email protected] ha scritto:
>
> Hi,
>>> Il giorno 17 apr 2018, alle ore 09:10, Jiang Biao <[email protected]> ha scritto:
>>>
>>> As described in the comment of blkcg_activate_policy(),
>>> *Update of each blkg is protected by both queue and blkcg locks so
>>> that holding either lock and testing blkcg_policy_enabled() is
>>> always enough for dereferencing policy data.*
>>> with queue lock held, there is no need to hold blkcg lock in
>>> blkcg_deactivate_policy(). Similar case is in
>>> blkcg_activate_policy(), which has removed holding of blkcg lock in
>>> commit 4c55f4f9ad3001ac1fefdd8d8ca7641d18558e23.
>>>
>>
>> Hi,
>> by chance, did you check whether this may cause problems with bfq,
>> being the latter not protected by the queue lock as cfq?
> Checked the bfq code, bfq seems never used blkcg lock derectly, and
> update of blkg in the common code is protected by both queue and
> blkcg locks, so IMHO this patch would not introduce any new problem
> with bfq, even though bfq is not protected by queue lock.
Thank you very much for checking. Then:
Acked-by: Paolo Valente <[email protected]>
Thanks,
Paolo
> On the other hand, the locks (queue lock/blkcg lock) used to protected
> the update of blkg seems a bit too heavyweight, especially the queue lock
> which is used too widely may cause races with other contexts. I wonder
> if there is any way to ease the case? e.g. add a new lock for blkg's own.:)
>
> Jiang,
> Regards
On 4/18/18 3:18 AM, [email protected] wrote:
> Hi,
>>> Il giorno 17 apr 2018, alle ore 09:10, Jiang Biao <[email protected]> ha scritto:
>>>
>>> As described in the comment of blkcg_activate_policy(),
>>> *Update of each blkg is protected by both queue and blkcg locks so
>>> that holding either lock and testing blkcg_policy_enabled() is
>>> always enough for dereferencing policy data.*
>>> with queue lock held, there is no need to hold blkcg lock in
>>> blkcg_deactivate_policy(). Similar case is in
>>> blkcg_activate_policy(), which has removed holding of blkcg lock in
>>> commit 4c55f4f9ad3001ac1fefdd8d8ca7641d18558e23.
>>>
>>
>> Hi,
>> by chance, did you check whether this may cause problems with bfq,
>> being the latter not protected by the queue lock as cfq?
> Checked the bfq code, bfq seems never used blkcg lock derectly, and
> update of blkg in the common code is protected by both queue and
> blkcg locks, so IMHO this patch would not introduce any new problem
> with bfq, even though bfq is not protected by queue lock.
> On the other hand, the locks (queue lock/blkcg lock) used to protected
> the update of blkg seems a bit too heavyweight, especially the queue lock
> which is used too widely may cause races with other contexts. I wonder
> if there is any way to ease the case? e.g. add a new lock for blkg's own.:)
It might make sense to lock it separately, but I would not worry
about it unless it shows up as hot in your testing.
I've applied your patch, thanks.
--
Jens Axboe
On 4/18/18 6:54 PM, [email protected] wrote:
>>>> by chance, did you check whether this may cause problems with bfq,
>>>> being the latter not protected by the queue lock as cfq?
>>> Checked the bfq code, bfq seems never used blkcg lock derectly, and
>>> update of blkg in the common code is protected by both queue and
>>> blkcg locks, so IMHO this patch would not introduce any new problem
>>> with bfq, even though bfq is not protected by queue lock.
>>> On the other hand, the locks (queue lock/blkcg lock) used to protected
>>> the update of blkg seems a bit too heavyweight, especially the queue lock
>>> which is used too widely may cause races with other contexts. I wonder
>>> if there is any way to ease the case? e.g. add a new lock for blkg's own.:)
>>
>> It might make sense to lock it separately, but I would not worry
>> about it unless it shows up as hot in your testing.
> Actually, we've met a triggering of nmi_watchdog, blocked at the queue lock
> in blkcg_print_blkgs(), caused by the slow serial console and too many printks.
> Related discussion is here,
> https://bugzilla.kernel.org/show_bug.cgi?id=199003
> Even though it's not caused by the queue lock directly, it would not happen
> without using queue lock. The queue lock is big and used too widely, using it
> would intensify the race, so we're trying to understand the locks using in blkg,
> and maybe could improve the situation.
The queue lock is only used widely on non blk-mq, where it is the only
lock really. Doing serial IO under a spinlock is always going to suck,
regardless of how contended it is.
--
Jens Axboe