Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
Sender: Tejun Heo <htejun@gmail.com>
Date:   Thu, 20 May 2021 17:54:49 -0400
From:   Tejun Heo <tj@kernel.org>
To:     Dmitry Monakhov <dmtrmonakhov@yandex-team.ru>
Cc:     linux-kernel@vger.kernel.org, axboe@kernel.dk
Subject: Re: [PATCH v2] blk-throttle: fix race between submitter and
 throttler thread
Message-ID: <YKbaqfTwdxkLaSUD@slm.duckdns.org>
References: <20210520194413.3977-1-dmtrmonakhov@yandex-team.ru>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20210520194413.3977-1-dmtrmonakhov@yandex-team.ru>
Precedence: bulk

On Thu, May 20, 2021 at 07:44:13PM +0000, Dmitry Monakhov wrote:
> Changes since v1:
>   - Improve comments
> 
> Currently we call bio_set_flag(bio, BIO_THROTTLED) unconditionally
> at the end of blk_throtl_bio w/o queue_lock. But once we drop queue_lock,
> bio may already be processed by thottler thread, so both threads
> may update bio->flags concurently
> 
> Dipite that race window is tiny, it happens in real life under heavy load.
> It looks like follows:
> 
> SUBMITTER_THREAD (CPU1)                  THROTTLER_THREAD (CPU2)
>  ->blk_throtl_bio
>    ->throtl_add_bio_tg
> (1)   bio_set_flag(bio, BIO_THROTTLED);
>    spin_unlock_irq(q->queue_lock);
>                                          ->blk_throtl_dispatch_work_fn
>                                           (2)spin_lock_irq(q->queue_lock);
> 					   ->generic_make_request
> 					     ->blk_queue_split
>                                                (3)bio_set_flag(bio, BIO_CHAINED)
> 
> (4) bio_set_flag(bio, BIO_THROTTLED);
> 
> Since bio->bi_flags is not atomic it will be cached on each CPU
> CPU1 will cache it at the step (1), and changes from step(3) is not visiable,
> so BIO_CHAINED flag will be lost and rewritten at step(4).
> This result in ->bi_end_io() will be called multiple times once for each
> chained bio and once for parent bio.
> 
> Bug#2: submit_bio_checks() call blkcg_bio_issue_init() for throttled bio,
> but at this moment bio may be already be completed and freed by throttler thread
> 
> In order to fix both issues we should modify throttled bio under queue_lock only.
> 
> Fixes: 111be88398174 ("block-throttle: avoid double charge")
> Signed-off-by: Dmitry Monakhov <dmtrmonakhov@yandex-team.ru>

Acked-by: Tejun Heo <tj@kernel.org>

Thanks.

-- 
tejun