Several distributions mount the "proper root" as ro during initrd and
then remount it as rw before pivot_root(2). Thus, if a rescan had been
aborted by a previous shutdown, the rescan would never be resumed.
This issue would manifest itself as several btrfs ioctl(2)s causing the
entire machine to hang when btrfs_qgroup_wait_for_completion was hit
(due to the fs_info->qgroup_rescan_running flag being set but the rescan
itself not being resumed). Notably, Docker's btrfs storage driver makes
regular use of BTRFS_QUOTA_CTL_DISABLE and BTRFS_IOC_QUOTA_RESCAN_WAIT
(causing this problem to be manifested on boot for some machines).
Cc: <[email protected]> # v3.11+
Cc: Jeff Mahoney <[email protected]>
Fixes: b382a324b60f ("Btrfs: fix qgroup rescan resume on mount")
Signed-off-by: Aleksa Sarai <[email protected]>
---
fs/btrfs/super.c | 2 ++
1 file changed, 2 insertions(+)
diff --git a/fs/btrfs/super.c b/fs/btrfs/super.c
index 6346876c97ea..ff6690389343 100644
--- a/fs/btrfs/super.c
+++ b/fs/btrfs/super.c
@@ -1821,6 +1821,8 @@ static int btrfs_remount(struct super_block *sb, int *flags, char *data)
goto restore;
}
+ btrfs_qgroup_rescan_resume(fs_info);
+
if (!fs_info->uuid_root) {
btrfs_info(fs_info, "creating UUID tree");
ret = btrfs_create_uuid_tree(fs_info);
--
2.13.2
On 4.07.2017 14:49, Aleksa Sarai wrote:
> Several distributions mount the "proper root" as ro during initrd and
> then remount it as rw before pivot_root(2). Thus, if a rescan had been
> aborted by a previous shutdown, the rescan would never be resumed.
>
> This issue would manifest itself as several btrfs ioctl(2)s causing the
> entire machine to hang when btrfs_qgroup_wait_for_completion was hit
> (due to the fs_info->qgroup_rescan_running flag being set but the rescan
> itself not being resumed). Notably, Docker's btrfs storage driver makes
> regular use of BTRFS_QUOTA_CTL_DISABLE and BTRFS_IOC_QUOTA_RESCAN_WAIT
> (causing this problem to be manifested on boot for some machines).
>
> Cc: <[email protected]> # v3.11+
> Cc: Jeff Mahoney <[email protected]>
> Fixes: b382a324b60f ("Btrfs: fix qgroup rescan resume on mount")
> Signed-off-by: Aleksa Sarai <[email protected]>
Indeed, looking at the code it seems that b382a324b60f ("Btrfs: fix
qgroup rescan resume on mount") missed adding the qgroup_rescan_resume
in the remount path. One thing which I couldn't verify though is whether
reading fs_info->qgroup_flags without any locking is safe from remount
context.
During remount I don't see any locks taken that prevent operations which
can modify qgroup_flags.
> ---
> fs/btrfs/super.c | 2 ++
> 1 file changed, 2 insertions(+)
>
> diff --git a/fs/btrfs/super.c b/fs/btrfs/super.c
> index 6346876c97ea..ff6690389343 100644
> --- a/fs/btrfs/super.c
> +++ b/fs/btrfs/super.c
> @@ -1821,6 +1821,8 @@ static int btrfs_remount(struct super_block *sb, int *flags, char *data)
> goto restore;
> }
>
> + btrfs_qgroup_rescan_resume(fs_info);
> +
> if (!fs_info->uuid_root) {
> btrfs_info(fs_info, "creating UUID tree");
> ret = btrfs_create_uuid_tree(fs_info);
>
On 10.07.2017 16:12, Nikolay Borisov wrote:
>
>
> On 4.07.2017 14:49, Aleksa Sarai wrote:
>> Several distributions mount the "proper root" as ro during initrd and
>> then remount it as rw before pivot_root(2). Thus, if a rescan had been
>> aborted by a previous shutdown, the rescan would never be resumed.
>>
>> This issue would manifest itself as several btrfs ioctl(2)s causing the
>> entire machine to hang when btrfs_qgroup_wait_for_completion was hit
>> (due to the fs_info->qgroup_rescan_running flag being set but the rescan
>> itself not being resumed). Notably, Docker's btrfs storage driver makes
>> regular use of BTRFS_QUOTA_CTL_DISABLE and BTRFS_IOC_QUOTA_RESCAN_WAIT
>> (causing this problem to be manifested on boot for some machines).
>>
>> Cc: <[email protected]> # v3.11+
>> Cc: Jeff Mahoney <[email protected]>
>> Fixes: b382a324b60f ("Btrfs: fix qgroup rescan resume on mount")
>> Signed-off-by: Aleksa Sarai <[email protected]>
>
> Indeed, looking at the code it seems that b382a324b60f ("Btrfs: fix
> qgroup rescan resume on mount") missed adding the qgroup_rescan_resume
> in the remount path. One thing which I couldn't verify though is whether
> reading fs_info->qgroup_flags without any locking is safe from remount
> context.
>
> During remount I don't see any locks taken that prevent operations which
> can modify qgroup_flags.
>
>
Further inspection reveals that the access rules to qgroup_flags are
somewhat broken so this patch doesn't really make things any worse than
they are. As such:
Reviewed-by: Nikolay Borisov <[email protected]>
Tested-by: Nikolay Borisov <[email protected]>
>
>> ---
>> fs/btrfs/super.c | 2 ++
>> 1 file changed, 2 insertions(+)
>>
>> diff --git a/fs/btrfs/super.c b/fs/btrfs/super.c
>> index 6346876c97ea..ff6690389343 100644
>> --- a/fs/btrfs/super.c
>> +++ b/fs/btrfs/super.c
>> @@ -1821,6 +1821,8 @@ static int btrfs_remount(struct super_block *sb, int *flags, char *data)
>> goto restore;
>> }
>>
>> + btrfs_qgroup_rescan_resume(fs_info);
>> +
>> if (!fs_info->uuid_root) {
>> btrfs_info(fs_info, "creating UUID tree");
>> ret = btrfs_create_uuid_tree(fs_info);
>>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
On Mon, Jul 10, 2017 at 04:56:36PM +0300, Nikolay Borisov wrote:
> On 10.07.2017 16:12, Nikolay Borisov wrote:
> > On 4.07.2017 14:49, Aleksa Sarai wrote:
> >> Several distributions mount the "proper root" as ro during initrd and
> >> then remount it as rw before pivot_root(2). Thus, if a rescan had been
> >> aborted by a previous shutdown, the rescan would never be resumed.
> >>
> >> This issue would manifest itself as several btrfs ioctl(2)s causing the
> >> entire machine to hang when btrfs_qgroup_wait_for_completion was hit
> >> (due to the fs_info->qgroup_rescan_running flag being set but the rescan
> >> itself not being resumed). Notably, Docker's btrfs storage driver makes
> >> regular use of BTRFS_QUOTA_CTL_DISABLE and BTRFS_IOC_QUOTA_RESCAN_WAIT
> >> (causing this problem to be manifested on boot for some machines).
> >>
> >> Cc: <[email protected]> # v3.11+
> >> Cc: Jeff Mahoney <[email protected]>
> >> Fixes: b382a324b60f ("Btrfs: fix qgroup rescan resume on mount")
> >> Signed-off-by: Aleksa Sarai <[email protected]>
> >
> > Indeed, looking at the code it seems that b382a324b60f ("Btrfs: fix
> > qgroup rescan resume on mount") missed adding the qgroup_rescan_resume
> > in the remount path. One thing which I couldn't verify though is whether
> > reading fs_info->qgroup_flags without any locking is safe from remount
> > context.
> >
> > During remount I don't see any locks taken that prevent operations which
> > can modify qgroup_flags.
>
> Further inspection reveals that the access rules to qgroup_flags are
> somewhat broken so this patch doesn't really make things any worse than
> they are.
The usage follows a pattern for a bitfield, updated by set_bit/clear_bit
etc. The updates to the state or inconsistency is not safe, so some
updates could get lost under some circumstances.
Patch added to devel queue, possibly will be submitted to 4.13 so stable
can pick it.
On 07/12/2017 03:03 AM, David Sterba wrote:
> On Mon, Jul 10, 2017 at 04:56:36PM +0300, Nikolay Borisov wrote:
>> On 10.07.2017 16:12, Nikolay Borisov wrote:
>>> On 4.07.2017 14:49, Aleksa Sarai wrote:
>>>> Several distributions mount the "proper root" as ro during initrd and
>>>> then remount it as rw before pivot_root(2). Thus, if a rescan had been
>>>> aborted by a previous shutdown, the rescan would never be resumed.
>>>>
>>>> This issue would manifest itself as several btrfs ioctl(2)s causing the
>>>> entire machine to hang when btrfs_qgroup_wait_for_completion was hit
>>>> (due to the fs_info->qgroup_rescan_running flag being set but the rescan
>>>> itself not being resumed). Notably, Docker's btrfs storage driver makes
>>>> regular use of BTRFS_QUOTA_CTL_DISABLE and BTRFS_IOC_QUOTA_RESCAN_WAIT
>>>> (causing this problem to be manifested on boot for some machines).
>>>>
>>>> Cc: <[email protected]> # v3.11+
>>>> Cc: Jeff Mahoney <[email protected]>
>>>> Fixes: b382a324b60f ("Btrfs: fix qgroup rescan resume on mount")
>>>> Signed-off-by: Aleksa Sarai <[email protected]>
>>>
>>> Indeed, looking at the code it seems that b382a324b60f ("Btrfs: fix
>>> qgroup rescan resume on mount") missed adding the qgroup_rescan_resume
>>> in the remount path. One thing which I couldn't verify though is whether
>>> reading fs_info->qgroup_flags without any locking is safe from remount
>>> context.
>>>
>>> During remount I don't see any locks taken that prevent operations which
>>> can modify qgroup_flags.
>>
>> Further inspection reveals that the access rules to qgroup_flags are
>> somewhat broken so this patch doesn't really make things any worse than
>> they are.
>
> The usage follows a pattern for a bitfield, updated by set_bit/clear_bit
> etc. The updates to the state or inconsistency is not safe, so some
> updates could get lost under some circumstances.
>
> Patch added to devel queue, possibly will be submitted to 4.13 so stable
> can pick it.
Looks like it wasn't merged in the 4.13 window (so stable hasn't picked
it), will this be submitted for 4.14? Thanks.
--
Aleksa Sarai
Software Engineer (Containers)
SUSE Linux GmbH
https://www.cyphar.com/
On Sat, Aug 26, 2017 at 04:39:02PM +1000, Aleksa Sarai wrote:
> On 07/12/2017 03:03 AM, David Sterba wrote:
> > On Mon, Jul 10, 2017 at 04:56:36PM +0300, Nikolay Borisov wrote:
> >> On 10.07.2017 16:12, Nikolay Borisov wrote:
> >>> On 4.07.2017 14:49, Aleksa Sarai wrote:
> >>>> Several distributions mount the "proper root" as ro during initrd and
> >>>> then remount it as rw before pivot_root(2). Thus, if a rescan had been
> >>>> aborted by a previous shutdown, the rescan would never be resumed.
> >>>>
> >>>> This issue would manifest itself as several btrfs ioctl(2)s causing the
> >>>> entire machine to hang when btrfs_qgroup_wait_for_completion was hit
> >>>> (due to the fs_info->qgroup_rescan_running flag being set but the rescan
> >>>> itself not being resumed). Notably, Docker's btrfs storage driver makes
> >>>> regular use of BTRFS_QUOTA_CTL_DISABLE and BTRFS_IOC_QUOTA_RESCAN_WAIT
> >>>> (causing this problem to be manifested on boot for some machines).
> >>>>
> >>>> Cc: <[email protected]> # v3.11+
> >>>> Cc: Jeff Mahoney <[email protected]>
> >>>> Fixes: b382a324b60f ("Btrfs: fix qgroup rescan resume on mount")
> >>>> Signed-off-by: Aleksa Sarai <[email protected]>
> >>>
> >>> Indeed, looking at the code it seems that b382a324b60f ("Btrfs: fix
> >>> qgroup rescan resume on mount") missed adding the qgroup_rescan_resume
> >>> in the remount path. One thing which I couldn't verify though is whether
> >>> reading fs_info->qgroup_flags without any locking is safe from remount
> >>> context.
> >>>
> >>> During remount I don't see any locks taken that prevent operations which
> >>> can modify qgroup_flags.
> >>
> >> Further inspection reveals that the access rules to qgroup_flags are
> >> somewhat broken so this patch doesn't really make things any worse than
> >> they are.
> >
> > The usage follows a pattern for a bitfield, updated by set_bit/clear_bit
> > etc. The updates to the state or inconsistency is not safe, so some
> > updates could get lost under some circumstances.
> >
> > Patch added to devel queue, possibly will be submitted to 4.13 so stable
> > can pick it.
>
> Looks like it wasn't merged in the 4.13 window (so stable hasn't picked
> it), will this be submitted for 4.14? Thanks.
Sorry this didn't make it to 4.13, the patch is in the queue for 4.14.