Received-SPF: pass (google.com: domain of linux-kernel+bounces-112638-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.199.223 as permitted sender) client-ip=147.75.199.223;
Message-ID: <b1d67dd6-4326-4247-96a3-283ab1975a50@molgen.mpg.de>
Date: Sun, 24 Mar 2024 11:50:47 +0100
Precedence: bulk
MIME-Version: 1.0
User-Agent: Mozilla Thunderbird
Subject: Re: possible 6.6 regression: Deadlock involving super_lock()
Content-Language: en-US
To: Christian Brauner <brauner@kernel.org>
Cc: Alexander Viro <viro@zeniv.linux.org.uk>, linux-fsdevel@vger.kernel.org,
 linux-kernel@vger.kernel.org, it+linux@molgen.mpg.de
References: <6e010dbb-f125-4f44-9b1a-9e6ac9bb66ff@molgen.mpg.de>
 <20240321-brecheisen-lasst-7ac15aff03b1@brauner>
From: Donald Buczek <buczek@molgen.mpg.de>
In-Reply-To: <20240321-brecheisen-lasst-7ac15aff03b1@brauner>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit

On 3/21/24 14:35, Christian Brauner wrote:
>> Also, one writeback thread was blocked. I mention that, because I
>> don't get how these these two threads could depend on each other:
> 
> Writeback holds s_umount. So writeback seems to not make progress and
> blocks the mount. So right now it seems unlikely that this is related.

Oh, yes, super_trylock_shared() in __writeback_inodes_wb(). Sorry, I've missed that and blamed the victim.

> Any chance you can try and reproduce this with v6.7 and newer?

We can't really reproduce anything yet, we can only wait and see if it happens again on the production system. We are willing to risk that, so we've just switched the system to 6.6.22 and made sure, that drgn is available to further analyze a possible frozen state. This was not the case on the reported freeze due to a configuration error. Although, currently, I would have no idea how to find out, why the folio is locked forever, even with drgn available.

We can switch to 6.7. Why do you propose it? Is there reason to believe that the problem might go away or would that help in analysis if we hit it again on that series?

I'm a bit hesitating, because this is a production system and we have the experience that switching from one mainline release to the next (or from one stable series to another) produces some new problems most of the times.

Thanks

   Donald

>> # # /proc/39359/task/39359: kworker/u268:5+flush-0:58 :
>> # cat /proc/39359/task/39359/stack
>>
>> [<0>] folio_wait_bit_common+0x135/0x350
>> [<0>] write_cache_pages+0x1a0/0x3a0
>> [<0>] nfs_writepages+0x12a/0x1e0 [nfs]
>> [<0>] do_writepages+0xcf/0x1e0
>> [<0>] __writeback_single_inode+0x46/0x3a0
>> [<0>] writeback_sb_inodes+0x1f5/0x4d0
>> [<0>] __writeback_inodes_wb+0x4c/0xf0
>> [<0>] wb_writeback+0x1f5/0x320
>> [<0>] wb_workfn+0x350/0x4f0
>> [<0>] process_one_work+0x142/0x300
>> [<0>] worker_thread+0x2f5/0x410
>> [<0>] kthread+0xe8/0x120
>> [<0>] ret_from_fork+0x34/0x50
>> [<0>] ret_from_fork_asm+0x1b/0x30

-- 
Donald Buczek
buczek@molgen.mpg.de
Tel: +49 30 8413 1433