Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:18 as permitted sender) client-ip=2620:137:e000::1:18;
Subject: Re: Tasks stuck jbd2 for a long time
Message-ID: <e716473e-7251-7a81-fa5e-6bf6ba34e49f@amazon.com>
Date:   Wed, 16 Aug 2023 11:32:47 -0700
MIME-Version: 1.0
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:102.0)
 Gecko/20100101 Thunderbird/102.14.0
Content-Language: en-US
To:     Jan Kara <jack@suse.cz>
CC:     Theodore Ts'o <tytso@mit.edu>, <jack@suse.com>,
        <linux-ext4@vger.kernel.org>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        "gregkh@linuxfoundation.org" <gregkh@linuxfoundation.org>,
        "Park, SeongJae" <sjpark@amazon.com>
References: <153d081d-e738-b916-4f72-364b2c1cc36a@amazon.com>
 <20230816022851.GH2247938@mit.edu>
 <17b6398c-859e-4ce7-b751-8688a7288b47@amazon.com>
 <20230816145310.giogco2nbzedgak2@quack3>
From:   "Bhatnagar, Rishabh" <risbhat@amazon.com>
In-Reply-To: <20230816145310.giogco2nbzedgak2@quack3>
Content-Type: text/plain; charset="UTF-8"; format=flowed
Content-Transfer-Encoding: 7bit
Precedence: bulk


On 8/16/23 7:53 AM, Jan Kara wrote:
> CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.
>
>
>
> On Tue 15-08-23 20:57:14, Bhatnagar, Rishabh wrote:
>> On 8/15/23 7:28 PM, Theodore Ts'o wrote:
>>> CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.
>>>
>>>
>>>
>>> It would be helpful if you can translate address in the stack trace to
>>> line numbers.  See [1] and the script in
>>> ./scripts/decode_stacktrace.sh in the kernel sources.  (It is
>>> referenced in the web page at [1].)
>>>
>>> [1] https://docs.kernel.org/admin-guide/bug-hunting.html
>>>
>>> Of course, in order to interpret the line numbers, we'll need a
>>> pointer to the git repo of your kernel sources and the git commit ID
>>> you were using that presumably corresponds to 5.10.184-175.731.amzn2.x86_64.
>>>
>>> The stack trace for which I am particularly interested is the one for
>>> the jbd2/md0-8 task, e.g.:
>> Thanks for checking Ted.
>>
>> We don't have fast_commit feature enabled. So it should correspond to this
>> line:
>> https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/tree/fs/jbd2/commit.c?h=linux-5.10.y#n496
>>
>>>>         Not tainted 5.10.184-175.731.amzn2.x86_64 #1
>>>> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
>>>> task:jbd2/md0-8      state:D stack:    0 pid: 8068 ppid:     2
>>>> flags:0x00004080
>>>> Call Trace:
>>>> __schedule+0x1f9/0x660
>>>>    schedule+0x46/0xb0
>>>>    jbd2_journal_commit_transaction+0x35d/0x1880 [jbd2]  <--------- line #?
>>>>    ? update_load_avg+0x7a/0x5d0
>>>>    ? add_wait_queue_exclusive+0x70/0x70
>>>>    ? lock_timer_base+0x61/0x80
>>>>    ? kjournald2+0xcf/0x360 [jbd2]
>>>>    kjournald2+0xcf/0x360 [jbd2]
>>> Most of the other stack traces you refenced are tasks that are waiting
>>> for the transaction commit to complete so they can proceed with some
>>> file system operation.  The stack traces which have
>>> start_this_handle() in them are examples of this going on.  Stack
>>> traces of tasks that do *not* have start_this_handle() would be
>>> specially interesting.
>> I see all other stacks apart from kjournald have "start_this_handle".
> That would be strange. Can you post full output of "echo w
>> /proc/sysrq-trigger" to dmesg, ideally passed through scripts/faddr2line as
> Ted suggests. Thanks!

Sure i'll try to collect that. The system freezes when such a situation 
happens and i'm not able
to collect much information. I'll try to crash the kernel and collect 
kdump and see if i can get that info.

Can low available memory be a reason for a thread to not be able to 
close the transaction handle for a long time?
Maybe some writeback thread starts the handle but is not able to 
complete writeback?

>
>                                                                  Honza
> --
> Jan Kara <jack@suse.com>
> SUSE Labs, CR