Received: by 2002:a25:1506:0:0:0:0:0 with SMTP id 6csp2479614ybv; Mon, 24 Feb 2020 06:04:25 -0800 (PST) X-Google-Smtp-Source: APXvYqxvfuUkURc3Kyzu6Kfv+Jvkw8TxPxIv18HcPbm+t1kkqns7SfogZO2SahfQ0LYpThaN3T9u X-Received: by 2002:aca:330b:: with SMTP id z11mr12183382oiz.96.1582553065140; Mon, 24 Feb 2020 06:04:25 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1582553065; cv=none; d=google.com; s=arc-20160816; b=tx+J4xPB25+mxRFhVo7O7BIATI7odZrKkw2rSrLJlSQqXt4JU+2QQsDjeKLvSso4El v1TEgIOBXxT6tVbZkshqrFG1a40dg9nUG8hBGB52VoXHtXa1WQirSiL3K61nh/pUpSKD wfPpD1cowhURWH/Yej5IPEfkvcp2PcF6R+NwQ/+JMyEFMdADBxID/AIWgUw4GyLJl5DE fmh+w08Xtoif8j8Upj0hcWzTEAm9CiKcWWrUKqR8aQwwzM4+ezaKUPW5PdB+o3jF0t7l t/i4fN0IJvFy0bUywK2UUHG0tXLdeFKqDysI8lfAO2/0Ks7uZvPgoSnbMUF203mHrmjP Oc+g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:in-reply-to:content-disposition :mime-version:references:mail-followup-to:message-id:subject:to:from :date:dkim-signature; bh=dcaS3AsvAds87+8Dqpw0cKPB/YFxfdgeQbxHZAQ+VXE=; b=jyqDUsPb6xvWPkJCJ2NUzZFZT4KIsgM5sFwN9jQ5wBqplr5S7+CVmyeQL/vmKbq0zo 3CNoQE4TxCvN8HifVnmbPpfzCykK4ZFW+VcQOhMu+H0qP32iWhuNIHILCOEvWp4Hcc4t oE8aJaCdQjzDZM3k2wqbyj9UHwxZaYUDvd2vhah0zAM5f8V/gMIdQc02DuPNUEsGjgNC gguqQQtEJEC+kfWrNrD41EBXYhj1/HE0bMzf80NXiQjj3XOx6iC/gg3XH9CDYp5jHv5l likB+bJCb3vqhSqcWpysCNMLSvW6p9kPeC984l4pLdchOuz7nuEgpS8ae4JvWTIr4QrP IrYw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@xff.cz header.s=mail header.b=lu1XHxvQ; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=xff.cz Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id x10si4660799oic.136.2020.02.24.06.04.12; Mon, 24 Feb 2020 06:04:25 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@xff.cz header.s=mail header.b=lu1XHxvQ; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=xff.cz Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727460AbgBXODw (ORCPT + 99 others); Mon, 24 Feb 2020 09:03:52 -0500 Received: from vps.xff.cz ([195.181.215.36]:57508 "EHLO vps.xff.cz" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727326AbgBXODw (ORCPT ); Mon, 24 Feb 2020 09:03:52 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=xff.cz; s=mail; t=1582553030; bh=yTwvIGrMVTD1/6h9dxU9dspNltcK6voxnhZxctNTAEc=; h=Date:From:To:Subject:References:X-My-GPG-KeyId:From; b=lu1XHxvQgCoUXZ/dUuFOOcRJIrqcQfm3bEICIKPWwOERcWS42DOFh/JYJVSz/kHBd GR4Abbmn3IHPaqsDGYpnBZXhEF6A9DZP4hh+yqKBn49bBgusowc/XxX/RejpLQGlB+ rjbbyfZhmt06ULyQKHKD7dDfBaOhkqKLenVhICxA= Date: Mon, 24 Feb 2020 15:03:49 +0100 From: =?utf-8?Q?Ond=C5=99ej?= Jirman To: Chao Yu , Jaegeuk Kim , linux-kernel@vger.kernel.org, linux-f2fs-devel@lists.sourceforge.net Subject: Re: [f2fs-dev] Writes stoped working on f2fs after the compression support was added Message-ID: <20200224140349.74yagjdwewmclx4v@core.my.home> Mail-Followup-To: Chao Yu , Jaegeuk Kim , linux-kernel@vger.kernel.org, linux-f2fs-devel@lists.sourceforge.net References: <20191209222345.1078-1-jaegeuk@kernel.org> <20200222044617.pfrhnz2iavkrtdn6@core.my.home> <20200222181721.tzrrohep5l3yklpf@core.my.home> <20200224135837.k54ke4ppca26ibec@core.my.home> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20200224135837.k54ke4ppca26ibec@core.my.home> X-My-GPG-KeyId: EBFBDDE11FB918D44D1F56C1F9F0A873BE9777ED Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Mon, Feb 24, 2020 at 02:58:37PM +0100, megi xff wrote: > Hello, > > On Mon, Feb 24, 2020 at 06:41:03PM +0800, Chao Yu wrote: > > On 2020/2/24 18:37, Chao Yu wrote: > > > Hi, > > > > > > Thanks for the report. > > > > > > Could you dump all other task stack info via "echo "t" > /proc/sysrq-trigger"? > > > > > >> > > >> [ 246.758021] INFO: task kworker/u16:1:58 blocked for more than 122 seconds. > > >> [ 246.758040] Not tainted 5.6.0-rc2-00590-g9983bdae4974e #11 > > >> [ 246.758044] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. > > >> [ 246.758052] kworker/u16:1 D 0 58 2 0x00000000 > > >> [ 246.758090] Workqueue: writeback wb_workfn (flush-179:0) > > >> [ 246.758099] Backtrace: > > >> [ 246.758121] [] (__schedule) from [] (schedule+0x78/0xf4) > > >> [ 246.758130] r10:da644000 r9:00000000 r8:da645a60 r7:da283e10 r6:00000002 r5:da644000 > > >> [ 246.758132] r4:da4d3600 > > >> [ 246.758148] [] (schedule) from [] (rwsem_down_write_slowpath+0x24c/0x4c0) > > >> [ 246.758152] r5:00000001 r4:da283e00 > > >> [ 246.758161] [] (rwsem_down_write_slowpath) from [] (down_write+0x6c/0x70) > > >> [ 246.758167] r10:da283e00 r9:da645d80 r8:d9ed0000 r7:00000001 r6:00000000 r5:eff213b0 > > >> [ 246.758169] r4:da283e00 > > >> [ 246.758187] [] (down_write) from [] (f2fs_write_single_data_page+0x608/0x7ac) > > > > > > I'm not sure what is this semaphore, I suspect this is F2FS_I(inode)->i_sem, in order to make > > > sure of this, can you help to add below function, and use them to replace > > > all {down,up}_{write,read}(&.i_sem) invoking? then reproduce this issue and catch the log. > > > > Sorry, just forgot attaching below function. > > > > void inode_down_write(struct inode *inode) > > { > > printk("%s from %pS\n", __func__, __builtin_return_address(0)); > > down_write(&F2FS_I(inode)->i_sem); > > } > > > > void inode_up_write(struct inode *inode) > > { > > up_write(&F2FS_I(inode)->i_sem); > > printk("%s from %pS\n", __func__, __builtin_return_address(0)); > > } > > > > void inode_down_read(struct inode *inode) > > { > > printk("%s from %pS\n", __func__, __builtin_return_address(0)); > > down_read(&F2FS_I(inode)->i_sem); > > } > > > > void inode_up_read(struct inode *inode) > > { > > up_read(&F2FS_I(inode)->i_sem); > > printk("%s from %pS\n", __func__, __builtin_return_address(0)); > > } > > > > Here's the log and vmlinux file that may help mapping the code addresses back to > code, hope it helps: > > https://megous.com/dl/tmp/f2fs-dmesg-log > https://megous.com/dl/tmp/f2fs-log-build-artifacts.tar.gz Just by a looks of it: root@tbs2[/proc/sys/kernel] # dmesg | grep up_write | wc -l 324 root@tbs2[/proc/sys/kernel] # dmesg | grep down_write | wc -l 347 there seems to be a mismatch of lock/unlock counts. root@tbs2[/proc/sys/kernel] # dmesg | grep down_read | wc -l 16 root@tbs2[/proc/sys/kernel] # dmesg | grep up_read | wc -l 16 regards, o. > thank you, > o. > > > > Thanks, > > > > > >> [ 246.758190] r5:eff213b0 r4:da283c60 > > >> [ 246.758198] [] (f2fs_write_single_data_page) from [] (f2fs_write_cache_pages+0x2b4/0x7c4) > > >> [ 246.758204] r10:da645c28 r9:da283d60 r8:da283c60 r7:0000000f r6:da645d80 r5:00000001 > > >> [ 246.758206] r4:eff213b0 > > >> [ 246.758214] [] (f2fs_write_cache_pages) from [] (f2fs_write_data_pages+0x344/0x35c) > > >> [ 246.758220] r10:00000000 r9:d9ed002c r8:d9ed0000 r7:00000004 r6:da283d60 r5:da283c60 > > >> [ 246.758223] r4:da645d80 > > >> [ 246.758238] [] (f2fs_write_data_pages) from [] (do_writepages+0x3c/0xd4) > > >> [ 246.758244] r10:0000000a r9:c0e03d00 r8:00000c00 r7:c0264ddc r6:da645d80 r5:da283d60 > > >> [ 246.758246] r4:da283c60 > > >> [ 246.758254] [] (do_writepages) from [] (__writeback_single_inode+0x44/0x454) > > >> [ 246.758259] r7:da283d60 r6:da645eac r5:da645d80 r4:da283c60 > > >> [ 246.758266] [] (__writeback_single_inode) from [] (writeback_sb_inodes+0x204/0x4b0) > > >> [ 246.758272] r10:0000000a r9:c0e03d00 r8:da283cc8 r7:da283c60 r6:da645eac r5:da283d08 > > >> [ 246.758274] r4:d9dc9848 > > >> [ 246.758281] [] (writeback_sb_inodes) from [] (__writeback_inodes_wb+0x50/0xe4) > > >> [ 246.758287] r10:da3797a8 r9:c0e03d00 r8:d9dc985c r7:da645eac r6:00000000 r5:d9dc9848 > > >> [ 246.758289] r4:da5a8800 > > >> [ 246.758296] [] (__writeback_inodes_wb) from [] (wb_writeback+0x294/0x338) > > >> [ 246.758302] r10:fffbf200 r9:da644000 r8:c0e04e64 r7:d9dc9848 r6:d9dc9874 r5:da645eac > > >> [ 246.758305] r4:d9dc9848 > > >> [ 246.758312] [] (wb_writeback) from [] (wb_workfn+0x35c/0x54c) > > >> [ 246.758318] r10:da5f2005 r9:d9dc984c r8:d9dc9948 r7:d9dc9848 r6:00000000 r5:d9dc9954 > > >> [ 246.758321] r4:000031e6 > > >> [ 246.758334] [] (wb_workfn) from [] (process_one_work+0x214/0x544) > > >> [ 246.758340] r10:da5f2005 r9:00000200 r8:00000000 r7:da5f2000 r6:ef044400 r5:da5eb000 > > >> [ 246.758343] r4:d9dc9954 > > >> [ 246.758350] [] (process_one_work) from [] (worker_thread+0x4c/0x574) > > >> [ 246.758357] r10:ef044400 r9:c0e03d00 r8:ef044418 r7:00000088 r6:ef044400 r5:da5eb014 > > >> [ 246.758359] r4:da5eb000 > > >> [ 246.758368] [] (worker_thread) from [] (kthread+0x144/0x170) > > >> [ 246.758374] r10:ec9e5e90 r9:dabf325c r8:da5eb000 r7:da644000 r6:00000000 r5:da5fe000 > > >> [ 246.758377] r4:dabf3240 > > >> [ 246.758386] [] (kthread) from [] (ret_from_fork+0x14/0x2c) > > >> [ 246.758391] Exception stack(0xda645fb0 to 0xda645ff8) > > >> [ 246.758397] 5fa0: 00000000 00000000 00000000 00000000 > > >> [ 246.758402] 5fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 > > >> [ 246.758407] 5fe0: 00000000 00000000 00000000 00000000 00000013 00000000 > > >> [ 246.758413] r10:00000000 r9:00000000 r8:00000000 r7:00000000 r6:00000000 r5:c01563b8 > > >> [ 246.758416] r4:da5fe000 > > >> . > > >> > > > > > > > > > _______________________________________________ > > > Linux-f2fs-devel mailing list > > > Linux-f2fs-devel@lists.sourceforge.net > > > https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel > > >