Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67;
From:   Song Liu <songliubraving@fb.com>
To:     Johannes Weiner <hannes@cmpxchg.org>
CC:     syzbot <syzbot+efb9e48b9fbdc49bb34a@syzkaller.appspotmail.com>,
        "Andrew Morton" <akpm@linux-foundation.org>,
        "amir73il@gmail.com" <amir73il@gmail.com>,
        "darrick.wong@oracle.com" <darrick.wong@oracle.com>,
        "hughd@google.com" <hughd@google.com>,
        "jack@suse.cz" <jack@suse.cz>,
        "jglisse@redhat.com" <jglisse@redhat.com>,
        Josef Bacik <josef@toxicpanda.com>,
        "kirill.shutemov@linux.intel.com" <kirill.shutemov@linux.intel.com>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        "linux-mm@kvack.org" <linux-mm@kvack.org>,
        "sfr@canb.auug.org.au" <sfr@canb.auug.org.au>,
        "syzkaller-bugs@googlegroups.com" <syzkaller-bugs@googlegroups.com>,
        "william.kucharski@oracle.com" <william.kucharski@oracle.com>,
        "willy@infradead.org" <willy@infradead.org>
Subject: Re: INFO: task hung in mpage_prepare_extent_to_map
Thread-Topic: INFO: task hung in mpage_prepare_extent_to_map
Thread-Index: AQHVjck0AXGPtEIKWEas0QCK1EXfO6dwfRoAgAAh3wA=
Date:   Mon, 28 Oct 2019 22:15:58 +0000
Message-ID: <70FEE01E-654E-4C4A-BF5C-B0A06A073A5C@fb.com>
References: <000000000000c50fd70595fdd5b2@google.com>
 <20191028201444.GA27425@cmpxchg.org>
In-Reply-To: <20191028201444.GA27425@cmpxchg.org>
Accept-Language: en-US
Content-Language: en-US
received-spf: None (protection.outlook.com: fb.com does not designate
 permitted sender hosts)
Content-Type: text/plain; charset="us-ascii"
Content-ID: <826D420DE97C16469A3EF6EA15375C83@namprd15.prod.outlook.com>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-MS-Exchange-CrossTenant-Network-Message-Id: 861a41c5-6be7-45a5-5da9-08d75bf4692e
X-MS-Exchange-CrossTenant-originalarrivaltime: 28 Oct 2019 22:15:58.8879
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 8ae927fe-1255-47a7-a2af-5f3a069daaa2
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: efOrsm3S4/8pQim2qnbt+rLueSneAVOp7mC7meI+kXVtUn7Dl/Tb/AcBwh1+1nSgY0Sp0eZ8pexby3j+mEAlMw==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: MWHPR15MB1711
X-OriginatorOrg: fb.com
X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.95,1.0.8
 definitions=2019-10-28_07:2019-10-28,2019-10-28 signatures=0
X-Proofpoint-Spam-Details: rule=fb_default_notspam policy=fb_default score=0 phishscore=0 mlxscore=0
 adultscore=0 bulkscore=0 lowpriorityscore=0 mlxlogscore=999
 priorityscore=1501 clxscore=1011 suspectscore=0 impostorscore=0
 malwarescore=0 spamscore=0 classifier=spam adjust=0 reason=mlx scancount=1
 engine=8.12.0-1908290000 definitions=main-1910280207
X-FB-Internal: deliver
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org


> On Oct 28, 2019, at 1:14 PM, Johannes Weiner <hannes@cmpxchg.org> wrote:
>=20
> On Mon, Oct 28, 2019 at 12:52:09PM -0700, syzbot wrote:
>> Hello,
>>=20
>> syzbot found the following crash on:
>>=20
>> HEAD commit:    12d61c69 Add linux-next specific files for 20191024
>> git tree:       linux-next
>> console output: https://urldefense.proofpoint.com/v2/url?u=3Dhttps-3A__s=
yzkaller.appspot.com_x_log.txt-3Fx-3D15a0fa97600000&d=3DDwIBAg&c=3D5VD0RTtN=
lTh3ycd41b3MUw&r=3DdR8692q0_uaizy0jkrBJQM5k2hfm4CiFxYT8KaysFrg&m=3DYEaOe5RP=
2hLXAC4tKPLehAQsea0_3k3tI4DL32BcA-8&s=3D6-TXLGQxJcK1GdwMwa51423Y221rRncNiC_=
T09O0OLc&e=3D=20
>> kernel config:  https://urldefense.proofpoint.com/v2/url?u=3Dhttps-3A__s=
yzkaller.appspot.com_x_.config-3Fx-3Dafb75fd8c9fd5ed8&d=3DDwIBAg&c=3D5VD0RT=
tNlTh3ycd41b3MUw&r=3DdR8692q0_uaizy0jkrBJQM5k2hfm4CiFxYT8KaysFrg&m=3DYEaOe5=
RP2hLXAC4tKPLehAQsea0_3k3tI4DL32BcA-8&s=3DGuFgLJZOb7jtjZ5mDbkVT_zqtiVW4Py13=
e6Oq5CFxgY&e=3D=20
>> dashboard link: https://urldefense.proofpoint.com/v2/url?u=3Dhttps-3A__s=
yzkaller.appspot.com_bug-3Fextid-3Defb9e48b9fbdc49bb34a&d=3DDwIBAg&c=3D5VD0=
RTtNlTh3ycd41b3MUw&r=3DdR8692q0_uaizy0jkrBJQM5k2hfm4CiFxYT8KaysFrg&m=3DYEaO=
e5RP2hLXAC4tKPLehAQsea0_3k3tI4DL32BcA-8&s=3DpF1hv-zGR8F378weGq9zxCE5ibI2_73=
qweMB_KuaZLM&e=3D=20
>> compiler:       gcc (GCC) 9.0.0 20181231 (experimental)
>> syz repro:      https://urldefense.proofpoint.com/v2/url?u=3Dhttps-3A__s=
yzkaller.appspot.com_x_repro.syz-3Fx-3D13a63dc4e00000&d=3DDwIBAg&c=3D5VD0RT=
tNlTh3ycd41b3MUw&r=3DdR8692q0_uaizy0jkrBJQM5k2hfm4CiFxYT8KaysFrg&m=3DYEaOe5=
RP2hLXAC4tKPLehAQsea0_3k3tI4DL32BcA-8&s=3DmI7ZOgrDWeG-p6vn2d_kj65a5g8J7exXJ=
2MIUUF84-w&e=3D=20
>>=20
>> The bug was bisected to:
>>=20
>> commit 9c61acffe2b8833152041f7b6a02d1d0a17fd378
>> Author: Song Liu <songliubraving@fb.com>
>> Date:   Wed Oct 23 00:24:28 2019 +0000
>>=20
>>    mm,thp: recheck each page before collapsing file THP
>>=20
>> bisection log:  https://urldefense.proofpoint.com/v2/url?u=3Dhttps-3A__s=
yzkaller.appspot.com_x_bisect.txt-3Fx-3D13eb6ec0e00000&d=3DDwIBAg&c=3D5VD0R=
TtNlTh3ycd41b3MUw&r=3DdR8692q0_uaizy0jkrBJQM5k2hfm4CiFxYT8KaysFrg&m=3DYEaOe=
5RP2hLXAC4tKPLehAQsea0_3k3tI4DL32BcA-8&s=3DYtSUy5Dtjo6tek7CvwzMTPL40BJwOC6r=
Eom-AkVx0SM&e=3D=20
>> final crash:    https://urldefense.proofpoint.com/v2/url?u=3Dhttps-3A__s=
yzkaller.appspot.com_x_report.txt-3Fx-3D101b6ec0e00000&d=3DDwIBAg&c=3D5VD0R=
TtNlTh3ycd41b3MUw&r=3DdR8692q0_uaizy0jkrBJQM5k2hfm4CiFxYT8KaysFrg&m=3DYEaOe=
5RP2hLXAC4tKPLehAQsea0_3k3tI4DL32BcA-8&s=3DBvPJx3QSPHgsN12jSZci_MqW_VxYp-MZ=
pQtogZjlJOo&e=3D=20
>> console output: https://urldefense.proofpoint.com/v2/url?u=3Dhttps-3A__s=
yzkaller.appspot.com_x_log.txt-3Fx-3D17eb6ec0e00000&d=3DDwIBAg&c=3D5VD0RTtN=
lTh3ycd41b3MUw&r=3DdR8692q0_uaizy0jkrBJQM5k2hfm4CiFxYT8KaysFrg&m=3DYEaOe5RP=
2hLXAC4tKPLehAQsea0_3k3tI4DL32BcA-8&s=3DYPvxWpQDpk9MI9W6QCtxME64wmxL2CZ5ZtE=
kCn0nI0c&e=3D=20
>>=20
>> IMPORTANT: if you fix the bug, please add the following tag to the commi=
t:
>> Reported-by: syzbot+efb9e48b9fbdc49bb34a@syzkaller.appspotmail.com
>> Fixes: 9c61acffe2b8 ("mm,thp: recheck each page before collapsing file T=
HP")
>>=20
>> INFO: task khugepaged:1084 blocked for more than 143 seconds.
>>      Not tainted 5.4.0-rc4-next-20191024 #0
>> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message=
.
>> khugepaged      D27568  1084      2 0x80004000
>> Call Trace:
>> context_switch kernel/sched/core.c:3384 [inline]
>> __schedule+0x94a/0x1e70 kernel/sched/core.c:4069
>> schedule+0xd9/0x260 kernel/sched/core.c:4136
>> io_schedule+0x1c/0x70 kernel/sched/core.c:5780
>> wait_on_page_bit_common mm/filemap.c:1175 [inline]
>> __lock_page+0x422/0xab0 mm/filemap.c:1383
>> lock_page include/linux/pagemap.h:480 [inline]
>> mpage_prepare_extent_to_map+0xb3f/0xf90 fs/ext4/inode.c:2668
>> ext4_writepages+0xb6a/0x2e70 fs/ext4/inode.c:2866
>> ? 0xffffffff81000000
>> do_writepages+0xfa/0x2a0 mm/page-writeback.c:2344
>> __filemap_fdatawrite_range+0x2bc/0x3b0 mm/filemap.c:421
>> __filemap_fdatawrite mm/filemap.c:429 [inline]
>> filemap_flush+0x24/0x30 mm/filemap.c:456
>=20
> This is a double locking deadlock. The page lock is already held when
> we call into filemap_flush() here, and does another lock_page() in
> write_cache_pages().
>=20
> To fix it, we have to either initiate flushing before acquiring the
> page lock, or simply skip over dirty pages.
>=20
> Maybe doing vfs_fsync_range() from the madvise(HUGEPAGE) call isn't a
> bad idea after all? (I had discussed this with Song off-list before.)

Thanks syzbot and Johannes!

I just sent a quick fix, that just removes filemap_flush().=20

I will work on a better mechanism to flush the file.=20

Thanks,
Song