Received: by 2002:ac0:8c9a:0:0:0:0:0 with SMTP id r26csp4294758ima; Mon, 4 Feb 2019 13:49:58 -0800 (PST) X-Google-Smtp-Source: AHgI3IZtKFjAecvZkJ+mf7pTs7RvDdGIyEnE3ozluXww57wY2jTiwLN58HjtvGe9gsOikLC+lupu X-Received: by 2002:a17:902:2aaa:: with SMTP id j39mr1559019plb.335.1549316998677; Mon, 04 Feb 2019 13:49:58 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1549316998; cv=none; d=google.com; s=arc-20160816; b=yfyrWQBKYeGRT0pWNg9t4viGzoV1mMssUP92hCk/kRURJeNNhOdNU7Je66JJ15AbVw ClLm9n+Wfd1maVbYp59ZbgutqpV5lHuU4LsdPrv0F+h/MMdO2uAI2ZmDDHGPWkReBAvL DwpGpNz0BzHieyqB2N8+tI0+MPj5vcGWi14qNxjzYx3u4qFGcbapd38kUQlJLec+YEzi PwQj0W0/5gtFLJV3r6wxXbO+We1SgL6nrOhY3fCsWzdrpCqZm24UfNfH1v+4MrHKL42E xCk2Oyx+MN0mtVhTv7xNk8XNXPc2hKpSs6HywSLlFN3N3NVbonQ+WwaELoUoloDcWiXy 00sA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:mime-version:user-agent:references :message-id:in-reply-to:subject:cc:to:from:date:dkim-signature; bh=kk6MwTpxjihccu+ylol4Y3Kbog2koihAsmNoJGT+zfQ=; b=X6Vl0DWzIRbmXWHROFBFEM++xrQY2BteYpHygqoFmON++ApQEjc4+2QPDpgMrUxknX jS9ynmTeGmoKeVbQ0gJZi2rWUYmxuR/2LeSc89/7+06ovmKUUqKJt5J+e5z5nZdUwzG6 fTjb39+eOj4AdI29CeoBCC3TrIS37efsSOgtvWCTJT57a91d8Ydv2yb037V2yIVofAwq TsI77fwA3qgfzkUcIrRSsTQ1eihlRasYVjabvOWzIGbjEOB0Ht1pD3QdgsZnOHNOXJhF +7Go9fUVKi1R9fq2KgPruKJJ7aUoByeMbcR748g0tTxrVUQU+4ZD8r1jdrfn72Utx3mX dI4w== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20161025 header.b=tFOHSsym; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id a2si1097057plm.86.2019.02.04.13.49.42; Mon, 04 Feb 2019 13:49:58 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20161025 header.b=tFOHSsym; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729933AbfBDUnA (ORCPT + 99 others); Mon, 4 Feb 2019 15:43:00 -0500 Received: from mail-pl1-f194.google.com ([209.85.214.194]:37875 "EHLO mail-pl1-f194.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1729611AbfBDUnA (ORCPT ); Mon, 4 Feb 2019 15:43:00 -0500 Received: by mail-pl1-f194.google.com with SMTP id b5so483682plr.4 for ; Mon, 04 Feb 2019 12:43:00 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:from:to:cc:subject:in-reply-to:message-id:references :user-agent:mime-version; bh=kk6MwTpxjihccu+ylol4Y3Kbog2koihAsmNoJGT+zfQ=; b=tFOHSsymn9owFLGajQZCJA1UVDbpvu5HgGVuP7S9Fum4DuxQz/3BkuGAXugdSGOWrd ZjYkbT9UT8y9IUknr8TOkX7iM+iF5Vu5dpFjLaDJxYZOY6rwqgeFPDJHTBBK/f6ccpjx vVta/Q3aVtoif+9WWH7sT70koFRtky5kPfD4Z5yn9qjhkyCLCRnSYG390uNF31gwZzz9 x5PPmXFfkuje9GpBRrsgLaiMjbFgDX5j0Qk4toB8ZSc8T0v3h+bALrLm4dTlr9nqcF8y Gmgp9jVAoM2DoyKKsgwm2qGe8bxLeRd5Hxp3GpeWYyidCuIAM8MgrtcZdLRs6A7yYfi5 W40Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:in-reply-to:message-id :references:user-agent:mime-version; bh=kk6MwTpxjihccu+ylol4Y3Kbog2koihAsmNoJGT+zfQ=; b=jHHPVTjsoydpaiXyGPO3ll64azalqus1VSTwNcFzL3jYhm4Fk/Vyn5/wzZtVvMZigD Exr5+EBjQbf1ACjVhk5IdGd85qK1EadYLEswwqvMO+o267E8XK2Q/4KeWVIzI02v55RZ RtvDAcyv+6iSTFjO7lWlLCc97qBn5KQXu/vJg6Lttd5nXM4wFJvpR2j3NJyCnwrDpS3q ziXWPKaJvo+wE7Yc5A54N8NWCleuGKKwykCNw+jL6Cq20vyoMoIjtqVrVrPNnOmIThv1 jSERZ9iculAgZLeE9NrlOpS6HOgkocNJa4zBuvXv/a9DBqescFWL0YefZx4ZQzDdn3u7 0RVw== X-Gm-Message-State: AHQUAua+vEJwjPcqimjSnUi29YsEtbwaONeAk8YIe8NhRIv1AnNzHTW7 7s9uROk4VfiJOBfe8CaozZJAsA== X-Received: by 2002:a17:902:6948:: with SMTP id k8mr1337453plt.2.1549312979106; Mon, 04 Feb 2019 12:42:59 -0800 (PST) Received: from [100.112.89.103] ([104.133.8.103]) by smtp.gmail.com with ESMTPSA id b9sm994459pgt.66.2019.02.04.12.42.57 (version=TLS1 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Mon, 04 Feb 2019 12:42:58 -0800 (PST) Date: Mon, 4 Feb 2019 12:42:50 -0800 (PST) From: Hugh Dickins X-X-Sender: hugh@eggly.anvils To: Artem Savkov cc: Hugh Dickins , Baoquan He , Qian Cai , Andrea Arcangeli , Michal Hocko , Vlastimil Babka , Linus Torvalds , Andrew Morton , linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: Re: mm: race in put_and_wait_on_page_locked() In-Reply-To: <20190204091300.GB13536@shodan.usersys.redhat.com> Message-ID: References: <20190204091300.GB13536@shodan.usersys.redhat.com> User-Agent: Alpine 2.11 (LSU 23 2013-08-11) MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Mon, 4 Feb 2019, Artem Savkov wrote: > Hi Hugh, > > Your recent patch 9a1ea439b16b "mm: put_and_wait_on_page_locked() while > page is migrated" seems to have introduced a race into page migration > process. I have a host that eagerly reproduces the following BUG under > stress: > > [ 302.847402] page:f000000000021700 count:0 mapcount:0 mapping:c0000000b2710bb0 index:0x19 > [ 302.848096] xfs_address_space_operations [xfs] > [ 302.848100] name:"libc-2.28.so" > [ 302.848244] flags: 0x3ffff800000006(referenced|uptodate) > [ 302.848521] raw: 003ffff800000006 5deadbeef0000100 5deadbeef0000200 0000000000000000 > [ 302.848724] raw: 0000000000000019 0000000000000000 00000001ffffffff c0000000bc0b1000 > [ 302.848919] page dumped because: VM_BUG_ON_PAGE(page_ref_count(page) == 0) > [ 302.849076] page->mem_cgroup:c0000000bc0b1000 > [ 302.849269] ------------[ cut here ]------------ > [ 302.849397] kernel BUG at include/linux/mm.h:546! > [ 302.849586] Oops: Exception in kernel mode, sig: 5 [#1] > [ 302.849711] LE SMP NR_CPUS=2048 NUMA pSeries > [ 302.849839] Modules linked in: pseries_rng sunrpc xts vmx_crypto virtio_balloon xfs libcrc32c virtio_net net_failover virtio_console failover virtio_blk > [ 302.850400] CPU: 3 PID: 8759 Comm: cc1 Not tainted 5.0.0-rc4+ #36 > [ 302.850571] NIP: c00000000039c8b8 LR: c00000000039c8b4 CTR: c00000000080a0e0 > [ 302.850758] REGS: c0000000b0d7f7e0 TRAP: 0700 Not tainted (5.0.0-rc4+) > [ 302.850952] MSR: 8000000000029033 CR: 48024422 XER: 00000000 > [ 302.851150] CFAR: c0000000003ff584 IRQMASK: 0 > [ 302.851150] GPR00: c00000000039c8b4 c0000000b0d7fa70 c000000001bcca00 0000000000000021 > [ 302.851150] GPR04: c0000000b044c628 0000000000000007 55555555555555a0 c000000001fc3760 > [ 302.851150] GPR08: 0000000000000007 0000000000000000 c0000000b0d7c000 c0000000b0d7f5ff > [ 302.851150] GPR12: 0000000000004400 c00000003fffae80 0000000000000000 0000000000000000 > [ 302.851150] GPR16: 0000000000000000 0000000000000000 0000000000000000 0000000000000000 > [ 302.851150] GPR20: c0000000689f5aa8 c00000002a13ee48 0000000000000000 c000000001da29b0 > [ 302.851150] GPR24: c000000001bf7d80 c0000000689f5a00 0000000000000000 0000000000000000 > [ 302.851150] GPR28: c000000001bf9e80 c0000000b0d7fab8 0000000000000001 f000000000021700 > [ 302.852914] NIP [c00000000039c8b8] put_and_wait_on_page_locked+0x398/0x3d0 > [ 302.853080] LR [c00000000039c8b4] put_and_wait_on_page_locked+0x394/0x3d0 > [ 302.853235] Call Trace: > [ 302.853305] [c0000000b0d7fa70] [c00000000039c8b4] put_and_wait_on_page_locked+0x394/0x3d0 (unreliable) > [ 302.853540] [c0000000b0d7fb10] [c00000000047b838] __migration_entry_wait+0x178/0x250 > [ 302.853738] [c0000000b0d7fb50] [c00000000040c928] do_swap_page+0xd78/0xf60 > [ 302.853997] [c0000000b0d7fbd0] [c000000000411078] __handle_mm_fault+0xbf8/0xe80 > [ 302.854187] [c0000000b0d7fcb0] [c000000000411548] handle_mm_fault+0x248/0x450 > [ 302.854379] [c0000000b0d7fd00] [c000000000078ca4] __do_page_fault+0x2d4/0xdf0 > [ 302.854877] [c0000000b0d7fde0] [c0000000000797f8] do_page_fault+0x38/0xf0 > [ 302.855057] [c0000000b0d7fe20] [c00000000000a7c4] handle_page_fault+0x18/0x38 > [ 302.855300] Instruction dump: > [ 302.855432] 4bfffcf0 60000000 3948ffff 4bfffd20 60000000 60000000 3c82ff36 7fe3fb78 > [ 302.855689] fb210068 38843b78 48062f09 60000000 <0fe00000> 60000000 3b400001 3b600001 > [ 302.855950] ---[ end trace a52140e0f9751ae0 ]--- > > What seems to be happening is migrate_page_move_mapping() calling > page_ref_freeze() on another cpu somewhere between __migration_entry_wait() > taking a reference and wait_on_page_bit_common() calling page_put(). Thank you for reporting, Artem. And see the mm thread https://marc.info/?l=linux-mm&m=154821775401218&w=2 That was on arm64, you are on power I think: both point towards xfs (Cai could not reproduce it on ext4), but that should not be taken too seriously - it could just be easier to reproduce on one than the other. Your description in your last paragraph is what I imagined happening too. And nothing wrong with that, except that the page_ref_freeze() should have failed, but succeeded. We believe that something has done an improper put_page(), on a libc-2.28.so page that's normally always in use, and the put_and_wait_on_page_locked() commit has exposed that by making its migration possible when it was almost impossible before (Cai has reproduced it without the put_and_wait_on_page_locked commit). I don't think any of us have made progress on this since the 25th. I'll wrap up what I'm working on in the next hour or two, and switch my attention to this. Even if put_and_wait_on_page_locked() happens to be correct, and just makes a pre-existing bug much easier to hit, we shall have to revert it from 5.0 if we cannot find the right answer in the next week or so. Which would be sad: I'll try to rescue it, but don't have great confidence that I'll be successful. I'll be looking through the source, thinking around it, and trying to find a surplus put_page(). I don't have any experiments in mind to try at this stage. Something I shall not be doing, is verifying the correctness of the low-level get_page_unless_zero() versus page_ref_freeze() protocol on arm64 and power - nobody has reported on x86, and I do wonder if there's a barrier missing somewhere, that could manifest in this way - but I'm unlikely to be the one to find that (and also think that any weakness there should have shown up long before now). Hugh