Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1756783AbcJWVcr (ORCPT ); Sun, 23 Oct 2016 17:32:47 -0400 Received: from mx0b-00082601.pphosted.com ([67.231.153.30]:42062 "EHLO mx0a-00082601.pphosted.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1756635AbcJWVco (ORCPT ); Sun, 23 Oct 2016 17:32:44 -0400 Subject: Re: bio linked list corruption. To: Dave Jones , Andy Lutomirski , Andy Lutomirski , Linus Torvalds , Jens Axboe , Al Viro , Josef Bacik , David Sterba , linux-btrfs , Linux Kernel References: <20161018233148.GA93792@clm-mbp.masoncoding.com> <20161018234248.GB93792@clm-mbp.masoncoding.com> <332c8e94-a969-093f-1fb4-30d89be8993e@kernel.org> <20161020225028.czodw54tjbiwwv3o@codemonkey.org.uk> <20161020230341.jsxpia2sy53xn5l5@codemonkey.org.uk> <20161021200245.kahjzgqzdfyoe3uz@codemonkey.org.uk> <20161022152033.gkmm3l75kqjzsije@codemonkey.org.uk> From: Chris Mason Message-ID: Date: Sun, 23 Oct 2016 17:32:21 -0400 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Thunderbird/45.3.0 MIME-Version: 1.0 In-Reply-To: <20161022152033.gkmm3l75kqjzsije@codemonkey.org.uk> Content-Type: text/plain; charset="windows-1252"; format=flowed Content-Transfer-Encoding: 7bit X-Originating-IP: [2620:10d:c091:180::773d] X-ClientProxiedBy: DM5PR19CA0001.namprd19.prod.outlook.com (10.175.226.139) To MWHPR15MB1247.namprd15.prod.outlook.com (10.175.3.9) X-MS-Office365-Filtering-Correlation-Id: 90677498-8f8a-4cef-5999-08d3fb8c1668 X-Microsoft-Exchange-Diagnostics: 1;MWHPR15MB1247;2:lI+Cc9pLUbQ9mN6rZxocICkwI0JUvuVGespuL+cDMLx5Ln2P93jX5Ds1tLcSKjM1yrjxNyf/512gSnsixmoI4a9WmpUZqwohYT8w+4CMZEOHdpFmbTSQ+YLXEhnJKAiLlK4AZBRs/H+8QueJ2pOGEfGICftrzF8GVh0h8IIeYZRAGqPaRW2KzIQVoVRL9mDtfbhxzeSuiBLxCs5P08e7BA==;3:5oXRYzUDCojBMvMOUJ0kuK4lJjYPQyqlBehyFAkSLaQ2Ad9MgM17Tt3pk8A7XhLj7kKjtG2fe6k8PYTy+2EeLkbEKmuVOlAwxrN5l/2tMwri+3Ri5qYu7/hKw+p5EhIA9t2mmatalAZRCGJR3pZpfw==;25:DWFFDobn0wdHl4d/c+aTb/GUPH83oM0gvrcflkvR7O0/Zv5/IuelpGzVsLgASnPFdaW5uE57fCNapE/HOxqyN107m5IsFlDMvYxVfMRNjelEa+TyqZ/UUsxeLqxeuBEBv9bjtgu8ewvgAEcTimvucTOqZ9DMbmEdDd+gs3uF8PE8SMWIUOGudJbaEC2Vh21M/HsP0Fz8gleKtYe68GJH5Z4uAUTfvU5t9DEx74cWUweQgZ+4NVfg4upX8ctt5H6WfSAY0+v51BsHRO0ddLKqAsByFHy2LJMYB5tq1t7e4tMFo7JXjUX9ZROYRtZc2B+hFDlJqnv+lHmyYDsv2oPUee6WPw3Z2U8dGQ2jBxwvj4Uy0XxDksXTMYfi49ftTndPu98Bdoz+wcAD/yhdz5bY1uknUGvm1rZNaGxXqA0zzEt2C0swN9hiOgAdwnU1p7+8 X-Microsoft-Antispam: UriScan:;BCL:0;PCL:0;RULEID:;SRVR:MWHPR15MB1247; X-Microsoft-Exchange-Diagnostics: 1;MWHPR15MB1247;31:NtUUmdam1NCqSQt3UL7QvLXQzbcHwyquyawP7pX4k1vTTE4mXyMzNCntuXSEJIrwTe74/CisY1+bjhdDVaTsQc/xzmSyTyZC4TXMb18Oh0baqkxu3fCJuIJmSk26AH0kwyDAU5ebytyZOurFBk9qzNEtMi85pjHlQ2IvJmPSqX6ycL4X3TDnhquYNCe340NcJ/BooLhWgWib4yzwrKqWhpjcLDVZhuXae52Flkb7tP1TLSTeh6kVAAl9zG00hwptxJG+x8qAZsu6djoDdpfdeHVTWDZcaWcTnT08JEbp5zE=;20:d2VF28Lpw1KtnYv35YBUwxTvinvRoCJwE5SeRTfw+9pDYfCAMu9FHf21CzVNzNgzhAAE2VNB8jGp5p4tq0ULyX6fTK0jG8Y8B19wOmHMPPT2YL/lOl7jqD4hALMaLGmiDIpo3fM0EFJ2xTCZLoxzB6UW1IAnQzOQ5BiZ9kNtOcE=;4:P3ySsBIPKrOtP7PCo3p5/QhJwo2LGK01HZoqLjGZ0lJOplzqa6Z60hJMASGqLCjI7IorYgPpDzHAHkgsow6lXuKgA4SPt4ScL5KyqFWwf8C+BjwfYzbPjwOCy6eIa9DTYfLQ+7KNWc2j4C4XFlFTf3LMb6TI61vB8w2w8RGB67MUKL/ByCdjQcKqUY6c1XeR7UwXDolcJZMRvFjXt5bjxCygJI76suGr+YxZWa815L2R95oEa4qu8z3EayvI9qtOxlxUA8tEpb3dZhPMGFwckJ9ynfZYmvCMEK8B9cBSZvVOeZ+uJvHMmoiJtbcFiORYe1Th4Zmx+M/evMbY+2jaDUyNTj9UBj8xPR4fqksLHb4aHPBevX56sW28vOapxArbvZF0i6YHxLk9XfrAH0zfi4PCrYF8ljaPs23rq5d+O9fFMEdiIc7RXJzGrYqThNZK X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:(84791874153150); X-Exchange-Antispam-Report-CFA-Test: BCL:0;PCL:0;RULEID:(6040176)(601004)(2401047)(8121501046)(5005006)(3002001)(10201501046);SRVR:MWHPR15MB1247;BCL:0;PCL:0;RULEID:;SRVR:MWHPR15MB1247; X-Forefront-PRVS: 0104247462 X-Forefront-Antispam-Report: SFV:NSPM;SFS:(10019020)(4630300001)(6009001)(7916002)(51234002)(189002)(24454002)(199003)(377454003)(107886002)(4001350100001)(1706002)(97736004)(5001770100001)(23746002)(6116002)(101416001)(68736007)(106356001)(189998001)(105586002)(31686004)(15975445007)(42186005)(76176999)(50986999)(586003)(2950100002)(7736002)(54356999)(86362001)(47776003)(64126003)(7846002)(19580395003)(81156014)(81166006)(83506001)(8676002)(92566002)(230700001)(305945005)(33646002)(3480700004)(31696002)(6666003)(5660300001)(50466002)(77096005)(36756003)(93886004)(2906002)(65806001)(65956001)(65826007)(921003)(3826002)(42262002)(1121003);DIR:OUT;SFP:1102;SCL:1;SRVR:MWHPR15MB1247;H:[IPv6:2620:10d:c0a1:1110:8000::204d];FPR:;SPF:None;PTR:InfoNoRecords;MX:1;A:1;LANG:en; X-Microsoft-Exchange-Diagnostics: =?Windows-1252?Q?1;MWHPR15MB1247;23:25tQh10dUyp3VqBYv3R1+HDVdGJoeDHGU87Ft?= =?Windows-1252?Q?u37Rnjf+QtGS43JOZHRGkHLlTPCCCzhiBm3lSJ6a1wKdQhM14iQp5euI?= =?Windows-1252?Q?GGV3yULM/Cd47D03soPJ1ciW4u/XaaD7WG79fqc1PD34ElCmjO5vsMqQ?= =?Windows-1252?Q?zpopBUaoSGWKD7edA0eGykNCyTR0RUP9eIXFlBffprqDzg3oMXyVTXCx?= =?Windows-1252?Q?xKKlYEg33Ps18SbR46s/o23hKSkhL1ISqHGq84gQ3UUG11RxHQSQxIzN?= =?Windows-1252?Q?DbrRSAkPW2cifvWxuswQ0vo1LdkPQ7MXkGaz32Wly3yrjGL/3+dMQTWd?= =?Windows-1252?Q?Dzl86WxCQEdHZ2OqlKbnfwJP3NNG5b8t/sxWnbqM+u3cLxQhRmHYByJQ?= =?Windows-1252?Q?gMjkVYcHiq94jvmIVvrkjNTczf8AsFsnGsSf1qZCa+4jm5tp4qBy0UJf?= =?Windows-1252?Q?FmVCS+bvC9hwN5CrTGtLhZSk+jJ3+rSiRM+Iss1zlqsqHW6mJmhyBzE9?= =?Windows-1252?Q?bOVTy7hN64/5SLeuCZwb0z9tqVs43IH02B5dOtPhNpOk9O4SY7LtH9Ms?= =?Windows-1252?Q?M26jUSbPNJ7uDYFXnzYyCtKT17Q1kbcbXindkigO7QSb9SOQlVAt+xsh?= =?Windows-1252?Q?1t2oIDB04k3ufsu8DvDKNwym0fpnU3hgI0D4q1mQcxAI/hHnGgHflBEh?= =?Windows-1252?Q?Xn4iUPNZN2kx2lgruoC6Vvt0TV5lwpuenf0dC5JOfj2g8FCwDqtmz21P?= =?Windows-1252?Q?9MUXtpFP/sWdPj9FOGRBLh25YhsauK6qMsiG1m6UMNSK6GW1xEgIPWdO?= =?Windows-1252?Q?bCDunDXhbbqclWJbmAXqFQO9XGuPIWd5BQKvaZADEo7udwpDBKi19lya?= =?Windows-1252?Q?aAJpYcPiORcraUvzXjJWFEAaoDh+ecbFSJdm0RDEiU/U/Xg/kJCPj8Ka?= =?Windows-1252?Q?0/y4u9ljX+1wMt7qpiMqC7Lij3Hfm2c06kEuAXxvdsIIItDIpACptJGZ?= =?Windows-1252?Q?CSZfoisNOtB4Z2eSm+nRXFubxMWPeFbVTlp+sn1gRBmeX9Y5bj8lBqqF?= =?Windows-1252?Q?/tVF+GwmeVnkMeG+gvM8wKZSoC80VE1IlCPyjDb7IvrODww5a/rJM+fb?= =?Windows-1252?Q?aCehWGMdVXXqyPNsFnNu/FEuo47bR8K9Ob4PV0KNvCSSVp2Gubv6pSQJ?= =?Windows-1252?Q?4jw/oTyFR2Mp9Zy5A1b7IacTswGheaZokOjeOO1slI/qzyplaCiKJdTx?= =?Windows-1252?Q?Mh4MCnfcgGZWD0HYHY9wjwO/rG46+GaYYIj7/mPFeOxv8j2Jm9z6XSLc?= =?Windows-1252?Q?QeoFXDRCNZD/8rpG4n8m6vCXyXxf0PHeDE5kxz4Xwp6lKtOxn0KlPD8y?= =?Windows-1252?Q?N4p7F7RVOe2htye5q1b/ZWSzWxGqRCakidNAJm2E9ItbYwfF6PsJh9zQ?= =?Windows-1252?Q?nrgUUdLdam2Tit6+INNvnh81JbVYMFIaMOKlamiof2YIQQT6dkp+Y6p2?= =?Windows-1252?Q?qD3T0S7vzoEVdsrmA6wqVfWF7+Jl+bIaUzucRk5LDnRhA9Z0Q=3D=3D?= X-Microsoft-Exchange-Diagnostics: 1;MWHPR15MB1247;6:uxpGgAw/4HId6qUvT0iNJPdrheF76bfBI1o5B3n2MLJqsXpANwIQoCi+a1YqSmyXLhPqs9kKoVCRrYdJK3xSie0TnNQE8WdwSXyYpsvmCYjB6umH3VZtLfyik/Jk15i9V6eLV9vDRnAa7HCbNZu3J6iTdW1x/f+RJr0RNewjDdcrsHjOrlAqWZ0qUCAaGXkWD+nMvn0nYZTnvWsCKnKbylU73dtBxU89++AJ64Sr1bJi9m/LM9K1W2K2V0WTaRTG6uMRZD4GJEDrmxyqRwRxbgRHYEGGLqV6MEp6sySF/PovCzG+4wQ2mfUnU5/+HLiK;5:gdsSRcMIxkD//GgGGApLJocfL/BmCrRi1XpczK+cDO5WPvHMKYqniDp8706sZJdzH+uq0USnYJUx6goC7y7qBNZH1aWee5OXL24nVAUnUn+9/z4p8H2/DXGs2aigq6vPH/TzMs4NSIUVtmygyFchrw==;24:dZ2KLrYFGmWMa88GBQsTiZCH0Jv2QNlIf28AWIKKkJPdA9jyLob5tDdEoH4jF+pcNl9ObFGLH29Alk45Q0PU5ghj7BoOG3rQ9QO8c0kHaLE=;7:7n2gh4EIU3k5mYNu1oJ2yKnTnxpgsPcSWKBuf6LVdBMD146WauoU/4eN8Bxtcv0RXx4oZ8lUEtscS6fYWzeCEgogF2h9SkAYSouBUfK003PBx6MdNS/x3UCTbYHR7hALvmXTVu5jLzzx/qtW5F3zaSzOAGOzvFjw2yrzI0Q1Lp2pP1sGPEttUneAULVeX+nrzglZP1LpLzJMQmhmMW3oaZbyzq4SQSOsi8hxAWERpsST/q7SnDf5kngvetpN+3EZ3B4Taun86/ZF2nUplgmDs+alcvuaokn7LEVzwTUWiLO3CJf28TejOam7dXWnv10r76KO0e3SU6g0+WLG5dIR3GOdj7KfZYJXZ5ukmMzwIU4= SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-Microsoft-Exchange-Diagnostics: 1;MWHPR15MB1247;20:Qw7WDDVT6/Fv0l8uID79RtcT/B/xLPN9kT7ZitT4SytE/tjst8Ixe/6+++l1hsZWVakveUAWod/A42DChF63czeJslJEJkNLMbWC0DnT7peBNy4rRza9GfET+QeimR3Ux21TyWeARtgeKopPAezxR6ROW3Jm8eha6keGlzBkENc= X-MS-Exchange-CrossTenant-OriginalArrivalTime: 23 Oct 2016 21:32:26.2308 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-Transport-CrossTenantHeadersStamped: MWHPR15MB1247 X-OriginatorOrg: fb.com X-Proofpoint-Spam-Reason: safe X-FB-Internal: Safe X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:,, definitions=2016-10-23_08:,, signatures=0 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 1575 Lines: 38 On 10/22/2016 11:20 AM, Dave Jones wrote: > On Fri, Oct 21, 2016 at 04:02:45PM -0400, Dave Jones wrote: > > > > It could be worth trying this, too: > > > > > > https://git.kernel.org/cgit/linux/kernel/git/luto/linux.git/commit/?h=x86/vmap_stack&id=174531fef4e8 > > > > > > It occurred to me that the current code is a little bit fragile. > > > > It's been nearly 24hrs with the above changes, and it's been pretty much > > silent the whole time. > > > > The only thing of note over that time period has been a btrfs lockdep > > warning that's been around for a while, and occasional btrfs checksum > > failures, which I've been seeing for a while, but seem to have gotten > > worse since 4.8. > > > > I'm pretty confident in the disk being ok in this machine, so I think > > the checksum warnings are bogus. Chris suggested they may be the result > > of memory corruption, but there's little else going on. > > The only interesting thing last nights run was this.. > > BUG: Bad page state in process kworker/u8:1 pfn:4e2b70 > page:ffffea00138adc00 count:0 mapcount:0 mapping:ffff88046e9fc2e0 index:0xdf0 > flags: 0x400000000000000c(referenced|uptodate) > page dumped because: non-NULL mapping > CPU: 3 PID: 24234 Comm: kworker/u8:1 Not tainted 4.9.0-rc1-think+ #11 > Workqueue: writeback wb_workfn (flush-btrfs-2) Well crud, we're back to wondering if this is Btrfs or the stack corruption. Since the pagevecs are on the stack and this is a new crash, my guess is you'll be able to trigger it on xfs/ext4 too. But we should make sure. -chris