Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S936357AbcJRBI1 (ORCPT ); Mon, 17 Oct 2016 21:08:27 -0400 Received: from mx0b-00082601.pphosted.com ([67.231.153.30]:50917 "EHLO mx0a-00082601.pphosted.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S934873AbcJRBIV (ORCPT ); Mon, 17 Oct 2016 21:08:21 -0400 Date: Mon, 17 Oct 2016 21:07:00 -0400 From: Chris Mason To: Dave Jones CC: Al Viro , Josef Bacik , "David Sterba" , , Linux Kernel Subject: Re: btrfs bio linked list corruption. Message-ID: <20161018010659.GA88177@clm-mbp.thefacebook.com> Mail-Followup-To: Chris Mason , Dave Jones , Al Viro , Josef Bacik , David Sterba , linux-btrfs@vger.kernel.org, Linux Kernel References: <20161011144507.okg6baqvodn2m2lh@codemonkey.org.uk> <20161012134717.n74tww5eywc7dqp7@codemonkey.org.uk> <20161012144012.7vvfehceoykswmun@codemonkey.org.uk> <20161013181622.qpi5puv6ivxvslnf@codemonkey.org.uk> <7b476728-75f8-e3d3-1261-b9b0d598ed10@fb.com> <20161016004240.hpstul32lb2f3u4g@codemonkey.org.uk> MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed Content-Disposition: inline In-Reply-To: <20161016004240.hpstul32lb2f3u4g@codemonkey.org.uk> User-Agent: Mutt/1.5.24 (2015-08-30) X-Originating-IP: [2620:10d:c091:180::3b56] X-ClientProxiedBy: YQBPR01CA0056.CANPRD01.PROD.OUTLOOK.COM (10.169.139.24) To MWHPR15MB1245.namprd15.prod.outlook.com (10.175.3.7) X-MS-Office365-Filtering-Correlation-Id: be339bcf-6644-4d76-c54a-08d3f6f31740 X-Microsoft-Exchange-Diagnostics: 1;MWHPR15MB1245;2:2ljDhwH/lPRbPD8CsWCAz6xsd6RM0BnwlUzj7oDrSJbZm68mI+MrfE1NdBI5d8AQdEEQisqw9fIwa7relyBqwBvkI4cswvRHZsioWppyX6g4+f7oZOZlYf+JrxcuyerCAHpur4NOZBssiFix0E0w7pkL1MVwwxQmgRGhIB+ZUPgIGpkISnvSvMFWC6E1r2IfRi90UeRdLM5X0+niemPVtw==;3:aFxY/7JMrub9e87ImnLwut5ounsjx8qSkf7v7Wimww/7zJJnyej1/3buHCafCrlFxIR1rIu99S/eYgBjaGIOAWlEhUCu//GwS4rYwfXorwFiqeohGpo+mqSNUstx5ES7JTmgyRIxyfLZWwtEp1dWDQ== X-Microsoft-Antispam: UriScan:;BCL:0;PCL:0;RULEID:;SRVR:MWHPR15MB1245; X-Microsoft-Exchange-Diagnostics: 1;MWHPR15MB1245;25:olkzfmwhIohy1d6IBEVtGxa4LrHaADyICcpoBXIy3Ki4zRomwMb5GuGjpy8Ir67SUn9rpc1Wr1fsR97NVj/AEzMw2WZMBfscaTAWfJDDPFgd2AzcK6GDl+6RlzAuiSjqH9Zq501qYtljJcT/I0LCx+iEQBiSobaZdTBE+5EHSluoeBwQNwlGPwoeNudcjX0xmlkFHPsUqwBQKHejenTME5ATgFyNVGbezVFeoFkbglluefZ+MFVF6+CF9C/Z8/2nwzRGmQsgalT2+80IphjMZBbOvnois5TlD3nCafMDkCpATfuxpOLZuDMAiXvXPQYJtZEHBFhrz6jJeSeQwggKdK7zRmpQjbjncHstS+ypOAydxh3poCvnpbVLcOHvrvHbwXZLS1xcqwCxIjg6mmY6WChz+7qaFE/jVprAzSzjiLcHEKDHvcYZDj02iix2aAUCYWgy1T1vO6NwH46pHyVc++4Ra4IRj7S2oCRZw1p6623CWn/dHnNrTB5GVo4a9I7bVI0kM6QcWwjH0G5hnjZQ9GQurv76QZ9753iWDLFsuSPRzVFWonxb0WOpQ/BxXTShMImf7bbBZjVxZypz051Bb213p4nmTzG4eB75fC9+HsMMjVY8c24R0C42WWnUW+RfijfA0SdM7JUY6DIbou+OcRbfCKmEWPrmR/vo+eGzpf3tUKBrIouwIXnzVSP1zlt4ZrjjyjBSdUFPSAZn855ArSLTk0Z8r97bn8R+g+8z7I4= X-Microsoft-Exchange-Diagnostics: 1;MWHPR15MB1245;31:20CRHh8cAHnk3iqc/BC9ZDbJPm17hxP356tCMOFxQZr2ahsGmEiFMMVEjB0duPGX65zuoRwibXsy8LQ2qFttEFxMXIvyYD3NcFGPiwJm9iD5ASQlKNWKpY27vdoUuEHYkhdus/dGdmLEHjvTUQZ9bOskGYz58IjGnT37bKbQ8IOmaWBlHm8huZch2Nn2JhGaztQxWmdiu9TmaRfaXmcb0h0NU8gAXw3Nz9EhObRbKrdknIoyeBNMhIhYN96TqEoGlH5Zyq7Liexq7xhQysYj1Q==;20:ZLIu6uV3Df94an0tU56Np/rzpKI9dSvg6gsgWuJ/I1pIhJRKb8I0y46QeWOLmlr+APcBsFSCtmtiP5Q/Ks7ggaJAca3CU9QM8+OphwGIgos7JPO89W1rumj6KxXhVr6HRcwjOGKKfG8eUBr5DCqNu5H84zCfsDladJS6T524jWU=;4:N5up6IaKMqdneXeGk/oaL3sonbfBcPKL6eCIvIKtw2IyDn3Nv+3/EihzrWn1bKf6UAcDSXm0lQkvKnjOzLmNP4b5sEJl2FjKTxmy0rpzMn2G1WL297SY/veRTOeWTDAEtgDMhEmhL6YmLksc60uM32wssUvwp/IdmcLuinUMBXc/TPSUv1BHlgqepJUgXFgdeYarsz6HZpSnA1NPlob/iPwWiIooPz0Bos8ZWDBUECUZYgnqY9bnsomRT41v4MxeClwZE4J9aDP/JywC6rR0+e+jKy+3vj6cZ2htk5WkgqzSfvwxxntB9ipbQCgeljvXHGSshRzb2ts1Q9WyL//+otZcNmHp6Gw/M1MQoAsXn4SdhmjNzr1ebTdBGBSan7eR29xdXnsxThz4C7I2gjNWRw== X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:; X-Exchange-Antispam-Report-CFA-Test: BCL:0;PCL:0;RULEID:(6040176)(601004)(2401047)(8121501046)(5005006)(10201501046)(3002001);SRVR:MWHPR15MB1245;BCL:0;PCL:0;RULEID:;SRVR:MWHPR15MB1245; X-Forefront-PRVS: 00997889E7 X-Forefront-Antispam-Report: SFV:NSPM;SFS:(10019020)(4630300001)(6009001)(7916002)(24454002)(199003)(189002)(81156014)(81166006)(1076002)(50466002)(86362001)(54356999)(189998001)(68736007)(6116002)(586003)(9686002)(42186005)(551934003)(33656002)(92566002)(93886004)(106356001)(105586002)(77096005)(83506001)(53416004)(110136003)(2950100002)(6916009)(6666003)(4326007)(97736004)(50986999)(5660300001)(23726003)(69596002)(101416001)(47776003)(4001350100001)(76176999)(2906002)(7736002)(7846002)(305945005)(8676002)(18370500001)(3826002)(42262002);DIR:OUT;SFP:1102;SCL:1;SRVR:MWHPR15MB1245;H:clm-mbp.thefacebook.com;FPR:;SPF:None;PTR:InfoNoRecords;A:1;MX:1;LANG:en; X-Microsoft-Exchange-Diagnostics: =?us-ascii?Q?1;MWHPR15MB1245;23:+sq3YPjxHisORzMWCVpSxgo3/s4l/JGZNxCpS5miI?= =?us-ascii?Q?U0MSHgqzHaSbMdiNiK2gwCoPYqfLhqvUAZAqdW/F2HzBcKOzCZJqYkrxggUO?= =?us-ascii?Q?YMjaQRYCboZEcwJsqVo2HfrsaaONKZf12Ma/Fm7ZVdHpQw+t1ed3Eid5iHvf?= =?us-ascii?Q?aT30oA2detpZPC34NEn/SBeyVIL4at0Lddf89MdosR+sfMPVvaPYuDJ0lRCI?= =?us-ascii?Q?E2WEZbkVmLBS3ob/PL4YSfDNgKpQD8OLZaZtzaFd9k+/7b2pxfbkscek9unO?= =?us-ascii?Q?TANvoHUqYxjm70M5AVGu1PhGjHAKEK2Eoy6yXddezaCqt86WvpReKZjPu+c3?= =?us-ascii?Q?C2qILstdSXYsIW7bSPrd4iCLGjSiaGzXkxy871DKM1gLARgYyC9IkuN7HEGB?= =?us-ascii?Q?dC9GrR+XZmK+L5Y3c3Ou2kI2gkPP2cstrOf2QAj2cUVv0f1qKWOpnRPhLozC?= =?us-ascii?Q?3U4hfm06WpLaXOP8cZ4G9AVIrzqgDeJhroQ4aGQlIX2JD4OI5tv2elpo2w/A?= =?us-ascii?Q?plCKsn6xVgIadDUxR7fMkoZU6we9G7piy+AHObNCf1JixhpYVa/PyG0zIZYx?= =?us-ascii?Q?h6Lr1+U824bsCKkFTNTDCFADq6d7NYCIpPKRNVlcZHX+so7QOyGVr/RS9F32?= =?us-ascii?Q?i59EmnzFsTdhh3UzWk93j33kIHyYQsycoLI4k1m9XI/9fSU4woC34G9D+p4m?= =?us-ascii?Q?5JeMTv/P2LWxUDESN8/b5bkwRULIYCujJOlTTx82Afjk83YZvASFcA5SfeCY?= =?us-ascii?Q?CzcbaBJq2GZ9IS6TWKAwYdp4E0hJkkQoKLmBGivg2tdPAVNDSiT2IBdh8V4x?= =?us-ascii?Q?s/pRVK8bKNxcToIHhZmKXUkfq4aun/W7uGnOZ5oJy1icLsB6gKPEIzMNHKuV?= =?us-ascii?Q?gn4CL9pK/sMlL4t31UbJW6Va7BRuloSSJA9ouZd6V4XQRv/jfZqme1VYjRu2?= =?us-ascii?Q?8to/NnaLA1u3JVmmzKJOBTirVqWngEbmhI6q6F2tj63mNqD3aAvhVSJhEUvx?= =?us-ascii?Q?v0lwAyBhRr5qSsC3qJVPrm1VwCrXvbtUpIxqcZVxoz+U5tj9kwT9wn+ibQpv?= =?us-ascii?Q?Lio5ohjmYg0iOw3MZ2oQeJD7yG28odyiXz+FvzK2N8HhiJclnnOgopViz8Dg?= =?us-ascii?Q?hXB3EPIhwi+6CH6GjY7SwuD+q18xG0c5C3q/7wlPeveMA5lfNUEVUYlG8bQx?= =?us-ascii?Q?51iyKKtRmVoi69k+sD+lgZ8qVwcyZRxMH7U?= X-Microsoft-Exchange-Diagnostics: 1;MWHPR15MB1245;6:AuMKti8qcIHRNXPuOUfQRuniMSG8szI18wLR/OUJpLftWXyPWytkSDIyg/kklgiSNqgZ6rDt5c5FYNzzbcWDKIrslVlognhl/wLQlUW2Wul0SK7ZGHsxVD/e+YCYCuns+h0eiGlh6BpF8ixWo+AGdncpvIx5Fkj806gCC/epNDMiUDq+4BTAUWCzr1dEXw37/dl0R/GeQ4LO/6dAJfLrADJrvKDE77lb4kEewFfdn8ulvSKix/vEsiC2yyrc6xkLliBCCyZSEl6nHydjE7KkL9e/4cP7eI4VbcEw+VoLbrjJaUlJcJwzzxo6wdHQk+fe;5:p4tHNPlnbIjWIepIJksKbjmSJr2lF1xXb3prs4CWLckwVTSkFGLL1ZTZDZHujXsovki7mi2PsofxklTn8V6Um24DVET0Zni8t15AYidRDngCT51xeFe4/PnsqxD0oL2I9Hx+o8fiolPdWjFbGZYjgS6BsWaVT+rzpmcurd5qfa4=;24:NUZD8e3wQhKG34MZvxhlpw6Jjd4R2jt9yu9SLzB27q1w1+IGTp37+M+BRvwalDXA8OTZVC+Q32LohU4h9PtVBkffFeKG3jJPRccKCqaNnIA=;7:dePyZE2F8n7OhcHV1EDnY/nFtmEipnDl4uP+An1bMIHihQROrg5QOhIBxRdxLtFdVzgypF1IQ1GhevGXB1Ity6NGFrKOBndJbbaAIQOdF6DZ8A1FMu+RDhwJRl4yL+igliow5auSgptFoe5RSW5HGPKITRIVQhTKv1WzC4d3ulq09USSD15z8iDSKgJERdEybufkC1k4hlh+nHLaXVVqJEzTLWrGNibHN59YhlFJaUYCoW6p3MwKhIr80NUJjVzM7UJcAncJYQaPMVGxVirqD74CPj6GicFXjloKySIJzSMf5wze0NEwbbisq5h1/KJVmlxt2sdgrstqZuQDHdJXV9BBw28xMckiyqQLx0+e94I= SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-Microsoft-Exchange-Diagnostics: 1;MWHPR15MB1245;20:SqCROR4f/3+kA126Jrc7qaU5YmQD80yiI2a2Hlg95CuoXQHUjh7h0BE2J2psZbTbARL6ytPzHU7NlN8wdkkg18uLpV9zlyeJGhbDIVDQW0tUzKwDYxYwhHyJE4lAIeCisYnjP7cA7OWjBqLq2iDVjlbq5LrQ1HgftaCp/2Oh3uU= X-MS-Exchange-CrossTenant-OriginalArrivalTime: 18 Oct 2016 01:07:10.2066 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-Transport-CrossTenantHeadersStamped: MWHPR15MB1245 X-OriginatorOrg: fb.com X-Proofpoint-Spam-Reason: safe X-FB-Internal: Safe X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:,, definitions=2016-10-17_09:,, signatures=0 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 1534 Lines: 37 On Sat, Oct 15, 2016 at 08:42:40PM -0400, Dave Jones wrote: >On Thu, Oct 13, 2016 at 05:18:46PM -0400, Chris Mason wrote: > > > > > > .. and of course the first thing that happens is a completely different > > > > > btrfs trace.. > > > > > > > > > > > > > > > WARNING: CPU: 1 PID: 21706 at fs/btrfs/transaction.c:489 start_transaction+0x40a/0x440 [btrfs] > > > > > CPU: 1 PID: 21706 Comm: trinity-c16 Not tainted 4.8.0-think+ #14 > > > > > ffffc900019076a8 ffffffffb731ff3c 0000000000000000 0000000000000000 > > > > > ffffc900019076e8 ffffffffb707a6c1 000001e9f5806ce0 ffff8804f74c4d98 > > > > > 0000000000000801 ffff880501cfa2a8 000000000000008a 000000000000008a > > > > > > > > This isn't even IO. Uuughhhh. We're going to need a fast enough test > > > > that we can bisect. > > > > > > Progress... > > > I've found that this combination of syscalls.. > > > > > > ./trinity -C64 -q -l off -a64 --enable-fds=testfile -c fsync -c fsetxattr -c lremovexattr -c pwritev2 > > > > > > hits one of these two bugs in a few minutes runtime. > > > > > > Just the xattr syscalls + fsync isn't enough, neither is just pwrite + fsync. > > > Mix them together though, and something goes awry. > > > > > Hasn't triggered here yet. I'll leave it running though. > >The hits keep coming.. > >BUG: Bad page state in process kworker/u8:12 pfn:4988fa >page:ffffea0012623e80 count:0 mapcount:0 mapping:ffff8804450456e0 index:0x9 Hmpf, I've had this running since Friday without failing. Can you send me your .config please? -chris