Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S933304AbcJUUnw (ORCPT ); Fri, 21 Oct 2016 16:43:52 -0400 Received: from mx0a-00082601.pphosted.com ([67.231.145.42]:50219 "EHLO mx0a-00082601.pphosted.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1753953AbcJUUns (ORCPT ); Fri, 21 Oct 2016 16:43:48 -0400 Subject: Re: bio linked list corruption. To: Chris Mason , Dave Jones , "Andy Lutomirski" , Andy Lutomirski , "Linus Torvalds" , Jens Axboe , Al Viro , David Sterba , linux-btrfs , Linux Kernel References: <20161018234248.GB93792@clm-mbp.masoncoding.com> <332c8e94-a969-093f-1fb4-30d89be8993e@kernel.org> <20161020225028.czodw54tjbiwwv3o@codemonkey.org.uk> <20161020230341.jsxpia2sy53xn5l5@codemonkey.org.uk> <20161021200245.kahjzgqzdfyoe3uz@codemonkey.org.uk> <20161021202325.q6uh7k2pgnv276rg@codemonkey.org.uk> <0cbc7ee4-9268-9765-dde7-1e28c4cdf8f0@fb.com> From: Josef Bacik Message-ID: <411b54cd-c23c-765a-9547-0b43bf422546@fb.com> Date: Fri, 21 Oct 2016 16:41:09 -0400 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Thunderbird/45.2.0 MIME-Version: 1.0 In-Reply-To: <0cbc7ee4-9268-9765-dde7-1e28c4cdf8f0@fb.com> Content-Type: text/plain; charset="windows-1252"; format=flowed Content-Transfer-Encoding: 7bit X-Originating-IP: [107.15.72.49] X-ClientProxiedBy: BY2PR07CA0032.namprd07.prod.outlook.com (10.166.107.27) To BN6PR15MB1313.namprd15.prod.outlook.com (10.172.206.139) X-MS-Office365-Filtering-Correlation-Id: 26fb361d-db7c-4975-18f1-08d3f9f29b91 X-Microsoft-Exchange-Diagnostics: 1;BN6PR15MB1313;2:6xMfQfUFrsTrQmn+6ZkZuwREq3jJH6mMB+go9ONbLGkKVHpm24XTYwvXz+bu84coVCq0LTOPor1/m41NxKUR1XbLfE/N/owWngdJvw//TCKSbOixLJt5C4QxhKr7cYwRs/eOJWqGb81X5xi+a5is0XteNIW8VihufTnagIjrEoLd7XxYIYgW9hdn6RQYhQvzNuF5gpgZpNJMuAYp7Xp2hQ==;3:8a7JwodRaOtqZ2M7TBiBPpB+28DcCu4AGyJtc2GVgUuYqzoPuHbUzJWdUBAaIFVfg5CIwAzdzfecnAsnpkpFgqVn5twKuv71XZ0ncxvwdoa7oMm3/QzM1ppZ1epEehCDO+DS1SvmGCrIIwLJUr2wJA== X-Microsoft-Antispam: UriScan:;BCL:0;PCL:0;RULEID:;SRVR:BN6PR15MB1313; X-Microsoft-Exchange-Diagnostics: 1;BN6PR15MB1313;25:SjmWwmFZLqViEbUquSHcCZxFu1BjfusBNKI2U/L01GMx2tBx7tuJXI6upKThgK5ByMoj0d0Am8ub7G3K1OsE5L0fsMIV5NnkFwYiuXIhv9pHf3nY2dIuICgLpqmqZVmn7eZdtkohZ2DATLkds2lJ0mKPamiLjeDL8qMBT4BqlMSUbAA/Z0nLYFsF7f2FLC2+Mu7+3yo5BJye2pxQ+nzOtXEj+P+dr37C9bF3Hd3wmlDM/huc4Tg8fVnsVBFjfH43RUCoi0kdAaKA2IKCH7lvcvY7+FOyX46xbzeqRJbRrnCuuYbWatKqWloAyh2DZtwsVmDAk++u2v+JSrXHz/vLRP0NHYtjodqkk9m+/LQluvbAHw9mlyseexJ6qxIXtNKjL3TCfeqyEVd694sNFp9+egX72iSUoxdNQBDP2hva4oyqYBxUag+AE74IF4nj38AYI9MMTErNFEjYsQfS2pdBMk8by/XeLbE85/XAOyRMUIvTwZGgciqBxoncuaPrDfKm0p9h/zP9236lrIMe2xELt+L1wvMOyz3mPFMFTdUT13Y4J49iqBcc4g6cmTv+nMmnApRdXmHiB+Tc4bkG3IMVwrMSylQDsyQ4nP/P3XQa4T7nrePUTG8+CRRNNBSgoegfB9DVcJ7q/gEnVNhOdAAOfEO4dW0Bqfegfr3KaA3hbjvG4EFy664OQMUNuxR1L5ry0SyZu+5375khAXPuMWPvlm7FwuNFDYphXn2FCnGADSGL0tFOlTJodGWwMiElv6pn X-Microsoft-Exchange-Diagnostics: 1;BN6PR15MB1313;31:v9Xsr7Xqi5dC6zxuFyW1I4T1+TEwi0vFFf39oRrQMJqNikbH/CMAvbN1Wo8sci9VdRH+pzGxZkDl6TWv6POx6WxCIvIHPswfp+74Glzu0RN/jxXDdYj3/pS1BTpcXCyAO55d5HKz4LR5m9896EzaxGOOS9uQzPHf44A6IdWjHF9oVjvAw6euQiRlGSWZ8rwgMxh0dVmGaFNMLuzcPpNGfqaxp3f8kDJNvUYGALEVZZDoOlScEzfft9Jq1HX9bPLO3giCYD0GViJ8fcGoWFxXzg==;20:6cbLnYt+8oPS615NCxrxFQ5ZuREsh4K/Q9FZzMdsNExYe0UcWdcudUjrmCtMyd3j+yvLNN5elRMN/nh9/DU+JwKHUxU3TPi+tHs6u/uFhTnanl6xueh06sDcJw/9VXEDgk5xZe2taUyl6dle9KZaaUH+yhc/SgfK1Cm5eY/6P4U=;4:IWVSAeoZhvQEcpznhNZmsNf+pZShcRW9pDLkS9cfS8IPAYbLexW1oGKnRkAWPRJgUUNW/89SVpkvGk+kzqco/yLPSWe702RJgDsKPPjdh24wlMZkQhEVkltN44UTCQIAJ9ikrixKg1sUVBJFF1PcBc8+VGuLJuhwm+jGij0OrzPw4qjz5zYaGiq5cBJMnNBSojJGJMKsJ6eQxkEqsbDgn4daqHnAJxStpjHHlj74fMSoj4b+9aJPExCCFZqQRbBJ1EEF3vbLii/NkQDvMGdq0b/SQfBQEHDqKWy4CX1SMJmLR2jWTzrLRwC2LZsTH1IqeMGwz/+G1S2AGTSzfhiofSEm6LeC3viHgRpN3WSX2Wet02yheZBWTs6nRKKK5lvo2hdirPEqHWEtqB3s3uISNQ== X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:; X-Exchange-Antispam-Report-CFA-Test: BCL:0;PCL:0;RULEID:(6040176)(601004)(2401047)(8121501046)(5005006)(3002001)(10201501046);SRVR:BN6PR15MB1313;BCL:0;PCL:0;RULEID:;SRVR:BN6PR15MB1313; X-Forefront-PRVS: 01026E1310 X-Forefront-Antispam-Report: SFV:NSPM;SFS:(10019020)(4630300001)(6009001)(6069001)(7916002)(24454002)(199003)(189002)(377454003)(66066001)(81166006)(230700001)(33646002)(65826007)(101416001)(36756003)(7736002)(31696002)(83506001)(305945005)(8676002)(86362001)(77096005)(7846002)(23746002)(81156014)(2950100002)(6666003)(92566002)(5001770100001)(2906002)(65956001)(42186005)(93886004)(50466002)(6116002)(4001350100001)(3480700004)(64126003)(97736004)(189998001)(54356999)(586003)(3846002)(107886002)(50986999)(76176999)(31686004)(105586002)(47776003)(65806001)(106356001)(68736007)(5660300001)(921003)(1121003);DIR:OUT;SFP:1102;SCL:1;SRVR:BN6PR15MB1313;H:localhost.localdomain;FPR:;SPF:None;PTR:InfoNoRecords;A:1;MX:1;LANG:en; X-Microsoft-Exchange-Diagnostics: =?Windows-1252?Q?1;BN6PR15MB1313;23:SSfshqR3Htd1sMEPtMsaxWRTC9cbAx8o7SV+e?= =?Windows-1252?Q?2dCInDK5pNgy4KwGn2rdVWZq+PnWGkZiFdpIMzitszw33im4dfDFmkRB?= =?Windows-1252?Q?Lxu5kJ0OpE2LcHK4z35DE1vyZa6s023B7/Q4riV6YhBt+iWSgK5txW/k?= =?Windows-1252?Q?t6exwexdsZs4DizTLi4wPBxCZtU+FyYiiT/ihs1w+4/YBsthigpvTqo9?= =?Windows-1252?Q?kb8I3zP/WrbjrVRrcF3ZPTHN94joilZT3JdcD69dkTEiab5Hz9tyH8xT?= =?Windows-1252?Q?xXUKrN+EuFy2Mesy4VjLQaU4ZecuvUJh/pqaZgHD+mzU/12ST9XoIFmG?= =?Windows-1252?Q?4+nWz5cHDyvc/aWMt/3Nv3qGg/OYFek49m9S3FKM60csYI9gvbmIX8mG?= =?Windows-1252?Q?INoWiAz+D03k+rY83CTzdTcIG3boW78KZ1KCqUiTqIyikx2sanmZs2IX?= =?Windows-1252?Q?PvOSgvLa5HokgP3u/f0bBcPxs+0rgP8sbbwKqA60OOPZeblhkWdUZAi5?= =?Windows-1252?Q?NTBW5NSOYettoamTMu3plXuXR2HGMg3smtyiuXfUE/VpECnWCyjspQ/S?= =?Windows-1252?Q?TPs7wdE5Eq/vfBkLagZ4xa4IfNppaTAxRGilBEqFILxCtbWfmORenmN2?= =?Windows-1252?Q?pQv8cUz14wge4VIoQYXjI03lFQPwdyi/xMoPHf1gWMg83SZc295M89SG?= =?Windows-1252?Q?mMg/y7mhffnCBYkyqI6nuDkmozi2KZDBI2HanXdAG0gn8Q/FftKZ3UMZ?= =?Windows-1252?Q?yTs4sx58/BNbCAypVQNNjBgZZA8Lgjjo6LeMqfXPsYrTHcwrmlizE2bC?= =?Windows-1252?Q?eq2wxsaj4lrxRseuq+WGGK8sBnTz6HgKI7uCjDs6ZJS0x5YuUwdfonCT?= =?Windows-1252?Q?L8GF7kXI5OiOSHlg6KRL/wIXISIqy/Q2L3UOh1+09frUx10ZE4/3sARV?= =?Windows-1252?Q?R4t8xZfjCys1MHJN4phFqAGyiB9FTXZiI7WofFFql3IwcveYxRS6Qn9j?= =?Windows-1252?Q?JsL7RueFJdnQfZql8/QJUpjYL76yoGuhe8m3csddBRFFwr1fQPwYvc0j?= =?Windows-1252?Q?EwQKk6gKFd8kBhZLuohjZDttGWw8lUvvzeECHkklZYUmEkdUaGs0ZV+N?= =?Windows-1252?Q?PrCSwZ7bYhBJ+L+vBeVf9R8+xzuZ6ny9uj8MM0JjQSzYORBmVXoEUGsy?= =?Windows-1252?Q?mjAk3DWV1QkBPsvJu0piOvI4VoT5Wl0Us48M9DDTGLVMo+EliNaS0VAD?= =?Windows-1252?Q?dv+u3masy645TPTmI7JdiR3USWTr31bqyRy5S9ID3H7h/1jtkD7Irier?= =?Windows-1252?Q?186HpmYJCTNLUDbZmEidtAEvwVbXN/kn4H7N6zdruJcT4WPIRMjWflAh?= =?Windows-1252?Q?yxMofC6cryMn6BAakPV6b1mLRLc0oQedDkByUJgqZCGyzZyCNR74SdjG?= =?Windows-1252?Q?Yp92rHd/Ezsk1fZSMiaBmqDKJLlNxFPfTCaoO5rGQ=3D=3D?= X-Microsoft-Exchange-Diagnostics: 1;BN6PR15MB1313;6:vmHIiO4436NGw3rNnzZIO07jtXHL1VfDjeyAw2byddUgE1Szjc+/H9yRh/yn0f4hbBb3EKMlDvJtNVIosfeciv+K+cBfBoQjnE+YAEH3ETqJghDPrrMQjDHShdlAvILZA67HV7fmP/IspLFyAFQdZ31kCvoTqi7OmUvJH6IxkfleN3LKDJimyAqBgSGzoBH73w0XnvVYo5mnZ5VMeekvJxoMBuh5NI2iCfCfHaebWcZY7PWNdVKohtzTy50QZlMIKGJMytIxCPNmoafn43VtcQZXg2zaI4FWT8fY4P0KPkaLEM/G/F59cveVbqRHoe1f;5:bkjNRtvZGDq1tW1FET+WArERoXtfjwQPcTlrkD/948jWRQK5UIQEQ+qErzCLqs0yElnLt8fyD0FDqJ6+uzpzU8ZyAf+S7aynEv/IJQzeNTnpXOdTHIFxh6pq/QYYiKPb0g9XwD8CBCj6QFCqu7hOMqnTWN1gnCyEoOh0Yxmp+xE=;24:SV87KOF3BCEKyon/Qq7DIxidNqDITn4WhcpC1NlHPe4TO9ODe8DeEeMZcSBXKXEAG9/jz3f7QY9cvvcp+vnDGTMFX2V0uxAq1OZBlx82eaE=;7:0wsa9E4wQvxYjhoVHDsysYuwQ5EWxBlffYJhE4QECLLDNNQasVv+Z7C6r48YZfqp6m2glhBRaVwrHKPWgOyuZz1iwHe7n3Ino+uF9t4J9a2zfX3+UXJyteDEm5ZvpyO9X4DMIzKegmSblWkrHN/36bRiZ3giDHctWwcwZd6M7UHawvzbFNXd2VUL6XmpS5r3YQV94THCMwAa1u0IibpgFjrfa6ol86pUwi2CbtUp8RCHxOX+Z7sJNtnuci++6Ey41jUDZTxDr2ELxeA/CnldAB6L3EEu4NZxNJwoP2LtClQ5eyhOTdSa0SyQZc6idFKlwVrZ6Xgyjmm9xiRbokb+iSO2S4nWj9dOfYGQ8HyMuoA= SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-Microsoft-Exchange-Diagnostics: 1;BN6PR15MB1313;20:zmUNpuHpcQsJ5Js2x8E14NnCvg4o+f5W4miX6JERqXhgmz4jRjBlnjVLKt3xx3gJRXN5ewa0reZqvFB15BiEecYlkczyVCjRjTB+SS36mhBw5ZbvBo51QaTEiCM0tIGry8XbJ7E10n40pBHFXxogK36Hm/h/1v76zmN2ClM2+os= X-MS-Exchange-CrossTenant-OriginalArrivalTime: 21 Oct 2016 20:41:15.8115 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN6PR15MB1313 X-OriginatorOrg: fb.com X-Proofpoint-Spam-Reason: safe X-FB-Internal: Safe X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:,, definitions=2016-10-21_12:,, signatures=0 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 2469 Lines: 59 On 10/21/2016 04:38 PM, Chris Mason wrote: > > > On 10/21/2016 04:23 PM, Dave Jones wrote: >> On Fri, Oct 21, 2016 at 04:17:48PM -0400, Chris Mason wrote: >> >> > > BTRFS warning (device sda3): csum failed ino 130654 off 0 csum 2566472073 >> expected csum 3008371513 >> > > BTRFS warning (device sda3): csum failed ino 131057 off 4096 csum >> 3563910319 expected csum 738595262 >> > > BTRFS warning (device sda3): csum failed ino 131176 off 4096 csum >> 1344477721 expected csum 441864825 >> > > BTRFS warning (device sda3): csum failed ino 131241 off 245760 csum >> 3576232181 expected csum 2566472073 >> > > BTRFS warning (device sda3): csum failed ino 131429 off 0 csum 1494450239 >> expected csum 2646577722 >> > > BTRFS warning (device sda3): csum failed ino 131471 off 0 csum 3949539320 >> expected csum 3828807800 >> > > BTRFS warning (device sda3): csum failed ino 131471 off 4096 csum >> 3475108475 expected csum 2566472073 >> > > BTRFS warning (device sda3): csum failed ino 131471 off 958464 csum >> 142982740 expected csum 2566472073 >> > > BTRFS warning (device sda3): csum failed ino 131471 off 0 csum 3949539320 >> expected csum 3828807800 >> > > BTRFS warning (device sda3): csum failed ino 131532 off 270336 csum >> 3138898528 expected csum 2566472073 >> > > BTRFS warning (device sda3): csum failed ino 131532 off 1249280 csum >> 2169165042 expected csum 2566472073 >> > > BTRFS warning (device sda3): csum failed ino 131649 off 16384 csum >> 2914965650 expected csum 1425742005 >> > > >> > > >> > > A curious thing: the expected csum 2566472073 turns up a number of times >> for different inodes, and gets >> > > differing actual csums each time. I suppose this could be something like >> a block of all zeros in multiple files, >> > > but it struck me as surprising. >> > > >> > > btrfs people: is there an easy way to map those inodes to a filename ? >> I'm betting those are the >> > > test files that trinity generates. If so, it might point to a race >> somewhere. >> > >> > btrfs inspect inode 130654 mntpoint >> >> Interesting, they all return >> >> ERROR: ino paths ioctl: No such file or directory >> >> So these files got deleted perhaps ? >> > Yeah, they must have. > So one thing that will cause spurious csum errors is if you do things like change the memory while it is in flight during O_DIRECT. Does trinity do that? If so then that would explain it. If not we should probably dig into it. Thanks, Josef