Subject: Re: bio linked list corruption.
To: Chris Mason <clm@fb.com>, Dave Jones <davej@codemonkey.org.uk>,
        "Andy Lutomirski" <luto@amacapital.net>,
        Andy Lutomirski <luto@kernel.org>,
        "Linus Torvalds" <torvalds@linux-foundation.org>,
        Jens Axboe <axboe@fb.com>, Al Viro <viro@zeniv.linux.org.uk>,
        David Sterba <dsterba@suse.com>,
        linux-btrfs <linux-btrfs@vger.kernel.org>,
        Linux Kernel <linux-kernel@vger.kernel.org>
References: <CA+55aFz37uBqo9LG6pwRCB1pU94o3yz1SXwj3xwzHRXtg5Cgyw@mail.gmail.com>
 <20161018234248.GB93792@clm-mbp.masoncoding.com>
 <CA+55aFyXi-iUYx6kOnQrCjzGj-uoOa+0voz0HZz7DAFPYK6ctg@mail.gmail.com>
 <332c8e94-a969-093f-1fb4-30d89be8993e@kernel.org>
 <20161020225028.czodw54tjbiwwv3o@codemonkey.org.uk>
 <CALCETrV5hr_QQ7eiqrac7huh3hX1Mp0ArrOmKKj_eKHw5gx76Q@mail.gmail.com>
 <20161020230341.jsxpia2sy53xn5l5@codemonkey.org.uk>
 <CALCETrVHXPw1PyKSQja07V+MxHJPcDkaLJ1rPF2Z3286tHY2Xw@mail.gmail.com>
 <20161021200245.kahjzgqzdfyoe3uz@codemonkey.org.uk>
 <e094eb48-3007-f28a-12f6-6a162a43ad32@fb.com>
 <20161021202325.q6uh7k2pgnv276rg@codemonkey.org.uk>
 <0cbc7ee4-9268-9765-dde7-1e28c4cdf8f0@fb.com>
From: Josef Bacik <jbacik@fb.com>
Message-ID: <411b54cd-c23c-765a-9547-0b43bf422546@fb.com>
Date: Fri, 21 Oct 2016 16:41:09 -0400
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101
 Thunderbird/45.2.0
MIME-Version: 1.0
In-Reply-To: <0cbc7ee4-9268-9765-dde7-1e28c4cdf8f0@fb.com>
Content-Type: text/plain; charset="windows-1252"; format=flowed
Content-Transfer-Encoding: 7bit
SpamDiagnosticOutput: 1:99
SpamDiagnosticMetadata: NSPM
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 21 Oct 2016 20:41:15.8115 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN6PR15MB1313
X-OriginatorOrg: fb.com
X-Proofpoint-Spam-Reason: safe
X-FB-Internal: Safe
X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:,, definitions=2016-10-21_12:,,
 signatures=0
Sender: linux-kernel-owner@vger.kernel.org
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org
Content-Length: 2469
Lines: 59

On 10/21/2016 04:38 PM, Chris Mason wrote:
>
>
> On 10/21/2016 04:23 PM, Dave Jones wrote:
>> On Fri, Oct 21, 2016 at 04:17:48PM -0400, Chris Mason wrote:
>>
>>  > > BTRFS warning (device sda3): csum failed ino 130654 off 0 csum 2566472073
>> expected csum 3008371513
>>  > > BTRFS warning (device sda3): csum failed ino 131057 off 4096 csum
>> 3563910319 expected csum 738595262
>>  > > BTRFS warning (device sda3): csum failed ino 131176 off 4096 csum
>> 1344477721 expected csum 441864825
>>  > > BTRFS warning (device sda3): csum failed ino 131241 off 245760 csum
>> 3576232181 expected csum 2566472073
>>  > > BTRFS warning (device sda3): csum failed ino 131429 off 0 csum 1494450239
>> expected csum 2646577722
>>  > > BTRFS warning (device sda3): csum failed ino 131471 off 0 csum 3949539320
>> expected csum 3828807800
>>  > > BTRFS warning (device sda3): csum failed ino 131471 off 4096 csum
>> 3475108475 expected csum 2566472073
>>  > > BTRFS warning (device sda3): csum failed ino 131471 off 958464 csum
>> 142982740 expected csum 2566472073
>>  > > BTRFS warning (device sda3): csum failed ino 131471 off 0 csum 3949539320
>> expected csum 3828807800
>>  > > BTRFS warning (device sda3): csum failed ino 131532 off 270336 csum
>> 3138898528 expected csum 2566472073
>>  > > BTRFS warning (device sda3): csum failed ino 131532 off 1249280 csum
>> 2169165042 expected csum 2566472073
>>  > > BTRFS warning (device sda3): csum failed ino 131649 off 16384 csum
>> 2914965650 expected csum 1425742005
>>  > >
>>  > >
>>  > > A curious thing: the expected csum 2566472073 turns up a number of times
>> for different inodes, and gets
>>  > > differing actual csums each time.  I suppose this could be something like
>> a block of all zeros in multiple files,
>>  > > but it struck me as surprising.
>>  > >
>>  > > btrfs people: is there an easy way to map those inodes to a filename ?
>> I'm betting those are the
>>  > > test files that trinity generates. If so, it might point to a race
>> somewhere.
>>  >
>>  > btrfs inspect inode 130654 mntpoint
>>
>> Interesting, they all return
>>
>> ERROR: ino paths ioctl: No such file or directory
>>
>> So these files got deleted perhaps ?
>>
> Yeah, they must have.
>

So one thing that will cause spurious csum errors is if you do things like 
change the memory while it is in flight during O_DIRECT.  Does trinity do that? 
If so then that would explain it.  If not we should probably dig into it.  Thanks,

Josef