Got a few of these in the logs this morning after an overnight rsync over nfs
to an exported btrfs volume.
Dave
==================================================================
BUG: KASAN: stack-out-of-bounds in setup_cluster_bitmap+0xc4/0x5a0 at addr ffff88039bef6828
Read of size 8 by task nfsd/1009
page:ffffea000e6fbd80 count:0 mapcount:0 mapping: (null) index:0x0
flags: 0x8000000000000000()
page dumped because: kasan: bad access detected
CPU: 1 PID: 1009 Comm: nfsd Tainted: G W 4.4.0-rc3-backup-debug+ #1
ffff880065647b50 000000006bb712c2 ffff88039bef6640 ffffffffa680a43e
0000004559c00000 ffff88039bef66c8 ffffffffa62638d1 ffffffffa61121c0
ffff8803a5769de8 0000000000000296 ffff8803a5769df0 0000000000046280
Call Trace:
[<ffffffffa680a43e>] dump_stack+0x4b/0x6d
[<ffffffffa62638d1>] kasan_report_error+0x501/0x520
[<ffffffffa61121c0>] ? debug_show_all_locks+0x1e0/0x1e0
[<ffffffffa6263948>] kasan_report+0x58/0x60
[<ffffffffa6814b00>] ? rb_last+0x10/0x40
[<ffffffffa66f8af4>] ? setup_cluster_bitmap+0xc4/0x5a0
[<ffffffffa6262ead>] __asan_load8+0x5d/0x70
[<ffffffffa66f8af4>] setup_cluster_bitmap+0xc4/0x5a0
[<ffffffffa66f675a>] ? setup_cluster_no_bitmap+0x6a/0x400
[<ffffffffa66fcd16>] btrfs_find_space_cluster+0x4b6/0x640
[<ffffffffa66fc860>] ? btrfs_alloc_from_cluster+0x4e0/0x4e0
[<ffffffffa66fc36e>] ? btrfs_return_cluster_to_free_space+0x9e/0xb0
[<ffffffffa702dc37>] ? _raw_spin_unlock+0x27/0x40
[<ffffffffa666a1a1>] find_free_extent+0xba1/0x1520
[<ffffffffa6669600>] ? btrfs_delalloc_reserve_space+0x70/0x70
[<ffffffffa6119276>] ? do_raw_spin_lock+0x116/0x1a0
[<ffffffffa6119407>] ? do_raw_spin_unlock+0x97/0x130
[<ffffffffa702dc37>] ? _raw_spin_unlock+0x27/0x40
[<ffffffffa6651555>] ? get_alloc_profile+0x1c5/0x320
[<ffffffffa666ab90>] ? btrfs_reserve_extent+0x70/0x1d0
[<ffffffffa666abe0>] btrfs_reserve_extent+0xc0/0x1d0
[<ffffffffa666b0af>] btrfs_alloc_tree_block+0x3bf/0x680
[<ffffffffa61121c0>] ? debug_show_all_locks+0x1e0/0x1e0
[<ffffffffa666acf0>] ? btrfs_reserve_extent+0x1d0/0x1d0
[<ffffffffa62633b6>] ? memcpy+0x36/0x40
[<ffffffffa66c3337>] ? read_extent_buffer+0xe7/0x160
[<ffffffffa6642c0f>] __btrfs_cow_block+0x28f/0x9b0
[<ffffffffa6208a28>] ? mark_page_accessed+0x18/0xd0
[<ffffffffa6642980>] ? update_ref_for_cow+0x540/0x540
[<ffffffffa6133335>] ? debug_lockdep_rcu_enabled+0x35/0x40
[<ffffffffa66e96af>] ? btrfs_try_tree_write_lock+0x5f/0xe0
[<ffffffffa66e90f0>] ? btrfs_set_lock_blocking_rw+0x110/0x160
[<ffffffffa66435cf>] btrfs_cow_block+0x1cf/0x380
[<ffffffffa6649773>] btrfs_search_slot+0x413/0x11e0
[<ffffffffa6649360>] ? split_leaf+0xc50/0xc50
[<ffffffffa6641686>] ? btrfs_alloc_path+0x26/0x30
[<ffffffffa625bba3>] ? set_track+0x83/0x140
[<ffffffffa610f66d>] ? mark_lock+0x6d/0x8a0
[<ffffffffa6671cea>] btrfs_lookup_csum+0xba/0x260
[<ffffffffa610d244>] ? __lock_is_held+0x84/0xc0
[<ffffffffa6671c30>] ? truncate_one_csum+0x1c0/0x1c0
[<ffffffffa613325a>] ? rcu_read_lock_sched_held+0x8a/0xa0
[<ffffffffa625fbc3>] ? kmem_cache_alloc+0x1c3/0x280
[<ffffffffa6673f8d>] btrfs_csum_file_blocks+0x2bd/0xac0
[<ffffffffa6673cd0>] ? btrfs_del_csums+0x490/0x490
[<ffffffffa6260b87>] ? kfree+0xb7/0x230
[<ffffffffa676aa5a>] ? copy_items+0x6ab/0xd2d
[<ffffffffa676aa5a>] ? copy_items+0x6ab/0xd2d
[<ffffffffa676aa89>] copy_items+0x6da/0xd2d
[<ffffffffa66e9001>] ? btrfs_set_lock_blocking_rw+0x21/0x160
[<ffffffffa676a3af>] ? assfail.constprop.22+0x1e/0x1e
[<ffffffffa664ec61>] ? btrfs_search_forward+0x541/0x600
[<ffffffffa66c3337>] ? read_extent_buffer+0xe7/0x160
[<ffffffffa66ec627>] ? btrfs_item_key_to_cpu+0xb7/0xf0
[<ffffffffa66ec570>] ? check_parent_dirs_for_sync+0x200/0x200
[<ffffffffa676c6e0>] btrfs_log_inode+0x7a9/0x11fa
[<ffffffffa676bf37>] ? btrfs_log_changed_extents+0x883/0x883
[<ffffffffa610f66d>] ? mark_lock+0x6d/0x8a0
[<ffffffffa610ff2e>] ? mark_held_locks+0x8e/0xc0
[<ffffffffa7027f95>] ? mutex_lock_nested+0x3a5/0x510
[<ffffffffa61100f2>] ? trace_hardirqs_on_caller+0x192/0x290
[<ffffffffa610f66d>] ? mark_lock+0x6d/0x8a0
[<ffffffffa6133335>] ? debug_lockdep_rcu_enabled+0x35/0x40
[<ffffffffa610ff2e>] ? mark_held_locks+0x8e/0xc0
[<ffffffffa7027a00>] ? __mutex_unlock_slowpath+0xe0/0x1c0
[<ffffffffa61100f2>] ? trace_hardirqs_on_caller+0x192/0x290
[<ffffffffa61101fd>] ? trace_hardirqs_on+0xd/0x10
[<ffffffffa66f1cf4>] btrfs_log_inode_parent+0x404/0x1440
[<ffffffffa61121c0>] ? debug_show_all_locks+0x1e0/0x1e0
[<ffffffffa61121c0>] ? debug_show_all_locks+0x1e0/0x1e0
[<ffffffffa66f18f0>] ? btrfs_end_log_trans+0x50/0x50
[<ffffffffa6133335>] ? debug_lockdep_rcu_enabled+0x35/0x40
[<ffffffffa612e93a>] ? debug_lockdep_rcu_enabled.part.36+0x1a/0x30
[<ffffffffa6133335>] ? debug_lockdep_rcu_enabled+0x35/0x40
[<ffffffffa6290aae>] ? dget_parent+0x8e/0x2f0
[<ffffffffa6290ade>] ? dget_parent+0xbe/0x2f0
[<ffffffffa66f46aa>] btrfs_log_dentry_safe+0x6a/0x90
[<ffffffffa66aca5f>] btrfs_sync_file+0x4df/0x690
[<ffffffffa66ac580>] ? start_ordered_ops+0x30/0x30
[<ffffffffa62d4830>] ? __fsnotify_update_child_dentry_flags+0x30/0x30
[<ffffffffa62bdc3d>] vfs_fsync_range+0x5d/0x120
[<ffffffffa66ac580>] ? start_ordered_ops+0x30/0x30
[<ffffffffa64ae7c6>] nfsd_vfs_write+0x356/0x650
[<ffffffffa64ae470>] ? nfsd_readv+0xa0/0xa0
[<ffffffffa6133335>] ? debug_lockdep_rcu_enabled+0x35/0x40
[<ffffffffa64b230f>] nfsd_write+0xff/0x120
[<ffffffffa6839e34>] ? __list_add+0x74/0xf0
[<ffffffffa64bb4f7>] nfsd3_proc_write+0x1c7/0x2d0
[<ffffffffa64b7bdf>] ? nfsd_cache_lookup+0x6ef/0xa90
[<ffffffffa64bb330>] ? nfsd3_proc_symlink+0x1f0/0x1f0
[<ffffffffa64a4b15>] nfsd_dispatch+0x185/0x370
[<ffffffffa64bb330>] ? nfsd3_proc_symlink+0x1f0/0x1f0
[<ffffffffa6fecd96>] svc_process_common+0x8c6/0xda0
[<ffffffffa64a4990>] ? nfsd_svc+0x770/0x770
[<ffffffffa6fec4d0>] ? svc_printk+0x180/0x180
[<ffffffffa610d1e5>] ? __lock_is_held+0x25/0xc0
[<ffffffffa6feefbb>] svc_process+0x22b/0x450
[<ffffffffa64a3cfc>] nfsd+0x23c/0x370
[<ffffffffa64a3ac5>] ? nfsd+0x5/0x370
[<ffffffffa64a3ac0>] ? nfsd_destroy+0x1f0/0x1f0
[<ffffffffa60ce496>] kthread+0x196/0x1c0
[<ffffffffa60ce300>] ? __kthread_parkme+0xe0/0xe0
[<ffffffffa610fec3>] ? mark_held_locks+0x23/0xc0
[<ffffffffa60ce300>] ? __kthread_parkme+0xe0/0xe0
[<ffffffffa702e82f>] ret_from_fork+0x3f/0x70
[<ffffffffa60ce300>] ? __kthread_parkme+0xe0/0xe0
Memory state around the buggy address:
ffff88039bef6700: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
ffff88039bef6780: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>ffff88039bef6800: 00 00 00 00 00 f1 f1 f1 f1 00 00 f4 f4 f3 f3 f3
^
ffff88039bef6880: f3 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
ffff88039bef6900: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
On 12/02/2015 09:59 AM, Dave Jones wrote:
> Got a few of these in the logs this morning after an overnight rsync over nfs
> to an exported btrfs volume.
That's probably us and not NFS, what line is that in
setup_cluster_bitmap? Thanks,
Josef
On Wed, Dec 02, 2015 at 10:11:28AM -0500, Josef Bacik wrote:
> On 12/02/2015 09:59 AM, Dave Jones wrote:
> > Got a few of these in the logs this morning after an overnight rsync over nfs
> > to an exported btrfs volume.
>
> That's probably us and not NFS, what line is that in
> setup_cluster_bitmap? Thanks,
If my math is correct, it's this..
if (entry->offset != bitmap_offset)
I don't seem to be able to trigger it on demand unfortunatly.
Dave
On Wed, Dec 02, 2015 at 11:09:43AM -0500, Dave Jones wrote:
> On Wed, Dec 02, 2015 at 10:11:28AM -0500, Josef Bacik wrote:
> > On 12/02/2015 09:59 AM, Dave Jones wrote:
> > > Got a few of these in the logs this morning after an overnight rsync over nfs
> > > to an exported btrfs volume.
> >
> > That's probably us and not NFS, what line is that in
> > setup_cluster_bitmap? Thanks,
>
> If my math is correct, it's this..
>
> if (entry->offset != bitmap_offset)
>
> I don't seem to be able to trigger it on demand unfortunatly.
Is it possible we're blowing the stack? It seems pretty tricky to get a
stack out of bounds out of this code without flat out blowing through
it.
-chris
On Wed, Dec 02, 2015 at 12:14:56PM -0500, Chris Mason wrote:
> On Wed, Dec 02, 2015 at 11:09:43AM -0500, Dave Jones wrote:
> > On Wed, Dec 02, 2015 at 10:11:28AM -0500, Josef Bacik wrote:
> > > On 12/02/2015 09:59 AM, Dave Jones wrote:
> > > > Got a few of these in the logs this morning after an overnight rsync over nfs
> > > > to an exported btrfs volume.
> > >
> > > That's probably us and not NFS, what line is that in
> > > setup_cluster_bitmap? Thanks,
> >
> > If my math is correct, it's this..
> >
> > if (entry->offset != bitmap_offset)
> >
> > I don't seem to be able to trigger it on demand unfortunatly.
>
> Is it possible we're blowing the stack? It seems pretty tricky to get a
> stack out of bounds out of this code without flat out blowing through
> it.
Hm, there is a lot of debug crap on the stack from lockdep etc, though I didn't
get any warnings from the other stack overflow checks.
Dave
2015-12-02 20:14 GMT+03:00 Chris Mason <[email protected]>:
> On Wed, Dec 02, 2015 at 11:09:43AM -0500, Dave Jones wrote:
>> On Wed, Dec 02, 2015 at 10:11:28AM -0500, Josef Bacik wrote:
>> > On 12/02/2015 09:59 AM, Dave Jones wrote:
>> > > Got a few of these in the logs this morning after an overnight rsync over nfs
>> > > to an exported btrfs volume.
>> >
>> > That's probably us and not NFS, what line is that in
>> > setup_cluster_bitmap? Thanks,
>>
>> If my math is correct, it's this..
>>
>> if (entry->offset != bitmap_offset)
>>
>> I don't seem to be able to trigger it on demand unfortunatly.
>
> Is it possible we're blowing the stack? It seems pretty tricky to get a
> stack out of bounds out of this code without flat out blowing through
> it.
>
I think it just empty bitmaps list.
list_first_entry() can't be used on empty list.
BTW, there is similar report
http://lkml.kernel.org/r/<trinity-c7a088d8-bb35-484e-bf27-dbd9a94a804c-1448959367092@3capp-webde-bs56>
On Wed, Dec 02, 2015 at 09:32:34PM +0300, Andrey Ryabinin wrote:
> 2015-12-02 20:14 GMT+03:00 Chris Mason <[email protected]>:
> > On Wed, Dec 02, 2015 at 11:09:43AM -0500, Dave Jones wrote:
> >> On Wed, Dec 02, 2015 at 10:11:28AM -0500, Josef Bacik wrote:
> >> > On 12/02/2015 09:59 AM, Dave Jones wrote:
> >> > > Got a few of these in the logs this morning after an overnight rsync over nfs
> >> > > to an exported btrfs volume.
> >> >
> >> > That's probably us and not NFS, what line is that in
> >> > setup_cluster_bitmap? Thanks,
> >>
> >> If my math is correct, it's this..
> >>
> >> if (entry->offset != bitmap_offset)
> >>
> >> I don't seem to be able to trigger it on demand unfortunatly.
> >
> > Is it possible we're blowing the stack? It seems pretty tricky to get a
> > stack out of bounds out of this code without flat out blowing through
> > it.
> >
>
> I think it just empty bitmaps list.
> list_first_entry() can't be used on empty list.
Ohh, I was so busy looking for free'd entries I missed that. Good
point.
-chris