Hi Linus,
Please pull my for-linus branch:
git://git.kernel.org/pub/scm/linux/kernel/git/mason/linux-btrfs.git for-linus
This is against 3.11-rc7, but was pulled and tested against your tree as
of yesterday. We do have two small incrementals queued up, but I wanted
to get this bunch out the door before I hop on an airplane.
This is a fairly large batch of fixes, performance improvements, and
cleanups from the usual Btrfs suspects.
We've included Stefan Behren's work to index subvolume UUIDs, which is
targeted at speeding up send/receive with many subvolumes or snapshots
in place. It closes a long standing performance issue that was built
in to the disk format.
Mark Fasheh's offline dedup work is also here. In this case offline
means the FS is mounted and active, but the dedup work is not done
inline during file IO. This is a building block where utilities are
able to ask the FS to dedup a series of extents. The kernel takes
care of verifying the data involved really is the same. Today this
involves reading both extents, but we'll continue to evolve the patches.
Anand Jain (3):
btrfs: fix get set label blocking against balance
btrfs: use BTRFS_SUPER_INFO_SIZE macro at btrfs_read_dev_super()
btrfs: return btrfs error code for dev excl ops err
Andy Shevchenko (1):
btrfs: reuse kbasename helper
Carey Underwood (1):
Btrfs: Release uuid_mutex for shrink during device delete
Dan Carpenter (1):
btrfs/raid56: fix and cleanup some error paths
Dave Jones (1):
Fix leak in __btrfs_map_block error path
David Sterba (2):
btrfs: make errors in btrfs_num_copies less noisy
btrfs: add mount option to set commit interval
Filipe David Borba Manana (18):
Btrfs: optimize btrfs_lookup_extent_info()
Btrfs: add missing error checks to add_data_references
Btrfs: optimize function btrfs_read_chunk_tree
Btrfs: add missing error check to find_parent_nodes
Btrfs: add missing error handling to read_tree_block
Btrfs: fix inode leak on kmalloc failure in tree-log.c
Btrfs: don't ignore errors from btrfs_run_delayed_items
Btrfs: return ENOSPC when target space is full
Btrfs: add missing error code to BTRFS_IOC_INO_LOOKUP handler
Btrfs: don't miss inode ref items in BTRFS_IOC_INO_LOOKUP
Btrfs: reset force_compress on btrfs_file_defrag failure
Btrfs: fix memory leak of orphan block rsv
Btrfs: fix printing of non NULL terminated string
Btrfs: fix race between removing a dev and writing sbs
Btrfs: fix race conditions in BTRFS_IOC_FS_INFO ioctl
Btrfs: fix memory leak of uuid_root in free_fs_info
Btrfs: fix deadlock in uuid scan kthread
Btrfs: optimize key searches in btrfs_search_slot
Geert Uytterhoeven (12):
Btrfs: Remove superfluous casts from u64 to unsigned long long
Btrfs: Make BTRFS_DEV_REPLACE_DEVID an unsigned long long constant
Btrfs: Format PAGE_SIZE as unsigned long
Btrfs: Format mirror_num as int
Btrfs: Make btrfs_device_uuid() return unsigned long
Btrfs: Make btrfs_device_fsid() return unsigned long
Btrfs: Make btrfs_dev_extent_chunk_tree_uuid() return unsigned long
Btrfs: Make btrfs_header_fsid() return unsigned long
Btrfs: Make btrfs_header_chunk_tree_uuid() return unsigned long
Btrfs: PAGE_CACHE_SIZE is already unsigned long
Btrfs: Do not truncate sector_t on 32-bit with CONFIG_LBDAF=y
Btrfs: Use %z to format size_t
Ilya Dryomov (5):
Btrfs: find_next_devid: root -> fs_info
Btrfs: add btrfs_alloc_device and switch to it
Btrfs: add alloc_fs_devices and switch to it
Btrfs: rollback btrfs_device fields on umount
Btrfs: stop refusing the relocation of chunk 0
Jeff Mahoney (1):
btrfs: fall back to global reservation when removing subvolumes
Josef Bacik (30):
Btrfs: stop using GFP_ATOMIC for the tree mod log allocations
Btrfs: set lockdep class before locking new extent buffer
Btrfs: reset ret in record_one_backref
Btrfs: cleanup reloc roots properly on error
Btrfs: don't bother autodefragging if our root is going away
Btrfs: cleanup arguments to extent_clear_unlock_delalloc
Btrfs: fix what bits we clear when erroring out from delalloc
Btrfs: check to see if we have an inline item properly
Btrfs: change how we queue blocks for backref checking
Btrfs: don't bug_on when we fail when cleaning up transactions
Btrfs: handle errors when doing slow caching
Btrfs: check our parent dir when doing a compare send
Btrfs: deal with enomem in the rewind path
Btrfs: stop using GFP_ATOMIC when allocating rewind ebs
Btrfs: skip subvol entries when checking if we've created a dir already
Btrfs: don't allow a subvol to be deleted if it is the default subovl
Btrfs: fix the error handling wrt orphan items
Btrfs: fix heavy delalloc related deadlock
Btrfs: avoid starting a transaction in the write path
Btrfs: separate out tests into their own directory
Btrfs: fix send issues related to inode number reuse
Btrfs: fix send to deal with sparse files properly
Btrfs: do not clear our orphan item runtime flag on eexist
Btrfs: remove ourselves from the cluster list under lock
Btrfs: adjust the fs_devices->missing count on unmount
Btrfs: add support for asserts
Btrfs: convert all bug_ons in free-space-cache.c
Btrfs: allow partial ordered extent completion
Btrfs: only update disk_i_size as we remove extents
Btrfs: don't use an async starter for most of our workers
Liu Bo (3):
Btrfs: make free space caching faster with many non-inline extent references
Btrfs/tracepoint: update delayed ref tracepoints
Btrfs: allow compressed extents to be merged during defragment
Mark Fasheh (4):
btrfs: abtract out range locking in clone ioctl()
btrfs_ioctl_clone: Move clone code into it's own function
btrfs: Introduce extent_read_full_page_nolock()
btrfs: offline dedupe
Miao Xie (7):
Btrfs, raid56: fix memory leak when allocating pages for p/q stripes failed
Btrfs: remove unnecessary argument of bio_readpage_error()
Btrfs: add branch prediction hints in the read page end IO function
Btrfs: don't cache the csum value into the extent state tree
Btrfs: batch the extent state operation in the end io handle of the read page
Btrfs: batch the extent state operation when reading pages
Btrfs: cache the extent map struct when reading several pages
Mike Frysinger (1):
Btrfs: use __u64 in exported user headers
Qu Wenruo (1):
btrfs: Cleanup for using BTRFS_SETGET_STACK instead of raw convert
Sergei Trofimovich (1):
btrfs: mark some local function as 'static'
Stefan Agner (1):
Btrfs: return -1 when lzo compression makes data bigger
Stefan Behrens (14):
Btrfs: Print key type in decimal everywhere
Btrfs: get rid of sparse warnings
Btrfs: introduce a tree for items that map UUIDs to something
Btrfs: support printing UUID tree elements
Btrfs: create UUID tree if required
Btrfs: maintain subvolume items in the UUID tree
Btrfs: fill UUID tree initially
Btrfs: introduce uuid-tree-gen field
Btrfs: check UUID tree during mount if required
Btrfs: add mount option to force UUID tree checking
Btrfs: don't allow the replace procedure on read only filesystems
Btrfs: get rid of one BUG() in write_all_supers()
Btrfs: fix for patch "cleanup: don't check the same thing twice"
Btrf: cleanup: don't check for root_refs == 0 twice
Wang Shilong (10):
Btrfs: set qgroup_ulist to be null after calling ulist_free()
Btrfs: add sanity checks regarding to parsing mount options
Btrfs: use u64 for subvolid when parsing mount options
Btrfs: add missing mounting options in btrfs_show_options()
Btrfs: fix possible memory leak in find_parent_nodes()
Btrfs: fix oops when writing dirty qgroups to disk
Btrfs: move btrfs_free_qgroup_config() out of spin_lock and fix comments
Btrfs: remove reduplicate check when disabling quota
Btrfs: pass gfp_t to __add_prelim_ref() to avoid always using GFP_ATOMIC
Btrfs: allocate prelim_ref with a slab allocater
chandan (1):
btrfs_read_block_groups: Use enums to index
fs/btrfs/Kconfig | 9 +
fs/btrfs/Makefile | 5 +-
fs/btrfs/backref.c | 93 +++--
fs/btrfs/backref.h | 2 +
fs/btrfs/btrfs_inode.h | 21 ++
fs/btrfs/check-integrity.c | 422 ++++++++-------------
fs/btrfs/compression.c | 11 +-
fs/btrfs/ctree.c | 289 +++++++-------
fs/btrfs/ctree.h | 161 ++++++--
fs/btrfs/delayed-inode.c | 46 +--
fs/btrfs/delayed-ref.c | 8 +-
fs/btrfs/dev-replace.c | 4 +-
fs/btrfs/disk-io.c | 174 ++++++---
fs/btrfs/extent-tree.c | 184 ++++++---
fs/btrfs/extent_io.c | 664 +++++++++++++++++++--------------
fs/btrfs/extent_io.h | 35 +-
fs/btrfs/file-item.c | 85 +++--
fs/btrfs/file.c | 11 +-
fs/btrfs/free-space-cache.c | 525 ++++----------------------
fs/btrfs/free-space-cache.h | 11 +-
fs/btrfs/inode.c | 615 ++++++++++++++----------------
fs/btrfs/ioctl.c | 745 ++++++++++++++++++++++++++++---------
fs/btrfs/lzo.c | 4 +-
fs/btrfs/ordered-data.c | 28 +-
fs/btrfs/ordered-data.h | 7 +
fs/btrfs/print-tree.c | 107 +++---
fs/btrfs/qgroup.c | 69 ++--
fs/btrfs/raid56.c | 14 +-
fs/btrfs/relocation.c | 43 ++-
fs/btrfs/root-tree.c | 21 +-
fs/btrfs/scrub.c | 42 +--
fs/btrfs/send.c | 240 +++++++++---
fs/btrfs/super.c | 145 ++++++--
fs/btrfs/tests/btrfs-tests.h | 34 ++
fs/btrfs/tests/free-space-tests.c | 395 ++++++++++++++++++++
fs/btrfs/transaction.c | 34 +-
fs/btrfs/transaction.h | 2 -
fs/btrfs/tree-log.c | 19 +-
fs/btrfs/uuid-tree.c | 358 ++++++++++++++++++
fs/btrfs/volumes.c | 613 ++++++++++++++++++++++--------
fs/btrfs/volumes.h | 12 +
include/trace/events/btrfs.h | 60 ++-
include/uapi/linux/btrfs.h | 30 +-
43 files changed, 4045 insertions(+), 2352 deletions(-)
On Thu, Sep 12, 2013 at 11:36 AM, Chris Mason <[email protected]> wrote:
> Mark Fasheh (4):
> btrfs: offline dedupe
This commit adds calls to __put_user_unaligned, which causes build
failures on ARM if btrfs is configured:
+ make -s ARCH=arm V=1 -j4 modules
fs/btrfs/ioctl.c: In function 'btrfs_ioctl_file_extent_same':
fs/btrfs/ioctl.c:2802:3: error: implicit declaration of function
'__put_user_unaligned' [-Werror=implicit-function-declaration]
if (__put_user_unaligned(info.status, &args->info[i].status) ||
^
cc1: some warnings being treated as errors
make[2]: *** [fs/btrfs/ioctl.o] Error 1
make[1]: *** [fs/btrfs] Error 2
make[1]: *** Waiting for unfinished jobs....
make: *** [fs] Error 2
make: *** Waiting for unfinished jobs....
josh
On Thu, Sep 12, 2013 at 10:38 PM, Josh Boyer <[email protected]> wrote:
> On Thu, Sep 12, 2013 at 11:36 AM, Chris Mason <[email protected]> wrote:
>> Mark Fasheh (4):
>> btrfs: offline dedupe
>
> This commit adds calls to __put_user_unaligned, which causes build
> failures on ARM if btrfs is configured:
>
> + make -s ARCH=arm V=1 -j4 modules
> fs/btrfs/ioctl.c: In function 'btrfs_ioctl_file_extent_same':
> fs/btrfs/ioctl.c:2802:3: error: implicit declaration of function
> '__put_user_unaligned' [-Werror=implicit-function-declaration]
> if (__put_user_unaligned(info.status, &args->info[i].status) ||
> ^
> cc1: some warnings being treated as errors
> make[2]: *** [fs/btrfs/ioctl.o] Error 1
> make[1]: *** [fs/btrfs] Error 2
> make[1]: *** Waiting for unfinished jobs....
> make: *** [fs] Error 2
> make: *** Waiting for unfinished jobs....
Cfr. my early warning 10 days ago:
"Btrfs is the first user of __put_user_unaligned() outside the compat code,
hence now all 32-bit architectures should make sure to implement this, too."
http://marc.info/?l=linux-arch&m=137820065929216&w=2
Gr{oetje,eeting}s,
Geert
--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- [email protected]
In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
-- Linus Torvalds
On Fri, Sep 13, 2013 at 2:44 AM, Geert Uytterhoeven
<[email protected]> wrote:
> On Thu, Sep 12, 2013 at 10:38 PM, Josh Boyer <[email protected]> wrote:
>> On Thu, Sep 12, 2013 at 11:36 AM, Chris Mason <[email protected]> wrote:
>>> Mark Fasheh (4):
>>> btrfs: offline dedupe
>>
>> This commit adds calls to __put_user_unaligned, which causes build
>> failures on ARM if btrfs is configured:
>>
>> + make -s ARCH=arm V=1 -j4 modules
>> fs/btrfs/ioctl.c: In function 'btrfs_ioctl_file_extent_same':
>> fs/btrfs/ioctl.c:2802:3: error: implicit declaration of function
>> '__put_user_unaligned' [-Werror=implicit-function-declaration]
>> if (__put_user_unaligned(info.status, &args->info[i].status) ||
>> ^
>> cc1: some warnings being treated as errors
>> make[2]: *** [fs/btrfs/ioctl.o] Error 1
>> make[1]: *** [fs/btrfs] Error 2
>> make[1]: *** Waiting for unfinished jobs....
>> make: *** [fs] Error 2
>> make: *** Waiting for unfinished jobs....
>
> Cfr. my early warning 10 days ago:
>
> "Btrfs is the first user of __put_user_unaligned() outside the compat code,
> hence now all 32-bit architectures should make sure to implement this, too."
>
> http://marc.info/?l=linux-arch&m=137820065929216&w=2
Indeed. I missed that as it was an m68k patch.
I'm not an ARM expert, so I don't know if ARM should use the
asm-generic implementations, or just use __get_user/__put_user in all
cases. I've CC'd rmk.
josh
On Fri, Sep 13, 2013 at 07:53:21AM -0400, Josh Boyer wrote:
> I'm not an ARM expert, so I don't know if ARM should use the
> asm-generic implementations, or just use __get_user/__put_user in all
> cases. I've CC'd rmk.
Why do we have uaccess-unaligned.h ? Normally, these kinds of things
are spawned by architectures which have problems with unaligned accesses,
ARM being one of them, but afaik we've never need this.
With the kernel-side trapping of unaligned accesses on older hardware,
we've always dealt with the normal accessor faulting.
>From what I can tell in the git history, these unaligned put_user and
get_user have existed all the way back to the dawn of git use.
Can someone enlighten me why we have them?
--
Russell King
On Fri, Sep 13, 2013 at 2:15 PM, Russell King <[email protected]> wrote:
> On Fri, Sep 13, 2013 at 07:53:21AM -0400, Josh Boyer wrote:
>> I'm not an ARM expert, so I don't know if ARM should use the
>> asm-generic implementations, or just use __get_user/__put_user in all
>> cases. I've CC'd rmk.
>
> Why do we have uaccess-unaligned.h ? Normally, these kinds of things
> are spawned by architectures which have problems with unaligned accesses,
> ARM being one of them, but afaik we've never need this.
>
> With the kernel-side trapping of unaligned accesses on older hardware,
> we've always dealt with the normal accessor faulting.
>
> From what I can tell in the git history, these unaligned put_user and
> get_user have existed all the way back to the dawn of git use.
>
> Can someone enlighten me why we have them?
You removed the answer when trimming the quoted part:
| "Btrfs is the first user of __put_user_unaligned() outside the compat code,
__put_user_unaligned() is used in fs/compat.c, presumably because
alignment restrictions may differ between 32- and 64-bit versions of the
same CPU family.
No one seems to actully use __get_user_unaligned().
Gr{oetje,eeting}s,
Geert
--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- [email protected]
In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
-- Linus Torvalds
On 09/12/2013 11:36 AM, Chris Mason wrote:
> Mark Fasheh's offline dedup work is also here. In this case offline
> means the FS is mounted and active, but the dedup work is not done
> inline during file IO. This is a building block where utilities are
> able to ask the FS to dedup a series of extents. The kernel takes
> care of verifying the data involved really is the same. Today this
> involves reading both extents, but we'll continue to evolve the patches.
Nice feature!
Just a note, the "offline" label is really confusing. In other storage products,
they typically call this "out of band" since you are online but not during the
actual write in a synchronous way :)
Ric
On Fri, Sep 13, 2013 at 09:07:36AM -0400, Ric Wheeler wrote:
> On 09/12/2013 11:36 AM, Chris Mason wrote:
> >Mark Fasheh's offline dedup work is also here. In this case offline
> >means the FS is mounted and active, but the dedup work is not done
> >inline during file IO. This is a building block where utilities are
> >able to ask the FS to dedup a series of extents. The kernel takes
> >care of verifying the data involved really is the same. Today this
> >involves reading both extents, but we'll continue to evolve the patches.
>
> Nice feature!
>
> Just a note, the "offline" label is really confusing. In other
> storage products, they typically call this "out of band" since you
> are online but not during the actual write in a synchronous way :)
I knew there was a specific term for this, but couldn't remember
what it was. I've now updated the btrfs website's description(s) of
the feature to include "out-of-band" and "in-band".
Hugo.
--
=== Hugo Mills: hugo@... carfax.org.uk | darksatanic.net | lug.org.uk ===
PGP key: 65E74AC0 from wwwkeys.eu.pgp.net or http://www.carfax.org.uk
--- Once is happenstance; twice is coincidence; three times ---
is enemy action.
On Fri, Sep 13, 2013 at 8:15 AM, Russell King <[email protected]> wrote:
> On Fri, Sep 13, 2013 at 07:53:21AM -0400, Josh Boyer wrote:
>> I'm not an ARM expert, so I don't know if ARM should use the
>> asm-generic implementations, or just use __get_user/__put_user in all
>> cases. I've CC'd rmk.
>
> Why do we have uaccess-unaligned.h ? Normally, these kinds of things
> are spawned by architectures which have problems with unaligned accesses,
> ARM being one of them, but afaik we've never need this.
>
> With the kernel-side trapping of unaligned accesses on older hardware,
> we've always dealt with the normal accessor faulting.
>
> From what I can tell in the git history, these unaligned put_user and
> get_user have existed all the way back to the dawn of git use.
>
> Can someone enlighten me why we have them?
So while that gets sorted out, would it be safe to just do as Geert
did on m68k and put:
#define __put_user_unaligned(x, ptr) __put_user((x), (ptr))
in arch/arm/include/asm/uaccess.h, and let the normal accessors and
kernel-side trapping deal with things? I'm thinking that's a local
fix until something gets sorted upstream, but I don't want to do it if
it's going to break things.
josh
On Fri, Sep 13, 2013 at 11:06 AM, Josh Boyer <[email protected]> wrote:
> On Fri, Sep 13, 2013 at 8:15 AM, Russell King <[email protected]> wrote:
>> On Fri, Sep 13, 2013 at 07:53:21AM -0400, Josh Boyer wrote:
>>> I'm not an ARM expert, so I don't know if ARM should use the
>>> asm-generic implementations, or just use __get_user/__put_user in all
>>> cases. I've CC'd rmk.
>>
>> Why do we have uaccess-unaligned.h ? Normally, these kinds of things
>> are spawned by architectures which have problems with unaligned accesses,
>> ARM being one of them, but afaik we've never need this.
>>
>> With the kernel-side trapping of unaligned accesses on older hardware,
>> we've always dealt with the normal accessor faulting.
>>
>> From what I can tell in the git history, these unaligned put_user and
>> get_user have existed all the way back to the dawn of git use.
>>
>> Can someone enlighten me why we have them?
I somehow fail at email and dropped Russell from CC on accident. Sigh.
> So while that gets sorted out, would it be safe to just do as Geert
> did on m68k and put:
>
> #define __put_user_unaligned(x, ptr) __put_user((x), (ptr))
>
> in arch/arm/include/asm/uaccess.h, and let the normal accessors and
> kernel-side trapping deal with things? I'm thinking that's a local
> fix until something gets sorted upstream, but I don't want to do it if
> it's going to break things.
>
> josh
On Fri, Sep 13, 2013 at 11:38:15AM -0400, Josh Boyer wrote:
> On Fri, Sep 13, 2013 at 11:06 AM, Josh Boyer <[email protected]> wrote:
> > On Fri, Sep 13, 2013 at 8:15 AM, Russell King <[email protected]> wrote:
> >> On Fri, Sep 13, 2013 at 07:53:21AM -0400, Josh Boyer wrote:
> >>> I'm not an ARM expert, so I don't know if ARM should use the
> >>> asm-generic implementations, or just use __get_user/__put_user in all
> >>> cases. I've CC'd rmk.
> >>
> >> Why do we have uaccess-unaligned.h ? Normally, these kinds of things
> >> are spawned by architectures which have problems with unaligned accesses,
> >> ARM being one of them, but afaik we've never need this.
> >>
> >> With the kernel-side trapping of unaligned accesses on older hardware,
> >> we've always dealt with the normal accessor faulting.
> >>
> >> From what I can tell in the git history, these unaligned put_user and
> >> get_user have existed all the way back to the dawn of git use.
> >>
> >> Can someone enlighten me why we have them?
>
> I somehow fail at email and dropped Russell from CC on accident. Sigh.
You're not the first to do that recently. I'm beginning to think it's
something someone has written into email clients to make them do in order
to piss me off. I mean, it's _hard_ to do - you have to manually edit the
recipients list to just drop one person.
> > So while that gets sorted out, would it be safe to just do as Geert
> > did on m68k and put:
> >
> > #define __put_user_unaligned(x, ptr) __put_user((x), (ptr))
> >
> > in arch/arm/include/asm/uaccess.h, and let the normal accessors and
> > kernel-side trapping deal with things? I'm thinking that's a local
> > fix until something gets sorted upstream, but I don't want to do it if
> > it's going to break things.
Yep, that should work just fine.
--
Russell King
ARM architecture Linux Kernel maintainer
On Fri, Sep 13, 2013 at 04:58:36PM +0100, Russell King wrote:
> On Fri, Sep 13, 2013 at 11:38:15AM -0400, Josh Boyer wrote:
> > On Fri, Sep 13, 2013 at 11:06 AM, Josh Boyer <[email protected]> wrote:
> > > On Fri, Sep 13, 2013 at 8:15 AM, Russell King <[email protected]> wrote:
> > >> On Fri, Sep 13, 2013 at 07:53:21AM -0400, Josh Boyer wrote:
> > >>> I'm not an ARM expert, so I don't know if ARM should use the
> > >>> asm-generic implementations, or just use __get_user/__put_user in all
> > >>> cases. I've CC'd rmk.
> > >>
> > >> Why do we have uaccess-unaligned.h ? Normally, these kinds of things
> > >> are spawned by architectures which have problems with unaligned accesses,
> > >> ARM being one of them, but afaik we've never need this.
> > >>
> > >> With the kernel-side trapping of unaligned accesses on older hardware,
> > >> we've always dealt with the normal accessor faulting.
> > >>
> > >> From what I can tell in the git history, these unaligned put_user and
> > >> get_user have existed all the way back to the dawn of git use.
> > >>
> > >> Can someone enlighten me why we have them?
> >
> > I somehow fail at email and dropped Russell from CC on accident. Sigh.
>
> You're not the first to do that recently. I'm beginning to think it's
> something someone has written into email clients to make them do in order
> to piss me off. I mean, it's _hard_ to do - you have to manually edit the
> recipients list to just drop one person.
You configured your mail client to generate a "Mail-Followup-To:" header
field which actively asks other mail clients to remove you from replies.
So you only get what you ask for... ;)
I think the mutt "metoo" variable will change that, but I don't know
for sure.