Hi,
there's a usual mix of performance improvements and new features. The
core change is reworking how checksums are processed, with followup
cleanups and simplifications. There are two minor changes in block layer
and iomap code.
Please pull, thanks.
Features:
- block group allocation class heuristics
- pack files by size (up to 128k, up to 8M, more) to avoid
fragmentation in block groups, assuming that file size and life time
is correlated, in particular this may help during balance
- with tracepoints and extensible in the future
Performance:
- send: cache directory utimes and only emit the command when necessary
- speedup up to 10x
- smaller final stream produced (no redundant utimes commands issued),
- compatibility not affected
- fiemap: skip backref checks for shared leaves
- speedup 3x on sample filesystem with all leaves shared (e.g. on
snapshots)
- micro optimized b-tree key lookup, speedup in metadata operations
(sample benchmark: fs_mark +10% of files/sec)
Core changes:
- change where checksumming is done in the io path
- checksum and read repair does verification at lower layer
- cascaded cleanups and simplifications
- raid56 refactoring and cleanups
Fixes:
- sysfs: make sure that a run-time change of a feature is correctly
tracked by the feature files
- scrub: better reporting of tree block errors
Other:
- locally enable -Wmaybe-uninitialized after fixing all warnings
- misc cleanups, spelling fixes
Other code:
- block: export bio_split_rw
- iomap: remove IOMAP_F_ZONE_APPEND
----------------------------------------------------------------
The following changes since commit ceaa837f96adb69c0df0397937cd74991d5d821a:
Linux 6.2-rc8 (2023-02-12 14:10:17 -0800)
are available in the Git repository at:
git://git.kernel.org/pub/scm/linux/kernel/git/kdave/linux.git for-6.3-tag
for you to fetch changes up to 964a54e5e1a0d70cd80bd5a0885a1938463625b1:
btrfs: make kobj_type structures constant (2023-02-15 19:38:55 +0100)
----------------------------------------------------------------
Boris Burkov (6):
btrfs: pass find_free_extent_ctl to allocator tracepoints
btrfs: add more find_free_extent tracepoints
btrfs: introduce size class to block group allocator
btrfs: load block group size class when caching
btrfs: don't use size classes for zoned file systems
btrfs: hold block group refcount during async discard
Christoph Hellwig (48):
btrfs: remove the wait argument to btrfs_start_ordered_extent
block: export bio_split_rw
btrfs: better document struct btrfs_bio
btrfs: add a btrfs_inode pointer to struct btrfs_bio
btrfs: remove the direct I/O read checksum lookup optimization
btrfs: simplify parameters of btrfs_lookup_bio_sums
btrfs: refactor error handling in btrfs_submit_bio
btrfs: save the bio iter for checksum validation in common code
btrfs: pre-load data checksum for reads in btrfs_submit_bio
btrfs: add a btrfs_data_csum_ok helper
btrfs: handle checksum validation and repair at the storage layer
btrfs: open code btrfs_bio_free_csum
btrfs: remove btrfs_bio_for_each_sector
btrfs: remove now unused checksumming helpers
btrfs: remove struct btrfs_bio::device field
btrfs: remove the io_failure_record infrastructure
btrfs: rename btrfs_bio::iter field
btrfs: remove struct btrfs_bio::is_metadata flag
btrfs: open code the submit_bio_start helpers
btrfs: simplify the btrfs_csum_one_bio calling convention
btrfs: handle checksum generation in the storage layer
btrfs: handle recording of zoned writes in the storage layer
btrfs: support cloned bios in btree_csum_one_bio
btrfs: allow btrfs_submit_bio to split bios
btrfs: pass the iomap bio to btrfs_submit_bio
btrfs: remove struct btrfs_io_geometry
btrfs: open code submit_encoded_read_bio
btrfs: remove the fs_info argument to btrfs_submit_bio
btrfs: remove now spurious bio submission helpers
btrfs: calculate file system wide queue limit for zoned mode
btrfs: split zone append bios in btrfs_submit_bio
iomap: remove IOMAP_F_ZONE_APPEND
btrfs: raid56: simplify error handling and code flow in raid56_parity_write
btrfs: raid56: simplify code flow in rmw_rbio
btrfs: raid56: wait for I/O completion in submit_read_bios
btrfs: raid56: add a bio_list_put helper
btrfs: raid56: fold recover_assemble_read_bios into recover_rbio
btrfs: raid56: fold rmw_read_wait_recover into rmw_read_bios
btrfs: raid56: submit the read bios from scrub_assemble_read_bios
btrfs: raid56: handle endio in rmw_rbio
btrfs: raid56: handle endio in recover_rbio
btrfs: raid56: handle endio in scrub_rbio
btrfs: use file_offset to limit bios size in calc_bio_boundaries
btrfs: set bbio->file_offset in alloc_new_bio
btrfs: pass a btrfs_bio to btrfs_use_append
btrfs: never return true for reads in btrfs_use_zone_append
btrfs: don't rely on unchanging ->bi_bdev for zone append remaps
btrfs: remove the bdev argument to btrfs_rmap_block
Colin Ian King (1):
btrfs: fix spelling mistakes found using codespell
Filipe Manana (24):
btrfs: use a negative value for BTRFS_LOG_FORCE_COMMIT
btrfs: use a single variable to track return value for log_dir_items()
btrfs: send: directly return from did_overwrite_ref() and simplify it
btrfs: send: avoid unnecessary generation search at did_overwrite_ref()
btrfs: send: directly return from will_overwrite_ref() and simplify it
btrfs: send: avoid extra b+tree searches when checking reference overrides
btrfs: send: remove send_progress argument from can_rmdir()
btrfs: send: avoid duplicated orphan dir allocation and initialization
btrfs: send: avoid unnecessary orphan dir rbtree search at can_rmdir()
btrfs: send: reduce searches on parent root when checking if dir can be removed
btrfs: send: iterate waiting dir move rbtree only once when processing refs
btrfs: send: initialize all the red black trees earlier
btrfs: send: genericize the backref cache to allow it to be reused
btrfs: adapt lru cache to allow for 64 bits keys on 32 bits systems
btrfs: send: cache information about created directories
btrfs: allow a generation number to be associated with lru cache entries
btrfs: add an api to delete a specific entry from the lru cache
btrfs: send: use the lru cache to implement the name cache
btrfs: send: update size of roots array for backref cache entries
btrfs: send: cache utimes operations for directories if possible
btrfs: assert commit root semaphore is held when accessing backref cache
btrfs: skip backref walking during fiemap if we know the leaf is shared
btrfs: eliminate extra call when doing binary search on extent buffer
btrfs: do unsigned integer division in the extent buffer binary search loop
Johannes Thumshirn (4):
btrfs: drop unused trans parameter of drop_delayed_ref
btrfs: remove trans parameter of merge_ref
btrfs: drop trans parameter of insert_delayed_ref
btrfs: directly pass in fs_info to btrfs_merge_delayed_refs
Josef Bacik (15):
btrfs: move btrfs_abort_transaction to transaction.c
btrfs: fix uninitialized variable warning in btrfs_cleanup_ordered_extents
btrfs: fix uninitialized variable warning in get_inode_gen
btrfs: fix uninitialized variable warning in btrfs_update_block_group
btrfs: fix uninitialized variable warnings in __set_extent_bit and convert_extent_bit
btrfs: fix uninitialized variable warning in btrfs_sb_log_location
btrfs: fix uninitialized variable warning in run_one_async_start
btrfs: turn on -Wmaybe-uninitialized
btrfs: always lock the block before calling btrfs_clean_tree_block
btrfs: add trans argument to btrfs_clean_tree_block
btrfs: replace clearing extent buffer dirty bit with btrfs_clean_block
btrfs: do not increment dirty_metadata_bytes in set_btree_ioerr
btrfs: rename btrfs_clean_tree_block to btrfs_clear_buffer_dirty
btrfs: combine btrfs_clear_buffer_dirty and clear_extent_buffer_dirty
btrfs: replace btrfs_wait_tree_block_writeback by wait_on_extent_buffer_writeback
Naohiro Aota (1):
btrfs: zoned: fix uninitialized variable warning in btrfs_get_dev_zones
Peng Hao (1):
btrfs: go to matching label when cleaning em in btrfs_submit_direct
Qu Wenruo (6):
btrfs: scrub: improve tree block error reporting
btrfs: sysfs: update fs features directory asynchronously
btrfs: raid56: reduce overhead to calculate the bio length
btrfs: remove stripe boundary calculation for buffered I/O
btrfs: remove stripe boundary calculation for compressed I/O
btrfs: remove stripe boundary calculation for encoded I/O
Thomas Weißschuh (1):
btrfs: make kobj_type structures constant
Yushan Zhou (1):
btrfs: use PAGE_{ALIGN, ALIGNED, ALIGN_DOWN} macro
ye xingchen (1):
btrfs: remove duplicate include header in extent-tree.c
block/blk-merge.c | 3 +-
fs/btrfs/Makefile | 6 +-
fs/btrfs/backref.c | 33 +-
fs/btrfs/bio.c | 557 +++++++++++++++++++++++++++----
fs/btrfs/bio.h | 67 +---
fs/btrfs/block-group.c | 273 +++++++++++++--
fs/btrfs/block-group.h | 24 +-
fs/btrfs/btrfs_inode.h | 22 +-
fs/btrfs/compression.c | 276 +++------------
fs/btrfs/compression.h | 3 -
fs/btrfs/ctree.c | 62 ++--
fs/btrfs/ctree.h | 15 +
fs/btrfs/defrag.c | 4 +-
fs/btrfs/delayed-ref.c | 24 +-
fs/btrfs/delayed-ref.h | 2 +-
fs/btrfs/discard.c | 41 ++-
fs/btrfs/disk-io.c | 225 ++-----------
fs/btrfs/disk-io.h | 14 +-
fs/btrfs/extent-io-tree.c | 10 +-
fs/btrfs/extent-io-tree.h | 1 -
fs/btrfs/extent-tree.c | 181 +++-------
fs/btrfs/extent-tree.h | 81 +++++
fs/btrfs/extent_io.c | 582 ++++----------------------------
fs/btrfs/extent_io.h | 36 +-
fs/btrfs/file-item.c | 72 ++--
fs/btrfs/file-item.h | 8 +-
fs/btrfs/file.c | 2 +-
fs/btrfs/free-space-tree.c | 2 +-
fs/btrfs/fs.c | 4 +
fs/btrfs/fs.h | 11 +-
fs/btrfs/inode.c | 641 +++++------------------------------
fs/btrfs/ioctl.c | 2 +-
fs/btrfs/lru_cache.c | 166 +++++++++
fs/btrfs/lru_cache.h | 80 +++++
fs/btrfs/lzo.c | 2 +-
fs/btrfs/messages.c | 30 --
fs/btrfs/messages.h | 34 --
fs/btrfs/ordered-data.c | 25 +-
fs/btrfs/ordered-data.h | 3 +-
fs/btrfs/qgroup.c | 2 +-
fs/btrfs/raid56.c | 334 +++++++------------
fs/btrfs/raid56.h | 4 +-
fs/btrfs/relocation.c | 2 +-
fs/btrfs/scrub.c | 51 ++-
fs/btrfs/send.c | 684 ++++++++++++++++++++------------------
fs/btrfs/super.c | 3 +-
fs/btrfs/sysfs.c | 41 +--
fs/btrfs/sysfs.h | 3 +-
fs/btrfs/tests/extent-map-tests.c | 2 +-
fs/btrfs/transaction.c | 34 ++
fs/btrfs/transaction.h | 31 ++
fs/btrfs/tree-log.c | 87 ++---
fs/btrfs/tree-log.h | 9 +-
fs/btrfs/volumes.c | 116 ++-----
fs/btrfs/volumes.h | 18 -
fs/btrfs/zoned.c | 146 ++++----
fs/btrfs/zoned.h | 20 +-
fs/iomap/direct-io.c | 10 +-
include/linux/bio.h | 4 +
include/linux/iomap.h | 3 +-
include/trace/events/btrfs.h | 127 +++++--
61 files changed, 2457 insertions(+), 2898 deletions(-)
create mode 100644 fs/btrfs/lru_cache.c
create mode 100644 fs/btrfs/lru_cache.h
On Mon, Feb 20, 2023 at 11:26 AM David Sterba <[email protected]> wrote:
>
> Other:
>
> - locally enable -Wmaybe-uninitialized after fixing all warnings
I've pulled this, but I strongly suspect this change will get reverted.
I bet neither you nor linux-next is testing even _remotely_ a big
chunk of the different compiler versions that are out there, and the
reason flags like '-Wmaybe-uninitialized' get undone is because some
random compiler version on some random config and target archiecture
gives completely nonsensical warnings for odd reasons.
But hey, maybe the btrfs code is special.
Linus
The pull request you sent on Mon, 20 Feb 2023 20:20:08 +0100:
> git://git.kernel.org/pub/scm/linux/kernel/git/kdave/linux.git for-6.3-tag
has been merged into torvalds/linux.git:
https://git.kernel.org/torvalds/c/885ce48739189fac6645ff42d736ee0de0b5917d
Thank you!
--
Deet-doot-dot, I am a bot.
https://korg.docs.kernel.org/prtracker.html
On 2/21/23 4:02 AM, Linus Torvalds wrote:
> On Mon, Feb 20, 2023 at 11:26 AM David Sterba <[email protected]> wrote:
>> Other:
>>
>> - locally enable -Wmaybe-uninitialized after fixing all warnings
>
> I've pulled this, but I strongly suspect this change will get reverted.
>
> I bet neither you nor linux-next is testing even _remotely_ a big
> chunk of the different compiler versions that are out there, and the
> reason flags like '-Wmaybe-uninitialized' get undone is because some
> random compiler version on some random config and target archiecture
> gives completely nonsensical warnings for odd reasons.
>
> But hey, maybe the btrfs code is special.
Maybe it's too late for 6.3. So please fix this in 6.4 and backport it to
6.3 stable. If someone fixes it, kindly add:
Reported-by: Memet <[email protected]>
Compiling with GCC 9.4:
gcc (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0
Copyright (C) 2019 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
Linux version 6.3-rc7:
CC [M] fs/btrfs/volumes.o
fs/btrfs/volumes.c: In function ‘btrfs_init_new_device’:
fs/btrfs/volumes.c:2524:2: error: ‘seed_devices’ may be used uninitialized in this function [-Werror=maybe-uninitialized]
2524 | list_add(&seed_devices->seed_list, &fs_devices->seed_list);
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
fs/btrfs/volumes.c:2621:27: note: ‘seed_devices’ was declared here
2621 | struct btrfs_fs_devices *seed_devices;
| ^~~~~~~~~~~~
cc1: all warnings being treated as errors
make[7]: *** [scripts/Makefile.build:252: fs/btrfs/volumes.o] Error 1
make[6]: *** [scripts/Makefile.build:494: fs/btrfs] Error 2
make[5]: *** [scripts/Makefile.build:494: fs] Error 2
make[4]: *** [Makefile:2025: .] Error 2
make[3]: *** [Makefile:357: __build_one_by_one] Error 2
make[2]: *** [debian/rules:8: build-arch] Error 2
dpkg-buildpackage: error: debian/rules binary subprocess returned exit status 2
make[1]: *** [scripts/Makefile.package:120: bindeb-pkg] Error 2
make: *** [Makefile:1656: bindeb-pkg] Error 2
fs/btrfs/send.c: In function ‘get_cur_inode_state’:
fs/btrfs/send.c:1909:13: error: ‘right_gen’ may be used uninitialized in this function [-Werror=maybe-uninitialized]
1909 | } else if (right_gen == gen) {
| ^
--
Ammar Faizi
On Sun, Apr 23, 2023 at 09:27:30AM +0700, Ammar Faizi wrote:
> On 2/21/23 4:02 AM, Linus Torvalds wrote:
> > On Mon, Feb 20, 2023 at 11:26 AM David Sterba <[email protected]> wrote:
> >> Other:
> >>
> >> - locally enable -Wmaybe-uninitialized after fixing all warnings
> >
> > I've pulled this, but I strongly suspect this change will get reverted.
> >
> > I bet neither you nor linux-next is testing even _remotely_ a big
> > chunk of the different compiler versions that are out there, and the
> > reason flags like '-Wmaybe-uninitialized' get undone is because some
> > random compiler version on some random config and target archiecture
> > gives completely nonsensical warnings for odd reasons.
> >
> > But hey, maybe the btrfs code is special.
>
> Maybe it's too late for 6.3. So please fix this in 6.4 and backport it to
> 6.3 stable. If someone fixes it, kindly add:
Fix for this warning is in 6.4 pull request, there's no CC:stable tag
but we can ask to add it once the code lands in master.
On Tue, Apr 25, 2023 at 04:54:12PM +0200, David Sterba wrote:
> On Sun, Apr 23, 2023 at 09:27:30AM +0700, Ammar Faizi wrote:
> > On 2/21/23 4:02 AM, Linus Torvalds wrote:
> > > On Mon, Feb 20, 2023 at 11:26 AM David Sterba <[email protected]> wrote:
> > >> Other:
> > >>
> > >> - locally enable -Wmaybe-uninitialized after fixing all warnings
> > >
> > > I've pulled this, but I strongly suspect this change will get reverted.
> > >
> > > I bet neither you nor linux-next is testing even _remotely_ a big
> > > chunk of the different compiler versions that are out there, and the
> > > reason flags like '-Wmaybe-uninitialized' get undone is because some
> > > random compiler version on some random config and target archiecture
> > > gives completely nonsensical warnings for odd reasons.
> > >
> > > But hey, maybe the btrfs code is special.
> >
> > Maybe it's too late for 6.3. So please fix this in 6.4 and backport it to
> > 6.3 stable.
>
> Fix for this warning is in 6.4 pull request, there's no CC:stable tag
> but we can ask to add it once the code lands in master.
It landed in master.
[ Adding stable team to the Cc list ]
Hi Greg and stable team, could you please backport:
commit 8ba7d5f5ba931be68a94b8c91bcced1622934e7a upstream
("btrfs: fix uninitialized variable warnings")
to v6.3 to fix gcc (10, 9, 7) build error?
The fs/btrfs/volumes.c hunk won't apply cleanly, but it can auto-merge:
$ git cherry-pick 8ba7d5f5ba931be68a94b8c91bcced1622934e7a
Auto-merging fs/btrfs/volumes.c
[detached HEAD 572410288a1070c1] btrfs: fix uninitialized variable warnings
Author: Genjian Zhang <[email protected]>
Date: Fri Mar 24 10:08:38 2023 +0800
2 files changed, 2 insertions(+), 2 deletions(-)
Thanks,
--
Ammar Faizi
On Fri, Apr 28, 2023 at 10:14:58AM +0700, Ammar Faizi wrote:
> On Tue, Apr 25, 2023 at 04:54:12PM +0200, David Sterba wrote:
> > On Sun, Apr 23, 2023 at 09:27:30AM +0700, Ammar Faizi wrote:
> > > On 2/21/23 4:02 AM, Linus Torvalds wrote:
> > > > On Mon, Feb 20, 2023 at 11:26 AM David Sterba <[email protected]> wrote:
> > > >> Other:
> > > >>
> > > >> - locally enable -Wmaybe-uninitialized after fixing all warnings
> > > >
> > > > I've pulled this, but I strongly suspect this change will get reverted.
> > > >
> > > > I bet neither you nor linux-next is testing even _remotely_ a big
> > > > chunk of the different compiler versions that are out there, and the
> > > > reason flags like '-Wmaybe-uninitialized' get undone is because some
> > > > random compiler version on some random config and target archiecture
> > > > gives completely nonsensical warnings for odd reasons.
> > > >
> > > > But hey, maybe the btrfs code is special.
> > >
> > > Maybe it's too late for 6.3. So please fix this in 6.4 and backport it to
> > > 6.3 stable.
> >
> > Fix for this warning is in 6.4 pull request, there's no CC:stable tag
> > but we can ask to add it once the code lands in master.
>
> It landed in master.
>
> [ Adding stable team to the Cc list ]
>
> Hi Greg and stable team, could you please backport:
>
> commit 8ba7d5f5ba931be68a94b8c91bcced1622934e7a upstream
> ("btrfs: fix uninitialized variable warnings")
>
> to v6.3 to fix gcc (10, 9, 7) build error?
>
> The fs/btrfs/volumes.c hunk won't apply cleanly, but it can auto-merge:
>
> $ git cherry-pick 8ba7d5f5ba931be68a94b8c91bcced1622934e7a
> Auto-merging fs/btrfs/volumes.c
> [detached HEAD 572410288a1070c1] btrfs: fix uninitialized variable warnings
> Author: Genjian Zhang <[email protected]>
> Date: Fri Mar 24 10:08:38 2023 +0800
> 2 files changed, 2 insertions(+), 2 deletions(-)
Now queued up, thanks.
greg k-h