Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67;
From:   David Sterba <dsterba@suse.com>
To:     torvalds@linux-foundation.org
Cc:     David Sterba <dsterba@suse.com>, clm@fb.com,
        linux-btrfs@vger.kernel.org, linux-kernel@vger.kernel.org
Subject: [GIT PULL] Btrfs updates for 4.21
Date:   Wed, 19 Dec 2018 16:11:04 +0100
Message-Id: <cover.1545231616.git.dsterba@suse.com>
MIME-Version: 1.0
Content-Transfer-Encoding: 8bit
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk

Hi,

I've planned to merge a few more patches to the 1st pull branch that belong to
a core change mentioned below as 'delayed refs reserve'. There are review
comments that I think should be addressed, so the patches are postponed.

Technically they're fixes and I presume it's ok to take them in the -rc. I
didn't want to touch the pull request branch a few days before sending it,
though the patches have been in linux-next for some time. The bugs fixed are
not critical for testing so this should not block anybody.

No merge conflicts, please pull. Thanks.

--

New features:

* swapfile support - after a long time it's here, with some limitations where
  COW design does not work well with the swap implementation (nodatacow file,
  no compression, cannot be snapshotted, not possible on multiple devices,
  ...), as this is the most restricted but working setup, we'll try to improve
  that in the future

* metadata uuid - an optional incompat feature to assign a new filesystem UUID
  without overwriting all metadata blocks, stored only in superblock

* more balance messages are printed to system log, initial is in the format of
  the command line that would be used to start it


Fixes:

* tag pages of a snapshot to better separate pages that are involved in the
  snapshot (and need to get synced) from newly dirtied pages that could slow
  down or even livelock the snapshot operation

* improved check of filesystem id associated with a device during scan to
  detect duplicate devices that could be mixed up during mount

* fix device replace state transitions, eg. when it ends up interrupted and
  reboot tries to restart balance too, or when start/cancel ioctls race

* fix a crash due to a race when quotas are enabled during snapshot creation

* GFP_NOFS/memalloc_nofs_* fixes due to GFP_KERNEL allocations in transaction
  context

* fix fsync of files with multiple hard links in new directories

* fix race of send with transaction commits that create snapshots


Core changes:

* cleanups
  * further removals of now-dead fsync code
  * core function for finding free extent has been split and provides a base
    for further cleanups to make the logic more understandable
  * removed lot of indirect callbacks for data and metadata inodes
  * simplified refcounting and locking for cloned extent buffers
  * removed redundant function arguments
  * defines converted to enums where appropriate

* separate reserve for delayed refs from global reserve, update logic to do
  less trickery and ad-hoc heuristics, move out some related expensive
  operations from transaction commit or file truncate

* dev-replace switched from custom locking scheme to semaphore

* remove first phase of balance that tried to make some space for the
  relocation by calling shrink and grow, this did not work as expected and only
  introduced more error states due to potential resize failures, slightly
  improves the runtime as the chunks on all devices are not needlessly
  enumerated

* clone and deduplication now use generic helper that adds a few more checks
  that were missing from the original btrfs implementation of the ioctls

----------------------------------------------------------------
The following changes since commit 7566ec393f4161572ba6f11ad5171fd5d59b0fbd:

  Linux 4.20-rc7 (2018-12-16 15:46:55 -0800)

are available in the Git repository at:

  git://git.kernel.org/pub/scm/linux/kernel/git/kdave/linux.git for-4.21-tag

for you to fetch changes up to 52042d8e82ff50d40e76a275ac0b97aa663328b0:

  btrfs: Fix typos in comments and strings (2018-12-17 14:51:50 +0100)

----------------------------------------------------------------
Anand Jain (14):
      btrfs: remove redundant replace_state init
      btrfs: harden agaist duplicate fsid on scanned devices
      btrfs: mark btrfs_dev_replace_start as static
      btrfs: dev-replace: go back to suspended state if target device is missing
      btrfs: dev-replace: go back to suspend state if another EXCL_OP is running
      btrfs: fix use-after-free due to race between replace start and cancel
      btrfs: dev-replace: set result code of cancel by status of scrub
      btrfs: dev-replace: replace's scrub must not be running in suspended state
      btrfs: dev-replace: add explicit check for replace result "no error"
      btrfs: silence warning if replace is canceled
      btrfs: don't report user-requested cancel as an error
      btrfs: add helper to describe block group flags
      btrfs: balance: print args during start and resume
      btrfs: balance: print to system log when balance ends or is paused

Andrea Gelmini (1):
      btrfs: Fix typos in comments and strings

David Sterba (18):
      btrfs: merge btrfs_submit_bio_done to its caller
      btrfs: replace async_cow::root with fs_info
      btrfs: remove redundant csum buffer in btrfs_io_bio
      btrfs: replace btrfs_io_bio::end_io with a simple helper
      btrfs: switch BTRFS_FS_STATE_* to enums
      btrfs: switch BTRFS_BLOCK_RSV_* to enums
      btrfs: switch BTRFS_FS_* to enums
      btrfs: switch BTRFS_ROOT_* to enums
      btrfs: switch EXTENT_BUFFER_* to enums
      btrfs: switch EXTENT_FLAG_* to enums
      btrfs: switch BTRFS_ORDERED_* to enums
      btrfs: drop extra enum initialization where using defaults
      btrfs: reada: reorder dev-replace locks before radix tree preload
      btrfs: dev-replace: swich locking to rw semaphore
      btrfs: dev-replace: remove custom read/write blocking scheme
      btrfs: dev-replace: open code trivial locking helpers
      btrfs: scrub: pass fs_info to scrub_setup_ctx
      btrfs: scrub: move scrub_setup_ctx allocation out of device_list_mutex

Ethan Lien (1):
      btrfs: use tagged writepage to mitigate livelock of snapshot

Filipe Manana (13):
      Btrfs: remove no longer used logged range variables when logging extents
      Btrfs: remove no longer used stuff for tracking pending ordered extents
      Btrfs: simpler and more efficient cleanup of a log tree's extent io tree
      Btrfs: remove no longer used io_err from btrfs_log_ctx
      Btrfs: allow clear_extent_dirty() to receive a cached extent state record
      Btrfs: fix access to available allocation bits when starting balance
      Btrfs: fix deadlock when enabling quotas due to concurrent snapshot creation
      Btrfs: fix deadlock with memory reclaim during scrub
      Btrfs: fix fsync of files with multiple hard links in new directories
      Btrfs: scrub, move setup of nofs contexts higher in the stack
      Btrfs: use nofs context when initializing security xattrs to avoid deadlock
      Btrfs: send, fix race with transaction commits that create snapshots
      Btrfs: use generic_remap_file_range_prep() for cloning and deduplication

Hans van Kranenburg (5):
      btrfs: alloc_chunk: do not refurbish num_bytes
      btrfs: alloc_chunk: improve chunk size variable name
      btrfs: alloc_chunk: fix more DUP stripe size handling
      btrfs: fix ncopies raid_attr for RAID56
      btrfs: introduce nparity raid_attr

Johannes Thumshirn (9):
      btrfs: remove unused drop_on_err in btrfs_mkdir
      btrfs: introduce EXPORT_FOR_TESTS macro
      btrfs: use EXPORT_FOR_TESTS for conditionally exported functions
      btrfs: remove btrfs_bio_end_io_t
      btrfs: don't initialize 'offset' in map_private_extent_buffer()
      btrfs: document extent mapping assumptions in checksum
      btrfs: use offset_in_page instead of open-coding it
      btrfs: use PAGE_ALIGNED instead of open-coding it
      btrfs: improve error handling of btrfs_add_link

Josef Bacik (12):
      btrfs: add btrfs_delete_ref_head helper
      btrfs: add cleanup_ref_head_accounting helper
      btrfs: cleanup extent_op handling
      btrfs: only track ref_heads in delayed_ref_updates
      btrfs: introduce delayed_refs_rsv
      btrfs: update may_commit_transaction to use the delayed refs rsv
      btrfs: add new flushing states for the delayed refs rsv
      btrfs: rework btrfs_check_space_for_delayed_refs
      btrfs: don't run delayed refs in the end transaction logic
      btrfs: fix truncate throttling
      btrfs: catch cow on deleting snapshots
      btrfs: run delayed items before dropping the snapshot

Lu Fengqi (4):
      btrfs: extent-tree: Detect bytes_pinned underflow earlier
      btrfs: cleanup the useless DEFINE_WAIT in cleanup_transaction
      btrfs: skip file_extent generation check for free_space_inode in run_delalloc_nocow
      btrfs: remove always true if branch in find_delalloc_range

Nikolay Borisov (36):
      btrfs: Add function to distinguish between data and btree inode
      btrfs: Remove extent_io_ops::fill_delalloc
      btrfs: Remove extent_io_ops::writepage_start_hook
      btrfs: Remove extent_io_ops::writepage_end_io_hook
      btrfs: Remove extent_io_ops::check_extent_io_range callback
      btrfs: Remove extent_io_ops::set_bit_hook extent_io callback
      btrfs: Remove extent_io_ops::clear_bit_hook callback
      btrfs: Remove extent_io_ops::merge_extent_hook callback
      btrfs: Remove extent_io_ops::split_extent_hook callback
      btrfs: Remove needless tree locking in iterate_inode_refs
      btrfs: Remove needless tree locking in iterate_inode_extrefs
      btrfs: Remove redundant extent_buffer_get in get_old_root
      btrfs: Remove extraneous extent_buffer_get from tree_mod_log_rewind
      btrfs: Remove extra reference count bumps in btrfs_compare_trees
      btrfs: Remove unnecessary tree locking code in qgroup_rescan_leaf
      btrfs: Remove special handling of EXTENT_BUFFER_UNMAPPED while freeing
      btrfs: Adjust loop in free_extent_buffer
      btrfs: Check for missing device before bio submission in btrfs_map_bio
      btrfs: Move epd::extent_locked check to writepage_delalloc's caller
      btrfs: Remove extent_page_data argument from writepage_delalloc
      btrfs: Remove unused extent_state argument from btrfs_writepage_endio_finish_ordered
      btrfs: Remove superfluous check form btrfs_remove_chunk
      btrfs: Sink find_lock_delalloc_range's 'max_bytes' argument
      btrfs: Replace BUG_ON with ASSERT in find_lock_delalloc_range
      btrfs: Introduce support for FSID change without metadata rewrite
      btrfs: Add sysfs support for metadata_uuid feature
      btrfs: Remove fsid/metadata_fsid fields from btrfs_info
      btrfs: Add handling for disk split-brain scenario during fsid change
      btrfs: add members to fs_devices to track fsid changes
      btrfs: Handle one more split-brain scenario during fsid change
      btrfs: Handle final split-brain possibility during fsid change
      btrfs: Remove extent_io_ops::readpage_io_failed_hook
      btrfs: Refactor btrfs_merge_bio_hook
      btrfs: Fix error handling in btrfs_cleanup_ordered_extents
      btrfs: Remove 1st shrink/grow phase from balance
      btrfs: Refactor main loop in extent_readpages

Omar Sandoval (3):
      Btrfs: prevent ioctls from interfering with a swap file
      Btrfs: rename and export get_chunk_map
      Btrfs: support swap files

Qu Wenruo (9):
      btrfs: extent-tree: Detect bytes_may_use underflow earlier
      btrfs: Introduce find_free_extent_ctl structure for later rework
      btrfs: Refactor clustered extent allocation into find_free_extent_clustered
      btrfs: Refactor unclustered extent allocation into find_free_extent_unclustered()
      btrfs: Refactor find_free_extent loops update into find_free_extent_update_loop
      btrfs: volumes: Make sure there is no overlap of dev extents at mount time
      btrfs: volumes: Make sure no dev extent is beyond device boundary
      btrfs: tests: Use BTRFS_MAX_EXTENT_SIZE to replace the intermediate number
      btrfs: extent-tree: cleanup one-shot usage of @blocksize in do_walk_down

 fs/btrfs/backref.c               |   13 +-
 fs/btrfs/btrfs_inode.h           |   14 +-
 fs/btrfs/check-integrity.c       |   24 +-
 fs/btrfs/compression.c           |   26 +-
 fs/btrfs/ctree.c                 |   46 +-
 fs/btrfs/ctree.h                 |  263 +++++---
 fs/btrfs/delayed-ref.c           |   61 +-
 fs/btrfs/delayed-ref.h           |    3 +-
 fs/btrfs/dev-replace.c           |  191 +++---
 fs/btrfs/dev-replace.h           |    8 -
 fs/btrfs/disk-io.c               |  117 ++--
 fs/btrfs/disk-io.h               |   10 +-
 fs/btrfs/extent-tree.c           | 1254 ++++++++++++++++++++++++--------------
 fs/btrfs/extent_io.c             |  414 ++++++-------
 fs/btrfs/extent_io.h             |   66 +-
 fs/btrfs/extent_map.c            |    3 +-
 fs/btrfs/extent_map.h            |   21 +-
 fs/btrfs/file-item.c             |   13 +-
 fs/btrfs/file.c                  |   29 +-
 fs/btrfs/free-space-tree.c       |   15 +-
 fs/btrfs/inode.c                 |  665 ++++++++++++++------
 fs/btrfs/ioctl.c                 |  643 +++++--------------
 fs/btrfs/lzo.c                   |    2 +-
 fs/btrfs/ordered-data.c          |   30 -
 fs/btrfs/ordered-data.h          |   47 +-
 fs/btrfs/qgroup.c                |   35 +-
 fs/btrfs/qgroup.h                |    6 +-
 fs/btrfs/raid56.c                |    2 +-
 fs/btrfs/reada.c                 |   16 +-
 fs/btrfs/ref-verify.c            |    6 +-
 fs/btrfs/relocation.c            |   50 +-
 fs/btrfs/scrub.c                 |   85 ++-
 fs/btrfs/send.c                  |    6 +-
 fs/btrfs/super.c                 |   10 +-
 fs/btrfs/sysfs.c                 |   14 +
 fs/btrfs/sysfs.h                 |    2 +-
 fs/btrfs/tests/btrfs-tests.c     |    4 +-
 fs/btrfs/tests/extent-io-tests.c |   29 +-
 fs/btrfs/tests/inode-tests.c     |    6 -
 fs/btrfs/transaction.c           |   93 ++-
 fs/btrfs/transaction.h           |   16 +-
 fs/btrfs/tree-checker.c          |    6 +-
 fs/btrfs/tree-log.c              |   44 +-
 fs/btrfs/tree-log.h              |    2 -
 fs/btrfs/volumes.c               |  779 +++++++++++++++++------
 fs/btrfs/volumes.h               |   25 +-
 fs/btrfs/xattr.c                 |    8 +
 include/trace/events/btrfs.h     |    4 +-
 include/uapi/linux/btrfs.h       |    1 +
 include/uapi/linux/btrfs_tree.h  |    1 +
 50 files changed, 3030 insertions(+), 2198 deletions(-)