Received-SPF: pass (google.com: best guess record for domain of linux-ext4-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67;
Subject: Re: [LFS/MM TOPIC] fs reflink issues, fs online scrub/check, etc
To:     Steve French <smfrench@gmail.com>,
        "Darrick J. Wong" <darrick.wong@oracle.com>
Cc:     lsf-pc@lists.linux-foundation.org, xfs <xfs@oss.sgi.com>,
        linux-fsdevel <linux-fsdevel@vger.kernel.org>,
        linux-ext4 <linux-ext4@vger.kernel.org>,
        linux-btrfs <linux-btrfs@vger.kernel.org>,
        ocfs2-devel@oss.oracle.com
References: <20160210191715.GB6339@birch.djwong.org>
 <20160210191848.GC6346@birch.djwong.org>
 <CAH2r5mtM2nCicTKGFAjYtOG92TKKQdTbZxaD-_-RsWYL=Tn2Nw@mail.gmail.com>
From:   Qu Wenruo <quwenruo.btrfs@gmx.com>
Autocrypt: addr=quwenruo.btrfs@gmx.com; prefer-encrypt=mutual; keydata=
 mQENBFnVga8BCACyhFP3ExcTIuB73jDIBA/vSoYcTyysFQzPvez64TUSCv1SgXEByR7fju3o
 8RfaWuHCnkkea5luuTZMqfgTXrun2dqNVYDNOV6RIVrc4YuG20yhC1epnV55fJCThqij0MRL
 1NxPKXIlEdHvN0Kov3CtWA+R1iNN0RCeVun7rmOrrjBK573aWC5sgP7YsBOLK79H3tmUtz6b
 9Imuj0ZyEsa76Xg9PX9Hn2myKj1hfWGS+5og9Va4hrwQC8ipjXik6NKR5GDV+hOZkktU81G5
 gkQtGB9jOAYRs86QG/b7PtIlbd3+pppT0gaS+wvwMs8cuNG+Pu6KO1oC4jgdseFLu7NpABEB
 AAG0IlF1IFdlbnJ1byA8cXV3ZW5ydW8uYnRyZnNAZ214LmNvbT6JAU4EEwEIADgCGwMFCwkI
 BwIGFQgJCgsCBBYCAwECHgECF4AWIQQt33LlpaVbqJ2qQuHCPZHzoSX+qAUCXZw1oQAKCRDC
 PZHzoSX+qCY6CACd+mWu3okGwRKXju6bou+7VkqCaHTdyXwWFTsr+/0ly5nUdDtT3yEVggPJ
 3VP70wjlrxUjNjFb6iIvGYxiPOrop1NGwGYvQktgRhaIhALG6rPoSSAhGNjwGVRw0km0PlIN
 D29BTj/lYEk+jVM1YL0QLgAE1AI3krihg/lp/fQT53wLhR8YZIF8ETXbClQG1vJ0cllPuEEv
 efKxRyiTSjB+PsozSvYWhXsPeJ+KKjFen7ebE5reQTPFzSHctCdPnoR/4jSPlnTlnEvLeqcD
 ZTuKfQe1gWrPeevQzgCtgBF/WjIOeJs41klnYzC3DymuQlmFubss0jShLOW8eSOOWhLRuQEN
 BFnVga8BCACqU+th4Esy/c8BnvliFAjAfpzhI1wH76FD1MJPmAhA3DnX5JDORcgaCbPEwhLj
 1xlwTgpeT+QfDmGJ5B5BlrrQFZVE1fChEjiJvyiSAO4yQPkrPVYTI7Xj34FnscPj/IrRUUka
 68MlHxPtFnAHr25VIuOS41lmYKYNwPNLRz9Ik6DmeTG3WJO2BQRNvXA0pXrJH1fNGSsRb+pK
 EKHKtL1803x71zQxCwLh+zLP1iXHVM5j8gX9zqupigQR/Cel2XPS44zWcDW8r7B0q1eW4Jrv
 0x19p4P923voqn+joIAostyNTUjCeSrUdKth9jcdlam9X2DziA/DHDFfS5eq4fEvABEBAAGJ
 ATwEGAEIACYCGwwWIQQt33LlpaVbqJ2qQuHCPZHzoSX+qAUCXZw1rgUJCWpOfwAKCRDCPZHz
 oSX+qFcEB/95cs8cM1OQdE/GgOfCGxwgckMeWyzOR7bkAWW0lDVp2hpgJuxBW/gyfmtBnUai
 fnggx3EE3ev8HTysZU9q0h+TJwwJKGv6sUc8qcTGFDtavnnl+r6xDUY7A6GvXEsSoCEEynby
 72byGeSovfq/4AWGNPBG1L61Exl+gbqfvbECP3ziXnob009+z9I4qXodHSYINfAkZkA523JG
 ap12LndJeLk3gfWNZfXEWyGnuciRGbqESkhIRav8ootsCIops/SqXm0/k+Kcl4gGUO/iD/T5
 oagaDh0QtOd8RWSMwLxwn8uIhpH84Q4X1LadJ5NCgGa6xPP5qqRuiC+9gZqbq4Nj
Message-ID: <0089aff3-c4d3-214e-30d7-012abf70623a@gmx.com>
Date:   Tue, 21 Jan 2020 09:18:28 +0800
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101
 Thunderbird/68.3.1
MIME-Version: 1.0
In-Reply-To: <CAH2r5mtM2nCicTKGFAjYtOG92TKKQdTbZxaD-_-RsWYL=Tn2Nw@mail.gmail.com>
Content-Type: multipart/signed; micalg=pgp-sha256;
 protocol="application/pgp-signature";
 boundary="Qq0KCP7NpJNIkStCipMQK0TD8KG42ST8M"
Sender: linux-ext4-owner@vger.kernel.org
Precedence: bulk

This is an OpenPGP/MIME signed message (RFC 4880 and 3156)
--Qq0KCP7NpJNIkStCipMQK0TD8KG42ST8M
Content-Type: multipart/mixed; boundary="XlLjsP7dgzWpBqYJzOXOyVoQQEB9Mc2Iu"

--XlLjsP7dgzWpBqYJzOXOyVoQQEB9Mc2Iu
Content-Type: text/plain; charset=utf-8
Content-Language: en-US
Content-Transfer-Encoding: quoted-printable

Didn't see the original mail, so reply here.

On 2020/1/21 =E4=B8=8A=E5=8D=888:58, Steve French wrote:
> Since SMB3 protocol has at least three ways to do copy offload (server
> side copy),
> the reflink topic is of interest to me and likely useful to discuss
> for Samba server as
> well as client (cifs.ko)
>=20
> On Wed, Feb 10, 2016 at 1:19 PM Darrick J. Wong <darrick.wong@oracle.co=
m> wrote:
>>
>> [resend, email exploded, sorry...]
>>
>> Hi,
>>
>> I want to discuss a few FS related topics that I haven't already seen =
on
>> the mailing lists:
>>
>>  * Shared pagecache pages for reflinked files (and by extension making=
 dax
>>    work with reflink on xfs)

IIRC Goldwyn Rodrigues <rgoldwyn@suse.com> is working on this, mostly
for btrfs, but should also apply to other fses.

>>
>>  * Providing a simple interface for scrubbing filesystem metadata in t=
he
>>    background (the online check thing).  Ideally we'd make it easy to =
discover
>>    what kind of metadata there is to check and provide a simple interf=
ace to
>>    check the metadata, once discovered.  This is a tricky interface to=
pic
>>    since FS design differs pretty widely.

Although btrfs has already implemented scrub for a long time, and btrfs
is coming over the point where kernel can detects more problems than
btrfs-check, I still hesitate to call it "check the metadata".

It looks more like "extended metadata sanity verification", other than
full cross-ref check implemented in user-space tools.

Currently, btrfs (and I guess xfs too) can detect corrupted metadata by:
- checksum
  All modern fses have similar mechanism.

- internal fields checking at read time
  At least btrfs is trying to do a byte-per-byte check for each fields.
  And latest such check has killed tons of fuzzed images (I guess that's
  why a lot of such CVE/fuzzed image reporters only want to report on
  older kernels).

  But this is mostly based on the fact that btrfs on-disk fields has a
  lot of redundancy. E.g. btrfs uses bytenr, not block count, allowing
  us to detect bit flips in lower bits easily.
  So not all fs could follow this step.

  But this provides us a good centralized place to validate most tree
  blocks.

  Maybe other fs devs could share such details too?

- runtime sanity check across metadata boundaries
  This is the traditional methods.

So is that xfs online scrub just a similar thing, or a full cross-ref che=
ck?

And if so, can we find a less confusing naming for the interface first?

>>
>>  * Rudimentary online repair and rebuilding (xfs) from secondary metad=
ata

My guess is, it's checksum and copy based.
Or just like btrfs, if checksum passes but internal checks still failed,
just try next copy?

>>
>>  * Working out the variances in the btrfs/xfs/ocfs2/nfs reflink implem=
entations
>>    and making sure they all get test coverage

That would be great!

Thanks,
Qu

>>
>> I would also like participate in some of the proposed discussions:
>>
>>  * The ext4 summit (and whatever meeting of XFS devs may happen)
>>
>>  * Integrating existing filesystems into pmem, or hallway bofs about d=
esigning
>>    new filesystems for pmem
>>
>>  * Actually seeing the fs developers (well, everyone!) in person again=
 :)
>>
>> --Darrick
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-fsdeve=
l" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>=20
>=20
>=20


--XlLjsP7dgzWpBqYJzOXOyVoQQEB9Mc2Iu--

--Qq0KCP7NpJNIkStCipMQK0TD8KG42ST8M
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: OpenPGP digital signature
Content-Disposition: attachment; filename="signature.asc"

-----BEGIN PGP SIGNATURE-----

iQEzBAEBCAAdFiEELd9y5aWlW6idqkLhwj2R86El/qgFAl4mUWQACgkQwj2R86El
/qg2hQf/bV2A+E7hSFyFmFewaDjGQSjWWiJvxOnW3FDWEufZ3w6hgWm0YrjoB1hu
j4lkKCmp9NUQzp9ZSE5OYrziQHq11HPj9TH1pLmjDi3AyKAC7Ep4jPQJCmnXSv+W
bzrheqRjqeGvICJJOh+ksc2YR4mH6bbNWn4MstBjDG6d3JvHwiNU6Bjq4mVaUN03
RNkB0JUeYC/0rZHK18CYVYrHs1c6uVIjmpKXr1KEgPFdDVhxAPjmjrKGU6C3icOv
D8HfGW4PgYnKTWFSA7kopeafkqf0OgPUNe61IM6jtXpqP9aF2QOhEriI8P9zdNW3
o36PossEk/B3pnurdUAMG6cBeDyBhQ==
=gYmi
-----END PGP SIGNATURE-----

--Qq0KCP7NpJNIkStCipMQK0TD8KG42ST8M--