OSDs are able to perform object copies across different pools. Thus,
there's no need to prevent copy_file_range from doing remote copies if the
source and destination superblocks are different. Only return -EXDEV if
they have different fsid (the cluster ID).
Signed-off-by: Luis Henriques <[email protected]>
---
fs/ceph/file.c | 18 ++++++++++++++----
1 file changed, 14 insertions(+), 4 deletions(-)
Hi,
Here's the patch changelog since initial submittion:
- Dropped have_fsid checks on client structs
- Use %pU to print the fsid instead of raw hex strings (%*ph)
- Fixed 'To:' field in email so that this time the patch hits vger
Cheers,
--
Luis
diff --git a/fs/ceph/file.c b/fs/ceph/file.c
index 685a03cc4b77..4a624a1dd0bb 100644
--- a/fs/ceph/file.c
+++ b/fs/ceph/file.c
@@ -1904,6 +1904,7 @@ static ssize_t __ceph_copy_file_range(struct file *src_file, loff_t src_off,
struct ceph_inode_info *src_ci = ceph_inode(src_inode);
struct ceph_inode_info *dst_ci = ceph_inode(dst_inode);
struct ceph_cap_flush *prealloc_cf;
+ struct ceph_fs_client *src_fsc = ceph_inode_to_client(src_inode);
struct ceph_object_locator src_oloc, dst_oloc;
struct ceph_object_id src_oid, dst_oid;
loff_t endoff = 0, size;
@@ -1915,8 +1916,17 @@ static ssize_t __ceph_copy_file_range(struct file *src_file, loff_t src_off,
if (src_inode == dst_inode)
return -EINVAL;
- if (src_inode->i_sb != dst_inode->i_sb)
- return -EXDEV;
+ if (src_inode->i_sb != dst_inode->i_sb) {
+ struct ceph_fs_client *dst_fsc = ceph_inode_to_client(dst_inode);
+
+ if (ceph_fsid_compare(&src_fsc->client->fsid,
+ &dst_fsc->client->fsid)) {
+ dout("Copying object across different clusters:");
+ dout(" src fsid: %pU dst fsid: %pU\n",
+ &src_fsc->client->fsid, &dst_fsc->client->fsid);
+ return -EXDEV;
+ }
+ }
if (ceph_snap(dst_inode) != CEPH_NOSNAP)
return -EROFS;
@@ -1928,7 +1938,7 @@ static ssize_t __ceph_copy_file_range(struct file *src_file, loff_t src_off,
* efficient).
*/
- if (ceph_test_mount_opt(ceph_inode_to_client(src_inode), NOCOPYFROM))
+ if (ceph_test_mount_opt(src_fsc, NOCOPYFROM))
return -EOPNOTSUPP;
if ((src_ci->i_layout.stripe_unit != dst_ci->i_layout.stripe_unit) ||
@@ -2044,7 +2054,7 @@ static ssize_t __ceph_copy_file_range(struct file *src_file, loff_t src_off,
dst_ci->i_vino.ino, dst_objnum);
/* Do an object remote copy */
err = ceph_osdc_copy_from(
- &ceph_inode_to_client(src_inode)->client->osdc,
+ &src_fsc->client->osdc,
src_ci->i_vino.snap, 0,
&src_oid, &src_oloc,
CEPH_OSD_OP_FLAG_FADVISE_SEQUENTIAL |
On Mon, 2019-09-09 at 11:28 +0100, Luis Henriques wrote:
> OSDs are able to perform object copies across different pools. Thus,
> there's no need to prevent copy_file_range from doing remote copies if the
> source and destination superblocks are different. Only return -EXDEV if
> they have different fsid (the cluster ID).
>
> Signed-off-by: Luis Henriques <[email protected]>
> ---
> fs/ceph/file.c | 18 ++++++++++++++----
> 1 file changed, 14 insertions(+), 4 deletions(-)
>
> Hi,
>
> Here's the patch changelog since initial submittion:
>
> - Dropped have_fsid checks on client structs
> - Use %pU to print the fsid instead of raw hex strings (%*ph)
> - Fixed 'To:' field in email so that this time the patch hits vger
>
> Cheers,
> --
> Luis
>
> diff --git a/fs/ceph/file.c b/fs/ceph/file.c
> index 685a03cc4b77..4a624a1dd0bb 100644
> --- a/fs/ceph/file.c
> +++ b/fs/ceph/file.c
> @@ -1904,6 +1904,7 @@ static ssize_t __ceph_copy_file_range(struct file *src_file, loff_t src_off,
> struct ceph_inode_info *src_ci = ceph_inode(src_inode);
> struct ceph_inode_info *dst_ci = ceph_inode(dst_inode);
> struct ceph_cap_flush *prealloc_cf;
> + struct ceph_fs_client *src_fsc = ceph_inode_to_client(src_inode);
> struct ceph_object_locator src_oloc, dst_oloc;
> struct ceph_object_id src_oid, dst_oid;
> loff_t endoff = 0, size;
> @@ -1915,8 +1916,17 @@ static ssize_t __ceph_copy_file_range(struct file *src_file, loff_t src_off,
>
> if (src_inode == dst_inode)
> return -EINVAL;
> - if (src_inode->i_sb != dst_inode->i_sb)
> - return -EXDEV;
> + if (src_inode->i_sb != dst_inode->i_sb) {
> + struct ceph_fs_client *dst_fsc = ceph_inode_to_client(dst_inode);
> +
> + if (ceph_fsid_compare(&src_fsc->client->fsid,
> + &dst_fsc->client->fsid)) {
> + dout("Copying object across different clusters:");
> + dout(" src fsid: %pU dst fsid: %pU\n",
> + &src_fsc->client->fsid, &dst_fsc->client->fsid);
> + return -EXDEV;
> + }
> + }
Just to be clear: what happens here if I mount two entirely separate
clusters, and their OSDs don't have any access to one another? Will this
fail at some later point with an error that we can catch so that we can
fall back?
> if (ceph_snap(dst_inode) != CEPH_NOSNAP)
> return -EROFS;
>
> @@ -1928,7 +1938,7 @@ static ssize_t __ceph_copy_file_range(struct file *src_file, loff_t src_off,
> * efficient).
> */
>
> - if (ceph_test_mount_opt(ceph_inode_to_client(src_inode), NOCOPYFROM))
> + if (ceph_test_mount_opt(src_fsc, NOCOPYFROM))
> return -EOPNOTSUPP;
>
> if ((src_ci->i_layout.stripe_unit != dst_ci->i_layout.stripe_unit) ||
> @@ -2044,7 +2054,7 @@ static ssize_t __ceph_copy_file_range(struct file *src_file, loff_t src_off,
> dst_ci->i_vino.ino, dst_objnum);
> /* Do an object remote copy */
> err = ceph_osdc_copy_from(
> - &ceph_inode_to_client(src_inode)->client->osdc,
> + &src_fsc->client->osdc,
> src_ci->i_vino.snap, 0,
> &src_oid, &src_oloc,
> CEPH_OSD_OP_FLAG_FADVISE_SEQUENTIAL |
--
Jeff Layton <[email protected]>
On Mon, Sep 9, 2019 at 12:29 PM Luis Henriques <[email protected]> wrote:
>
> OSDs are able to perform object copies across different pools. Thus,
> there's no need to prevent copy_file_range from doing remote copies if the
> source and destination superblocks are different. Only return -EXDEV if
> they have different fsid (the cluster ID).
>
> Signed-off-by: Luis Henriques <[email protected]>
> ---
> fs/ceph/file.c | 18 ++++++++++++++----
> 1 file changed, 14 insertions(+), 4 deletions(-)
>
> Hi,
>
> Here's the patch changelog since initial submittion:
>
> - Dropped have_fsid checks on client structs
> - Use %pU to print the fsid instead of raw hex strings (%*ph)
> - Fixed 'To:' field in email so that this time the patch hits vger
>
> Cheers,
> --
> Luis
>
> diff --git a/fs/ceph/file.c b/fs/ceph/file.c
> index 685a03cc4b77..4a624a1dd0bb 100644
> --- a/fs/ceph/file.c
> +++ b/fs/ceph/file.c
> @@ -1904,6 +1904,7 @@ static ssize_t __ceph_copy_file_range(struct file *src_file, loff_t src_off,
> struct ceph_inode_info *src_ci = ceph_inode(src_inode);
> struct ceph_inode_info *dst_ci = ceph_inode(dst_inode);
> struct ceph_cap_flush *prealloc_cf;
> + struct ceph_fs_client *src_fsc = ceph_inode_to_client(src_inode);
> struct ceph_object_locator src_oloc, dst_oloc;
> struct ceph_object_id src_oid, dst_oid;
> loff_t endoff = 0, size;
> @@ -1915,8 +1916,17 @@ static ssize_t __ceph_copy_file_range(struct file *src_file, loff_t src_off,
>
> if (src_inode == dst_inode)
> return -EINVAL;
> - if (src_inode->i_sb != dst_inode->i_sb)
> - return -EXDEV;
> + if (src_inode->i_sb != dst_inode->i_sb) {
> + struct ceph_fs_client *dst_fsc = ceph_inode_to_client(dst_inode);
> +
> + if (ceph_fsid_compare(&src_fsc->client->fsid,
> + &dst_fsc->client->fsid)) {
> + dout("Copying object across different clusters:");
> + dout(" src fsid: %pU dst fsid: %pU\n",
> + &src_fsc->client->fsid, &dst_fsc->client->fsid);
Hi Luis,
This should be a single dout.
Thanks,
Ilya
On Mon, 2019-09-09 at 06:35 -0400, Jeff Layton wrote:
> On Mon, 2019-09-09 at 11:28 +0100, Luis Henriques wrote:
> > OSDs are able to perform object copies across different pools. Thus,
> > there's no need to prevent copy_file_range from doing remote copies if the
> > source and destination superblocks are different. Only return -EXDEV if
> > they have different fsid (the cluster ID).
> >
> > Signed-off-by: Luis Henriques <[email protected]>
> > ---
> > fs/ceph/file.c | 18 ++++++++++++++----
> > 1 file changed, 14 insertions(+), 4 deletions(-)
> >
> > Hi,
> >
> > Here's the patch changelog since initial submittion:
> >
> > - Dropped have_fsid checks on client structs
> > - Use %pU to print the fsid instead of raw hex strings (%*ph)
> > - Fixed 'To:' field in email so that this time the patch hits vger
> >
> > Cheers,
> > --
> > Luis
> >
> > diff --git a/fs/ceph/file.c b/fs/ceph/file.c
> > index 685a03cc4b77..4a624a1dd0bb 100644
> > --- a/fs/ceph/file.c
> > +++ b/fs/ceph/file.c
> > @@ -1904,6 +1904,7 @@ static ssize_t __ceph_copy_file_range(struct file *src_file, loff_t src_off,
> > struct ceph_inode_info *src_ci = ceph_inode(src_inode);
> > struct ceph_inode_info *dst_ci = ceph_inode(dst_inode);
> > struct ceph_cap_flush *prealloc_cf;
> > + struct ceph_fs_client *src_fsc = ceph_inode_to_client(src_inode);
> > struct ceph_object_locator src_oloc, dst_oloc;
> > struct ceph_object_id src_oid, dst_oid;
> > loff_t endoff = 0, size;
> > @@ -1915,8 +1916,17 @@ static ssize_t __ceph_copy_file_range(struct file *src_file, loff_t src_off,
> >
> > if (src_inode == dst_inode)
> > return -EINVAL;
> > - if (src_inode->i_sb != dst_inode->i_sb)
> > - return -EXDEV;
> > + if (src_inode->i_sb != dst_inode->i_sb) {
> > + struct ceph_fs_client *dst_fsc = ceph_inode_to_client(dst_inode);
> > +
> > + if (ceph_fsid_compare(&src_fsc->client->fsid,
> > + &dst_fsc->client->fsid)) {
> > + dout("Copying object across different clusters:");
> > + dout(" src fsid: %pU dst fsid: %pU\n",
> > + &src_fsc->client->fsid, &dst_fsc->client->fsid);
> > + return -EXDEV;
> > + }
> > + }
>
> Just to be clear: what happens here if I mount two entirely separate
> clusters, and their OSDs don't have any access to one another? Will this
> fail at some later point with an error that we can catch so that we can
> fall back?
>
Duh, sorry I asked before I had a cup of coffee this morning. The whole
point is to skip that case.
That said...I wonder if it's possible to have an fsid collision across
two separate clusters and this fail to catch that case? Aren't these
things just allocated via a simple counter increment?
Probably not worth worrying about overmuch, but might be good to
understand what would happen in that case if only to field mailing list
reports.
Other than that, this looks fine, modulo Ilya's comment about the two
dout messages.
>
> > if (ceph_snap(dst_inode) != CEPH_NOSNAP)
> > return -EROFS;
> >
> > @@ -1928,7 +1938,7 @@ static ssize_t __ceph_copy_file_range(struct file *src_file, loff_t src_off,
> > * efficient).
> > */
> >
> > - if (ceph_test_mount_opt(ceph_inode_to_client(src_inode), NOCOPYFROM))
> > + if (ceph_test_mount_opt(src_fsc, NOCOPYFROM))
> > return -EOPNOTSUPP;
> >
> > if ((src_ci->i_layout.stripe_unit != dst_ci->i_layout.stripe_unit) ||
> > @@ -2044,7 +2054,7 @@ static ssize_t __ceph_copy_file_range(struct file *src_file, loff_t src_off,
> > dst_ci->i_vino.ino, dst_objnum);
> > /* Do an object remote copy */
> > err = ceph_osdc_copy_from(
> > - &ceph_inode_to_client(src_inode)->client->osdc,
> > + &src_fsc->client->osdc,
> > src_ci->i_vino.snap, 0,
> > &src_oid, &src_oloc,
> > CEPH_OSD_OP_FLAG_FADVISE_SEQUENTIAL |
--
Jeff Layton <[email protected]>
"Jeff Layton" <[email protected]> writes:
> On Mon, 2019-09-09 at 11:28 +0100, Luis Henriques wrote:
>> OSDs are able to perform object copies across different pools. Thus,
>> there's no need to prevent copy_file_range from doing remote copies if the
>> source and destination superblocks are different. Only return -EXDEV if
>> they have different fsid (the cluster ID).
>>
>> Signed-off-by: Luis Henriques <[email protected]>
>> ---
>> fs/ceph/file.c | 18 ++++++++++++++----
>> 1 file changed, 14 insertions(+), 4 deletions(-)
>>
>> Hi,
>>
>> Here's the patch changelog since initial submittion:
>>
>> - Dropped have_fsid checks on client structs
>> - Use %pU to print the fsid instead of raw hex strings (%*ph)
>> - Fixed 'To:' field in email so that this time the patch hits vger
>>
>> Cheers,
>> --
>> Luis
>>
>> diff --git a/fs/ceph/file.c b/fs/ceph/file.c
>> index 685a03cc4b77..4a624a1dd0bb 100644
>> --- a/fs/ceph/file.c
>> +++ b/fs/ceph/file.c
>> @@ -1904,6 +1904,7 @@ static ssize_t __ceph_copy_file_range(struct file *src_file, loff_t src_off,
>> struct ceph_inode_info *src_ci = ceph_inode(src_inode);
>> struct ceph_inode_info *dst_ci = ceph_inode(dst_inode);
>> struct ceph_cap_flush *prealloc_cf;
>> + struct ceph_fs_client *src_fsc = ceph_inode_to_client(src_inode);
>> struct ceph_object_locator src_oloc, dst_oloc;
>> struct ceph_object_id src_oid, dst_oid;
>> loff_t endoff = 0, size;
>> @@ -1915,8 +1916,17 @@ static ssize_t __ceph_copy_file_range(struct file *src_file, loff_t src_off,
>>
>> if (src_inode == dst_inode)
>> return -EINVAL;
>> - if (src_inode->i_sb != dst_inode->i_sb)
>> - return -EXDEV;
>> + if (src_inode->i_sb != dst_inode->i_sb) {
>> + struct ceph_fs_client *dst_fsc = ceph_inode_to_client(dst_inode);
>> +
>> + if (ceph_fsid_compare(&src_fsc->client->fsid,
>> + &dst_fsc->client->fsid)) {
>> + dout("Copying object across different clusters:");
>> + dout(" src fsid: %pU dst fsid: %pU\n",
>> + &src_fsc->client->fsid, &dst_fsc->client->fsid);
>> + return -EXDEV;
>> + }
>> + }
>
> Just to be clear: what happens here if I mount two entirely separate
> clusters, and their OSDs don't have any access to one another? Will this
> fail at some later point with an error that we can catch so that we can
> fall back?
This is exactly what this check prevents: if we have two CephFS from two
unrelated clusters mounted and we try to copy a file across them, the
operation will fail with -EXDEV[1] because the FSIDs for these two
ceph_fs_client will be different. OTOH, if these two filesystems are
within the same cluster (and thus with the same FSID), then the OSDs are
able to do 'copy-from' operations between them.
I've tested all these scenarios and they seem to be handled correctly.
Now, I'm assuming that *all* OSDs within the same ceph cluster can
communicate between themselves; if this assumption is false, then this
patch is broken. But again, I'm not aware of any mechanism that
prevents 2 OSDs from communicating between them.
[1] Actually, the files will still be copied because we'll fallback into
the default VFS generic_copy_file_range behaviour, which is to do
reads+writes operations.
Cheers,
--
Luis
>
>
>> if (ceph_snap(dst_inode) != CEPH_NOSNAP)
>> return -EROFS;
>>
>> @@ -1928,7 +1938,7 @@ static ssize_t __ceph_copy_file_range(struct file *src_file, loff_t src_off,
>> * efficient).
>> */
>>
>> - if (ceph_test_mount_opt(ceph_inode_to_client(src_inode), NOCOPYFROM))
>> + if (ceph_test_mount_opt(src_fsc, NOCOPYFROM))
>> return -EOPNOTSUPP;
>>
>> if ((src_ci->i_layout.stripe_unit != dst_ci->i_layout.stripe_unit) ||
>> @@ -2044,7 +2054,7 @@ static ssize_t __ceph_copy_file_range(struct file *src_file, loff_t src_off,
>> dst_ci->i_vino.ino, dst_objnum);
>> /* Do an object remote copy */
>> err = ceph_osdc_copy_from(
>> - &ceph_inode_to_client(src_inode)->client->osdc,
>> + &src_fsc->client->osdc,
>> src_ci->i_vino.snap, 0,
>> &src_oid, &src_oloc,
>> CEPH_OSD_OP_FLAG_FADVISE_SEQUENTIAL |
"Jeff Layton" <[email protected]> writes:
> On Mon, 2019-09-09 at 06:35 -0400, Jeff Layton wrote:
>> On Mon, 2019-09-09 at 11:28 +0100, Luis Henriques wrote:
>> > OSDs are able to perform object copies across different pools. Thus,
>> > there's no need to prevent copy_file_range from doing remote copies if the
>> > source and destination superblocks are different. Only return -EXDEV if
>> > they have different fsid (the cluster ID).
>> >
>> > Signed-off-by: Luis Henriques <[email protected]>
>> > ---
>> > fs/ceph/file.c | 18 ++++++++++++++----
>> > 1 file changed, 14 insertions(+), 4 deletions(-)
>> >
>> > Hi,
>> >
>> > Here's the patch changelog since initial submittion:
>> >
>> > - Dropped have_fsid checks on client structs
>> > - Use %pU to print the fsid instead of raw hex strings (%*ph)
>> > - Fixed 'To:' field in email so that this time the patch hits vger
>> >
>> > Cheers,
>> > --
>> > Luis
>> >
>> > diff --git a/fs/ceph/file.c b/fs/ceph/file.c
>> > index 685a03cc4b77..4a624a1dd0bb 100644
>> > --- a/fs/ceph/file.c
>> > +++ b/fs/ceph/file.c
>> > @@ -1904,6 +1904,7 @@ static ssize_t __ceph_copy_file_range(struct file *src_file, loff_t src_off,
>> > struct ceph_inode_info *src_ci = ceph_inode(src_inode);
>> > struct ceph_inode_info *dst_ci = ceph_inode(dst_inode);
>> > struct ceph_cap_flush *prealloc_cf;
>> > + struct ceph_fs_client *src_fsc = ceph_inode_to_client(src_inode);
>> > struct ceph_object_locator src_oloc, dst_oloc;
>> > struct ceph_object_id src_oid, dst_oid;
>> > loff_t endoff = 0, size;
>> > @@ -1915,8 +1916,17 @@ static ssize_t __ceph_copy_file_range(struct file *src_file, loff_t src_off,
>> >
>> > if (src_inode == dst_inode)
>> > return -EINVAL;
>> > - if (src_inode->i_sb != dst_inode->i_sb)
>> > - return -EXDEV;
>> > + if (src_inode->i_sb != dst_inode->i_sb) {
>> > + struct ceph_fs_client *dst_fsc = ceph_inode_to_client(dst_inode);
>> > +
>> > + if (ceph_fsid_compare(&src_fsc->client->fsid,
>> > + &dst_fsc->client->fsid)) {
>> > + dout("Copying object across different clusters:");
>> > + dout(" src fsid: %pU dst fsid: %pU\n",
>> > + &src_fsc->client->fsid, &dst_fsc->client->fsid);
>> > + return -EXDEV;
>> > + }
>> > + }
>>
>> Just to be clear: what happens here if I mount two entirely separate
>> clusters, and their OSDs don't have any access to one another? Will this
>> fail at some later point with an error that we can catch so that we can
>> fall back?
>>
>
> Duh, sorry I asked before I had a cup of coffee this morning. The whole
> point is to skip that case.
>
> That said...I wonder if it's possible to have an fsid collision across
> two separate clusters and this fail to catch that case? Aren't these
> things just allocated via a simple counter increment?
My understanding is that this is some sort of UUID. Looking at
doc/install/manual-deployment.rst it says that the fsid is a unique ID
that should be generated using uuidgen (I believe that's what vstart.sh
clusters use).
That said, it's obviously possible to reuse an fsid in two clusters.
And mounting both filesystems with the same fsid on the same client may
already cause some troubles without even trying to copy_file_range files
across them (for ex., fscache code seems to assume unique fsids). But I
have never tested such sort of things (probably no one did) and I really
don't know what are the consequences. In this specific case, I would
expect the 'copy-from' operation to fail with some error from the OSDs.
> Probably not worth worrying about overmuch, but might be good to
> understand what would happen in that case if only to field mailing list
> reports.
If there are concerns regarding this, I'm OK simply dropping the patch
for now and continue forbidding object copies when superblocks are
different. I just thought this was a low-hanging fruit, and didn't
realized that it's not very easy to ensure that 2 cephfs instances
actually belong to the same cluster. Maybe there are other checks that
could be done...?
Cheers,
--
Luis
> Other than that, this looks fine, modulo Ilya's comment about the two
> dout messages.
>
>>
>> > if (ceph_snap(dst_inode) != CEPH_NOSNAP)
>> > return -EROFS;
>> >
>> > @@ -1928,7 +1938,7 @@ static ssize_t __ceph_copy_file_range(struct file *src_file, loff_t src_off,
>> > * efficient).
>> > */
>> >
>> > - if (ceph_test_mount_opt(ceph_inode_to_client(src_inode), NOCOPYFROM))
>> > + if (ceph_test_mount_opt(src_fsc, NOCOPYFROM))
>> > return -EOPNOTSUPP;
>> >
>> > if ((src_ci->i_layout.stripe_unit != dst_ci->i_layout.stripe_unit) ||
>> > @@ -2044,7 +2054,7 @@ static ssize_t __ceph_copy_file_range(struct file *src_file, loff_t src_off,
>> > dst_ci->i_vino.ino, dst_objnum);
>> > /* Do an object remote copy */
>> > err = ceph_osdc_copy_from(
>> > - &ceph_inode_to_client(src_inode)->client->osdc,
>> > + &src_fsc->client->osdc,
>> > src_ci->i_vino.snap, 0,
>> > &src_oid, &src_oloc,
>> > CEPH_OSD_OP_FLAG_FADVISE_SEQUENTIAL |
On Mon, Sep 9, 2019 at 4:15 AM Luis Henriques <[email protected]> wrote:
>
> "Jeff Layton" <[email protected]> writes:
>
> > On Mon, 2019-09-09 at 11:28 +0100, Luis Henriques wrote:
> >> OSDs are able to perform object copies across different pools. Thus,
> >> there's no need to prevent copy_file_range from doing remote copies if the
> >> source and destination superblocks are different. Only return -EXDEV if
> >> they have different fsid (the cluster ID).
> >>
> >> Signed-off-by: Luis Henriques <[email protected]>
> >> ---
> >> fs/ceph/file.c | 18 ++++++++++++++----
> >> 1 file changed, 14 insertions(+), 4 deletions(-)
> >>
> >> Hi,
> >>
> >> Here's the patch changelog since initial submittion:
> >>
> >> - Dropped have_fsid checks on client structs
> >> - Use %pU to print the fsid instead of raw hex strings (%*ph)
> >> - Fixed 'To:' field in email so that this time the patch hits vger
> >>
> >> Cheers,
> >> --
> >> Luis
> >>
> >> diff --git a/fs/ceph/file.c b/fs/ceph/file.c
> >> index 685a03cc4b77..4a624a1dd0bb 100644
> >> --- a/fs/ceph/file.c
> >> +++ b/fs/ceph/file.c
> >> @@ -1904,6 +1904,7 @@ static ssize_t __ceph_copy_file_range(struct file *src_file, loff_t src_off,
> >> struct ceph_inode_info *src_ci = ceph_inode(src_inode);
> >> struct ceph_inode_info *dst_ci = ceph_inode(dst_inode);
> >> struct ceph_cap_flush *prealloc_cf;
> >> + struct ceph_fs_client *src_fsc = ceph_inode_to_client(src_inode);
> >> struct ceph_object_locator src_oloc, dst_oloc;
> >> struct ceph_object_id src_oid, dst_oid;
> >> loff_t endoff = 0, size;
> >> @@ -1915,8 +1916,17 @@ static ssize_t __ceph_copy_file_range(struct file *src_file, loff_t src_off,
> >>
> >> if (src_inode == dst_inode)
> >> return -EINVAL;
> >> - if (src_inode->i_sb != dst_inode->i_sb)
> >> - return -EXDEV;
> >> + if (src_inode->i_sb != dst_inode->i_sb) {
> >> + struct ceph_fs_client *dst_fsc = ceph_inode_to_client(dst_inode);
> >> +
> >> + if (ceph_fsid_compare(&src_fsc->client->fsid,
> >> + &dst_fsc->client->fsid)) {
> >> + dout("Copying object across different clusters:");
> >> + dout(" src fsid: %pU dst fsid: %pU\n",
> >> + &src_fsc->client->fsid, &dst_fsc->client->fsid);
> >> + return -EXDEV;
> >> + }
> >> + }
> >
> > Just to be clear: what happens here if I mount two entirely separate
> > clusters, and their OSDs don't have any access to one another? Will this
> > fail at some later point with an error that we can catch so that we can
> > fall back?
>
> This is exactly what this check prevents: if we have two CephFS from two
> unrelated clusters mounted and we try to copy a file across them, the
> operation will fail with -EXDEV[1] because the FSIDs for these two
> ceph_fs_client will be different. OTOH, if these two filesystems are
> within the same cluster (and thus with the same FSID), then the OSDs are
> able to do 'copy-from' operations between them.
>
> I've tested all these scenarios and they seem to be handled correctly.
> Now, I'm assuming that *all* OSDs within the same ceph cluster can
> communicate between themselves; if this assumption is false, then this
> patch is broken. But again, I'm not aware of any mechanism that
> prevents 2 OSDs from communicating between them.
Your assumption is correct: all OSDs in a Ceph cluster can communicate
with each other. I'm not aware of any plans to change this.
I spent a bit of time trying to figure out how this could break
security models and things and didn't come up with anything, so I
think functionally it's fine even though I find it a bit scary.
Also, yes, cluster FSIDs are UUIDs so they shouldn't collide.
-Greg
>
> [1] Actually, the files will still be copied because we'll fallback into
> the default VFS generic_copy_file_range behaviour, which is to do
> reads+writes operations.
>
> Cheers,
> --
> Luis
>
>
> >
> >
> >> if (ceph_snap(dst_inode) != CEPH_NOSNAP)
> >> return -EROFS;
> >>
> >> @@ -1928,7 +1938,7 @@ static ssize_t __ceph_copy_file_range(struct file *src_file, loff_t src_off,
> >> * efficient).
> >> */
> >>
> >> - if (ceph_test_mount_opt(ceph_inode_to_client(src_inode), NOCOPYFROM))
> >> + if (ceph_test_mount_opt(src_fsc, NOCOPYFROM))
> >> return -EOPNOTSUPP;
> >>
> >> if ((src_ci->i_layout.stripe_unit != dst_ci->i_layout.stripe_unit) ||
> >> @@ -2044,7 +2054,7 @@ static ssize_t __ceph_copy_file_range(struct file *src_file, loff_t src_off,
> >> dst_ci->i_vino.ino, dst_objnum);
> >> /* Do an object remote copy */
> >> err = ceph_osdc_copy_from(
> >> - &ceph_inode_to_client(src_inode)->client->osdc,
> >> + &src_fsc->client->osdc,
> >> src_ci->i_vino.snap, 0,
> >> &src_oid, &src_oloc,
> >> CEPH_OSD_OP_FLAG_FADVISE_SEQUENTIAL |
Gregory Farnum <[email protected]> writes:
> On Mon, Sep 9, 2019 at 4:15 AM Luis Henriques <[email protected]> wrote:
>>
>> "Jeff Layton" <[email protected]> writes:
>>
>> > On Mon, 2019-09-09 at 11:28 +0100, Luis Henriques wrote:
>> >> OSDs are able to perform object copies across different pools. Thus,
>> >> there's no need to prevent copy_file_range from doing remote copies if the
>> >> source and destination superblocks are different. Only return -EXDEV if
>> >> they have different fsid (the cluster ID).
>> >>
>> >> Signed-off-by: Luis Henriques <[email protected]>
>> >> ---
>> >> fs/ceph/file.c | 18 ++++++++++++++----
>> >> 1 file changed, 14 insertions(+), 4 deletions(-)
>> >>
>> >> Hi,
>> >>
>> >> Here's the patch changelog since initial submittion:
>> >>
>> >> - Dropped have_fsid checks on client structs
>> >> - Use %pU to print the fsid instead of raw hex strings (%*ph)
>> >> - Fixed 'To:' field in email so that this time the patch hits vger
>> >>
>> >> Cheers,
>> >> --
>> >> Luis
>> >>
>> >> diff --git a/fs/ceph/file.c b/fs/ceph/file.c
>> >> index 685a03cc4b77..4a624a1dd0bb 100644
>> >> --- a/fs/ceph/file.c
>> >> +++ b/fs/ceph/file.c
>> >> @@ -1904,6 +1904,7 @@ static ssize_t __ceph_copy_file_range(struct file *src_file, loff_t src_off,
>> >> struct ceph_inode_info *src_ci = ceph_inode(src_inode);
>> >> struct ceph_inode_info *dst_ci = ceph_inode(dst_inode);
>> >> struct ceph_cap_flush *prealloc_cf;
>> >> + struct ceph_fs_client *src_fsc = ceph_inode_to_client(src_inode);
>> >> struct ceph_object_locator src_oloc, dst_oloc;
>> >> struct ceph_object_id src_oid, dst_oid;
>> >> loff_t endoff = 0, size;
>> >> @@ -1915,8 +1916,17 @@ static ssize_t __ceph_copy_file_range(struct file *src_file, loff_t src_off,
>> >>
>> >> if (src_inode == dst_inode)
>> >> return -EINVAL;
>> >> - if (src_inode->i_sb != dst_inode->i_sb)
>> >> - return -EXDEV;
>> >> + if (src_inode->i_sb != dst_inode->i_sb) {
>> >> + struct ceph_fs_client *dst_fsc = ceph_inode_to_client(dst_inode);
>> >> +
>> >> + if (ceph_fsid_compare(&src_fsc->client->fsid,
>> >> + &dst_fsc->client->fsid)) {
>> >> + dout("Copying object across different clusters:");
>> >> + dout(" src fsid: %pU dst fsid: %pU\n",
>> >> + &src_fsc->client->fsid, &dst_fsc->client->fsid);
>> >> + return -EXDEV;
>> >> + }
>> >> + }
>> >
>> > Just to be clear: what happens here if I mount two entirely separate
>> > clusters, and their OSDs don't have any access to one another? Will this
>> > fail at some later point with an error that we can catch so that we can
>> > fall back?
>>
>> This is exactly what this check prevents: if we have two CephFS from two
>> unrelated clusters mounted and we try to copy a file across them, the
>> operation will fail with -EXDEV[1] because the FSIDs for these two
>> ceph_fs_client will be different. OTOH, if these two filesystems are
>> within the same cluster (and thus with the same FSID), then the OSDs are
>> able to do 'copy-from' operations between them.
>>
>> I've tested all these scenarios and they seem to be handled correctly.
>> Now, I'm assuming that *all* OSDs within the same ceph cluster can
>> communicate between themselves; if this assumption is false, then this
>> patch is broken. But again, I'm not aware of any mechanism that
>> prevents 2 OSDs from communicating between them.
>
> Your assumption is correct: all OSDs in a Ceph cluster can communicate
> with each other. I'm not aware of any plans to change this.
>
> I spent a bit of time trying to figure out how this could break
> security models and things and didn't come up with anything, so I
> think functionally it's fine even though I find it a bit scary.
>
> Also, yes, cluster FSIDs are UUIDs so they shouldn't collide.
Awesome, thanks for clarifying these points!
Cheers,
--
Luis
> -Greg
>
>>
>> [1] Actually, the files will still be copied because we'll fallback into
>> the default VFS generic_copy_file_range behaviour, which is to do
>> reads+writes operations.
>>
>> Cheers,
>> --
>> Luis
>>
>>
>> >
>> >
>> >> if (ceph_snap(dst_inode) != CEPH_NOSNAP)
>> >> return -EROFS;
>> >>
>> >> @@ -1928,7 +1938,7 @@ static ssize_t __ceph_copy_file_range(struct file *src_file, loff_t src_off,
>> >> * efficient).
>> >> */
>> >>
>> >> - if (ceph_test_mount_opt(ceph_inode_to_client(src_inode), NOCOPYFROM))
>> >> + if (ceph_test_mount_opt(src_fsc, NOCOPYFROM))
>> >> return -EOPNOTSUPP;
>> >>
>> >> if ((src_ci->i_layout.stripe_unit != dst_ci->i_layout.stripe_unit) ||
>> >> @@ -2044,7 +2054,7 @@ static ssize_t __ceph_copy_file_range(struct file *src_file, loff_t src_off,
>> >> dst_ci->i_vino.ino, dst_objnum);
>> >> /* Do an object remote copy */
>> >> err = ceph_osdc_copy_from(
>> >> - &ceph_inode_to_client(src_inode)->client->osdc,
>> >> + &src_fsc->client->osdc,
>> >> src_ci->i_vino.snap, 0,
>> >> &src_oid, &src_oloc,
>> >> CEPH_OSD_OP_FLAG_FADVISE_SEQUENTIAL |
>
On Mon, 2019-09-09 at 14:55 +0100, Luis Henriques wrote:
> "Jeff Layton" <[email protected]> writes:
>
> > On Mon, 2019-09-09 at 06:35 -0400, Jeff Layton wrote:
> > > On Mon, 2019-09-09 at 11:28 +0100, Luis Henriques wrote:
> > > > OSDs are able to perform object copies across different pools. Thus,
> > > > there's no need to prevent copy_file_range from doing remote copies if the
> > > > source and destination superblocks are different. Only return -EXDEV if
> > > > they have different fsid (the cluster ID).
> > > >
> > > > Signed-off-by: Luis Henriques <[email protected]>
> > > > ---
> > > > fs/ceph/file.c | 18 ++++++++++++++----
> > > > 1 file changed, 14 insertions(+), 4 deletions(-)
> > > >
> > > > Hi,
> > > >
> > > > Here's the patch changelog since initial submittion:
> > > >
> > > > - Dropped have_fsid checks on client structs
> > > > - Use %pU to print the fsid instead of raw hex strings (%*ph)
> > > > - Fixed 'To:' field in email so that this time the patch hits vger
> > > >
> > > > Cheers,
> > > > --
> > > > Luis
> > > >
> > > > diff --git a/fs/ceph/file.c b/fs/ceph/file.c
> > > > index 685a03cc4b77..4a624a1dd0bb 100644
> > > > --- a/fs/ceph/file.c
> > > > +++ b/fs/ceph/file.c
> > > > @@ -1904,6 +1904,7 @@ static ssize_t __ceph_copy_file_range(struct file *src_file, loff_t src_off,
> > > > struct ceph_inode_info *src_ci = ceph_inode(src_inode);
> > > > struct ceph_inode_info *dst_ci = ceph_inode(dst_inode);
> > > > struct ceph_cap_flush *prealloc_cf;
> > > > + struct ceph_fs_client *src_fsc = ceph_inode_to_client(src_inode);
> > > > struct ceph_object_locator src_oloc, dst_oloc;
> > > > struct ceph_object_id src_oid, dst_oid;
> > > > loff_t endoff = 0, size;
> > > > @@ -1915,8 +1916,17 @@ static ssize_t __ceph_copy_file_range(struct file *src_file, loff_t src_off,
> > > >
> > > > if (src_inode == dst_inode)
> > > > return -EINVAL;
> > > > - if (src_inode->i_sb != dst_inode->i_sb)
> > > > - return -EXDEV;
> > > > + if (src_inode->i_sb != dst_inode->i_sb) {
> > > > + struct ceph_fs_client *dst_fsc = ceph_inode_to_client(dst_inode);
> > > > +
> > > > + if (ceph_fsid_compare(&src_fsc->client->fsid,
> > > > + &dst_fsc->client->fsid)) {
> > > > + dout("Copying object across different clusters:");
> > > > + dout(" src fsid: %pU dst fsid: %pU\n",
> > > > + &src_fsc->client->fsid, &dst_fsc->client->fsid);
> > > > + return -EXDEV;
> > > > + }
> > > > + }
> > >
> > > Just to be clear: what happens here if I mount two entirely separate
> > > clusters, and their OSDs don't have any access to one another? Will this
> > > fail at some later point with an error that we can catch so that we can
> > > fall back?
> > >
> >
> > Duh, sorry I asked before I had a cup of coffee this morning. The whole
> > point is to skip that case.
> >
> > That said...I wonder if it's possible to have an fsid collision across
> > two separate clusters and this fail to catch that case? Aren't these
> > things just allocated via a simple counter increment?
>
> My understanding is that this is some sort of UUID. Looking at
> doc/install/manual-deployment.rst it says that the fsid is a unique ID
> that should be generated using uuidgen (I believe that's what vstart.sh
> clusters use).
>
> That said, it's obviously possible to reuse an fsid in two clusters.
> And mounting both filesystems with the same fsid on the same client may
> already cause some troubles without even trying to copy_file_range files
> across them (for ex., fscache code seems to assume unique fsids). But I
> have never tested such sort of things (probably no one did) and I really
> don't know what are the consequences. In this specific case, I would
> expect the 'copy-from' operation to fail with some error from the OSDs.
>
Makes sense. I suppose the worst possible case is data corruption due to
copying to/from the wrong object, but the risk here seems quite low.
> > Probably not worth worrying about overmuch, but might be good to
> > understand what would happen in that case if only to field mailing list
> > reports.
>
> If there are concerns regarding this, I'm OK simply dropping the patch
> for now and continue forbidding object copies when superblocks are
> different. I just thought this was a low-hanging fruit, and didn't
> realized that it's not very easy to ensure that 2 cephfs instances
> actually belong to the same cluster. Maybe there are other checks that
> could be done...?
>
I'm not really concerned about it, particularly if these values are
usually generated as uuids. If we get reports that involve collisions
here, then we can revisit it then.
IMO, it's up to the admin to guarantee that the fsid is unique within a
multi-cluster environment.
--
Jeff Layton <[email protected]>