Hi!
I started seeing fstest generic/123 failing in ceph fscrypt, when running it
with 'test_dummy_encryption'. This test is quite simple:
1. Creates a directory with write permissions for root only
2. Writes into a file in that directory
3. Uses 'su' to try to modify that file as a different user, and
gets -EPERM
All the test steps succeed, but the test fails to cleanup: 'rm -rf <dir>'
will fail with -ENOTEMPTY. 'strace' shows that calling unlinkat() to remove
the file got a -ENOENT and then -ENOTEMPTY for the directory.
This is because 'su' does a drop_caches ('su (874): drop_caches: 2' in
dmesg), and ceph's atomic open will do:
if (IS_ENCRYPTED(dir)) {
set_bit(CEPH_MDS_R_FSCRYPT_FILE, &req->r_req_flags);
if (!fscrypt_has_encryption_key(dir)) {
spin_lock(&dentry->d_lock);
dentry->d_flags |= DCACHE_NOKEY_NAME;
spin_unlock(&dentry->d_lock);
}
}
Although 'dir' has the encryption key available, fscrypt_has_encryption_key()
will return 'false' because fscrypt info isn't yet set after the cache
cleanup.
The first patch will add a new helper for the atomic_open that will force
the fscrypt info to be loaded into an inode that has been evicted recently
but for which the key is still available.
The second patch switches ceph atomic_open to use the new fscrypt helper.
Cheers,
--
Luís
Changes since initial RFC (after Eric's review):
- Added kerneldoc comments to the new fscrypt helper
- Dropped '__' from helper name (now fscrypt_prepare_atomic_open())
- Added IS_ENCRYPTED() check in helper
- DCACHE_NOKEY_NAME is not set if fscrypt_get_encryption_info() returns an
error
- Fixed helper for !CONFIG_FS_ENCRYPTION (now defined 'static inline')
Luís Henriques (2):
fscrypt: new helper function - fscrypt_prepare_atomic_open()
ceph: switch atomic open to use new fscrypt helper
fs/ceph/file.c | 8 +++-----
fs/crypto/hooks.c | 35 +++++++++++++++++++++++++++++++++++
include/linux/fscrypt.h | 7 +++++++
3 files changed, 45 insertions(+), 5 deletions(-)
This patch introduces a new helper function which prepares an atomic_open.
Because atomic open can act as a lookup if handed a dentry that is negative,
we need to set DCACHE_NOKEY_NAME if the key for the parent isn't available.
The reason for getting the encryption info before checking if the directory
has the encryption key is because we may have the key available but the
encryption info isn't yet set (maybe due to a drop_caches). The regular
open path will use fscrypt_file_open for that but in the atomic open a
different approach is required.
Signed-off-by: Luís Henriques <[email protected]>
---
fs/crypto/hooks.c | 35 +++++++++++++++++++++++++++++++++++
include/linux/fscrypt.h | 7 +++++++
2 files changed, 42 insertions(+)
diff --git a/fs/crypto/hooks.c b/fs/crypto/hooks.c
index 7b8c5a1104b5..8be1e35984f1 100644
--- a/fs/crypto/hooks.c
+++ b/fs/crypto/hooks.c
@@ -117,6 +117,41 @@ int __fscrypt_prepare_readdir(struct inode *dir)
}
EXPORT_SYMBOL_GPL(__fscrypt_prepare_readdir);
+/**
+ * fscrypt_prepare_atomic_open() - prepare an atomic open on an encrypted directory
+ * @dir: inode of parent directory
+ * @dentry: dentry being open
+ *
+ * Because atomic open can act as a lookup if handed a dentry that is negative,
+ * we need to set DCACHE_NOKEY_NAME if the key for the parent isn't available.
+ *
+ * The reason for getting the encryption info before checking if the directory
+ * has the encryption key is because the key may be available but the encryption
+ * info isn't yet set (maybe due to a drop_caches). The regular open path will
+ * use fscrypt_file_open for that, but in the atomic open a different approach
+ * is required.
+ *
+ * Return: 0 on success, or an error code if fscrypt_get_encryption_info()
+ * fails.
+ */
+int fscrypt_prepare_atomic_open(struct inode *dir, struct dentry *dentry)
+{
+ int err;
+
+ if (!IS_ENCRYPTED(dir))
+ return 0;
+
+ err = fscrypt_get_encryption_info(dir, true);
+ if (!err && !fscrypt_has_encryption_key(dir)) {
+ spin_lock(&dentry->d_lock);
+ dentry->d_flags |= DCACHE_NOKEY_NAME;
+ spin_unlock(&dentry->d_lock);
+ }
+
+ return err;
+}
+EXPORT_SYMBOL_GPL(fscrypt_prepare_atomic_open);
+
int __fscrypt_prepare_setattr(struct dentry *dentry, struct iattr *attr)
{
if (attr->ia_valid & ATTR_SIZE)
diff --git a/include/linux/fscrypt.h b/include/linux/fscrypt.h
index 4f5f8a651213..c70acb2a737a 100644
--- a/include/linux/fscrypt.h
+++ b/include/linux/fscrypt.h
@@ -362,6 +362,7 @@ int __fscrypt_prepare_rename(struct inode *old_dir, struct dentry *old_dentry,
int __fscrypt_prepare_lookup(struct inode *dir, struct dentry *dentry,
struct fscrypt_name *fname);
int __fscrypt_prepare_readdir(struct inode *dir);
+int fscrypt_prepare_atomic_open(struct inode *dir, struct dentry *dentry);
int __fscrypt_prepare_setattr(struct dentry *dentry, struct iattr *attr);
int fscrypt_prepare_setflags(struct inode *inode,
unsigned int oldflags, unsigned int flags);
@@ -688,6 +689,12 @@ static inline int __fscrypt_prepare_readdir(struct inode *dir)
return -EOPNOTSUPP;
}
+static inline int fscrypt_prepare_atomic_open(struct inode *dir,
+ struct dentry *dentry)
+{
+ return -EOPNOTSUPP;
+}
+
static inline int __fscrypt_prepare_setattr(struct dentry *dentry,
struct iattr *attr)
{
On Mon, 2023-03-13 at 12:33 +0000, Lu?s Henriques wrote:
> Hi!
>
> I started seeing fstest generic/123 failing in ceph fscrypt, when running it
> with 'test_dummy_encryption'. This test is quite simple:
>
> 1. Creates a directory with write permissions for root only
> 2. Writes into a file in that directory
> 3. Uses 'su' to try to modify that file as a different user, and
> gets -EPERM
>
> All the test steps succeed, but the test fails to cleanup: 'rm -rf <dir>'
> will fail with -ENOTEMPTY. 'strace' shows that calling unlinkat() to remove
> the file got a -ENOENT and then -ENOTEMPTY for the directory.
>
> This is because 'su' does a drop_caches ('su (874): drop_caches: 2' in
> dmesg), and ceph's atomic open will do:
>
> if (IS_ENCRYPTED(dir)) {
> set_bit(CEPH_MDS_R_FSCRYPT_FILE, &req->r_req_flags);
> if (!fscrypt_has_encryption_key(dir)) {
> spin_lock(&dentry->d_lock);
> dentry->d_flags |= DCACHE_NOKEY_NAME;
> spin_unlock(&dentry->d_lock);
> }
> }
>
> Although 'dir' has the encryption key available, fscrypt_has_encryption_key()
> will return 'false' because fscrypt info isn't yet set after the cache
> cleanup.
>
> The first patch will add a new helper for the atomic_open that will force
> the fscrypt info to be loaded into an inode that has been evicted recently
> but for which the key is still available.
>
> The second patch switches ceph atomic_open to use the new fscrypt helper.
>
> Cheers,
> --
> Lu?s
>
> Changes since initial RFC (after Eric's review):
> - Added kerneldoc comments to the new fscrypt helper
> - Dropped '__' from helper name (now fscrypt_prepare_atomic_open())
> - Added IS_ENCRYPTED() check in helper
> - DCACHE_NOKEY_NAME is not set if fscrypt_get_encryption_info() returns an
> error
> - Fixed helper for !CONFIG_FS_ENCRYPTION (now defined 'static inline')
>
> Lu?s Henriques (2):
> fscrypt: new helper function - fscrypt_prepare_atomic_open()
> ceph: switch atomic open to use new fscrypt helper
>
> fs/ceph/file.c | 8 +++-----
> fs/crypto/hooks.c | 35 +++++++++++++++++++++++++++++++++++
> include/linux/fscrypt.h | 7 +++++++
> 3 files changed, 45 insertions(+), 5 deletions(-)
>
Looks like a nice cleanup.
Reviewed-by: Jeff Layton <[email protected]>
On Mon, Mar 13, 2023 at 12:33:09PM +0000, Lu?s Henriques wrote:
> + * The regular open path will use fscrypt_file_open for that, but in the
> + * atomic open a different approach is required.
This should actually be fscrypt_prepare_lookup, not fscrypt_file_open, right?
> +int fscrypt_prepare_atomic_open(struct inode *dir, struct dentry *dentry)
> +{
> + int err;
> +
> + if (!IS_ENCRYPTED(dir))
> + return 0;
> +
> + err = fscrypt_get_encryption_info(dir, true);
> + if (!err && !fscrypt_has_encryption_key(dir)) {
> + spin_lock(&dentry->d_lock);
> + dentry->d_flags |= DCACHE_NOKEY_NAME;
> + spin_unlock(&dentry->d_lock);
> + }
> +
> + return err;
> +}
> +EXPORT_SYMBOL_GPL(fscrypt_prepare_atomic_open);
[...]
> +static inline int fscrypt_prepare_atomic_open(struct inode *dir,
> + struct dentry *dentry)
> +{
> + return -EOPNOTSUPP;
> +}
This has different behavior on unencrypted directories depending on whether
CONFIG_FS_ENCRYPTION is enabled or not. That's bad.
In patch 2, the caller you are introducing has already checked IS_ENCRYPTED().
Also, your kerneldoc comment for fscrypt_prepare_atomic_open() says it is for
*encrypted* directories.
So IMO, just remove the IS_ENCRYPTED() check from the CONFIG_FS_ENCRYPTION
version of fscrypt_prepare_atomic_open().
- Eric
On 14/03/2023 02:09, Eric Biggers wrote:
> On Mon, Mar 13, 2023 at 12:33:09PM +0000, Luís Henriques wrote:
>> + * The regular open path will use fscrypt_file_open for that, but in the
>> + * atomic open a different approach is required.
> This should actually be fscrypt_prepare_lookup, not fscrypt_file_open, right?
>
>> +int fscrypt_prepare_atomic_open(struct inode *dir, struct dentry *dentry)
>> +{
>> + int err;
>> +
>> + if (!IS_ENCRYPTED(dir))
>> + return 0;
>> +
>> + err = fscrypt_get_encryption_info(dir, true);
>> + if (!err && !fscrypt_has_encryption_key(dir)) {
>> + spin_lock(&dentry->d_lock);
>> + dentry->d_flags |= DCACHE_NOKEY_NAME;
>> + spin_unlock(&dentry->d_lock);
>> + }
>> +
>> + return err;
>> +}
>> +EXPORT_SYMBOL_GPL(fscrypt_prepare_atomic_open);
> [...]
>> +static inline int fscrypt_prepare_atomic_open(struct inode *dir,
>> + struct dentry *dentry)
>> +{
>> + return -EOPNOTSUPP;
>> +}
> This has different behavior on unencrypted directories depending on whether
> CONFIG_FS_ENCRYPTION is enabled or not. That's bad.
>
> In patch 2, the caller you are introducing has already checked IS_ENCRYPTED().
>
> Also, your kerneldoc comment for fscrypt_prepare_atomic_open() says it is for
> *encrypted* directories.
>
> So IMO, just remove the IS_ENCRYPTED() check from the CONFIG_FS_ENCRYPTION
> version of fscrypt_prepare_atomic_open().
IMO we should keep this check in fscrypt_prepare_atomic_open() to make
it consistent with the existing fscrypt_prepare_open(). And we can just
remove the check from ceph instead.
- Xiubo
> - Eric
>
--
Best Regards,
Xiubo Li (李秀波)
Email: [email protected]/[email protected]
Slack: @Xiubo Li
On Tue, Mar 14, 2023 at 08:53:51AM +0800, Xiubo Li wrote:
>
> On 14/03/2023 02:09, Eric Biggers wrote:
> > On Mon, Mar 13, 2023 at 12:33:09PM +0000, Lu?s Henriques wrote:
> > > + * The regular open path will use fscrypt_file_open for that, but in the
> > > + * atomic open a different approach is required.
> > This should actually be fscrypt_prepare_lookup, not fscrypt_file_open, right?
> >
> > > +int fscrypt_prepare_atomic_open(struct inode *dir, struct dentry *dentry)
> > > +{
> > > + int err;
> > > +
> > > + if (!IS_ENCRYPTED(dir))
> > > + return 0;
> > > +
> > > + err = fscrypt_get_encryption_info(dir, true);
> > > + if (!err && !fscrypt_has_encryption_key(dir)) {
> > > + spin_lock(&dentry->d_lock);
> > > + dentry->d_flags |= DCACHE_NOKEY_NAME;
> > > + spin_unlock(&dentry->d_lock);
> > > + }
> > > +
> > > + return err;
> > > +}
> > > +EXPORT_SYMBOL_GPL(fscrypt_prepare_atomic_open);
> > [...]
> > > +static inline int fscrypt_prepare_atomic_open(struct inode *dir,
> > > + struct dentry *dentry)
> > > +{
> > > + return -EOPNOTSUPP;
> > > +}
> > This has different behavior on unencrypted directories depending on whether
> > CONFIG_FS_ENCRYPTION is enabled or not. That's bad.
> >
> > In patch 2, the caller you are introducing has already checked IS_ENCRYPTED().
> >
> > Also, your kerneldoc comment for fscrypt_prepare_atomic_open() says it is for
> > *encrypted* directories.
> >
> > So IMO, just remove the IS_ENCRYPTED() check from the CONFIG_FS_ENCRYPTION
> > version of fscrypt_prepare_atomic_open().
>
> IMO we should keep this check in fscrypt_prepare_atomic_open() to make it
> consistent with the existing fscrypt_prepare_open(). And we can just remove
> the check from ceph instead.
>
Well, then the !CONFIG_FS_ENCRYPTION version would need to return 0 if
IS_ENCRYPTED() too.
Either way would be okay, but please don't do a mix of both approaches within a
single function, as this patch currently does.
Note that there are other fscrypt_* functions, such as fscrypt_get_symlink(),
that require an IS_ENCRYPTED() inode, so that pattern is not new.
- Eric
On 14/03/2023 10:25, Eric Biggers wrote:
> On Tue, Mar 14, 2023 at 08:53:51AM +0800, Xiubo Li wrote:
>> On 14/03/2023 02:09, Eric Biggers wrote:
>>> On Mon, Mar 13, 2023 at 12:33:09PM +0000, Luís Henriques wrote:
>>>> + * The regular open path will use fscrypt_file_open for that, but in the
>>>> + * atomic open a different approach is required.
>>> This should actually be fscrypt_prepare_lookup, not fscrypt_file_open, right?
>>>
>>>> +int fscrypt_prepare_atomic_open(struct inode *dir, struct dentry *dentry)
>>>> +{
>>>> + int err;
>>>> +
>>>> + if (!IS_ENCRYPTED(dir))
>>>> + return 0;
>>>> +
>>>> + err = fscrypt_get_encryption_info(dir, true);
>>>> + if (!err && !fscrypt_has_encryption_key(dir)) {
>>>> + spin_lock(&dentry->d_lock);
>>>> + dentry->d_flags |= DCACHE_NOKEY_NAME;
>>>> + spin_unlock(&dentry->d_lock);
>>>> + }
>>>> +
>>>> + return err;
>>>> +}
>>>> +EXPORT_SYMBOL_GPL(fscrypt_prepare_atomic_open);
>>> [...]
>>>> +static inline int fscrypt_prepare_atomic_open(struct inode *dir,
>>>> + struct dentry *dentry)
>>>> +{
>>>> + return -EOPNOTSUPP;
>>>> +}
>>> This has different behavior on unencrypted directories depending on whether
>>> CONFIG_FS_ENCRYPTION is enabled or not. That's bad.
>>>
>>> In patch 2, the caller you are introducing has already checked IS_ENCRYPTED().
>>>
>>> Also, your kerneldoc comment for fscrypt_prepare_atomic_open() says it is for
>>> *encrypted* directories.
>>>
>>> So IMO, just remove the IS_ENCRYPTED() check from the CONFIG_FS_ENCRYPTION
>>> version of fscrypt_prepare_atomic_open().
>> IMO we should keep this check in fscrypt_prepare_atomic_open() to make it
>> consistent with the existing fscrypt_prepare_open(). And we can just remove
>> the check from ceph instead.
>>
> Well, then the !CONFIG_FS_ENCRYPTION version would need to return 0 if
> IS_ENCRYPTED() too.
For the !CONFIG_FS_ENCRYPTION version I think you mean:
static inline int fscrypt_prepare_atomic_open(struct inode *dir,
struct dentry *dentry)
{
if (IS_ENCRYPTED(dir))
return -EOPNOTSUPP;
return 0;
}
> Either way would be okay, but please don't do a mix of both approaches within a
> single function, as this patch currently does.
>
> Note that there are other fscrypt_* functions, such as fscrypt_get_symlink(),
> that require an IS_ENCRYPTED() inode, so that pattern is not new.
Yeah, correct, I didn't notice that.
- Xiubo
> - Eric
>
--
Best Regards,
Xiubo Li (李秀波)
Email: [email protected]/[email protected]
Slack: @Xiubo Li
Xiubo Li <[email protected]> writes:
> On 14/03/2023 10:25, Eric Biggers wrote:
>> On Tue, Mar 14, 2023 at 08:53:51AM +0800, Xiubo Li wrote:
>>> On 14/03/2023 02:09, Eric Biggers wrote:
>>>> On Mon, Mar 13, 2023 at 12:33:09PM +0000, Luís Henriques wrote:
>>>>> + * The regular open path will use fscrypt_file_open for that, but in the
>>>>> + * atomic open a different approach is required.
>>>> This should actually be fscrypt_prepare_lookup, not fscrypt_file_open, right?
>>>>
>>>>> +int fscrypt_prepare_atomic_open(struct inode *dir, struct dentry *dentry)
>>>>> +{
>>>>> + int err;
>>>>> +
>>>>> + if (!IS_ENCRYPTED(dir))
>>>>> + return 0;
>>>>> +
>>>>> + err = fscrypt_get_encryption_info(dir, true);
>>>>> + if (!err && !fscrypt_has_encryption_key(dir)) {
>>>>> + spin_lock(&dentry->d_lock);
>>>>> + dentry->d_flags |= DCACHE_NOKEY_NAME;
>>>>> + spin_unlock(&dentry->d_lock);
>>>>> + }
>>>>> +
>>>>> + return err;
>>>>> +}
>>>>> +EXPORT_SYMBOL_GPL(fscrypt_prepare_atomic_open);
>>>> [...]
>>>>> +static inline int fscrypt_prepare_atomic_open(struct inode *dir,
>>>>> + struct dentry *dentry)
>>>>> +{
>>>>> + return -EOPNOTSUPP;
>>>>> +}
>>>> This has different behavior on unencrypted directories depending on whether
>>>> CONFIG_FS_ENCRYPTION is enabled or not. That's bad.
>>>>
>>>> In patch 2, the caller you are introducing has already checked IS_ENCRYPTED().
>>>>
>>>> Also, your kerneldoc comment for fscrypt_prepare_atomic_open() says it is for
>>>> *encrypted* directories.
>>>>
>>>> So IMO, just remove the IS_ENCRYPTED() check from the CONFIG_FS_ENCRYPTION
>>>> version of fscrypt_prepare_atomic_open().
>>> IMO we should keep this check in fscrypt_prepare_atomic_open() to make it
>>> consistent with the existing fscrypt_prepare_open(). And we can just remove
>>> the check from ceph instead.
>>>
>> Well, then the !CONFIG_FS_ENCRYPTION version would need to return 0 if
>> IS_ENCRYPTED() too.
>
> For the !CONFIG_FS_ENCRYPTION version I think you mean:
>
> static inline int fscrypt_prepare_atomic_open(struct inode *dir, struct dentry
> *dentry)
>
> {
> if (IS_ENCRYPTED(dir))
> return -EOPNOTSUPP;
> return 0;
> }
>
>
>> Either way would be okay, but please don't do a mix of both approaches within a
>> single function, as this patch currently does.
>>
>> Note that there are other fscrypt_* functions, such as fscrypt_get_symlink(),
>> that require an IS_ENCRYPTED() inode, so that pattern is not new.
>
> Yeah, correct, I didn't notice that.
OK, thank you both for the feedback. I'll send out v2 in a few hours.
But my preference will be to drop the IS_ENCRYPTED() from
fscrypt_prepare_atomic_open(). The reason is that we still need to keep
it in the caller function anyway, because we need to set the MDS flags
accordingly (see patch 2):
if (IS_ENCRYPTED(dir)) {
set_bit(CEPH_MDS_R_FSCRYPT_FILE, &req->r_req_flags);
err = fscrypt_prepare_atomic_open(dir, dentry);
if (err)
goto out_req;
}
Cheers,
--
Luís
Eric Biggers <[email protected]> writes:
> On Mon, Mar 13, 2023 at 12:33:09PM +0000, Luís Henriques wrote:
>> + * The regular open path will use fscrypt_file_open for that, but in the
>> + * atomic open a different approach is required.
>
> This should actually be fscrypt_prepare_lookup, not fscrypt_file_open, right?
Ups, I missed this comment.
I was comparing the regular open() with the atomic_open() paths. I think
I really mean fscrypt_file_open() because that's where the encryption info
is (or may be) set by calling fscrypt_require_key(). atomic_open needs
something similar, but combined with a lookup.
Maybe I can rephrase it to:
The reason for getting the encryption info before checking if the
directory has the encryption key is because the key may be available but
the encryption info isn't yet set (maybe due to a drop_caches). The
regular open path will call fscrypt_file_open which uses function
fscrypt_require_key for setting the encryption info if needed. The
atomic open needs to do something similar.
Cheers,
--
Luís
>> +int fscrypt_prepare_atomic_open(struct inode *dir, struct dentry *dentry)
>> +{
>> + int err;
>> +
>> + if (!IS_ENCRYPTED(dir))
>> + return 0;
>> +
>> + err = fscrypt_get_encryption_info(dir, true);
>> + if (!err && !fscrypt_has_encryption_key(dir)) {
>> + spin_lock(&dentry->d_lock);
>> + dentry->d_flags |= DCACHE_NOKEY_NAME;
>> + spin_unlock(&dentry->d_lock);
>> + }
>> +
>> + return err;
>> +}
>> +EXPORT_SYMBOL_GPL(fscrypt_prepare_atomic_open);
> [...]
>> +static inline int fscrypt_prepare_atomic_open(struct inode *dir,
>> + struct dentry *dentry)
>> +{
>> + return -EOPNOTSUPP;
>> +}
>
> This has different behavior on unencrypted directories depending on whether
> CONFIG_FS_ENCRYPTION is enabled or not. That's bad.
>
> In patch 2, the caller you are introducing has already checked IS_ENCRYPTED().
>
> Also, your kerneldoc comment for fscrypt_prepare_atomic_open() says it is for
> *encrypted* directories.
>
> So IMO, just remove the IS_ENCRYPTED() check from the CONFIG_FS_ENCRYPTION
> version of fscrypt_prepare_atomic_open().
>
> - Eric
On Tue, Mar 14, 2023 at 10:15:11AM +0000, Lu?s Henriques wrote:
> Eric Biggers <[email protected]> writes:
>
> > On Mon, Mar 13, 2023 at 12:33:09PM +0000, Lu?s Henriques wrote:
> >> + * The regular open path will use fscrypt_file_open for that, but in the
> >> + * atomic open a different approach is required.
> >
> > This should actually be fscrypt_prepare_lookup, not fscrypt_file_open, right?
>
> Ups, I missed this comment.
>
> I was comparing the regular open() with the atomic_open() paths. I think
> I really mean fscrypt_file_open() because that's where the encryption info
> is (or may be) set by calling fscrypt_require_key(). atomic_open needs
> something similar, but combined with a lookup.
>
> Maybe I can rephrase it to:
>
> The reason for getting the encryption info before checking if the
> directory has the encryption key is because the key may be available but
> the encryption info isn't yet set (maybe due to a drop_caches). The
> regular open path will call fscrypt_file_open which uses function
> fscrypt_require_key for setting the encryption info if needed. The
> atomic open needs to do something similar.
>
No, regular open is two parts: ->lookup and ->open. fscrypt_prepare_lookup()
sets up the directory's key, whereas fscrypt_file_open() sets up the file's key.
Your proposed fscrypt_prepare_atomic_open() sets up the directory's key. So it
is really fscrypt_prepare_lookup() that is its equivalent.
However, that raises the question of why doesn't ceph just use
fscrypt_prepare_lookup()? It seems the answer is that ceph wants to handle the
filenames encryption and no-key name encoding itself. And for that reason, its
->lookup() does the following and does *not* use fscrypt_prepare_lookup():
if (IS_ENCRYPTED(dir)) {
err = ceph_fscrypt_prepare_readdir(dir);
if (err < 0)
return ERR_PTR(err);
if (!fscrypt_has_encryption_key(dir)) {
spin_lock(&dentry->d_lock);
dentry->d_flags |= DCACHE_NOKEY_NAME;
spin_unlock(&dentry->d_lock);
}
}
So, actually I think this patch doesn't make sense. If ceph is doing the above
in its ->lookup() anyway, then it just should do the exact same thing in its
->atomic_open() too.
If you want to add a new fscrypt_* helper function which *just* sets up the
given directory's key and sets the NOKEY_NAME flag on the given dentry
accordingly, that could make sense. However, it should be called from *both*
->lookup() and ->atomic_open(), not just ->atomic_open().
It's also worth mentioning that setting up the filename separately from the
NOKEY_NAME flag makes ceph have the same race condition that I had fixed for the
other filesystems in commit b01531db6cec ("fscrypt: fix race where ->lookup()
marks plaintext dentry as ciphertext"). It's not a huge deal, but it can cause
some odd behavior, so it's worth thinking about whether it can be solved.
- Eric
Eric Biggers <[email protected]> writes:
> On Tue, Mar 14, 2023 at 10:15:11AM +0000, Luís Henriques wrote:
>> Eric Biggers <[email protected]> writes:
>>
>> > On Mon, Mar 13, 2023 at 12:33:09PM +0000, Luís Henriques wrote:
>> >> + * The regular open path will use fscrypt_file_open for that, but in the
>> >> + * atomic open a different approach is required.
>> >
>> > This should actually be fscrypt_prepare_lookup, not fscrypt_file_open, right?
>>
>> Ups, I missed this comment.
>>
>> I was comparing the regular open() with the atomic_open() paths. I think
>> I really mean fscrypt_file_open() because that's where the encryption info
>> is (or may be) set by calling fscrypt_require_key(). atomic_open needs
>> something similar, but combined with a lookup.
>>
>> Maybe I can rephrase it to:
>>
>> The reason for getting the encryption info before checking if the
>> directory has the encryption key is because the key may be available but
>> the encryption info isn't yet set (maybe due to a drop_caches). The
>> regular open path will call fscrypt_file_open which uses function
>> fscrypt_require_key for setting the encryption info if needed. The
>> atomic open needs to do something similar.
>>
>
> No, regular open is two parts: ->lookup and ->open. fscrypt_prepare_lookup()
> sets up the directory's key, whereas fscrypt_file_open() sets up the file's key.
>
> Your proposed fscrypt_prepare_atomic_open() sets up the directory's key. So it
> is really fscrypt_prepare_lookup() that is its equivalent.
Oh, I see what you mean now, and you're obviously correct. Thanks for the
detailed explanation.
> However, that raises the question of why doesn't ceph just use
> fscrypt_prepare_lookup()? It seems the answer is that ceph wants to handle the
> filenames encryption and no-key name encoding itself. And for that reason, its
> ->lookup() does the following and does *not* use fscrypt_prepare_lookup():
>
> if (IS_ENCRYPTED(dir)) {
> err = ceph_fscrypt_prepare_readdir(dir);
> if (err < 0)
> return ERR_PTR(err);
> if (!fscrypt_has_encryption_key(dir)) {
> spin_lock(&dentry->d_lock);
> dentry->d_flags |= DCACHE_NOKEY_NAME;
> spin_unlock(&dentry->d_lock);
> }
> }
Ugh, I tend to forget all the details behind these decisions. If I
remember correctly, we had to work around the fact that the cephfs client
handle directory data in a cumbersome way. We may not have the full data
for a readdir, for example, and that has to be handled by a lookup.
> So, actually I think this patch doesn't make sense. If ceph is doing the above
> in its ->lookup() anyway, then it just should do the exact same thing in its
> ->atomic_open() too.
In fact, my initial fix for the cephfs bug was doing just that. It was a
single patch to ceph_atomic_open() that would simply do:
if (IS_ENCRYPTED(dir)) {
set_bit(CEPH_MDS_R_FSCRYPT_FILE, &req->r_req_flags);
err = __fscrypt_prepare_readdir(dir);
if (!err && !fscrypt_has_encryption_key(dir)) {
spin_lock(&dentry->d_lock);
dentry->d_flags |= DCACHE_NOKEY_NAME;
spin_unlock(&dentry->d_lock);
}
}
What made me want to create a new helper was that I simply needed to call
fscrypt_get_encryption_info() to force the encryption info to be set in
the parent directory. But this function was only accessible through
__fscrypt_prepare_readdir(), which isn't really a great function name for
what I need here.
Since __fscrypt_prepare_readdir() doesn't seem to be used anywhere else,
maybe it could be removed and fscrypt_get_encryption_info() be exported
instead?
> If you want to add a new fscrypt_* helper function which *just* sets up the
> given directory's key and sets the NOKEY_NAME flag on the given dentry
> accordingly, that could make sense. However, it should be called from *both*
> ->lookup() and ->atomic_open(), not just ->atomic_open().
>
> It's also worth mentioning that setting up the filename separately from the
> NOKEY_NAME flag makes ceph have the same race condition that I had fixed for the
> other filesystems in commit b01531db6cec ("fscrypt: fix race where ->lookup()
> marks plaintext dentry as ciphertext"). It's not a huge deal, but it can cause
> some odd behavior, so it's worth thinking about whether it can be solved.
Hmm... OK, looks like we'll need to have a look into this. Thanks for the
heads-up.
Cheers,
--
Luís
On Wed, Mar 15, 2023 at 11:08:23AM +0000, Lu?s Henriques wrote:
> > So, actually I think this patch doesn't make sense. If ceph is doing the above
> > in its ->lookup() anyway, then it just should do the exact same thing in its
> > ->atomic_open() too.
>
> In fact, my initial fix for the cephfs bug was doing just that. It was a
> single patch to ceph_atomic_open() that would simply do:
>
> if (IS_ENCRYPTED(dir)) {
> set_bit(CEPH_MDS_R_FSCRYPT_FILE, &req->r_req_flags);
> err = __fscrypt_prepare_readdir(dir);
> if (!err && !fscrypt_has_encryption_key(dir)) {
> spin_lock(&dentry->d_lock);
> dentry->d_flags |= DCACHE_NOKEY_NAME;
> spin_unlock(&dentry->d_lock);
> }
> }
>
> What made me want to create a new helper was that I simply needed to call
> fscrypt_get_encryption_info() to force the encryption info to be set in
> the parent directory. But this function was only accessible through
> __fscrypt_prepare_readdir(), which isn't really a great function name for
> what I need here.
>
> Since __fscrypt_prepare_readdir() doesn't seem to be used anywhere else,
> maybe it could be removed and fscrypt_get_encryption_info() be exported
> instead?
Well, fscrypt_get_encryption_info() *used* to be exported, but it was hard to
keep track of its use cases (some of which were not actually necessary), which
is why it eventually got replaced with use-case oriented helper functions.
Maybe just use fscrypt_prepare_lookup_partial() for the name of your new helper
function (instead of fscrypt_prepare_atomic_open())?
- Eric
Eric Biggers <[email protected]> writes:
> On Wed, Mar 15, 2023 at 11:08:23AM +0000, Luís Henriques wrote:
>> > So, actually I think this patch doesn't make sense. If ceph is doing the above
>> > in its ->lookup() anyway, then it just should do the exact same thing in its
>> > ->atomic_open() too.
>>
>> In fact, my initial fix for the cephfs bug was doing just that. It was a
>> single patch to ceph_atomic_open() that would simply do:
>>
>> if (IS_ENCRYPTED(dir)) {
>> set_bit(CEPH_MDS_R_FSCRYPT_FILE, &req->r_req_flags);
>> err = __fscrypt_prepare_readdir(dir);
>> if (!err && !fscrypt_has_encryption_key(dir)) {
>> spin_lock(&dentry->d_lock);
>> dentry->d_flags |= DCACHE_NOKEY_NAME;
>> spin_unlock(&dentry->d_lock);
>> }
>> }
>>
>> What made me want to create a new helper was that I simply needed to call
>> fscrypt_get_encryption_info() to force the encryption info to be set in
>> the parent directory. But this function was only accessible through
>> __fscrypt_prepare_readdir(), which isn't really a great function name for
>> what I need here.
>>
>> Since __fscrypt_prepare_readdir() doesn't seem to be used anywhere else,
>> maybe it could be removed and fscrypt_get_encryption_info() be exported
>> instead?
>
> Well, fscrypt_get_encryption_info() *used* to be exported, but it was hard to
> keep track of its use cases (some of which were not actually necessary), which
> is why it eventually got replaced with use-case oriented helper functions.
>
> Maybe just use fscrypt_prepare_lookup_partial() for the name of your new helper
> function (instead of fscrypt_prepare_atomic_open())?
OK, thanks for the name suggestion (naming is *indeed* hard). I'll go try
to get a new helper that can be used in both open_atomic and lookup.
That'll require a bit more of testing so that I don't end up breaking
something else.
Cheers,
--
Luís