On Wed, 5 Aug 2020, James Bottomley wrote:
> I'll leave Mimi to answer, but really this is exactly the question that
> should have been asked before writing IPE. However, since we have the
> cart before the horse, let me break the above down into two specific
> questions.
The question is valid and it was asked. We decided to first prototype what
we needed and then evaluate if it should be integrated with IMA. We
discussed this plan in person with Mimi (at LSS-NA in 2019), and presented
a more mature version of IPE to LSS-NA in 2020, with the expectation that
such a discussion may come up (it did not).
These patches are still part of this process and 'RFC' status.
> 1. Could we implement IPE in IMA (as in would extensions to IMA cover
> everything). I think the answers above indicate this is a "yes".
It could be done, if needed.
> 2. Should we extend IMA to implement it? This is really whether from a
> usability standpoint two seperate LSMs would make sense to cover the
> different use cases.
One issue here is that IMA is fundamentally a measurement & appraisal
scheme which has been extended to include integrity enforcement. IPE was
designed from scratch to only perform integrity enforcement. As such, it
is a cleaner design -- "do one thing and do it well" is a good design
pattern.
In our use-case, we utilize _both_ IMA and IPE, for attestation and code
integrity respectively. It is useful to be able to separate these
concepts. They really are different:
- Code integrity enforcement ensures that code running locally is of known
provenance and has not been modified prior to execution.
- Attestation is about measuring the health of a system and having that
measurement validated by a remote system. (Local attestation is useless).
I'm not sure there is value in continuing to shoe-horn both of these into
IMA.
> I've got to say the least attractive thing
> about separation is the fact that you now both have a policy parser.
> You've tried to differentiate yours by making it more Kconfig
> based, but policy has a way of becoming user space supplied because
> the distros hate config options, so I think you're going to end up
> with a policy parser very like IMAs.
--
James Morris
<[email protected]>
On Wed, 2020-08-05 at 09:59 -0700, James Morris wrote:
> On Wed, 5 Aug 2020, James Bottomley wrote:
>
> > I'll leave Mimi to answer, but really this is exactly the question that
> > should have been asked before writing IPE. However, since we have the
> > cart before the horse, let me break the above down into two specific
> > questions.
>
> The question is valid and it was asked. We decided to first prototype what
> we needed and then evaluate if it should be integrated with IMA. We
> discussed this plan in person with Mimi (at LSS-NA in 2019), and presented
> a more mature version of IPE to LSS-NA in 2020, with the expectation that
> such a discussion may come up (it did not).
When we first spoke the concepts weren't fully formulated, at least to
me.
>
> These patches are still part of this process and 'RFC' status.
>
> > 1. Could we implement IPE in IMA (as in would extensions to IMA cover
> > everything). I think the answers above indicate this is a "yes".
>
> It could be done, if needed.
>
> > 2. Should we extend IMA to implement it? This is really whether from a
> > usability standpoint two seperate LSMs would make sense to cover the
> > different use cases.
>
> One issue here is that IMA is fundamentally a measurement & appraisal
> scheme which has been extended to include integrity enforcement. IPE was
> designed from scratch to only perform integrity enforcement. As such, it
> is a cleaner design -- "do one thing and do it well" is a good design
> pattern.
>
> In our use-case, we utilize _both_ IMA and IPE, for attestation and code
> integrity respectively. It is useful to be able to separate these
> concepts. They really are different:
>
> - Code integrity enforcement ensures that code running locally is of known
> provenance and has not been modified prior to execution.
>
> - Attestation is about measuring the health of a system and having that
> measurement validated by a remote system. (Local attestation is useless).
>
> I'm not sure there is value in continuing to shoe-horn both of these into
> IMA.
True, IMA was originally limited to measurement and attestation, but
most of the original EVM concepts were subsequently included in IMA.
(Remember, Reiner Sailer wrote the original IMA, which I inherited. I
was originially working on EVM code integrity.) From a naming
perspective including EVM code integrity in IMA was a mistake. My
thinking at the time was that as IMA was already calculating the file
hash, instead of re-calculating the file hash for integrity, calculate
the file hash once and re-use it for multiple things - measurement,
integrity, and audit. At the same time define a single system wide
policy.
When we first started working on IMA, EVM, trusted, and encrypted keys,
the general kernel community didn't see a need for any of it. Thus, a
lot of what was accomplished has been accomplished without the backing
of the real core filesystem people.
If block layer integrity was enough, there wouldn't have been a need
for fs-verity. Even fs-verity is limited to read only filesystems,
which makes validating file integrity so much easier. From the
beginning, we've said that fs-verity signatures should be included in
the measurement list. (I thought someone signed on to add that support
to IMA, but have not yet seen anything.)
Going forward I see a lot of what we've accomplished being incorporated
into the filesystems. When IMA will be limited to defining a system
wide policy, I'll have completed my job.
Mimi
>
> > I've got to say the least attractive thing
> > about separation is the fact that you now both have a policy parser.
> > You've tried to differentiate yours by making it more Kconfig
> > based, but policy has a way of becoming user space supplied because
> > the distros hate config options, so I think you're going to end up
> > with a policy parser very like IMAs.
On Wed, 5 Aug 2020, Mimi Zohar wrote:
> If block layer integrity was enough, there wouldn't have been a need
> for fs-verity. Even fs-verity is limited to read only filesystems,
> which makes validating file integrity so much easier. From the
> beginning, we've said that fs-verity signatures should be included in
> the measurement list. (I thought someone signed on to add that support
> to IMA, but have not yet seen anything.)
>
> Going forward I see a lot of what we've accomplished being incorporated
> into the filesystems. When IMA will be limited to defining a system
> wide policy, I'll have completed my job.
What are your thoughts on IPE being a standalone LSM? Would you prefer to
see its functionality integrated into IMA?
On Thu, 2020-08-06 at 09:51 +1000, James Morris wrote:
> On Wed, 5 Aug 2020, Mimi Zohar wrote:
>
> > If block layer integrity was enough, there wouldn't have been a need
> > for fs-verity. Even fs-verity is limited to read only filesystems,
> > which makes validating file integrity so much easier. From the
> > beginning, we've said that fs-verity signatures should be included in
> > the measurement list. (I thought someone signed on to add that support
> > to IMA, but have not yet seen anything.)
> >
> > Going forward I see a lot of what we've accomplished being incorporated
> > into the filesystems. When IMA will be limited to defining a system
> > wide policy, I'll have completed my job.
>
> What are your thoughts on IPE being a standalone LSM? Would you prefer to
> see its functionality integrated into IMA?
Improving the integrity subsystem would be preferred.
Mimi
On Thu, 6 Aug 2020, Mimi Zohar wrote:
> On Thu, 2020-08-06 at 09:51 +1000, James Morris wrote:
> > On Wed, 5 Aug 2020, Mimi Zohar wrote:
> >
> > > If block layer integrity was enough, there wouldn't have been a need
> > > for fs-verity. Even fs-verity is limited to read only filesystems,
> > > which makes validating file integrity so much easier. From the
> > > beginning, we've said that fs-verity signatures should be included in
> > > the measurement list. (I thought someone signed on to add that support
> > > to IMA, but have not yet seen anything.)
> > >
> > > Going forward I see a lot of what we've accomplished being incorporated
> > > into the filesystems. When IMA will be limited to defining a system
> > > wide policy, I'll have completed my job.
> >
> > What are your thoughts on IPE being a standalone LSM? Would you prefer to
> > see its functionality integrated into IMA?
>
> Improving the integrity subsystem would be preferred.
>
Are you planning to attend Plumbers? Perhaps we could propose a BoF
session on this topic.
--
James Morris
<[email protected]>
On Sat, 2020-08-08 at 02:41 +1000, James Morris wrote:
> On Thu, 6 Aug 2020, Mimi Zohar wrote:
>
> > On Thu, 2020-08-06 at 09:51 +1000, James Morris wrote:
> > > On Wed, 5 Aug 2020, Mimi Zohar wrote:
> > >
> > > > If block layer integrity was enough, there wouldn't have been a need
> > > > for fs-verity. Even fs-verity is limited to read only filesystems,
> > > > which makes validating file integrity so much easier. From the
> > > > beginning, we've said that fs-verity signatures should be included in
> > > > the measurement list. (I thought someone signed on to add that support
> > > > to IMA, but have not yet seen anything.)
> > > >
> > > > Going forward I see a lot of what we've accomplished being incorporated
> > > > into the filesystems. When IMA will be limited to defining a system
> > > > wide policy, I'll have completed my job.
> > >
> > > What are your thoughts on IPE being a standalone LSM? Would you prefer to
> > > see its functionality integrated into IMA?
> >
> > Improving the integrity subsystem would be preferred.
> >
>
> Are you planning to attend Plumbers? Perhaps we could propose a BoF
> session on this topic.
That sounds like a good idea.
Mimi
On Fri, 2020-08-07 at 13:31 -0400, Mimi Zohar wrote:
> On Sat, 2020-08-08 at 02:41 +1000, James Morris wrote:
> > On Thu, 6 Aug 2020, Mimi Zohar wrote:
> >
> > > On Thu, 2020-08-06 at 09:51 +1000, James Morris wrote:
> > > > On Wed, 5 Aug 2020, Mimi Zohar wrote:
> > > >
> > > > > If block layer integrity was enough, there wouldn't have been a need
> > > > > for fs-verity. Even fs-verity is limited to read only filesystems,
> > > > > which makes validating file integrity so much easier. From the
> > > > > beginning, we've said that fs-verity signatures should be included in
> > > > > the measurement list. (I thought someone signed on to add that support
> > > > > to IMA, but have not yet seen anything.)
> > > > >
> > > > > Going forward I see a lot of what we've accomplished being incorporated
> > > > > into the filesystems. When IMA will be limited to defining a system
> > > > > wide policy, I'll have completed my job.
> > > >
> > > > What are your thoughts on IPE being a standalone LSM? Would you prefer to
> > > > see its functionality integrated into IMA?
> > >
> > > Improving the integrity subsystem would be preferred.
> > >
> >
> > Are you planning to attend Plumbers? Perhaps we could propose a BoF
> > session on this topic.
>
> That sounds like a good idea.
Other than it is already sold out.
Mimi
> On Aug 5, 2020, at 2:15 PM, Mimi Zohar <[email protected]> wrote:
>
> On Wed, 2020-08-05 at 09:59 -0700, James Morris wrote:
>> On Wed, 5 Aug 2020, James Bottomley wrote:
>>
>>> I'll leave Mimi to answer, but really this is exactly the question that
>>> should have been asked before writing IPE. However, since we have the
>>> cart before the horse, let me break the above down into two specific
>>> questions.
>>
>> The question is valid and it was asked. We decided to first prototype what
>> we needed and then evaluate if it should be integrated with IMA. We
>> discussed this plan in person with Mimi (at LSS-NA in 2019), and presented
>> a more mature version of IPE to LSS-NA in 2020, with the expectation that
>> such a discussion may come up (it did not).
>
> When we first spoke the concepts weren't fully formulated, at least to
> me.
>>
>> These patches are still part of this process and 'RFC' status.
>>
>>> 1. Could we implement IPE in IMA (as in would extensions to IMA cover
>>> everything). I think the answers above indicate this is a "yes".
>>
>> It could be done, if needed.
>>
>>> 2. Should we extend IMA to implement it? This is really whether from a
>>> usability standpoint two seperate LSMs would make sense to cover the
>>> different use cases.
>>
>> One issue here is that IMA is fundamentally a measurement & appraisal
>> scheme which has been extended to include integrity enforcement. IPE was
>> designed from scratch to only perform integrity enforcement. As such, it
>> is a cleaner design -- "do one thing and do it well" is a good design
>> pattern.
>>
>> In our use-case, we utilize _both_ IMA and IPE, for attestation and code
>> integrity respectively. It is useful to be able to separate these
>> concepts. They really are different:
>>
>> - Code integrity enforcement ensures that code running locally is of known
>> provenance and has not been modified prior to execution.
My interest is in code integrity enforcement for executables stored
in NFS files.
My struggle with IPE is that due to its dependence on dm-verity, it
does not seem to able to protect content that is stored separately
from its execution environment and accessed via a file access
protocol (FUSE, SMB, NFS, etc).
>> - Attestation is about measuring the health of a system and having that
>> measurement validated by a remote system. (Local attestation is useless).
>>
>> I'm not sure there is value in continuing to shoe-horn both of these into
>> IMA.
>
> True, IMA was originally limited to measurement and attestation, but
> most of the original EVM concepts were subsequently included in IMA.
> (Remember, Reiner Sailer wrote the original IMA, which I inherited. I
> was originially working on EVM code integrity.) From a naming
> perspective including EVM code integrity in IMA was a mistake. My
> thinking at the time was that as IMA was already calculating the file
> hash, instead of re-calculating the file hash for integrity, calculate
> the file hash once and re-use it for multiple things - measurement,
> integrity, and audit. At the same time define a single system wide
> policy.
>
> When we first started working on IMA, EVM, trusted, and encrypted keys,
> the general kernel community didn't see a need for any of it. Thus, a
> lot of what was accomplished has been accomplished without the backing
> of the real core filesystem people.
>
> If block layer integrity was enough, there wouldn't have been a need
> for fs-verity. Even fs-verity is limited to read only filesystems,
> which makes validating file integrity so much easier. From the
> beginning, we've said that fs-verity signatures should be included in
> the measurement list. (I thought someone signed on to add that support
> to IMA, but have not yet seen anything.)
Mimi, when you and I discussed this during LSS NA 2019, I didn't fully
understand that you expected me to implement signed Merkle trees for all
filesystems. At the time, it sounded to me like you wanted signed Merkle
trees only for NFS files. Is that still the case?
The first priority (for me, anyway) therefore is getting the ability to
move IMA metadata between NFS clients and servers shoveled into the NFS
protocol, but that's been blocked for various legal reasons.
IMO we need agreement from everyone (integrity developers, FS
implementers, and Linux distributors) that a signed Merkle tree IMA
metadata format, stored in either an xattr or appended to an executable
file, will be the way forward for IMA in all filesystems.
> Going forward I see a lot of what we've accomplished being incorporated
> into the filesystems. When IMA will be limited to defining a system
> wide policy, I'll have completed my job.
>
> Mimi
>
>>
>>> I've got to say the least attractive thing
>>> about separation is the fact that you now both have a policy parser.
>>> You've tried to differentiate yours by making it more Kconfig
>>> based, but policy has a way of becoming user space supplied because
>>> the distros hate config options, so I think you're going to end up
>>> with a policy parser very like IMAs.
On Sat, 2020-08-08 at 13:47 -0400, Chuck Lever wrote:
> > On Aug 5, 2020, at 2:15 PM, Mimi Zohar <[email protected]> wrote:
<snip>
> > If block layer integrity was enough, there wouldn't have been a need
> > for fs-verity. Even fs-verity is limited to read only filesystems,
> > which makes validating file integrity so much easier. From the
> > beginning, we've said that fs-verity signatures should be included in
> > the measurement list. (I thought someone signed on to add that support
> > to IMA, but have not yet seen anything.)
>
> Mimi, when you and I discussed this during LSS NA 2019, I didn't fully
> understand that you expected me to implement signed Merkle trees for all
> filesystems. At the time, it sounded to me like you wanted signed Merkle
> trees only for NFS files. Is that still the case?
I definitely do not expect you to support signed Merkle trees for all
filesystems. My interested is from an IMA perspective of measuring and
verifying the fs-verity Merkle tree root (and header info) signature.
This is independent of which filesystems support it.
>
> The first priority (for me, anyway) therefore is getting the ability to
> move IMA metadata between NFS clients and servers shoveled into the NFS
> protocol, but that's been blocked for various legal reasons.
Up to now, verifying remote filesystem file integrity has been out of
scope for IMA. With fs-verity file signatures I can at least grasp
how remote file integrity could possibly work. I don't understand how
remote file integrity with existing IMA formats could be supported. You
might want to consider writing a whitepaper, which could later be used
as the basis for a patch set cover letter.
Mimi
>
> IMO we need agreement from everyone (integrity developers, FS
> implementers, and Linux distributors) that a signed Merkle tree IMA
> metadata format, stored in either an xattr or appended to an executable
> file, will be the way forward for IMA in all filesystems.
On Sun, 2020-08-09 at 13:16 -0400, Mimi Zohar wrote:
> On Sat, 2020-08-08 at 13:47 -0400, Chuck Lever wrote:
> > > On Aug 5, 2020, at 2:15 PM, Mimi Zohar <[email protected]>
> > > wrote:
>
> <snip>
>
> > > If block layer integrity was enough, there wouldn't have been a
> > > need for fs-verity. Even fs-verity is limited to read only
> > > filesystems, which makes validating file integrity so much
> > > easier. From the beginning, we've said that fs-verity signatures
> > > should be included in the measurement list. (I thought someone
> > > signed on to add that support to IMA, but have not yet seen
> > > anything.)
> >
> > Mimi, when you and I discussed this during LSS NA 2019, I didn't
> > fully understand that you expected me to implement signed Merkle
> > trees for all filesystems. At the time, it sounded to me like you
> > wanted signed Merkle trees only for NFS files. Is that still the
> > case?
>
> I definitely do not expect you to support signed Merkle trees for all
> filesystems. My interested is from an IMA perspective of measuring
> and verifying the fs-verity Merkle tree root (and header info)
> signature. This is independent of which filesystems support it.
>
> >
> > The first priority (for me, anyway) therefore is getting the
> > ability to move IMA metadata between NFS clients and servers
> > shoveled into the NFS protocol, but that's been blocked for various
> > legal reasons.
>
> Up to now, verifying remote filesystem file integrity has been out of
> scope for IMA. With fs-verity file signatures I can at least grasp
> how remote file integrity could possibly work. I don't understand
> how remote file integrity with existing IMA formats could be
> supported. You might want to consider writing a whitepaper, which
> could later be used as the basis for a patch set cover letter.
I think, before this, we can help with the basics (and perhaps we
should sort them out before we start documenting what we'll do). The
first basic is that a merkle tree allows unit at a time verification.
First of all we should agree on the unit. Since we always fault a page
at a time, I think our merkle tree unit should be a page not a block.
Next, we should agree where the check gates for the per page accesses
should be ... definitely somewhere in readpage, I suspect and finally
we should agree how the merkle tree is presented at the gate. I think
there are three ways:
1. Ahead of time transfer: The merkle tree is transferred and verified
at some time before the accesses begin, so we already have a
verified copy and can compare against the lower leaf.
2. Async transfer: We provide an async mechanism to transfer the
necessary components, so when presented with a unit, we check the
log n components required to get to the root
3. The protocol actually provides the capability of 2 (like the SCSI
DIF/DIX), so to IMA all the pieces get presented instead of IMA
having to manage the tree
There are also a load of minor things like how we get the head hash,
which must be presented and verified ahead of time for each of the
above 3.
James
On Mon, 2020-08-10 at 08:35 -0700, James Bottomley wrote:
> On Sun, 2020-08-09 at 13:16 -0400, Mimi Zohar wrote:
> > On Sat, 2020-08-08 at 13:47 -0400, Chuck Lever wrote:
> > > > On Aug 5, 2020, at 2:15 PM, Mimi Zohar <[email protected]>
> > > > wrote:
> >
> > <snip>
> >
> > > > If block layer integrity was enough, there wouldn't have been a
> > > > need for fs-verity. Even fs-verity is limited to read only
> > > > filesystems, which makes validating file integrity so much
> > > > easier. From the beginning, we've said that fs-verity signatures
> > > > should be included in the measurement list. (I thought someone
> > > > signed on to add that support to IMA, but have not yet seen
> > > > anything.)
> > >
> > > Mimi, when you and I discussed this during LSS NA 2019, I didn't
> > > fully understand that you expected me to implement signed Merkle
> > > trees for all filesystems. At the time, it sounded to me like you
> > > wanted signed Merkle trees only for NFS files. Is that still the
> > > case?
> >
> > I definitely do not expect you to support signed Merkle trees for all
> > filesystems. My interested is from an IMA perspective of measuring
> > and verifying the fs-verity Merkle tree root (and header info)
> > signature. This is independent of which filesystems support it.
> >
> > > The first priority (for me, anyway) therefore is getting the
> > > ability to move IMA metadata between NFS clients and servers
> > > shoveled into the NFS protocol, but that's been blocked for various
> > > legal reasons.
> >
> > Up to now, verifying remote filesystem file integrity has been out of
> > scope for IMA. With fs-verity file signatures I can at least grasp
> > how remote file integrity could possibly work. I don't understand
> > how remote file integrity with existing IMA formats could be
> > supported. You might want to consider writing a whitepaper, which
> > could later be used as the basis for a patch set cover letter.
>
> I think, before this, we can help with the basics (and perhaps we
> should sort them out before we start documenting what we'll do).
I'm not opposed to doing that, but you're taking this discussion in a
totally different direction. The current discussion is about NFSv4
supporting the existing IMA signatures, not only fs-verity signatures.
I'd like to understand how that is possible and for the community to
weigh in on whether it makes sense.
> The
> first basic is that a merkle tree allows unit at a time verification.
> First of all we should agree on the unit. Since we always fault a page
> at a time, I think our merkle tree unit should be a page not a block.
> Next, we should agree where the check gates for the per page accesses
> should be ... definitely somewhere in readpage, I suspect and finally
> we should agree how the merkle tree is presented at the gate. I think
> there are three ways:
>
> 1. Ahead of time transfer: The merkle tree is transferred and verified
> at some time before the accesses begin, so we already have a
> verified copy and can compare against the lower leaf.
> 2. Async transfer: We provide an async mechanism to transfer the
> necessary components, so when presented with a unit, we check the
> log n components required to get to the root
> 3. The protocol actually provides the capability of 2 (like the SCSI
> DIF/DIX), so to IMA all the pieces get presented instead of IMA
> having to manage the tree
>
> There are also a load of minor things like how we get the head hash,
> which must be presented and verified ahead of time for each of the
> above 3.
I was under the impression that IMA support for fs-verity signatures
would be limited to including the fs-verity signature in the
measurement list and verifying the fs-verity signature. As fs-verity
is limited to immutable files, this could be done on file open. fs-
verity would be responsible for enforcing the block/page data
integrity. From a local filesystem perspective, I think that is all
that is necessary.
In terms of remote file systems, the main issue is transporting and
storing the Merkle tree. As fs-verity is limited to immutable files,
this could still be done on file open.
Mimi
On Mon, 2020-08-10 at 12:35 -0400, Mimi Zohar wrote:
> On Mon, 2020-08-10 at 08:35 -0700, James Bottomley wrote:
[...]
> > > Up to now, verifying remote filesystem file integrity has been
> > > out of scope for IMA. With fs-verity file signatures I can at
> > > least grasp how remote file integrity could possibly work. I
> > > don't understand how remote file integrity with existing IMA
> > > formats could be supported. You might want to consider writing a
> > > whitepaper, which could later be used as the basis for a patch
> > > set cover letter.
> >
> > I think, before this, we can help with the basics (and perhaps we
> > should sort them out before we start documenting what we'll do).
>
> I'm not opposed to doing that, but you're taking this discussion in a
> totally different direction. The current discussion is about NFSv4
> supporting the existing IMA signatures, not only fs-verity
> signatures. I'd like to understand how that is possible and for the
> community to weigh in on whether it makes sense.
Well, I see the NFS problem as being chunk at a time, right, which is
merkle tree, or is there a different chunk at a time mechanism we want
to use? IMA currently verifies signature on open/exec and then
controls updates. Since for NFS we only control the client, we can't
do that on an NFS server, so we really do need verification at read
time ... unless we're threading IMA back to the NFS server?
> > The first basic is that a merkle tree allows unit at a time
> > verification. First of all we should agree on the unit. Since we
> > always fault a page at a time, I think our merkle tree unit should
> > be a page not a block. Next, we should agree where the check gates
> > for the per page accesses should be ... definitely somewhere in
> > readpage, I suspect and finally we should agree how the merkle tree
> > is presented at the gate. I think there are three ways:
> >
> > 1. Ahead of time transfer: The merkle tree is transferred and
> > verified
> > at some time before the accesses begin, so we already have a
> > verified copy and can compare against the lower leaf.
> > 2. Async transfer: We provide an async mechanism to transfer
> > the
> > necessary components, so when presented with a unit, we check
> > the
> > log n components required to get to the root
> > 3. The protocol actually provides the capability of 2 (like the
> > SCSI
> > DIF/DIX), so to IMA all the pieces get presented instead of
> > IMA
> > having to manage the tree
> >
> > There are also a load of minor things like how we get the head
> > hash, which must be presented and verified ahead of time for each
> > of the above 3.
>
>
> I was under the impression that IMA support for fs-verity signatures
> would be limited to including the fs-verity signature in the
> measurement list and verifying the fs-verity signature. As fs-
> verity is limited to immutable files, this could be done on file
> open. fs-verity would be responsible for enforcing the block/page
> data integrity. From a local filesystem perspective, I think that
> is all that is necessary.
The fs-verity use case is a bit of a crippled one because it's
immutable. I think NFS represents more the general case where you
can't rely on immutability and have to verify at chunk read time. If
we get chunk at a time working for NFS, it should work also for fs-
verity and we wouldn't need to have two special paths.
I think, even for NFS we would only really need to log the open, so
same as you imagine for fs-verity. As long as the chunk read hashes
match, we can be silent because everything is going OK, so we only need
to determine what to do and log on mismatch (which isn't expected to
happen for fs-verity).
> In terms of remote file systems, the main issue is transporting and
> storing the Merkle tree. As fs-verity is limited to immutable files,
> this could still be done on file open.
Right, I mentioned that in my options ... we need some "supply
integrity" hook ... or possibly multiple hooks for a variety of
possible methods.
James
On Mon, 2020-08-10 at 10:13 -0700, James Bottomley wrote:
> On Mon, 2020-08-10 at 12:35 -0400, Mimi Zohar wrote:
> > On Mon, 2020-08-10 at 08:35 -0700, James Bottomley wrote:
> [...]
> > > > Up to now, verifying remote filesystem file integrity has been
> > > > out of scope for IMA. With fs-verity file signatures I can at
> > > > least grasp how remote file integrity could possibly work. I
> > > > don't understand how remote file integrity with existing IMA
> > > > formats could be supported. You might want to consider writing a
> > > > whitepaper, which could later be used as the basis for a patch
> > > > set cover letter.
> > >
> > > I think, before this, we can help with the basics (and perhaps we
> > > should sort them out before we start documenting what we'll do).
> >
> > I'm not opposed to doing that, but you're taking this discussion in a
> > totally different direction. The current discussion is about NFSv4
> > supporting the existing IMA signatures, not only fs-verity
> > signatures. I'd like to understand how that is possible and for the
> > community to weigh in on whether it makes sense.
>
> Well, I see the NFS problem as being chunk at a time, right, which is
> merkle tree, or is there a different chunk at a time mechanism we want
> to use? IMA currently verifies signature on open/exec and then
> controls updates. Since for NFS we only control the client, we can't
> do that on an NFS server, so we really do need verification at read
> time ... unless we're threading IMA back to the NFS server?
Yes. I still don't see how we can support the existing IMA signatures,
which is based on the file data hash, unless the "chunk at a time
mechanism" is not a tree, but linear.
Mimi
>
> > > The first basic is that a merkle tree allows unit at a time
> > > verification. First of all we should agree on the unit. Since we
> > > always fault a page at a time, I think our merkle tree unit should
> > > be a page not a block. Next, we should agree where the check gates
> > > for the per page accesses should be ... definitely somewhere in
> > > readpage, I suspect and finally we should agree how the merkle tree
> > > is presented at the gate. I think there are three ways:
> > >
> > > 1. Ahead of time transfer: The merkle tree is transferred and
> > > verified
> > > at some time before the accesses begin, so we already have a
> > > verified copy and can compare against the lower leaf.
> > > 2. Async transfer: We provide an async mechanism to transfer
> > > the
> > > necessary components, so when presented with a unit, we check
> > > the
> > > log n components required to get to the root
> > > 3. The protocol actually provides the capability of 2 (like the
> > > SCSI
> > > DIF/DIX), so to IMA all the pieces get presented instead of
> > > IMA
> > > having to manage the tree
> > >
> > > There are also a load of minor things like how we get the head
> > > hash, which must be presented and verified ahead of time for each
> > > of the above 3.
> >
> >
> > I was under the impression that IMA support for fs-verity signatures
> > would be limited to including the fs-verity signature in the
> > measurement list and verifying the fs-verity signature. As fs-
> > verity is limited to immutable files, this could be done on file
> > open. fs-verity would be responsible for enforcing the block/page
> > data integrity. From a local filesystem perspective, I think that
> > is all that is necessary.
>
> The fs-verity use case is a bit of a crippled one because it's
> immutable. I think NFS represents more the general case where you
> can't rely on immutability and have to verify at chunk read time. If
> we get chunk at a time working for NFS, it should work also for fs-
> verity and we wouldn't need to have two special paths.
>
> I think, even for NFS we would only really need to log the open, so
> same as you imagine for fs-verity. As long as the chunk read hashes
> match, we can be silent because everything is going OK, so we only need
> to determine what to do and log on mismatch (which isn't expected to
> happen for fs-verity).
>
> > In terms of remote file systems, the main issue is transporting and
> > storing the Merkle tree. As fs-verity is limited to immutable files,
> > this could still be done on file open.
>
> Right, I mentioned that in my options ... we need some "supply
> integrity" hook ... or possibly multiple hooks for a variety of
> possible methods.
On Fri, 7 Aug 2020, Mimi Zohar wrote:
> > > Are you planning to attend Plumbers? Perhaps we could propose a BoF
> > > session on this topic.
> >
> > That sounds like a good idea.
>
> Other than it is already sold out.
Mimi advised me off-list that she is able to attend, so I've submitted a
BoF proposal:
https://www.linuxplumbersconf.org/event/7/abstracts/732/
--
James Morris
<[email protected]>
> On Aug 10, 2020, at 11:35 AM, James Bottomley <[email protected]> wrote:
>
> On Sun, 2020-08-09 at 13:16 -0400, Mimi Zohar wrote:
>> On Sat, 2020-08-08 at 13:47 -0400, Chuck Lever wrote:
>>>> On Aug 5, 2020, at 2:15 PM, Mimi Zohar <[email protected]>
>>>> wrote:
>>
>> <snip>
>>
>>>> If block layer integrity was enough, there wouldn't have been a
>>>> need for fs-verity. Even fs-verity is limited to read only
>>>> filesystems, which makes validating file integrity so much
>>>> easier. From the beginning, we've said that fs-verity signatures
>>>> should be included in the measurement list. (I thought someone
>>>> signed on to add that support to IMA, but have not yet seen
>>>> anything.)
>>>
>>> Mimi, when you and I discussed this during LSS NA 2019, I didn't
>>> fully understand that you expected me to implement signed Merkle
>>> trees for all filesystems. At the time, it sounded to me like you
>>> wanted signed Merkle trees only for NFS files. Is that still the
>>> case?
>>
>> I definitely do not expect you to support signed Merkle trees for all
>> filesystems. My interested is from an IMA perspective of measuring
>> and verifying the fs-verity Merkle tree root (and header info)
>> signature. This is independent of which filesystems support it.
>>
>>>
>>> The first priority (for me, anyway) therefore is getting the
>>> ability to move IMA metadata between NFS clients and servers
>>> shoveled into the NFS protocol, but that's been blocked for various
>>> legal reasons.
>>
>> Up to now, verifying remote filesystem file integrity has been out of
>> scope for IMA. With fs-verity file signatures I can at least grasp
>> how remote file integrity could possibly work. I don't understand
>> how remote file integrity with existing IMA formats could be
>> supported. You might want to consider writing a whitepaper, which
>> could later be used as the basis for a patch set cover letter.
>
> I think, before this, we can help with the basics (and perhaps we
> should sort them out before we start documenting what we'll do).
Thanks for the help! I just want to emphasize that documentation
(eg, a specification) will be critical for remote filesystems.
If any of this is to be supported by a remote filesystem, then we
need an unencumbered description of the new metadata format rather
than code. GPL-encumbered formats cannot be contributed to the NFS
standard, and are probably difficult for other filesystems that are
not Linux-native, like SMB, as well.
> The
> first basic is that a merkle tree allows unit at a time verification.
> First of all we should agree on the unit. Since we always fault a page
> at a time, I think our merkle tree unit should be a page not a block.
Remote filesystems will need to agree that the size of that unit is
the same everywhere, or the unit size could be stored in the per-file
metadata.
> Next, we should agree where the check gates for the per page accesses
> should be ... definitely somewhere in readpage, I suspect and finally
> we should agree how the merkle tree is presented at the gate. I think
> there are three ways:
>
> 1. Ahead of time transfer: The merkle tree is transferred and verified
> at some time before the accesses begin, so we already have a
> verified copy and can compare against the lower leaf.
> 2. Async transfer: We provide an async mechanism to transfer the
> necessary components, so when presented with a unit, we check the
> log n components required to get to the root
> 3. The protocol actually provides the capability of 2 (like the SCSI
> DIF/DIX), so to IMA all the pieces get presented instead of IMA
> having to manage the tree
A Merkle tree is potentially large enough that it cannot be stored in
an extended attribute. In addition, an extended attribute is not a
byte stream that you can seek into or read small parts of, it is
retrieved in a single shot.
For this reason, the idea was to save only the signature of the tree's
root on durable storage. The client would retrieve that signature
possibly at open time, and reconstruct the tree at that time.
Or the tree could be partially constructed on-demand at the time each
unit is to be checked (say, as part of 2. above).
The client would have to reconstruct that tree again if memory pressure
caused some or all of the tree to be evicted, so perhaps an on-demand
mechanism is preferable.
> There are also a load of minor things like how we get the head hash,
> which must be presented and verified ahead of time for each of the
> above 3.
Also, changes to a file's content and its tree signature are not
atomic. If a file is mutable, then there is the period between when
the file content has changed and when the signature is updated.
Some discussion of how a client is to behave in those situations will
be necessary.
--
Chuck Lever
[email protected]
On Mon, 2020-08-10 at 19:36 -0400, Chuck Lever wrote:
> > On Aug 10, 2020, at 11:35 AM, James Bottomley
> > <[email protected]> wrote:
> > On Sun, 2020-08-09 at 13:16 -0400, Mimi Zohar wrote:
> > > On Sat, 2020-08-08 at 13:47 -0400, Chuck Lever wrote:
[...]
> > > > The first priority (for me, anyway) therefore is getting the
> > > > ability to move IMA metadata between NFS clients and servers
> > > > shoveled into the NFS protocol, but that's been blocked for
> > > > various legal reasons.
> > >
> > > Up to now, verifying remote filesystem file integrity has been
> > > out of scope for IMA. With fs-verity file signatures I can at
> > > least grasp how remote file integrity could possibly work. I
> > > don't understand how remote file integrity with existing IMA
> > > formats could be supported. You might want to consider writing a
> > > whitepaper, which could later be used as the basis for a patch
> > > set cover letter.
> >
> > I think, before this, we can help with the basics (and perhaps we
> > should sort them out before we start documenting what we'll do).
>
> Thanks for the help! I just want to emphasize that documentation
> (eg, a specification) will be critical for remote filesystems.
>
> If any of this is to be supported by a remote filesystem, then we
> need an unencumbered description of the new metadata format rather
> than code. GPL-encumbered formats cannot be contributed to the NFS
> standard, and are probably difficult for other filesystems that are
> not Linux-native, like SMB, as well.
I don't understand what you mean by GPL encumbered formats. The GPL is
a code licence not a data or document licence. The way the spec
process works in Linux is that we implement or evolve a data format
under a GPL implementaiton, but that implementation doesn't implicate
the later standardisation of the data format and people are free to
reimplement under any licence they choose.
> > The first basic is that a merkle tree allows unit at a time
> > verification. First of all we should agree on the unit. Since we
> > always fault a page at a time, I think our merkle tree unit should
> > be a page not a block.
>
> Remote filesystems will need to agree that the size of that unit is
> the same everywhere, or the unit size could be stored in the per-file
> metadata.
>
>
> > Next, we should agree where the check gates for the per page
> > accesses should be ... definitely somewhere in readpage, I suspect
> > and finally we should agree how the merkle tree is presented at the
> > gate. I think there are three ways:
> >
> > 1. Ahead of time transfer: The merkle tree is transferred and
> > verified
> > at some time before the accesses begin, so we already have a
> > verified copy and can compare against the lower leaf.
> > 2. Async transfer: We provide an async mechanism to transfer the
> > necessary components, so when presented with a unit, we check
> > the
> > log n components required to get to the root
> > 3. The protocol actually provides the capability of 2 (like the
> > SCSI
> > DIF/DIX), so to IMA all the pieces get presented instead of
> > IMA
> > having to manage the tree
>
> A Merkle tree is potentially large enough that it cannot be stored in
> an extended attribute. In addition, an extended attribute is not a
> byte stream that you can seek into or read small parts of, it is
> retrieved in a single shot.
Well you wouldn't store the tree would you, just the head hash. The
rest of the tree can be derived from the data. You need to distinguish
between what you *must* have to verify integrity (the head hash,
possibly signed) and what is nice to have to speed up the verification
process. The choice for the latter is cache or reconstruct depending
on the resources available. If the tree gets cached on the server,
that would be a server implementation detail invisible to the client.
> For this reason, the idea was to save only the signature of the
> tree's root on durable storage. The client would retrieve that
> signature possibly at open time, and reconstruct the tree at that
> time.
Right that's the integrity data you must have.
> Or the tree could be partially constructed on-demand at the time each
> unit is to be checked (say, as part of 2. above).
Whether it's reconstructed or cached can be an implementation detail.
You clearly have to reconstruct once, but whether you have to do it
again depends on the memory available for caching and all the other
resource calls in the system.
> The client would have to reconstruct that tree again if memory
> pressure caused some or all of the tree to be evicted, so perhaps an
> on-demand mechanism is preferable.
Right, but I think that's implementation detail. Probably what we need
is a way to get the log(N) verification hashes from the server and it's
up to the client whether it caches them or not.
> > There are also a load of minor things like how we get the head
> > hash, which must be presented and verified ahead of time for each
> > of the above 3.
>
> Also, changes to a file's content and its tree signature are not
> atomic. If a file is mutable, then there is the period between when
> the file content has changed and when the signature is updated.
> Some discussion of how a client is to behave in those situations will
> be necessary.
For IMA, if you write to a checked file, it gets rechecked the next
time the gate (open/exec/mmap) is triggered. This means you must
complete the update and have the new integrity data in-place before
triggering the check. I think this could apply equally to a merkel
tree based system. It's a sort of Doctor, Doctor it hurts when I do
this situation.
James
> On Aug 11, 2020, at 1:43 AM, James Bottomley <[email protected]> wrote:
>
> On Mon, 2020-08-10 at 19:36 -0400, Chuck Lever wrote:
>>> On Aug 10, 2020, at 11:35 AM, James Bottomley
>>> <[email protected]> wrote:
>>> On Sun, 2020-08-09 at 13:16 -0400, Mimi Zohar wrote:
>>>> On Sat, 2020-08-08 at 13:47 -0400, Chuck Lever wrote:
> [...]
>>>>> The first priority (for me, anyway) therefore is getting the
>>>>> ability to move IMA metadata between NFS clients and servers
>>>>> shoveled into the NFS protocol, but that's been blocked for
>>>>> various legal reasons.
>>>>
>>>> Up to now, verifying remote filesystem file integrity has been
>>>> out of scope for IMA. With fs-verity file signatures I can at
>>>> least grasp how remote file integrity could possibly work. I
>>>> don't understand how remote file integrity with existing IMA
>>>> formats could be supported. You might want to consider writing a
>>>> whitepaper, which could later be used as the basis for a patch
>>>> set cover letter.
>>>
>>> I think, before this, we can help with the basics (and perhaps we
>>> should sort them out before we start documenting what we'll do).
>>
>> Thanks for the help! I just want to emphasize that documentation
>> (eg, a specification) will be critical for remote filesystems.
>>
>> If any of this is to be supported by a remote filesystem, then we
>> need an unencumbered description of the new metadata format rather
>> than code. GPL-encumbered formats cannot be contributed to the NFS
>> standard, and are probably difficult for other filesystems that are
>> not Linux-native, like SMB, as well.
>
> I don't understand what you mean by GPL encumbered formats. The GPL is
> a code licence not a data or document licence.
IETF contributions occur under a BSD-style license incompatible
with the GPL.
https://trustee.ietf.org/trust-legal-provisions.html
Non-Linux implementers (of OEM storage devices) rely on such standards
processes to indemnify them against licensing claims.
Today, there is no specification for existing IMA metadata formats,
there is only code. My lawyer tells me that because the code that
implements these formats is under GPL, the formats themselves cannot
be contributed to, say, the IETF without express permission from the
authors of that code. There are a lot of authors of the Linux IMA
code, so this is proving to be an impediment to contribution. That
blocks the ability to provide a fully-specified NFS protocol
extension to support IMA metadata formats.
> The way the spec
> process works in Linux is that we implement or evolve a data format
> under a GPL implementaiton, but that implementation doesn't implicate
> the later standardisation of the data format and people are free to
> reimplement under any licence they choose.
That technology transfer can happen only if all the authors of that
prototype agree to contribute to a standard. That's much easier if
that agreement comes before an implementation is done. The current
IMA code base is more than a decade old, and there are more than a
hundred authors who have contributed to that base.
Thus IMO we want an unencumbered description of any IMA metadata
format that is to be contributed to an open standards body (as it
would have to be to extend, say, the NFS protocol).
I'm happy to write that specification, as long as any contributions
here are unencumbered and acknowledged. In fact, I have been working
on a (so far, flawed) NFS extension:
https://datatracker.ietf.org/doc/draft-ietf-nfsv4-integrity-measurement/
Now, for example, the components of a putative Merkle-based IMA
metadata format are all already open:
- The unit size is just an integer
- A certificate fingerprint is a de facto standard, and the
fingerprint digest algorithms are all standardized
- Merkle trees are public domain, I believe, and we're not adding
any special sauce here
- Digital signing algorithms are all standardized
Wondering if we want to hash and save the file's mtime and size too.
>>> The first basic is that a merkle tree allows unit at a time
>>> verification. First of all we should agree on the unit. Since we
>>> always fault a page at a time, I think our merkle tree unit should
>>> be a page not a block.
>>
>> Remote filesystems will need to agree that the size of that unit is
>> the same everywhere, or the unit size could be stored in the per-file
>> metadata.
>>
>>
>>> Next, we should agree where the check gates for the per page
>>> accesses should be ... definitely somewhere in readpage, I suspect
>>> and finally we should agree how the merkle tree is presented at the
>>> gate. I think there are three ways:
>>>
>>> 1. Ahead of time transfer: The merkle tree is transferred and
>>> verified
>>> at some time before the accesses begin, so we already have a
>>> verified copy and can compare against the lower leaf.
>>> 2. Async transfer: We provide an async mechanism to transfer the
>>> necessary components, so when presented with a unit, we check
>>> the
>>> log n components required to get to the root
>>> 3. The protocol actually provides the capability of 2 (like the
>>> SCSI
>>> DIF/DIX), so to IMA all the pieces get presented instead of
>>> IMA
>>> having to manage the tree
>>
>> A Merkle tree is potentially large enough that it cannot be stored in
>> an extended attribute. In addition, an extended attribute is not a
>> byte stream that you can seek into or read small parts of, it is
>> retrieved in a single shot.
>
> Well you wouldn't store the tree would you, just the head hash. The
> rest of the tree can be derived from the data. You need to distinguish
> between what you *must* have to verify integrity (the head hash,
> possibly signed)
We're dealing with an untrusted storage device, and for a remote
filesystem, an untrusted network.
Mimi's earlier point is that any IMA metadata format that involves
unsigned digests is exposed to an alteration attack at rest or in
transit, thus will not provide a robust end-to-end integrity
guarantee.
Therefore, tree root digests must be cryptographically signed to be
properly protected in these environments. Verifying that signature
should be done infrequently relative to reading a file's content.
> and what is nice to have to speed up the verification
> process. The choice for the latter is cache or reconstruct depending
> on the resources available. If the tree gets cached on the server,
> that would be a server implementation detail invisible to the client.
We assume that storage targets (for block or file) are not trusted.
Therefore storage clients cannot rely on intermediate results (eg,
middle nodes in a Merkle tree) unless those results are generated
within the client's trust envelope.
So: if the storage target is considered inside the client's trust
envelope, it can cache or store durably any intermediate parts of
the verification process. If not, the network and file storage is
considered untrusted, and the client has to rely on nothing but the
signed digest of the tree root.
We could build a scheme around, say, fscache, that might save the
intermediate results durably and locally.
>> For this reason, the idea was to save only the signature of the
>> tree's root on durable storage. The client would retrieve that
>> signature possibly at open time, and reconstruct the tree at that
>> time.
>
> Right that's the integrity data you must have.
>
>> Or the tree could be partially constructed on-demand at the time each
>> unit is to be checked (say, as part of 2. above).
>
> Whether it's reconstructed or cached can be an implementation detail.
> You clearly have to reconstruct once, but whether you have to do it
> again depends on the memory available for caching and all the other
> resource calls in the system.
>
>> The client would have to reconstruct that tree again if memory
>> pressure caused some or all of the tree to be evicted, so perhaps an
>> on-demand mechanism is preferable.
>
> Right, but I think that's implementation detail. Probably what we need
> is a way to get the log(N) verification hashes from the server and it's
> up to the client whether it caches them or not.
Agreed, these are implementation details. But see above about the
trustworthiness of the intermediate hashes. If they are conveyed
on an untrusted network, then they can't be trusted either.
>>> There are also a load of minor things like how we get the head
>>> hash, which must be presented and verified ahead of time for each
>>> of the above 3.
>>
>> Also, changes to a file's content and its tree signature are not
>> atomic. If a file is mutable, then there is the period between when
>> the file content has changed and when the signature is updated.
>> Some discussion of how a client is to behave in those situations will
>> be necessary.
>
> For IMA, if you write to a checked file, it gets rechecked the next
> time the gate (open/exec/mmap) is triggered. This means you must
> complete the update and have the new integrity data in-place before
> triggering the check. I think this could apply equally to a merkel
> tree based system. It's a sort of Doctor, Doctor it hurts when I do
> this situation.
I imagine it's a common situation where a "yum update" process is
modifying executables while clients are running them. To prevent
a read from pulling refreshed content before the new tree root is
available, it would have to block temporarily until the verification
process succeeds with the updated tree root.
--
Chuck Lever
[email protected]
On Tue, 2020-08-11 at 10:48 -0400, Chuck Lever wrote:
> > On Aug 11, 2020, at 1:43 AM, James Bottomley
> > <[email protected]> wrote:
> > On Mon, 2020-08-10 at 19:36 -0400, Chuck Lever wrote:
[...]
> > > Thanks for the help! I just want to emphasize that documentation
> > > (eg, a specification) will be critical for remote filesystems.
> > >
> > > If any of this is to be supported by a remote filesystem, then we
> > > need an unencumbered description of the new metadata format
> > > rather than code. GPL-encumbered formats cannot be contributed to
> > > the NFS standard, and are probably difficult for other
> > > filesystems that are not Linux-native, like SMB, as well.
> >
> > I don't understand what you mean by GPL encumbered formats. The
> > GPL is a code licence not a data or document licence.
>
> IETF contributions occur under a BSD-style license incompatible
> with the GPL.
>
> https://trustee.ietf.org/trust-legal-provisions.html
>
> Non-Linux implementers (of OEM storage devices) rely on such
> standards processes to indemnify them against licensing claims.
Well, that simply means we won't be contributing the Linux
implementation, right? However, IETF doesn't require BSD for all
implementations, so that's OK.
> Today, there is no specification for existing IMA metadata formats,
> there is only code. My lawyer tells me that because the code that
> implements these formats is under GPL, the formats themselves cannot
> be contributed to, say, the IETF without express permission from the
> authors of that code. There are a lot of authors of the Linux IMA
> code, so this is proving to be an impediment to contribution. That
> blocks the ability to provide a fully-specified NFS protocol
> extension to support IMA metadata formats.
Well, let me put the counterpoint: I can write a book about how linux
device drivers work (which includes describing the data formats), for
instance, without having to get permission from all the authors ... or
is your lawyer taking the view we should be suing Jonathan Corbet,
Alessandro Rubini, and Greg Kroah-Hartman for licence infringement? In
fact do they think we now have a huge class action possibility against
O'Reilly and a host of other publishers ...
> > The way the spec process works in Linux is that we implement or
> > evolve a data format under a GPL implementaiton, but that
> > implementation doesn't implicate the later standardisation of the
> > data format and people are free to reimplement under any licence
> > they choose.
>
> That technology transfer can happen only if all the authors of that
> prototype agree to contribute to a standard. That's much easier if
> that agreement comes before an implementation is done. The current
> IMA code base is more than a decade old, and there are more than a
> hundred authors who have contributed to that base.
>
> Thus IMO we want an unencumbered description of any IMA metadata
> format that is to be contributed to an open standards body (as it
> would have to be to extend, say, the NFS protocol).
Fine, good grief, people who take a sensible view of this can write the
data format down and publish it under any licence you like then you can
pick it up again safely. Would CC0 be OK? ... neither GPL nor BSD are
document licences and we shouldn't perpetuate bad practice by licensing
documentation under them.
James
On Tue, 2020-08-11 at 10:48 -0400, Chuck Lever wrote:
> > On Aug 11, 2020, at 1:43 AM, James Bottomley <James.Bottomley@Hanse
> > nPartnership.com> wrote:
> >
> > On Mon, 2020-08-10 at 19:36 -0400, Chuck Lever wrote:
> > > > On Aug 10, 2020, at 11:35 AM, James Bottomley
> > > > <[email protected]> wrote:
[...]
> > > > The first basic is that a merkle tree allows unit at a time
> > > > verification. First of all we should agree on the unit. Since
> > > > we always fault a page at a time, I think our merkle tree unit
> > > > should be a page not a block.
> > >
> > > Remote filesystems will need to agree that the size of that unit
> > > is the same everywhere, or the unit size could be stored in the
> > > per-filemetadata.
> > >
> > >
> > > > Next, we should agree where the check gates for the per page
> > > > accesses should be ... definitely somewhere in readpage, I
> > > > suspect and finally we should agree how the merkle tree is
> > > > presented at the gate. I think there are three ways:
> > > >
> > > > 1. Ahead of time transfer: The merkle tree is transferred and
> > > > verified
> > > > at some time before the accesses begin, so we already have
> > > > a
> > > > verified copy and can compare against the lower leaf.
> > > > 2. Async transfer: We provide an async mechanism to transfer
> > > > the
> > > > necessary components, so when presented with a unit, we
> > > > check the
> > > > log n components required to get to the root
> > > > 3. The protocol actually provides the capability of 2 (like
> > > > the SCSI
> > > > DIF/DIX), so to IMA all the pieces get presented instead of
> > > > IMA
> > > > having to manage the tree
> > >
> > > A Merkle tree is potentially large enough that it cannot be
> > > stored in an extended attribute. In addition, an extended
> > > attribute is not a byte stream that you can seek into or read
> > > small parts of, it is retrieved in a single shot.
> >
> > Well you wouldn't store the tree would you, just the head
> > hash. The rest of the tree can be derived from the data. You need
> > to distinguish between what you *must* have to verify integrity
> > (the head hash, possibly signed)
>
> We're dealing with an untrusted storage device, and for a remote
> filesystem, an untrusted network.
>
> Mimi's earlier point is that any IMA metadata format that involves
> unsigned digests is exposed to an alteration attack at rest or in
> transit, thus will not provide a robust end-to-end integrity
> guarantee.
>
> Therefore, tree root digests must be cryptographically signed to be
> properly protected in these environments. Verifying that signature
> should be done infrequently relative to reading a file's content.
I'm not disagreeing there has to be a way for the relying party to
trust the root hash.
> > and what is nice to have to speed up the verification
> > process. The choice for the latter is cache or reconstruct
> > depending on the resources available. If the tree gets cached on
> > the server, that would be a server implementation detail invisible
> > to the client.
>
> We assume that storage targets (for block or file) are not trusted.
> Therefore storage clients cannot rely on intermediate results (eg,
> middle nodes in a Merkle tree) unless those results are generated
> within the client's trust envelope.
Yes, they can ... because supplied nodes can be verified. That's the
whole point of a merkle tree. As long as I'm sure of the root hash I
can verify all the rest even if supplied by an untrusted source. If
you consider a simple merkle tree covering 4 blocks:
R
/ \
H11 H12
/ \ / \
H21 H22 H23 H24
| | | |
B1 B2 B3 B4
Assume I have the verified root hash R. If you supply B3 you also
supply H24 and H11 as proof. I verify by hashing B3 to produce H23
then hash H23 and H24 to produce H12 and if H12 and your supplied H11
hash to R the tree is correct and the B3 you supplied must likewise be
correct.
> So: if the storage target is considered inside the client's trust
> envelope, it can cache or store durably any intermediate parts of
> the verification process. If not, the network and file storage is
> considered untrusted, and the client has to rely on nothing but the
> signed digest of the tree root.
>
> We could build a scheme around, say, fscache, that might save the
> intermediate results durably and locally.
I agree we want caching on the client, but we can always page in from
the remote as long as we page enough to verify up to R, so we're always
sure the remote supplied genuine information.
> > > For this reason, the idea was to save only the signature of the
> > > tree's root on durable storage. The client would retrieve that
> > > signature possibly at open time, and reconstruct the tree at that
> > > time.
> >
> > Right that's the integrity data you must have.
> >
> > > Or the tree could be partially constructed on-demand at the time
> > > each unit is to be checked (say, as part of 2. above).
> >
> > Whether it's reconstructed or cached can be an implementation
> > detail. You clearly have to reconstruct once, but whether you have
> > to do it again depends on the memory available for caching and all
> > the other resource calls in the system.
> >
> > > The client would have to reconstruct that tree again if memory
> > > pressure caused some or all of the tree to be evicted, so perhaps
> > > an on-demand mechanism is preferable.
> >
> > Right, but I think that's implementation detail. Probably what we
> > need is a way to get the log(N) verification hashes from the server
> > and it's up to the client whether it caches them or not.
>
> Agreed, these are implementation details. But see above about the
> trustworthiness of the intermediate hashes. If they are conveyed
> on an untrusted network, then they can't be trusted either.
Yes, they can, provided enough of them are asked for to verify. If you
look at the simple example above, suppose I have cached H11 and H12,
but I've lost the entire H2X layer. I want to verify B3 so I also ask
you for your copy of H24. Then I generate H23 from B3 and Hash H23 and
H24. If this doesn't hash to H12 I know either you supplied me the
wrong block or lied about H24. However, if it all hashes correctly I
know you supplied me with both the correct B3 and the correct H24.
> > > > There are also a load of minor things like how we get the head
> > > > hash, which must be presented and verified ahead of time for
> > > > each of the above 3.
> > >
> > > Also, changes to a file's content and its tree signature are not
> > > atomic. If a file is mutable, then there is the period between
> > > when the file content has changed and when the signature is
> > > updated. Some discussion of how a client is to behave in those
> > > situations will be necessary.
> >
> > For IMA, if you write to a checked file, it gets rechecked the next
> > time the gate (open/exec/mmap) is triggered. This means you must
> > complete the update and have the new integrity data in-place before
> > triggering the check. I think this could apply equally to a merkel
> > tree based system. It's a sort of Doctor, Doctor it hurts when I
> > do this situation.
>
> I imagine it's a common situation where a "yum update" process is
> modifying executables while clients are running them. To prevent
> a read from pulling refreshed content before the new tree root is
> available, it would have to block temporarily until the verification
> process succeeds with the updated tree root.
No ... it's not. Yum specifically worries about that today because if
you update running binaries, it causes a crash. Yum constructs the
entire new file then atomically links it into place and deletes the old
inode to prevent these crashes. It never allows you to get into the
situation where you can execute something that will be modified.
That's also why you have to restart stuff after a yum update because if
you didn't it would still be attached to the deleted inode.
James
On Tue, 2020-08-11 at 10:48 -0400, Chuck Lever wrote:
> Mimi's earlier point is that any IMA metadata format that involves
> unsigned digests is exposed to an alteration attack at rest or in
> transit, thus will not provide a robust end-to-end integrity
> guarantee.
I don't believe that is Mimi's point, because it's mostly not correct:
the xattr mechanism does provide this today. The point is the
mechanism we use for storing IMA hashes and signatures today is xattrs
because they have robust security properties for local filesystems that
the kernel enforces. This use goes beyond IMA, selinux labels for
instance use this property as well.
What I think you're saying is that NFS can't provide the robust
security for xattrs we've been relying on, so you need some other
mechanism for storing them.
I think Mimi's other point is actually that IMA uses a flat hash which
we derive by reading the entire file and then watching for mutations.
Since you cannot guarantee we get notice of mutation with NFS, the
entire IMA mechanism can't really be applied in its current form and we
have to resort to chunk at a time verifications that a Merkel tree
would provide. Doesn't this make moot any thinking about
standardisation in NFS for the current IMA flat hash mechanism because
we simply can't use it ... If I were to construct a prototype I'd have
to work out and securely cache the hash of ever chunk when verifying
the flat hash so I could recheck on every chunk read. I think that's
infeasible for large files.
James
Hi!
> > > > (eg, a specification) will be critical for remote filesystems.
> > > >
> > > > If any of this is to be supported by a remote filesystem, then we
> > > > need an unencumbered description of the new metadata format
> > > > rather than code. GPL-encumbered formats cannot be contributed to
> > > > the NFS standard, and are probably difficult for other
> > > > filesystems that are not Linux-native, like SMB, as well.
> > >
> > > I don't understand what you mean by GPL encumbered formats. The
> > > GPL is a code licence not a data or document licence.
> >
> > IETF contributions occur under a BSD-style license incompatible
> > with the GPL.
> >
> > https://trustee.ietf.org/trust-legal-provisions.html
> >
> > Non-Linux implementers (of OEM storage devices) rely on such
> > standards processes to indemnify them against licensing claims.
>
> Well, that simply means we won't be contributing the Linux
> implementation, right? However, IETF doesn't require BSD for all
> implementations, so that's OK.
>
> > Today, there is no specification for existing IMA metadata formats,
> > there is only code. My lawyer tells me that because the code that
> > implements these formats is under GPL, the formats themselves cannot
> > be contributed to, say, the IETF without express permission from the
> > authors of that code. There are a lot of authors of the Linux IMA
> > code, so this is proving to be an impediment to contribution. That
> > blocks the ability to provide a fully-specified NFS protocol
> > extension to support IMA metadata formats.
>
> Well, let me put the counterpoint: I can write a book about how
> linux
You should probably talk to your lawyer.
> device drivers work (which includes describing the data formats), for
> instance, without having to get permission from all the authors ... or
> is your lawyer taking the view we should be suing Jonathan Corbet,
> Alessandro Rubini, and Greg Kroah-Hartman for licence infringement? In
> fact do they think we now have a huge class action possibility against
> O'Reilly and a host of other publishers ...
Because yes, you can reverse engineer for compatibility reasons --
doing clean room re-implementation (BIOS binary -> BIOS documentation
-> BIOS sources under different license), but that was only tested in
the US, is expensive, and I understand people might be uncomfortable
doing that.
Best regards,
Pavel
--
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
On Sat, 8 Aug 2020, Chuck Lever wrote:
> My interest is in code integrity enforcement for executables stored
> in NFS files.
>
> My struggle with IPE is that due to its dependence on dm-verity, it
> does not seem to able to protect content that is stored separately
> from its execution environment and accessed via a file access
> protocol (FUSE, SMB, NFS, etc).
It's not dependent on DM-Verity, that's just one possible integrity
verification mechanism, and one of two supported in this initial
version. The other is 'boot_verified' for a verified or otherwise trusted
rootfs. Future versions will support FS-Verity, at least.
IPE was designed to be extensible in this way, with a strong separation of
mechanism and policy.
Whatever is implemented for NFS should be able to plug in to IPE pretty
easily.
--
James Morris
<[email protected]>
> On Aug 11, 2020, at 2:28 PM, James Bottomley <[email protected]> wrote:
>
> On Tue, 2020-08-11 at 10:48 -0400, Chuck Lever wrote:
>> Mimi's earlier point is that any IMA metadata format that involves
>> unsigned digests is exposed to an alteration attack at rest or in
>> transit, thus will not provide a robust end-to-end integrity
>> guarantee.
>
> I don't believe that is Mimi's point, because it's mostly not correct:
> the xattr mechanism does provide this today. The point is the
> mechanism we use for storing IMA hashes and signatures today is xattrs
> because they have robust security properties for local filesystems that
> the kernel enforces. This use goes beyond IMA, selinux labels for
> instance use this property as well.
I don't buy this for a second. If storing a security label in a
local xattr is so secure, we wouldn't have any need for EVM.
> What I think you're saying is that NFS can't provide the robust
> security for xattrs we've been relying on, so you need some other
> mechanism for storing them.
For NFS, there's a network traversal which is an attack surface.
A local xattr can be attacked as well: a device or bus malfunction
can corrupt the content of an xattr, or a privileged user can modify
it.
How does that metadata get from the software provider to the end
user? It's got to go over a network, stored in various ways, some
of which will not be trusted. To attain an unbroken chain of
provenance, that metadata has to be signed.
I don't think the question is the storage mechanism, but rather the
protection mechanism. Signing the metadata protects it in all of
these cases.
> I think Mimi's other point is actually that IMA uses a flat hash which
> we derive by reading the entire file and then watching for mutations.
> Since you cannot guarantee we get notice of mutation with NFS, the
> entire IMA mechanism can't really be applied in its current form and we
> have to resort to chunk at a time verifications that a Merkel tree
> would provide.
I'm not sure what you mean by this. An NFS client relies on notification
of mutation to maintain the integrity of its cache of NFS file content,
and it's done that since the 1980s.
In addition to examining a file's mtime and ctime as maintained by
the NFS server, a client can rely on the file's NFSv4 change attribute
or an NFSv4 delegation.
> Doesn't this make moot any thinking about
> standardisation in NFS for the current IMA flat hash mechanism because
> we simply can't use it ... If I were to construct a prototype I'd have
> to work out and securely cache the hash of ever chunk when verifying
> the flat hash so I could recheck on every chunk read. I think that's
> infeasible for large files.
>
> James
>
--
Chuck Lever
[email protected]
> On Aug 11, 2020, at 11:53 AM, James Bottomley <[email protected]> wrote:
>
> On Tue, 2020-08-11 at 10:48 -0400, Chuck Lever wrote:
>>> On Aug 11, 2020, at 1:43 AM, James Bottomley <James.Bottomley@Hanse
>>> nPartnership.com> wrote:
>>>
>>> On Mon, 2020-08-10 at 19:36 -0400, Chuck Lever wrote:
>>>>> On Aug 10, 2020, at 11:35 AM, James Bottomley
>>>>> <[email protected]> wrote:
> [...]
>>>>> The first basic is that a merkle tree allows unit at a time
>>>>> verification. First of all we should agree on the unit. Since
>>>>> we always fault a page at a time, I think our merkle tree unit
>>>>> should be a page not a block.
>>>>
>>>> Remote filesystems will need to agree that the size of that unit
>>>> is the same everywhere, or the unit size could be stored in the
>>>> per-filemetadata.
>>>>
>>>>
>>>>> Next, we should agree where the check gates for the per page
>>>>> accesses should be ... definitely somewhere in readpage, I
>>>>> suspect and finally we should agree how the merkle tree is
>>>>> presented at the gate. I think there are three ways:
>>>>>
>>>>> 1. Ahead of time transfer: The merkle tree is transferred and
>>>>> verified
>>>>> at some time before the accesses begin, so we already have
>>>>> a
>>>>> verified copy and can compare against the lower leaf.
>>>>> 2. Async transfer: We provide an async mechanism to transfer
>>>>> the
>>>>> necessary components, so when presented with a unit, we
>>>>> check the
>>>>> log n components required to get to the root
>>>>> 3. The protocol actually provides the capability of 2 (like
>>>>> the SCSI
>>>>> DIF/DIX), so to IMA all the pieces get presented instead of
>>>>> IMA
>>>>> having to manage the tree
>>>>
>>>> A Merkle tree is potentially large enough that it cannot be
>>>> stored in an extended attribute. In addition, an extended
>>>> attribute is not a byte stream that you can seek into or read
>>>> small parts of, it is retrieved in a single shot.
>>>
>>> Well you wouldn't store the tree would you, just the head
>>> hash. The rest of the tree can be derived from the data. You need
>>> to distinguish between what you *must* have to verify integrity
>>> (the head hash, possibly signed)
>>
>> We're dealing with an untrusted storage device, and for a remote
>> filesystem, an untrusted network.
>>
>> Mimi's earlier point is that any IMA metadata format that involves
>> unsigned digests is exposed to an alteration attack at rest or in
>> transit, thus will not provide a robust end-to-end integrity
>> guarantee.
>>
>> Therefore, tree root digests must be cryptographically signed to be
>> properly protected in these environments. Verifying that signature
>> should be done infrequently relative to reading a file's content.
>
> I'm not disagreeing there has to be a way for the relying party to
> trust the root hash.
>
>>> and what is nice to have to speed up the verification
>>> process. The choice for the latter is cache or reconstruct
>>> depending on the resources available. If the tree gets cached on
>>> the server, that would be a server implementation detail invisible
>>> to the client.
>>
>> We assume that storage targets (for block or file) are not trusted.
>> Therefore storage clients cannot rely on intermediate results (eg,
>> middle nodes in a Merkle tree) unless those results are generated
>> within the client's trust envelope.
>
> Yes, they can ... because supplied nodes can be verified. That's the
> whole point of a merkle tree. As long as I'm sure of the root hash I
> can verify all the rest even if supplied by an untrusted source. If
> you consider a simple merkle tree covering 4 blocks:
>
> R
> / \
> H11 H12
> / \ / \
> H21 H22 H23 H24
> | | | |
> B1 B2 B3 B4
>
> Assume I have the verified root hash R. If you supply B3 you also
> supply H24 and H11 as proof. I verify by hashing B3 to produce H23
> then hash H23 and H24 to produce H12 and if H12 and your supplied H11
> hash to R the tree is correct and the B3 you supplied must likewise be
> correct.
I'm not sure what you are proving here. Obviously this has to work
in order for a client to reconstruct the file's Merkle tree given
only R and the file content.
It's the construction of the tree and verification of the hashes that
are potentially expensive. The point of caching intermediate hashes
is so that the client verifies them as few times as possible. I
don't see value in caching those hashes on an untrusted server --
the client will have to reverify them anyway, and there will be no
savings.
Cache once, as close as you can to where the data will be used.
>> So: if the storage target is considered inside the client's trust
>> envelope, it can cache or store durably any intermediate parts of
>> the verification process. If not, the network and file storage is
>> considered untrusted, and the client has to rely on nothing but the
>> signed digest of the tree root.
>>
>> We could build a scheme around, say, fscache, that might save the
>> intermediate results durably and locally.
>
> I agree we want caching on the client, but we can always page in from
> the remote as long as we page enough to verify up to R, so we're always
> sure the remote supplied genuine information.
Agreed.
>>>> For this reason, the idea was to save only the signature of the
>>>> tree's root on durable storage. The client would retrieve that
>>>> signature possibly at open time, and reconstruct the tree at that
>>>> time.
>>>
>>> Right that's the integrity data you must have.
>>>
>>>> Or the tree could be partially constructed on-demand at the time
>>>> each unit is to be checked (say, as part of 2. above).
>>>
>>> Whether it's reconstructed or cached can be an implementation
>>> detail. You clearly have to reconstruct once, but whether you have
>>> to do it again depends on the memory available for caching and all
>>> the other resource calls in the system.
>>>
>>>> The client would have to reconstruct that tree again if memory
>>>> pressure caused some or all of the tree to be evicted, so perhaps
>>>> an on-demand mechanism is preferable.
>>>
>>> Right, but I think that's implementation detail. Probably what we
>>> need is a way to get the log(N) verification hashes from the server
>>> and it's up to the client whether it caches them or not.
>>
>> Agreed, these are implementation details. But see above about the
>> trustworthiness of the intermediate hashes. If they are conveyed
>> on an untrusted network, then they can't be trusted either.
>
> Yes, they can, provided enough of them are asked for to verify. If you
> look at the simple example above, suppose I have cached H11 and H12,
> but I've lost the entire H2X layer. I want to verify B3 so I also ask
> you for your copy of H24. Then I generate H23 from B3 and Hash H23 and
> H24. If this doesn't hash to H12 I know either you supplied me the
> wrong block or lied about H24. However, if it all hashes correctly I
> know you supplied me with both the correct B3 and the correct H24.
My point is there is a difference between a trusted cache and an
untrusted cache. I argue there is not much value in a cache where
the hashes have to be verified again.
>>>>> There are also a load of minor things like how we get the head
>>>>> hash, which must be presented and verified ahead of time for
>>>>> each of the above 3.
>>>>
>>>> Also, changes to a file's content and its tree signature are not
>>>> atomic. If a file is mutable, then there is the period between
>>>> when the file content has changed and when the signature is
>>>> updated. Some discussion of how a client is to behave in those
>>>> situations will be necessary.
>>>
>>> For IMA, if you write to a checked file, it gets rechecked the next
>>> time the gate (open/exec/mmap) is triggered. This means you must
>>> complete the update and have the new integrity data in-place before
>>> triggering the check. I think this could apply equally to a merkel
>>> tree based system. It's a sort of Doctor, Doctor it hurts when I
>>> do this situation.
>>
>> I imagine it's a common situation where a "yum update" process is
>> modifying executables while clients are running them. To prevent
>> a read from pulling refreshed content before the new tree root is
>> available, it would have to block temporarily until the verification
>> process succeeds with the updated tree root.
>
> No ... it's not. Yum specifically worries about that today because if
> you update running binaries, it causes a crash. Yum constructs the
> entire new file then atomically links it into place and deletes the old
> inode to prevent these crashes. It never allows you to get into the
> situation where you can execute something that will be modified.
> That's also why you have to restart stuff after a yum update because if
> you didn't it would still be attached to the deleted inode.
Fair enough.
--
Chuck Lever
[email protected]
> On Aug 11, 2020, at 5:03 PM, James Morris <[email protected]> wrote:
>
> On Sat, 8 Aug 2020, Chuck Lever wrote:
>
>> My interest is in code integrity enforcement for executables stored
>> in NFS files.
>>
>> My struggle with IPE is that due to its dependence on dm-verity, it
>> does not seem to able to protect content that is stored separately
>> from its execution environment and accessed via a file access
>> protocol (FUSE, SMB, NFS, etc).
>
> It's not dependent on DM-Verity, that's just one possible integrity
> verification mechanism, and one of two supported in this initial
> version. The other is 'boot_verified' for a verified or otherwise trusted
> rootfs. Future versions will support FS-Verity, at least.
>
> IPE was designed to be extensible in this way, with a strong separation of
> mechanism and policy.
I got that, but it looked to me like the whole system relied on having
access to the block device under the filesystem. That's not possible
for a remote filesystem like Ceph or NFS.
I'm happy to take a closer look if someone can point me the right way.
--
Chuck Lever
[email protected]
> On Aug 11, 2020, at 11:32 AM, James Bottomley <[email protected]> wrote:
>
> On Tue, 2020-08-11 at 10:48 -0400, Chuck Lever wrote:
>>> On Aug 11, 2020, at 1:43 AM, James Bottomley
>>> <[email protected]> wrote:
>>> On Mon, 2020-08-10 at 19:36 -0400, Chuck Lever wrote:
> [...]
>>>> Thanks for the help! I just want to emphasize that documentation
>>>> (eg, a specification) will be critical for remote filesystems.
>>>>
>>>> If any of this is to be supported by a remote filesystem, then we
>>>> need an unencumbered description of the new metadata format
>>>> rather than code. GPL-encumbered formats cannot be contributed to
>>>> the NFS standard, and are probably difficult for other
>>>> filesystems that are not Linux-native, like SMB, as well.
>>>
>>> I don't understand what you mean by GPL encumbered formats. The
>>> GPL is a code licence not a data or document licence.
>>
>> IETF contributions occur under a BSD-style license incompatible
>> with the GPL.
>>
>> https://trustee.ietf.org/trust-legal-provisions.html
>>
>> Non-Linux implementers (of OEM storage devices) rely on such
>> standards processes to indemnify them against licensing claims.
>
> Well, that simply means we won't be contributing the Linux
> implementation, right?
At the present time, there is nothing but the Linux implementation.
There's no English description, there's no specification of the
formats, the format is described only by source code.
The only way to contribute current IMA metadata formats to an open
standards body like the IETF is to look at encumbered code first.
We would effectively be contributing an implementation in this case.
(I'm not saying the current formats should or should not be
contributed; merely that there is a legal stumbling block to doing
so that can be avoided for newly defined formats).
> Well, let me put the counterpoint: I can write a book about how linux
> device drivers work (which includes describing the data formats)
Our position is that someone who reads that book and implements those
formats under a non-GPL-compatible license would be in breach of the
GPL.
The point of the standards process is to indemnify implementing
and distributing under _any_ license what has been published by the
standards body. That legally enables everyone to use the published
protocol/format in their own code no matter how it happens to be
licensed.
> Fine, good grief, people who take a sensible view of this can write the
> data format down and publish it under any licence you like then you can
> pick it up again safely.
That's what I proposed. Write it down under the IETF Trust legal
provisions license. And I volunteered to do that.
All I'm saying is that description needs to come before code.
--
Chuck Lever
[email protected]
On Wed, 2020-08-12 at 09:56 -0400, Chuck Lever wrote:
> > On Aug 11, 2020, at 2:28 PM, James Bottomley <James.Bottomley@Hanse
> > nPartnership.com> wrote:
> >
> > On Tue, 2020-08-11 at 10:48 -0400, Chuck Lever wrote:
> > > Mimi's earlier point is that any IMA metadata format that
> > > involves unsigned digests is exposed to an alteration attack at
> > > rest or in transit, thus will not provide a robust end-to-end
> > > integrity guarantee.
> >
> > I don't believe that is Mimi's point, because it's mostly not
> > correct: the xattr mechanism does provide this today. The point is
> > the mechanism we use for storing IMA hashes and signatures today is
> > xattrs because they have robust security properties for local
> > filesystems that the kernel enforces. This use goes beyond IMA,
> > selinux labels for instance use this property as well.
>
> I don't buy this for a second. If storing a security label in a
> local xattr is so secure, we wouldn't have any need for EVM.
What don't you buy? Security xattrs can only be updated by local root.
If you trust local root, the xattr mechanism is fine ... it's the only
one a lot of LSMs use, for instance. If you don't trust local root or
worry about offline backups, you use EVM. A thing isn't secure or
insecure, it depends on the threat model. However, if you don't trust
the NFS server it doesn't matter whether you do or don't trust local
root, you can't believe the contents of the xattr.
> > What I think you're saying is that NFS can't provide the robust
> > security for xattrs we've been relying on, so you need some other
> > mechanism for storing them.
>
> For NFS, there's a network traversal which is an attack surface.
>
> A local xattr can be attacked as well: a device or bus malfunction
> can corrupt the content of an xattr, or a privileged user can modify
> it.
>
> How does that metadata get from the software provider to the end
> user? It's got to go over a network, stored in various ways, some
> of which will not be trusted. To attain an unbroken chain of
> provenance, that metadata has to be signed.
>
> I don't think the question is the storage mechanism, but rather the
> protection mechanism. Signing the metadata protects it in all of
> these cases.
I think we're saying about the same thing. For most people the
security mechanism of local xattrs is sufficient. If you're paranoid,
you don't believe it is and you use EVM.
> > I think Mimi's other point is actually that IMA uses a flat hash
> > which we derive by reading the entire file and then watching for
> > mutations. Since you cannot guarantee we get notice of mutation
> > with NFS, the entire IMA mechanism can't really be applied in its
> > current form and we have to resort to chunk at a time verifications
> > that a Merkel tree would provide.
>
> I'm not sure what you mean by this. An NFS client relies on
> notification of mutation to maintain the integrity of its cache of
> NFS file content, and it's done that since the 1980s.
Mutation detection is part of the current IMA security model. If IMA
sees a file mutate it has to be rehashed the next time it passes the
gate. If we can't trust the NFS server, we can't trust the NFS
mutation notification and we have to have a different mechanism to
check the file.
> In addition to examining a file's mtime and ctime as maintained by
> the NFS server, a client can rely on the file's NFSv4 change
> attribute or an NFSv4 delegation.
And that's secure in the face of a malicious or compromised server?
The bottom line is still, I think we can't use linear hashes with an
open/exec/mmap gate with NFS and we have to move to chunk at a time
verification like that provided by a merkel tree.
James
On Wed, 2020-08-12 at 10:15 -0400, Chuck Lever wrote:
> > On Aug 11, 2020, at 11:53 AM, James Bottomley
> > <[email protected]> wrote:
> >
> > On Tue, 2020-08-11 at 10:48 -0400, Chuck Lever wrote:
[...]
> > > >
> > > > and what is nice to have to speed up the verification
> > > > process. The choice for the latter is cache or reconstruct
> > > > depending on the resources available. If the tree gets cached
> > > > on the server, that would be a server implementation detail
> > > > invisible to the client.
> > >
> > > We assume that storage targets (for block or file) are not
> > > trusted. Therefore storage clients cannot rely on intermediate
> > > results (eg, middle nodes in a Merkle tree) unless those results
> > > are generated within the client's trust envelope.
> >
> > Yes, they can ... because supplied nodes can be verified. That's
> > the whole point of a merkle tree. As long as I'm sure of the root
> > hash I can verify all the rest even if supplied by an untrusted
> > source. If you consider a simple merkle tree covering 4 blocks:
> >
> > R
> > / \
> > H11 H12
> > / \ / \
> > H21 H22 H23 H24
> > > | | |
> >
> > B1 B2 B3 B4
> >
> > Assume I have the verified root hash R. If you supply B3 you also
> > supply H24 and H11 as proof. I verify by hashing B3 to produce H23
> > then hash H23 and H24 to produce H12 and if H12 and your supplied
> > H11 hash to R the tree is correct and the B3 you supplied must
> > likewise be correct.
>
> I'm not sure what you are proving here. Obviously this has to work
> in order for a client to reconstruct the file's Merkle tree given
> only R and the file content.
You implied the server can't be trusted to generate the merkel tree.
I'm showing above it can because of the tree path based verification.
> It's the construction of the tree and verification of the hashes that
> are potentially expensive. The point of caching intermediate hashes
> is so that the client verifies them as few times as possible. I
> don't see value in caching those hashes on an untrusted server --
> the client will have to reverify them anyway, and there will be no
> savings.
I'm not making any claim about server caching, I'm just saying the
client can request pieces of the tree from the server without having to
reconstruct the whole thing itself because it can verify their
correctness.
> Cache once, as close as you can to where the data will be used.
>
>
> > > So: if the storage target is considered inside the client's trust
> > > envelope, it can cache or store durably any intermediate parts of
> > > the verification process. If not, the network and file storage is
> > > considered untrusted, and the client has to rely on nothing but
> > > the signed digest of the tree root.
> > >
> > > We could build a scheme around, say, fscache, that might save the
> > > intermediate results durably and locally.
> >
> > I agree we want caching on the client, but we can always page in
> > from the remote as long as we page enough to verify up to R, so
> > we're always sure the remote supplied genuine information.
>
> Agreed.
>
>
> > > > > For this reason, the idea was to save only the signature of
> > > > > the tree's root on durable storage. The client would retrieve
> > > > > that signature possibly at open time, and reconstruct the
> > > > > tree at that time.
> > > >
> > > > Right that's the integrity data you must have.
> > > >
> > > > > Or the tree could be partially constructed on-demand at the
> > > > > time each unit is to be checked (say, as part of 2. above).
> > > >
> > > > Whether it's reconstructed or cached can be an implementation
> > > > detail. You clearly have to reconstruct once, but whether you
> > > > have to do it again depends on the memory available for caching
> > > > and all the other resource calls in the system.
> > > >
> > > > > The client would have to reconstruct that tree again if
> > > > > memory pressure caused some or all of the tree to be evicted,
> > > > > so perhaps an on-demand mechanism is preferable.
> > > >
> > > > Right, but I think that's implementation detail. Probably what
> > > > we need is a way to get the log(N) verification hashes from the
> > > > server and it's up to the client whether it caches them or not.
> > >
> > > Agreed, these are implementation details. But see above about the
> > > trustworthiness of the intermediate hashes. If they are conveyed
> > > on an untrusted network, then they can't be trusted either.
> >
> > Yes, they can, provided enough of them are asked for to verify. If
> > you look at the simple example above, suppose I have cached H11 and
> > H12, but I've lost the entire H2X layer. I want to verify B3 so I
> > also ask you for your copy of H24. Then I generate H23 from B3 and
> > Hash H23 and H24. If this doesn't hash to H12 I know either you
> > supplied me the wrong block or lied about H24. However, if it all
> > hashes correctly I know you supplied me with both the correct B3
> > and the correct H24.
>
> My point is there is a difference between a trusted cache and an
> untrusted cache. I argue there is not much value in a cache where
> the hashes have to be verified again.
And my point isn't about caching, it's about where the tree comes from.
I claim and you agree the client can get the tree from the server a
piece at a time (because it can path verify it) and doesn't have to
generate it itself. How much of the tree the client has to store and
whether the server caches, reads it in from somewhere or reconstructs
it is an implementation detail.
James
On 8/12/2020 7:18 AM, Chuck Lever wrote:
>
>
>> On Aug 11, 2020, at 5:03 PM, James Morris <[email protected]> wrote:
>>
>> On Sat, 8 Aug 2020, Chuck Lever wrote:
>>
>>> My interest is in code integrity enforcement for executables stored
>>> in NFS files.
>>>
>>> My struggle with IPE is that due to its dependence on dm-verity, it
>>> does not seem to able to protect content that is stored separately
>>> from its execution environment and accessed via a file access
>>> protocol (FUSE, SMB, NFS, etc).
>>
>> It's not dependent on DM-Verity, that's just one possible integrity
>> verification mechanism, and one of two supported in this initial
>> version. The other is 'boot_verified' for a verified or otherwise trusted
>> rootfs. Future versions will support FS-Verity, at least.
>>
>> IPE was designed to be extensible in this way, with a strong separation of
>> mechanism and policy.
>
> I got that, but it looked to me like the whole system relied on having
> access to the block device under the filesystem. That's not possible
> for a remote filesystem like Ceph or NFS.
Block device structure no, (though that's what the currently used, to be
fair). It really has a hard dependency on the file structure,
specifically the ability to determine whether that file structure can be
used to navigate back to the integrity claim provided by the mechanism.
In the current world of IPE, the integrity claim is the root-hash or
root-hash-signature on the block device, provided by dm-verity's
setsecurity hooks (also introduced in this series).
>
> I'm happy to take a closer look if someone can point me the right way.
>
Sure, if you look at the 2nd patch, you want to look at the file
"security/ipe/ipe-property.h", it defines what methods are required to
be implemented by a mechanism to work with IPE. It passes the engine
context which is defined as:
struct ipe_engine_ctx {
enum ipe_op op;
enum ipe_hook hook;
const struct file *file;
const char *audit_pathname;
const struct ipe_bdev_blob *sec_bdev;
};
Now, if the security blob existed for the block_device, it would be
in sec_bdev, but that may be NULL, as well to be fair.
If you want a more worked example of how integration works, patches 8
and 10 introduce the dm-verity properties mentioned in this patch.
> On Aug 12, 2020, at 11:42 AM, James Bottomley <[email protected]> wrote:
>
> On Wed, 2020-08-12 at 09:56 -0400, Chuck Lever wrote:
>>> On Aug 11, 2020, at 2:28 PM, James Bottomley <James.Bottomley@Hanse
>>> nPartnership.com> wrote:
>>>
>>> On Tue, 2020-08-11 at 10:48 -0400, Chuck Lever wrote:
>>>> Mimi's earlier point is that any IMA metadata format that
>>>> involves unsigned digests is exposed to an alteration attack at
>>>> rest or in transit, thus will not provide a robust end-to-end
>>>> integrity guarantee.
>>>
>>> I don't believe that is Mimi's point, because it's mostly not
>>> correct: the xattr mechanism does provide this today. The point is
>>> the mechanism we use for storing IMA hashes and signatures today is
>>> xattrs because they have robust security properties for local
>>> filesystems that the kernel enforces. This use goes beyond IMA,
>>> selinux labels for instance use this property as well.
>>
>> I don't buy this for a second. If storing a security label in a
>> local xattr is so secure, we wouldn't have any need for EVM.
>
> What don't you buy? Security xattrs can only be updated by local root.
> If you trust local root, the xattr mechanism is fine ... it's the only
> one a lot of LSMs use, for instance. If you don't trust local root or
> worry about offline backups, you use EVM. A thing isn't secure or
> insecure, it depends on the threat model. However, if you don't trust
> the NFS server it doesn't matter whether you do or don't trust local
> root, you can't believe the contents of the xattr.
>
>>> What I think you're saying is that NFS can't provide the robust
>>> security for xattrs we've been relying on, so you need some other
>>> mechanism for storing them.
>>
>> For NFS, there's a network traversal which is an attack surface.
>>
>> A local xattr can be attacked as well: a device or bus malfunction
>> can corrupt the content of an xattr, or a privileged user can modify
>> it.
>>
>> How does that metadata get from the software provider to the end
>> user? It's got to go over a network, stored in various ways, some
>> of which will not be trusted. To attain an unbroken chain of
>> provenance, that metadata has to be signed.
>>
>> I don't think the question is the storage mechanism, but rather the
>> protection mechanism. Signing the metadata protects it in all of
>> these cases.
>
> I think we're saying about the same thing.
Roughly.
> For most people the
> security mechanism of local xattrs is sufficient. If you're paranoid,
> you don't believe it is and you use EVM.
When IMA metadata happens to be stored in local filesystems in
a trusted xattr, it's going to enjoy the protection you describe
without needing the addition of a cryptographic signature.
However, that metadata doesn't live its whole life there. It
can reside in a tar file, it can cross a network, it can live
on a back-up tape. I think we agree that any time that metadata
is in transit or at rest outside of a Linux local filesystem, it
is exposed.
Thus I'm interested in a metadata protection mechanism that does
not rely on the security characteristics of a particular storage
container. For me, a cryptographic signature fits that bill
nicely.
>>> I think Mimi's other point is actually that IMA uses a flat hash
>>> which we derive by reading the entire file and then watching for
>>> mutations. Since you cannot guarantee we get notice of mutation
>>> with NFS, the entire IMA mechanism can't really be applied in its
>>> current form and we have to resort to chunk at a time verifications
>>> that a Merkel tree would provide.
>>
>> I'm not sure what you mean by this. An NFS client relies on
>> notification of mutation to maintain the integrity of its cache of
>> NFS file content, and it's done that since the 1980s.
>
> Mutation detection is part of the current IMA security model. If IMA
> sees a file mutate it has to be rehashed the next time it passes the
> gate. If we can't trust the NFS server, we can't trust the NFS
> mutation notification and we have to have a different mechanism to
> check the file.
When an NFS server lies about mtime and ctime, then NFS is completely
broken. Untrusted NFS server doesn't mean "broken behavior" -- I
would think that local filesystems will have the same problem if
they can't trust a local block device to store filesystem metadata
like indirect blocks and timestamps.
It's not clear to me that IMA as currently implemented can protect
against broken storage devices or incorrect filesystem behavior.
>> In addition to examining a file's mtime and ctime as maintained by
>> the NFS server, a client can rely on the file's NFSv4 change
>> attribute or an NFSv4 delegation.
>
> And that's secure in the face of a malicious or compromised server?
>
> The bottom line is still, I think we can't use linear hashes with an
> open/exec/mmap gate with NFS and we have to move to chunk at a time
> verification like that provided by a merkel tree.
That's fine until we claim that remote filesystems require one form of
metadata and local filesystems use some other form.
To guarantee an unbroken chain of provenance, everyone has to use the
same portable metadata format that is signed once by the content creator.
That's essentially why I believe the Merkle-based metadata format must
require that the tree root is signed.
--
Chuck Lever
[email protected]
On Thu, 2020-08-13 at 10:21 -0400, Chuck Lever wrote:
> > On Aug 12, 2020, at 11:42 AM, James Bottomley <James.Bottomley@Hans
> > enPartnership.com> wrote:
[...]
> > For most people the security mechanism of local xattrs is
> > sufficient. If you're paranoid, you don't believe it is and you
> > use EVM.
>
> When IMA metadata happens to be stored in local filesystems in
> a trusted xattr, it's going to enjoy the protection you describe
> without needing the addition of a cryptographic signature.
>
> However, that metadata doesn't live its whole life there. It
> can reside in a tar file, it can cross a network, it can live
> on a back-up tape. I think we agree that any time that metadata
> is in transit or at rest outside of a Linux local filesystem, it
> is exposed.
>
> Thus I'm interested in a metadata protection mechanism that does
> not rely on the security characteristics of a particular storage
> container. For me, a cryptographic signature fits that bill
> nicely.
Sure, but one of the points about IMA is a separation of mechanism from
policy. Signed hashes (called appraisal in IMA terms) is just one
policy you can decide to require or not or even make it conditional on
other things.
> > > > I think Mimi's other point is actually that IMA uses a flat
> > > > hash which we derive by reading the entire file and then
> > > > watching for mutations. Since you cannot guarantee we get
> > > > notice of mutation with NFS, the entire IMA mechanism can't
> > > > really be applied in its current form and we have to resort to
> > > > chunk at a time verifications that a Merkel tree would provide.
> > >
> > > I'm not sure what you mean by this. An NFS client relies on
> > > notification of mutation to maintain the integrity of its cache
> > > of NFS file content, and it's done that since the 1980s.
> >
> > Mutation detection is part of the current IMA security model. If
> > IMA sees a file mutate it has to be rehashed the next time it
> > passes the gate. If we can't trust the NFS server, we can't trust
> > the NFS mutation notification and we have to have a different
> > mechanism to check the file.
>
> When an NFS server lies about mtime and ctime, then NFS is completely
> broken. Untrusted NFS server doesn't mean "broken behavior" -- I
> would think that local filesystems will have the same problem if
> they can't trust a local block device to store filesystem metadata
> like indirect blocks and timestamps.
>
> It's not clear to me that IMA as currently implemented can protect
> against broken storage devices or incorrect filesystem behavior.
IMA doesn't really care about the storage. The gate check will fail if
the storage corrupts the file because the hashes won't match. The
mechanism for modification notification is the province of the
filesystem and there are definitely some which don't do it (or other fs
features) correctly and thus can't use IMA.
> > > In addition to examining a file's mtime and ctime as maintained
> > > by the NFS server, a client can rely on the file's NFSv4 change
> > > attribute or an NFSv4 delegation.
> >
> > And that's secure in the face of a malicious or compromised server?
> >
> > The bottom line is still, I think we can't use linear hashes with
> > an open/exec/mmap gate with NFS and we have to move to chunk at a
> > time verification like that provided by a merkel tree.
>
> That's fine until we claim that remote filesystems require one form
> of metadata and local filesystems use some other form.
>
> To guarantee an unbroken chain of provenance, everyone has to use the
> same portable metadata format that is signed once by the content
> creator. That's essentially why I believe the Merkle-based metadata
> format must require that the tree root is signed.
Well, no, that would be optional policy. We should certainly support
signed head hashes and require it if the policy said so, but we
shouldn't enforce it without the policy.
Suppose I'm a cloud service provider exporting files over NFS on the
control (private) network. I use IMA to measure untrusted tenants to
get a feel for what they're doing, but since I control the NFS server,
the client and the private network, I wouldn't feel the requirement to
have signed hashes because I trust other mechanisms for the security.
James
> On Aug 12, 2020, at 11:51 AM, James Bottomley <[email protected]> wrote:
>
> On Wed, 2020-08-12 at 10:15 -0400, Chuck Lever wrote:
>>> On Aug 11, 2020, at 11:53 AM, James Bottomley
>>> <[email protected]> wrote:
>>>
>>> On Tue, 2020-08-11 at 10:48 -0400, Chuck Lever wrote:
> [...]
>>>>>
>>>>> and what is nice to have to speed up the verification
>>>>> process. The choice for the latter is cache or reconstruct
>>>>> depending on the resources available. If the tree gets cached
>>>>> on the server, that would be a server implementation detail
>>>>> invisible to the client.
>>>>
>>>> We assume that storage targets (for block or file) are not
>>>> trusted. Therefore storage clients cannot rely on intermediate
>>>> results (eg, middle nodes in a Merkle tree) unless those results
>>>> are generated within the client's trust envelope.
>>>
>>> Yes, they can ... because supplied nodes can be verified. That's
>>> the whole point of a merkle tree. As long as I'm sure of the root
>>> hash I can verify all the rest even if supplied by an untrusted
>>> source. If you consider a simple merkle tree covering 4 blocks:
>>>
>>> R
>>> / \
>>> H11 H12
>>> / \ / \
>>> H21 H22 H23 H24
>>>> | | |
>>>
>>> B1 B2 B3 B4
>>>
>>> Assume I have the verified root hash R. If you supply B3 you also
>>> supply H24 and H11 as proof. I verify by hashing B3 to produce H23
>>> then hash H23 and H24 to produce H12 and if H12 and your supplied
>>> H11 hash to R the tree is correct and the B3 you supplied must
>>> likewise be correct.
>>
>> I'm not sure what you are proving here. Obviously this has to work
>> in order for a client to reconstruct the file's Merkle tree given
>> only R and the file content.
>
> You implied the server can't be trusted to generate the merkel tree.
> I'm showing above it can because of the tree path based verification.
What I was implying is that clients can't trust intermediate Merkle
tree content that is not also signed. So far we are talking about
signing only the tree root.
The storage server can store the tree durably, but if the intermediate
parts of the tree are not signed, the client has to verify them anyway,
and that reduces the value of storing potentially large data structures.
>> It's the construction of the tree and verification of the hashes that
>> are potentially expensive. The point of caching intermediate hashes
>> is so that the client verifies them as few times as possible. I
>> don't see value in caching those hashes on an untrusted server --
>> the client will have to reverify them anyway, and there will be no
>> savings.
>
> I'm not making any claim about server caching, I'm just saying the
> client can request pieces of the tree from the server without having to
> reconstruct the whole thing itself because it can verify their
> correctness.
To be clear, my concern is about how much of the tree might be stored
in a Merkle-based metadata format. I just don't see that it has much
value to store more than the signed tree root, because the client will
have to reconstitute or verify some tree contents on most every read.
For sufficiently large files, the tree itself can be larger than what
can be stored in an xattr. This is the same problem that fs-verity
faces. And, as I stated earlier, xattr objects are read in their
entirety, they can't be seeked into or read piecemeal.
What it seemed to me that you were suggesting was an offloaded cache
of the Merkle tree. Either the whole tree is stored on the storage
server, or the storage server provides a service that reconstitutes
that tree on behalf of clients. (Please correct me if I misunderstood).
I just don't think that will be practicable or provide the kind of
benefit you might want.
>> Cache once, as close as you can to where the data will be used.
>>
>>
>>>> So: if the storage target is considered inside the client's trust
>>>> envelope, it can cache or store durably any intermediate parts of
>>>> the verification process. If not, the network and file storage is
>>>> considered untrusted, and the client has to rely on nothing but
>>>> the signed digest of the tree root.
>>>>
>>>> We could build a scheme around, say, fscache, that might save the
>>>> intermediate results durably and locally.
>>>
>>> I agree we want caching on the client, but we can always page in
>>> from the remote as long as we page enough to verify up to R, so
>>> we're always sure the remote supplied genuine information.
>>
>> Agreed.
>>
>>
>>>>>> For this reason, the idea was to save only the signature of
>>>>>> the tree's root on durable storage. The client would retrieve
>>>>>> that signature possibly at open time, and reconstruct the
>>>>>> tree at that time.
>>>>>
>>>>> Right that's the integrity data you must have.
>>>>>
>>>>>> Or the tree could be partially constructed on-demand at the
>>>>>> time each unit is to be checked (say, as part of 2. above).
>>>>>
>>>>> Whether it's reconstructed or cached can be an implementation
>>>>> detail. You clearly have to reconstruct once, but whether you
>>>>> have to do it again depends on the memory available for caching
>>>>> and all the other resource calls in the system.
>>>>>
>>>>>> The client would have to reconstruct that tree again if
>>>>>> memory pressure caused some or all of the tree to be evicted,
>>>>>> so perhaps an on-demand mechanism is preferable.
>>>>>
>>>>> Right, but I think that's implementation detail. Probably what
>>>>> we need is a way to get the log(N) verification hashes from the
>>>>> server and it's up to the client whether it caches them or not.
>>>>
>>>> Agreed, these are implementation details. But see above about the
>>>> trustworthiness of the intermediate hashes. If they are conveyed
>>>> on an untrusted network, then they can't be trusted either.
>>>
>>> Yes, they can, provided enough of them are asked for to verify. If
>>> you look at the simple example above, suppose I have cached H11 and
>>> H12, but I've lost the entire H2X layer. I want to verify B3 so I
>>> also ask you for your copy of H24. Then I generate H23 from B3 and
>>> Hash H23 and H24. If this doesn't hash to H12 I know either you
>>> supplied me the wrong block or lied about H24. However, if it all
>>> hashes correctly I know you supplied me with both the correct B3
>>> and the correct H24.
>>
>> My point is there is a difference between a trusted cache and an
>> untrusted cache. I argue there is not much value in a cache where
>> the hashes have to be verified again.
>
> And my point isn't about caching, it's about where the tree comes from.
> I claim and you agree the client can get the tree from the server a
> piece at a time (because it can path verify it) and doesn't have to
> generate it itself.
OK, let's focus on where the tree comes from. It is certainly
possible to build protocol to exchange parts of a Merkle tree. The
question is how it might be stored on the server. There are some
underlying assumptions about the metadata storage mechanism that
should be stated up front.
Current forms of IMA metadata are limited in size and stored in a
container that is read and written in a single operation. If we stick
with that container format, I don't see a way to store a Merkle tree
in there for all file sizes.
Thus it seems to me that we cannot begin to consider the tree-on-the-
server model unless there is a proposed storage mechanism for that
whole tree. Otherwise, the client must have the primary role in
unpacking and verifying the tree.
Storing only the tree root in the metadata means the metadata format
is nicely bounded in size.
> How much of the tree the client has to store and
> whether the server caches, reads it in from somewhere or reconstructs
> it is an implementation detail.
Sure.
--
Chuck Lever
[email protected]
> On Aug 13, 2020, at 10:42 AM, James Bottomley <[email protected]> wrote:
>
> On Thu, 2020-08-13 at 10:21 -0400, Chuck Lever wrote:
>>> On Aug 12, 2020, at 11:42 AM, James Bottomley <James.Bottomley@Hans
>>> enPartnership.com> wrote:
> [...]
>>> For most people the security mechanism of local xattrs is
>>> sufficient. If you're paranoid, you don't believe it is and you
>>> use EVM.
>>
>> When IMA metadata happens to be stored in local filesystems in
>> a trusted xattr, it's going to enjoy the protection you describe
>> without needing the addition of a cryptographic signature.
>>
>> However, that metadata doesn't live its whole life there. It
>> can reside in a tar file, it can cross a network, it can live
>> on a back-up tape. I think we agree that any time that metadata
>> is in transit or at rest outside of a Linux local filesystem, it
>> is exposed.
>>
>> Thus I'm interested in a metadata protection mechanism that does
>> not rely on the security characteristics of a particular storage
>> container. For me, a cryptographic signature fits that bill
>> nicely.
>
> Sure, but one of the points about IMA is a separation of mechanism from
> policy. Signed hashes (called appraisal in IMA terms) is just one
> policy you can decide to require or not or even make it conditional on
> other things.
AFAICT, the current EVM_IMA_DIGSIG and EVM_PORTABLE_DIGSIG formats are
always signed. The policy choice is whether or not to verify the
signature, not whether or not the metadata format is signed.
--
Chuck Lever
[email protected]
On Thu, 2020-08-13 at 10:42 -0400, Chuck Lever wrote:
> > On Aug 12, 2020, at 11:51 AM, James Bottomley <James.Bottomley@Hans
> > enPartnership.com> wrote:
> > On Wed, 2020-08-12 at 10:15 -0400, Chuck Lever wrote:
> > > > On Aug 11, 2020, at 11:53 AM, James Bottomley
> > > > <[email protected]> wrote:
> > > > On Tue, 2020-08-11 at 10:48 -0400, Chuck Lever wrote:
[...]
> > > > > > > The client would have to reconstruct that tree again if
> > > > > > > memory pressure caused some or all of the tree to be
> > > > > > > evicted, so perhaps an on-demand mechanism is preferable.
> > > > > >
> > > > > > Right, but I think that's implementation detail. Probably
> > > > > > what we need is a way to get the log(N) verification hashes
> > > > > > from the server and it's up to the client whether it caches
> > > > > > them or not.
> > > > >
> > > > > Agreed, these are implementation details. But see above about
> > > > > the trustworthiness of the intermediate hashes. If they are
> > > > > conveyed on an untrusted network, then they can't be trusted
> > > > > either.
> > > >
> > > > Yes, they can, provided enough of them are asked for to
> > > > verify. If you look at the simple example above, suppose I
> > > > have cached H11 and H12, but I've lost the entire H2X layer. I
> > > > want to verify B3 so I also ask you for your copy of H24. Then
> > > > I generate H23 from B3 and Hash H23 and H24. If this doesn't
> > > > hash to H12 I know either you supplied me the wrong block or
> > > > lied about H24. However, if it all hashes correctly I know you
> > > > supplied me with both the correct B3 and the correct H24.
> > >
> > > My point is there is a difference between a trusted cache and an
> > > untrusted cache. I argue there is not much value in a cache where
> > > the hashes have to be verified again.
> >
> > And my point isn't about caching, it's about where the tree comes
> > from. I claim and you agree the client can get the tree from the
> > server a piece at a time (because it can path verify it) and
> > doesn't have to generate it itself.
>
> OK, let's focus on where the tree comes from. It is certainly
> possible to build protocol to exchange parts of a Merkle tree.
Which is what I think we need to extend IMA to do.
> The question is how it might be stored on the server.
I think the only thing the server has to guarantee to store is the head
hash, possibly signed.
> There are some underlying assumptions about the metadata storage
> mechanism that should be stated up front.
>
> Current forms of IMA metadata are limited in size and stored in a
> container that is read and written in a single operation. If we stick
> with that container format, I don't see a way to store a Merkle tree
> in there for all file sizes.
Well, I don't think you need to. The only thing that needs to be
stored is the head hash. Everything else can be reconstructed. If you
asked me to implement it locally, I'd probably put the head hash in an
xattr but use a CAM based cache for the merkel trees and construct the
tree on first access if it weren't already in the cache.
However, the above isn't what fs-verity does: it stores the tree in a
hidden section of the file. That's why I don't think we'd mandate
anything about tree storage. Just describe the partial retrieval
properties we'd like and leave the rest as an implementation detail.
> Thus it seems to me that we cannot begin to consider the tree-on-the-
> server model unless there is a proposed storage mechanism for that
> whole tree. Otherwise, the client must have the primary role in
> unpacking and verifying the tree.
Well, as I said, I don't think you need to store the tree. You
certainly could decide to store the entire tree (as fs-verity does) if
it fitted your use case, but it's not required. Perhaps even in my
case I'd make the CAM based cache persistent, like android's dalvik
cache.
James
> Storing only the tree root in the metadata means the metadata format
> is nicely bounded in size.
> On Aug 13, 2020, at 11:10 AM, James Bottomley <[email protected]> wrote:
>
> On Thu, 2020-08-13 at 10:42 -0400, Chuck Lever wrote:
>>> On Aug 12, 2020, at 11:51 AM, James Bottomley <James.Bottomley@Hans
>>> enPartnership.com> wrote:
>>> On Wed, 2020-08-12 at 10:15 -0400, Chuck Lever wrote:
>>>>> On Aug 11, 2020, at 11:53 AM, James Bottomley
>>>>> <[email protected]> wrote:
>>>>> On Tue, 2020-08-11 at 10:48 -0400, Chuck Lever wrote:
> [...]
>>>>>>>> The client would have to reconstruct that tree again if
>>>>>>>> memory pressure caused some or all of the tree to be
>>>>>>>> evicted, so perhaps an on-demand mechanism is preferable.
>>>>>>>
>>>>>>> Right, but I think that's implementation detail. Probably
>>>>>>> what we need is a way to get the log(N) verification hashes
>>>>>>> from the server and it's up to the client whether it caches
>>>>>>> them or not.
>>>>>>
>>>>>> Agreed, these are implementation details. But see above about
>>>>>> the trustworthiness of the intermediate hashes. If they are
>>>>>> conveyed on an untrusted network, then they can't be trusted
>>>>>> either.
>>>>>
>>>>> Yes, they can, provided enough of them are asked for to
>>>>> verify. If you look at the simple example above, suppose I
>>>>> have cached H11 and H12, but I've lost the entire H2X layer. I
>>>>> want to verify B3 so I also ask you for your copy of H24. Then
>>>>> I generate H23 from B3 and Hash H23 and H24. If this doesn't
>>>>> hash to H12 I know either you supplied me the wrong block or
>>>>> lied about H24. However, if it all hashes correctly I know you
>>>>> supplied me with both the correct B3 and the correct H24.
>>>>
>>>> My point is there is a difference between a trusted cache and an
>>>> untrusted cache. I argue there is not much value in a cache where
>>>> the hashes have to be verified again.
>>>
>>> And my point isn't about caching, it's about where the tree comes
>>> from. I claim and you agree the client can get the tree from the
>>> server a piece at a time (because it can path verify it) and
>>> doesn't have to generate it itself.
>>
>> OK, let's focus on where the tree comes from. It is certainly
>> possible to build protocol to exchange parts of a Merkle tree.
>
> Which is what I think we need to extend IMA to do.
>
>> The question is how it might be stored on the server.
>
> I think the only thing the server has to guarantee to store is the head
> hash, possibly signed.
>
>> There are some underlying assumptions about the metadata storage
>> mechanism that should be stated up front.
>>
>> Current forms of IMA metadata are limited in size and stored in a
>> container that is read and written in a single operation. If we stick
>> with that container format, I don't see a way to store a Merkle tree
>> in there for all file sizes.
>
> Well, I don't think you need to. The only thing that needs to be
> stored is the head hash. Everything else can be reconstructed. If you
> asked me to implement it locally, I'd probably put the head hash in an
> xattr but use a CAM based cache for the merkel trees and construct the
> tree on first access if it weren't already in the cache.
The contents of the security.ima xattr might be modeled after
EVM_IMA_DIGSIG:
- a format enumerator (used by all IMA metadata formats)
- the tree's unit size
- a fingerprint of the signer's certificate
- digest algorithm name and full digest
- the root hash, always signed
- signing algorithm name and signature
The rest of the hash tree is always stored somewhere else or
constructed on-demand.
My experience of security communities both within and outside the
IETF is that they would insist on always having a signature.
If one doesn't care about signing, a self-signed certificate can be
automatically provisioned when ima-evm-utils is installed that can
be used for those cases. That would make the signature process
invisible to any administrator who doesn't care about signed
metadata.
Because storage in NFS would cross trust boundaries, it would have
to require the use of a signed root hash. I don't want to be in the
position where copying a file with an unsigned root hash into NFS
makes it unreadable because of a change in policy.
> However, the above isn't what fs-verity does: it stores the tree in a
> hidden section of the file. That's why I don't think we'd mandate
> anything about tree storage. Just describe the partial retrieval
> properties we'd like and leave the rest as an implementation detail.
I'm starting to consider how much compatibility with fs-verity is
required. There are several forms of hash-tree, and a specification
of the IMA metadata format would need to describe exactly how to
form the tree root. If we want compatibility with fs-verity, then
it is reasonable to assume that this IMA metadata format might be
required to use the same hash tree construction algorithm that
fs-verity uses.
The original Merkle tree concept was patented 40 years ago. I'm not
clear yet on whether the patent encumbers the use of Merkle trees
in any way, but since their usage seems pretty widespread in P2P
and BitCoin applications, I'm guessing the answer to that is
favorable. More research needed.
There is an implementation used by several GNU utilities that is
available as a piece of GPL code. It could be a potential blocker
if that was the tree algorithm that fs-verity uses -- as discussed
in the other thread.
Apparently there are some known weaknesses in older hash tree
algorithms, including at least one CVE. We could choose a recent
algorithm, but perhaps there needs to be a degree of extensibility
in case that algorithm needs to be updated due to a subsequent
security issue.
Tree construction could include a few items besides file content to
help secure the hash further. For instance the file's size and mtime,
as well as the depth of the tree, could be included in the signature.
But that depends on whether it can be done while maintaining
compatibility with fs-verity.
I would feel better if someone with more domain expertise chimed in.
>> Thus it seems to me that we cannot begin to consider the tree-on-the-
>> server model unless there is a proposed storage mechanism for that
>> whole tree. Otherwise, the client must have the primary role in
>> unpacking and verifying the tree.
>
> Well, as I said, I don't think you need to store the tree.
We basically agree there.
> You certainly could decide to store the entire tree (as fs-verity does) if
> it fitted your use case, but it's not required. Perhaps even in my
> case I'd make the CAM based cache persistent, like android's dalvik
> cache.
>
> James
>
>
>> Storing only the tree root in the metadata means the metadata format
>> is nicely bounded in size.
--
Chuck Lever
[email protected]