On 2015-11-17 14:30, Al Viro wrote:
> On Tue, Nov 17, 2015 at 02:02:09PM -0500, Austin S Hemmelgarn wrote:
>
>>> _Static_ attacks, or change-image-under-mounted-fs attacks?
>> To properly protect against attacks on mounted filesystems, we'd
>> need some new concept of a userspace immutable file (that is, one
>> where nobody can write to it except the kernel, and only the kernel
>> can change it between regular access and this new state), and then
>> have the kernel set an image (or block device) to this state when a
>> filesystem is mounted from it (this introduces all kinds of other
>> issues too however, for example stuff that allows an online fsck on
>> the device will stop working, as will many un-deletion tools).
>>
>> The only other option would be to force the FS to cache all metadata
>> in memory, and validate between the cache and what's on disk on
>> every access, which is not realistic for any real world system.
>
> Doctor, it hurt when I do it...
>
> IOW, the other option is to refuse attempting this insanity. Fuse probably
> can be handled, but being able to mount (with kernel-space drivera) an
> arbitrary ext4 image is equivalent to being able to do anything and it's
> going to stay that way for the forseeable future. You are talking about
> a large pile of code that deals with rather convoluted data structure,
> had not been written with validation in mind *and* keeps being developed.
> What's more, that code runs with maximal priveleges there are.
Without factoring in unprivileged mounts, for cases when mounting from a
block device, you shouldn't have to worry about a malicious third party,
because their ability to modify it under the filesystem implies that
they either already have root privileges, or they have direct access to
hardware, and in both cases, your system is already compromised to a
degree that makes the reliability of your filesystem irrelevant. Under
the same circumstances with filesystem images, the same statement applies.

However, while unprivileged mounts make validation more important, there
is still the fact that if you don't trust someone, then you shouldn't be
letting them have access to your system. No amount of sandboxing short
of full isolation can solve that, period. Yes people will try to crack
the system, but no matter how much sandboxing there is, it will not be
unbreakable unless nobody can access it.

The attack surface is already there, it's just hard to get to. There's
a reason I never mount anything I didn't create myself and can't prove
chain of custody on without at least running fsck on it first, and then
only mount it with the kernel if there isn't a FUSE driver for it that I
trust (and GRUB provides a lot of those).
>
> This is absolutely insane, no matter how much LSM snake oil you slatter on
> the whole thing. All of a sudden you are exposing a huge attack surface
> in the place where it would hurt most and as the consolation we are offered
> basically "Ted is willing to fix holes when they are found".
For the context of static image attacks, anything that's found _needs_
to be fixed regardless, and unless you can find some way to actually
prevent attacks on mounted filesystems that doesn't involve a complete
re-write of the filesystem drivers, then there's not much we can do
about it. Yes, unprivileged mounts expose an attack surface, but so
does userspace access to the network stack, and so do a lot of other
features that are considered essential in a modern general purpose
operating system.

Attachments:

smime.p7s (2.95 kB)
S/MIME Cryptographic Signature

2015-11-17 20:55:14

On Wed, Nov 18, 2015 at 11:12:51AM +1100, James Morris wrote:
> On Tue, 17 Nov 2015, Seth Forshee wrote:
>
> > + sbsp = inode->i_sb->s_security;
> > + if ((sbsp->smk_flags & SMK_SB_UNTRUSTED) &&
>
> Where is SMK_SB_UNTRUSTED defined?
>
> I can't see it in this patch series, mainline or security next.

Eric's already applied a few of my patches to a branch in his tree,
including the one that defines this. See the for-testing branch of
git://git.kernel.org/pub/scm/linux/kernel/git/ebiederm/user-namespace.git,
specifically
https://git.kernel.org/cgit/linux/kernel/git/ebiederm/user-namespace.git/commit/?id=36979551f2345d4e20714c1fc53d6a15c87f8b64.

Thanks,
Seth

2015-11-18 12:24:17

by Austin S Hemmelgarn

[permalink] [raw]

Subject: Re: [PATCH v3 0/7] User namespace mount updates

On 2015-11-17 16:32, Seth Forshee wrote:
> On Tue, Nov 17, 2015 at 03:54:50PM -0500, Austin S Hemmelgarn wrote:
>> On 2015-11-17 14:16, Seth Forshee wrote:
>>> On Tue, Nov 17, 2015 at 02:02:09PM -0500, Austin S Hemmelgarn wrote:
>>>> On 2015-11-17 12:55, Al Viro wrote:
>>>>> On Tue, Nov 17, 2015 at 11:25:51AM -0600, Seth Forshee wrote:
>>>>>
>>>>>> Shortly after that I plan to follow with support for ext4. I've been
>>>>>> fuzzing ext4 for a while now and it has held up well, and I'm currently
>>>>>> working on hand-crafted attacks. Ted has commented privately (to others,
>>>>>> not to me personally) that he will fix bugs for such attacks, though I
>>>>>> haven't seen any public comments to that effect.
>>>>>
>>>>> _Static_ attacks, or change-image-under-mounted-fs attacks?
>>>> To properly protect against attacks on mounted filesystems, we'd
>>>> need some new concept of a userspace immutable file (that is, one
>>>> where nobody can write to it except the kernel, and only the kernel
>>>> can change it between regular access and this new state), and then
>>>> have the kernel set an image (or block device) to this state when a
>>>> filesystem is mounted from it (this introduces all kinds of other
>>>> issues too however, for example stuff that allows an online fsck on
>>>> the device will stop working, as will many un-deletion tools).
>>>
>>> Yeah, Serge and I were just tossing that idea around on irc. If we can
>>> make that work then it's probably the best solution.
>>>
>>> From a naive perspective it seems like all we really have to do is make
>>> the block device inode immutable to userspace when it is mounted. And
>>> the parent block device if it's a partition, which might be a bit
>>> troublesome. We'd have to ensure that writes couldn't happen via any fds
>>> already open when the device was mounted too.
>>>
>>> We'd need some cooperation from the loop driver too I suppose, to make
>>> sure the file backing the loop device is also immutable.
>>>
>> From a completeness perspective, you'd also need to hook into DM,
>> MD, and bcache to handle their backing devices. There's not much we
>> could do about iSCSI/ATAoE/NBD devices, and I think being able to
>
> But really no one would be able to mount any of those without
> intervention from a privileged user anyway. The same is true today of
> loop devices, but I have some patches to change that.
Um, no, depending on how your system is configured, it's fully possible
to mount those as a regular user with no administrative interaction
required. All that needs done is some udev rules (or something
equivalent) to add ACL's to the device nodes allowing regular users to
access them (and there are systems I've seen that are naive enough to do
this). And I can almost assure you that there will be someone out there
who decides to directly expose such block devices to a user namespace.

Attachments:

smime.p7s (2.95 kB)
S/MIME Cryptographic Signature

2015-11-18 12:47:20

On 2015-11-18 09:58, Al Viro wrote:
> On Wed, Nov 18, 2015 at 08:22:38AM -0600, Seth Forshee wrote:
>
>> But it still requires the admin set it up that way, no? And aren't
>> privileges required to set up those devices in the first place?
>>
>> I'm not saying that it wouldn't be a good idea to lock down the backing
>> stores for those types of devices too, just that it isn't something that
>> a regular user could exploit without an admin doing something to
>> facilitate it.
>
> Sigh... If it boils down to "all admins within all containers must be
> trusted not to try and break out" (along with "roothole in any container
> escalates to kernel-mode code execution on host"), then what the fuck
> is the *point* of bothering with containers, userns, etc. in the first
> place? If your model is basically "you want isolation, just use kvm",
> fine, but where's the place for userns in all that?
In this case, Seth is referring to the host admin, not the container admin.
>
> And if you are talking about the _host_ admin, then WTF not have him just
> mount what's needed as part of setup and to hell with mounting those
> inside the container?
This is decidedly non-trivial to handle in some cases. IIRC, one of the
particular things that sparked this in the first place was the Chrome
Native Client having to have CAP_SYS_ADMIN or SUID set on it to handle
setting up it's own sandbox, which is not something that should ever be
set on an executable that runs untrusted code (which is the whole point
of NaCl).
>
> Look at that from the hosting company POV - they are offering a bunch of
> virtual machines on one physical system. And you want the admins on those
> virtual machines independent from the host admin. Fine, but then you
> really need to keep them unable to screw each other or gain kernel-mode
> execution on the host.
>
> Again, what's the point of all that? I assumed the model where containers
> do, you know, contain what's in them, regardless of trust. You guys seem
> to assume something different and I really wonder what it _is_...
Yes, hosting and isolation of untrusted code are valid uses for
containers, which is why I suggested the ability to disallow mounts
other than FUSE, and make that the default state. There are other
perfectly valid uses for them as well, and for me the two I'm
particularly interested in are safe deployment of a new system from an
existing system (ala Gentoo or Arch installation, or manual installation
of *BSD), and running non-native distros without virtualization (On a
single user system, virtualization is overkill when all you want is a
Debian or Fedora or Arch testing environment and don't care about their
specific kernel features).

Attachments:

smime.p7s (2.95 kB)
S/MIME Cryptographic Signature

2015-11-18 15:36:57

by Angel Shtilianov

[permalink] [raw]

Subject: Re: [PATCH v3 0/7] User namespace mount updates

On 11/18/2015 04:58 PM, Al Viro wrote:
> On Wed, Nov 18, 2015 at 08:22:38AM -0600, Seth Forshee wrote:
>
>> But it still requires the admin set it up that way, no? And aren't
>> privileges required to set up those devices in the first place?
>>
>> I'm not saying that it wouldn't be a good idea to lock down the backing
>> stores for those types of devices too, just that it isn't something that
>> a regular user could exploit without an admin doing something to
>> facilitate it.
>
> Sigh... If it boils down to "all admins within all containers must be
> trusted not to try and break out" (along with "roothole in any container
> escalates to kernel-mode code execution on host"), then what the fuck
> is the *point* of bothering with containers, userns, etc. in the first
> place? If your model is basically "you want isolation, just use kvm",
> fine, but where's the place for userns in all that?
>
> And if you are talking about the _host_ admin, then WTF not have him just
> mount what's needed as part of setup and to hell with mounting those
> inside the container?
>
> Look at that from the hosting company POV - they are offering a bunch of
> virtual machines on one physical system. And you want the admins on those
> virtual machines independent from the host admin. Fine, but then you
> really need to keep them unable to screw each other or gain kernel-mode
> execution on the host.

Actually from the POV of a hosting company there's also the use case of
wanting to use container as substitutes for virtual machines (of course
we are a long way from that). But being able to do what those patches
enable (i.e. what Seth has pointed to with mount -o loop) is beneficial
and desirable.

>
> Again, what's the point of all that? I assumed the model where containers
> do, you know, contain what's in them, regardless of trust. You guys seem
> to assume something different and I really wonder what it _is_...
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at http://www.tux.org/lkml/
>

2015-11-18 15:39:27

On Thu, Nov 19, 2015 at 06:19:10PM +0200, Octavian Purdila wrote:
> On Thu, Nov 19, 2015 at 5:23 PM, Seth Forshee
> > And for the second:
> >
> > stress-ng --class filesystem --seq 0 -v -t 60
> >
> > There really wasn't anything interesting in the lklfuse output for the
> > first run, but for the second run I pasted the output here:
> > http://paste.ubuntu.com/13346993/
>
> lklfuse allocates a fixed 100MB to the kernel and this is probably not
> enough. For the short term I can add a parameter to lklfuse that
> allows the user to specify the amount of memory to allocate to lkl. A
> better fix would probably be to dynamically adjust the memory size of
> lkl. I am thinking of using the ballon virtio driver or the memory
> hotplug infrastructure. Any other suggestions?

Hi Octavian,

Like Seth said this was very nice to install and run. But I think I'm
doing something wrong. I cloned the lkl/linux git tree into a 9G ext4
loopback file, both one mounted with lkl and one mounted as loopback.
The results were worse than I expected - loopback was 0m10.078s for
clone and 0m0.877s for rm -rf; lkl was 5m5.625s for clone and 0m44.332s
for rm -rf. I assume this has to do with the 100MB buffer and subsequent
thrashing? How should I tune this to make it perform better?

No crashes from this, though.

thanks,
-serge