Hello,
as of linux 3.6 hardlink restrictions to the Linux VFS have been enabled
by default. This breaks the application AFD [1] of which I am the author.
Internally it uses hardlink to distribute files. The reason for hardlinks
is that AFD can distribute one file to many destinations and for each
distributing process it creates a directory with hardlinks to the original
file. That way AFD itself never needs to copy the content of a file. Another
nice feature about hardlinks was that there is no need to have any logic in
the code needing AFD to know where the original file was, each distributing
process could delete its hardlink and the last one would delete the real
file. This way AFD could distribute files at rates of more then 20000 files
per second (in benchmarks). This has worked from the first linux kernel
up to 3.5.7 and with solaris, hpux, aix, ftx, irix. As of 3.6 this does
not work for files where AFD does not have write permissions. It was always
sufficient to just have read permission on a file it wants to distribute.
The fix for the "at" daemon [2] mentioned in the commitdiff [3] cannot
be used for AFD since it is not run with root privileges. Is there any
other way I can "fix" my application? I currently can see no other way
then doing it via: echo 0 > /proc/sys/fs/protected_hardlinks
Why is such a fundamentally change to the linux kernel activated by default?
Would it not be better if it is the other way around, that the system
administrator or distributions enable this?
Regards,
Holger
PS: Please CC me as I am not on the list.
[1] http://www.dwd.de/AFD
[2] http://anonscm.debian.org/gitweb/?p=collab-maint/at.git;a=commitdiff;h=f4114656c3a6c6f6070e315ffdf940a49eda3279
[3] https://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=800179c9b8a1e796e441674776d11cd4c05d61d7
Hi Holger,
On Thu, Oct 25, 2012 at 12:13:40PM +0000, Holger Kiehl wrote:
> as of linux 3.6 hardlink restrictions to the Linux VFS have been enabled
> by default. This breaks the application AFD [1] of which I am the author.
Sorry this created a problem for you!
> Internally it uses hardlink to distribute files. The reason for hardlinks
> is that AFD can distribute one file to many destinations and for each
> distributing process it creates a directory with hardlinks to the original
> file. That way AFD itself never needs to copy the content of a file. Another
> nice feature about hardlinks was that there is no need to have any logic in
> the code needing AFD to know where the original file was, each distributing
> process could delete its hardlink and the last one would delete the real
> file. This way AFD could distribute files at rates of more then 20000 files
> per second (in benchmarks). This has worked from the first linux kernel
> up to 3.5.7 and with solaris, hpux, aix, ftx, irix. As of 3.6 this does
> not work for files where AFD does not have write permissions. It was always
> sufficient to just have read permission on a file it wants to distribute.
Just to clarify, not even read access was needed for hardlinks:
$ whoami
kees
$ ls -l /etc/shadow
-r--r----- 1 root shadow 3112 Oct 22 17:02 /etc/shadow
$ ln /etc/shadow /tmp/ohai
$ ls -l /tmp/ohai
-r--r----- 2 root shadow 3112 Oct 22 17:02 ohai
You mention "the last one would delete the real file". That would have
required AFD to have write permission to the directory where the original
file existed? Maybe there is something in your architecture that could
take advantage of that? Directory group-write set-gid? I haven't taken
a look at AFD's code.
> The fix for the "at" daemon [2] mentioned in the commitdiff [3] cannot
> be used for AFD since it is not run with root privileges. Is there any
> other way I can "fix" my application? I currently can see no other way
> then doing it via: echo 0 > /proc/sys/fs/protected_hardlinks
You said you have read access to these files, so perhaps you can make
a copy when you have read but not write, and then all the subsequent
duplication would be able to hardlink?
If you wanted to turn off the sysctl, you could have AFD could ship
files in /etc/sysctl.d/ (or your distro equivalent) to turn it off.
I'm sure there are plenty of options available.
> Why is such a fundamentally change to the linux kernel activated by default?
Based on about two years of testing in Ubuntu, the number of problems
was vanishingly small, so the security benefit is seen to outweigh
the downside.
> Would it not be better if it is the other way around, that the system
> administrator or distributions enable this?
Virtually all distributions would have turned this on by default,
so it seemed better to many people to just make it the default in the
kernel. Only unusual corner-cases would need it disabled.
I hope that helps!
-Kees
--
Kees Cook @outflux.net
Hello Kees,
first, many thanks for trying to help!
On Thu, 25 Oct 2012, Kees Cook wrote:
> Hi Holger,
>
> On Thu, Oct 25, 2012 at 12:13:40PM +0000, Holger Kiehl wrote:
>> as of linux 3.6 hardlink restrictions to the Linux VFS have been enabled
>> by default. This breaks the application AFD [1] of which I am the author.
>
> Sorry this created a problem for you!
>
>> Internally it uses hardlink to distribute files. The reason for hardlinks
>> is that AFD can distribute one file to many destinations and for each
>> distributing process it creates a directory with hardlinks to the original
>> file. That way AFD itself never needs to copy the content of a file. Another
>> nice feature about hardlinks was that there is no need to have any logic in
>> the code needing AFD to know where the original file was, each distributing
>> process could delete its hardlink and the last one would delete the real
>> file. This way AFD could distribute files at rates of more then 20000 files
>> per second (in benchmarks). This has worked from the first linux kernel
>> up to 3.5.7 and with solaris, hpux, aix, ftx, irix. As of 3.6 this does
>> not work for files where AFD does not have write permissions. It was always
>> sufficient to just have read permission on a file it wants to distribute.
>
> Just to clarify, not even read access was needed for hardlinks:
>
> $ whoami
> kees
> $ ls -l /etc/shadow
> -r--r----- 1 root shadow 3112 Oct 22 17:02 /etc/shadow
> $ ln /etc/shadow /tmp/ohai
> $ ls -l /tmp/ohai
> -r--r----- 2 root shadow 3112 Oct 22 17:02 ohai
>
Correct, but when AFD wants to distribute the file via for example FTP
it must have read access on the file. Because it needs to read the file
when it wants to send it on a socket.
> You mention "the last one would delete the real file". That would have
> required AFD to have write permission to the directory where the original
> file existed? Maybe there is something in your architecture that could
> take advantage of that? Directory group-write set-gid? I haven't taken
> a look at AFD's code.
>
Right, it must have write permission on the directory that is monitored
by AFD. When it detects a file it moves (rename()) it to an internal
directory where AFD works. So this step still works. But from there it
creates hardlinks for each distributing job. But this no longer works if
AFD does not have write access on the file itself. So even if set-gid
is set, this would still not work if the file does not have write
permission for the group.
>> The fix for the "at" daemon [2] mentioned in the commitdiff [3] cannot
>> be used for AFD since it is not run with root privileges. Is there any
>> other way I can "fix" my application? I currently can see no other way
>> then doing it via: echo 0 > /proc/sys/fs/protected_hardlinks
>
> You said you have read access to these files, so perhaps you can make
> a copy when you have read but not write, and then all the subsequent
> duplication would be able to hardlink?
>
This is exactly what AFD tries to avoid. AFD is used on systems where it
distributes Terabytes of data daily and if it would need to copy the file
first imagine the strain it imposes on those servers.
> If you wanted to turn off the sysctl, you could have AFD could ship
> files in /etc/sysctl.d/ (or your distro equivalent) to turn it off.
>
Yes, that could be done. However, I do not want as a maintainer of one
software package by default disable or enable anything in the kernel.
I do not think the system administrators would like this.
> I'm sure there are plenty of options available.
>
Sorry, I cannot see them. But please if you or others have more ideas
I am certainly open to change AFD if it can be done efficiently.
>> Why is such a fundamentally change to the linux kernel activated by default?
>
> Based on about two years of testing in Ubuntu, the number of problems
> was vanishingly small, so the security benefit is seen to outweigh
> the downside.
>
Ubuntu is known to be very user friendly and mostly used by users on their
laptops/pc's and not so common in the server environment such as Redhat,
SLES, etc. So I question the statement "vanishingly small", when you
enable it in those environments by default.
And I think there is a real benefit in that one can do hardlinks on a
file that one does not own, which I think was not seen by those that
disable this feature now by default.
>> Would it not be better if it is the other way around, that the system
>> administrator or distributions enable this?
>
> Virtually all distributions would have turned this on by default,
> so it seemed better to many people to just make it the default in the
> kernel. Only unusual corner-cases would need it disabled.
>
So you too would say not all distributions would enable it by default.
Would it then not be better for them to first try this and see if the number
of problems is really "vanishingly small". And then if all distributions
enable this by default one can do it in the kernel by default as well.
Has it not always worked that way?
Again many thanks for trying to help!
Regards,
Holger
On Thu, Oct 25, 2012 at 5:13 AM, Holger Kiehl <[email protected]> wrote:
>
> as of linux 3.6 hardlink restrictions to the Linux VFS have been enabled
> by default. This breaks the application AFD [1] of which I am the author.
Ok, we had a previous report of breakage, but that was just local
scripting. Since that was just a single user (Nick Bowler), and he was
ok with just fixing his setup, I let it go, waiting to see if anybody
else reacted.
There may well have been other users that had odd breakage, but didn't
realize what the cause was.
Regardless, clearly this does break things, and as such needs to be
undone. We do not cause regressions that people notice in the kernel.
So I've defaulted these things to off, and marked it for stable. See
commit 561ec64ae67e ("VFS: don't do protected {sym,hard}links by
default"). Either distributions can enable it with some security
setting (along with the other security things they do, like the whole
selinux thing), or we might at some future date make some config
option for "boot up in hard-*ss mode that may break things", but for
now we clearly cannot enable it by default.
I've added people from the original commit and the previous discussion
to the cc, and marked the commit for stable too.
Thanks,
Linus
On Fri, Oct 26, 2012 at 10:22 AM, Linus Torvalds
<[email protected]> wrote:
> On Thu, Oct 25, 2012 at 5:13 AM, Holger Kiehl <[email protected]> wrote:
>>
>> as of linux 3.6 hardlink restrictions to the Linux VFS have been enabled
>> by default. This breaks the application AFD [1] of which I am the author.
>
> Ok, we had a previous report of breakage, but that was just local
> scripting. Since that was just a single user (Nick Bowler), and he was
> ok with just fixing his setup, I let it go, waiting to see if anybody
> else reacted.
>
> There may well have been other users that had odd breakage, but didn't
> realize what the cause was.
>
> Regardless, clearly this does break things, and as such needs to be
> undone. We do not cause regressions that people notice in the kernel.
>
> So I've defaulted these things to off, and marked it for stable. See
> commit 561ec64ae67e ("VFS: don't do protected {sym,hard}links by
> default"). Either distributions can enable it with some security
> setting (along with the other security things they do, like the whole
> selinux thing), or we might at some future date make some config
> option for "boot up in hard-*ss mode that may break things", but for
> now we clearly cannot enable it by default.
>
> I've added people from the original commit and the previous discussion
> to the cc, and marked the commit for stable too.
Ok, seems fair. I've sent a patch to add the config options.
-Kees
--
Kees Cook
Chrome OS Security