Hi,
exactly this[0] happened today, on a 5.6.5 kernel:
process '/usr/bin/rsync' started with executable stack
But I can't reproduce this message, and rsync (v3.2.0, not exactly
abandonware) runs several times a day, so to repeat Andrew's questions[0]
from last year:
> What are poor users supposed to do if this message comes out?
> Hopefully google the message and end up at this thread. What do you
> want to tell them?
Also, the PID is missing from that message. I had some long running rsync
processes running earlier, maybe the RWE status would have been visible in
/proc/$PID/map, or somewhere else maybe?
Please advise? :-)
Thanks,
Christian.
[0] https://lore.kernel.org/patchwork/patch/1164047/#1362722
$ checksec --format=json --extended --file=`which rsync` | jq
{
"/usr/bin/rsync": {
"relro": "full",
"canary": "yes",
"nx": "no",
"pie": "yes",
"clangcfi": "no",
"safestack": "no",
"rpath": "no",
"runpath": "no",
"symbols": "no",
"fortify_source": "yes",
"fortified": "10",
"fortify-able": "19"
}
}
--
BOFH excuse #244:
Your cat tried to eat the mouse.
On Tue, Jun 23, 2020 at 10:39:26AM -0700, Christian Kujau wrote:
> Hi,
>
> exactly this[0] happened today, on a 5.6.5 kernel:
>
> process '/usr/bin/rsync' started with executable stack
>
> But I can't reproduce this message, and rsync (v3.2.0, not exactly
> abandonware) runs several times a day, so to repeat Andrew's questions[0]
> from last year:
>
> > What are poor users supposed to do if this message comes out?
> > Hopefully google the message and end up at this thread. What do you
> > want to tell them?
>
> Also, the PID is missing from that message. I had some long running rsync
> processes running earlier, maybe the RWE status would have been visible in
> /proc/$PID/map, or somewhere else maybe?
>
> Please advise? :-)
>
> Thanks,
> Christian.
>
> [0] https://lore.kernel.org/patchwork/patch/1164047/#1362722
>
>
> $ checksec --format=json --extended --file=`which rsync` | jq
> {
> "/usr/bin/rsync": {
> "relro": "full",
> "canary": "yes",
> "nx": "no",
^^^^^^^^^^^^^^^^^^
It is, indeed, marked executable, it seems. What distro is this?
-Kees
> "pie": "yes",
> "clangcfi": "no",
> "safestack": "no",
> "rpath": "no",
> "runpath": "no",
> "symbols": "no",
> "fortify_source": "yes",
> "fortified": "10",
> "fortify-able": "19"
> }
> }
>
> --
> BOFH excuse #244:
>
> Your cat tried to eat the mouse.
--
Kees Cook
On Tue, Jun 23, 2020 at 10:39:26AM -0700, Christian Kujau wrote:
> Hi,
>
> exactly this[0] happened today, on a 5.6.5 kernel:
>
> process '/usr/bin/rsync' started with executable stack
>
> But I can't reproduce this message, and rsync (v3.2.0, not exactly
> abandonware) runs several times a day, so to repeat Andrew's questions[0]
> from last year:
>
> > What are poor users supposed to do if this message comes out?
> > Hopefully google the message and end up at this thread. What do you
> > want to tell them?
>
> Also, the PID is missing from that message.
That's intentional. I for one hate pids.
> I had some long running rsync
> processes running earlier, maybe the RWE status would have been visible in
> /proc/$PID/map, or somewhere else maybe?
If you think process is still running, /proc/*/maps should have 'rwxp'
indeed. You can do quick
$ grep -e '\[stack\]' /proc/*/maps'
to find it.
> $ checksec --format=json --extended --file=`which rsync` | jq
> {
> "/usr/bin/rsync": {
> "relro": "full",
> "canary": "yes",
> "nx": "no",
> "pie": "yes",
> "clangcfi": "no",
> "safestack": "no",
> "rpath": "no",
> "runpath": "no",
> "symbols": "no",
> "fortify_source": "yes",
> "fortified": "10",
> "fortify-able": "19"
> }
> }
$ readelf -l /usr/bin/rsync | grep GNU_STACK -A1
GNU_STACK 0x0000000000000000 0x0000000000000000 0x0000000000000000
0x0000000000000000 0x0000000000000000 RW 0x10
On Wed, Jun 24, 2020 at 12:12:18AM +0300, Alexey Dobriyan wrote:
> On Tue, Jun 23, 2020 at 10:39:26AM -0700, Christian Kujau wrote:
> > Hi,
> >
> > exactly this[0] happened today, on a 5.6.5 kernel:
> >
> > process '/usr/bin/rsync' started with executable stack
> >
> > But I can't reproduce this message,
This message is once-per-reboot.
If you run something with exec stack after the message
you shouldn't get it second time.
If you rebooted and rsync starts clean, then it is separate story.
> > and rsync (v3.2.0, not exactly
> > abandonware) runs several times a day, so to repeat Andrew's questions[0]
> > from last year:
> >
> > > What are poor users supposed to do if this message comes out?
> > > Hopefully google the message and end up at this thread. What do you
> > > want to tell them?
> >
> > Also, the PID is missing from that message.
>
> That's intentional. I for one hate pids.
>
> > I had some long running rsync
> > processes running earlier, maybe the RWE status would have been visible in
> > /proc/$PID/map, or somewhere else maybe?
>
> If you think process is still running, /proc/*/maps should have 'rwxp'
> indeed. You can do quick
>
> $ grep -e '\[stack\]' /proc/*/maps'
>
> to find it.
Run as root obviously, or you won't get full picture.
On Tue, 23 Jun 2020, Kees Cook wrote:
> > $ checksec --format=json --extended --file=`which rsync` | jq
> > {
> > "/usr/bin/rsync": {
> > "relro": "full",
> > "canary": "yes",
> > "nx": "no",
> ^^^^^^^^^^^^^^^^^^
>
> It is, indeed, marked executable, it seems. What distro is this?
Arch Linux (x86-64) with 5.6.5.a-1-hardened[0], running in a Xen DomU.
Christian.
[0] https://git.archlinux.org/svntogit/packages.git/tree/trunk?h=packages/linux-hardened
--
BOFH excuse #211:
Lightning strikes.
On Tue, Jun 23, 2020 at 02:33:50PM -0700, Christian Kujau wrote:
> On Tue, 23 Jun 2020, Kees Cook wrote:
> > > $ checksec --format=json --extended --file=`which rsync` | jq
> > > {
> > > "/usr/bin/rsync": {
> > > "relro": "full",
> > > "canary": "yes",
> > > "nx": "no",
> > ^^^^^^^^^^^^^^^^^^
> >
> > It is, indeed, marked executable, it seems. What distro is this?
>
> Arch Linux (x86-64) with 5.6.5.a-1-hardened[0], running in a Xen DomU.
Congratulations!
https://www.archlinux.org/packages/extra/x86_64/rsync/
GNU_STACK 0x0000000000000000 0x0000000000000000 0x0000000000000000
0x0000000000000000 0x0000000000000000 RWE 0x10
On Tue, Jun 23, 2020 at 02:33:50PM -0700, Christian Kujau wrote:
> On Tue, 23 Jun 2020, Kees Cook wrote:
> > > $ checksec --format=json --extended --file=`which rsync` | jq
> > > {
> > > "/usr/bin/rsync": {
> > > "relro": "full",
> > > "canary": "yes",
> > > "nx": "no",
> > ^^^^^^^^^^^^^^^^^^
> >
> > It is, indeed, marked executable, it seems. What distro is this?
>
> Arch Linux (x86-64) with 5.6.5.a-1-hardened[0], running in a Xen DomU.
>
> Christian.
>
> [0] https://git.archlinux.org/svntogit/packages.git/tree/trunk?h=packages/linux-hardened
Fixed in testing (rsync-3.2.1-1-x86_64.pkg)
GNU_STACK 0x0000000000000000 0x0000000000000000 0x0000000000000000
0x0000000000000000 0x0000000000000000 RW 0x10
On Wed, 24 Jun 2020, Alexey Dobriyan wrote:
> > > process '/usr/bin/rsync' started with executable stack
> > > But I can't reproduce this message,
>
> This message is once-per-reboot.
Interesting, thanks. Now I know why I cannot reproduce this. I still
wonder what made rsync trigger this message today. The machine is running
for some weeks, rsync is run a few times an hour the whole day, regularly
and automatically, with always the same parameters. But oh, now I see,
rsync had been upgraded (automatically) over night:
> [ALPM] upgraded rsync (3.1.3-3 -> 3.2.0-1)
And indeed, the _older_ version had NX enabled:
$ wget https://archive.archlinux.org/packages/.all/rsync-3.1.3-3-x86_64.pkg.tar.zst
$ zstd -dc rsync-3.1.3-3-x86_64.pkg.tar.zst | tar -xf - usr/bin/rsync
$ checksec --format=json --extended --file=usr/bin/rsync | jq
{
"usr/bin/rsync": {
"relro": "full",
"canary": "yes",
"nx": "yes",
"pie": "yes",
"clangcfi": "no",
"safestack": "no",
"rpath": "no",
"runpath": "no",
"symbols": "no",
"fortify_source": "yes",
"fortified": "10",
"fortify-able": "19"
}
}
So, while I still think a PID would have been nice, now I know that it's
pr_warn_once and won't be printed again until after the next reboot. Going
to ask the Arch folks why NX has been disabled...
Thanks,
Christian.
--
BOFH excuse #211:
Lightning strikes.
On Tue, Jun 23, 2020 at 02:33:50PM -0700, Christian Kujau wrote:
> On Tue, 23 Jun 2020, Kees Cook wrote:
> > > $ checksec --format=json --extended --file=`which rsync` | jq
> > > {
> > > "/usr/bin/rsync": {
> > > "relro": "full",
> > > "canary": "yes",
> > > "nx": "no",
> > ^^^^^^^^^^^^^^^^^^
> >
> > It is, indeed, marked executable, it seems. What distro is this?
>
> Arch Linux (x86-64) with 5.6.5.a-1-hardened[0], running in a Xen DomU.
BTW this bug was exactly the one described in the changelog:
compiling assembly brings executable stack by default:
$ git-show 73faaab26d7db19ae6e04396a6e9d6372ed8e4ad
commit 73faaab26d7db19ae6e04396a6e9d6372ed8e4ad
Pass --noexecstack to assembler.
--- a/Makefile.in
+++ b/Makefile.in
@@ -135,7 +135,7 @@ simd-checksum-x86_64.o: simd-checksum-x86_64.cpp
$(CXX) -I. $(CXXFLAGS) $(CPPFLAGS) -c -o $@ $(srcdir)/simd-checksum-x86_64.cpp
lib/md5-asm-x86_64.o: lib/md5-asm-x86_64.S config.h lib/md-defines.h
- $(CC) -I. -c -o $@ $(srcdir)/lib/md5-asm-x86_64.S
+ $(CC) -I. -Wa,--noexecstack -c -o $@ $(srcdir)/lib/md5-asm-x86_64.S
On Wed, 24 Jun 2020, Alexey Dobriyan wrote:
> BTW this bug was exactly the one described in the changelog:
> compiling assembly brings executable stack by default:
Great, thanks for the pointer, will wait until this lands in Arch. My
search engine brought up the lkml discussion though, no the thread[0] on
rsync-cvs ;-)
Christian.
[0] https://lists.samba.org/archive/rsync-cvs/2020-June/007661.html
--
BOFH excuse #211:
Lightning strikes.
On Wed, Jun 24, 2020 at 12:22:14AM +0300, Alexey Dobriyan wrote:
> On Wed, Jun 24, 2020 at 12:12:18AM +0300, Alexey Dobriyan wrote:
> > On Tue, Jun 23, 2020 at 10:39:26AM -0700, Christian Kujau wrote:
> > > Hi,
> > >
> > > exactly this[0] happened today, on a 5.6.5 kernel:
> > >
> > > process '/usr/bin/rsync' started with executable stack
> > >
> > > But I can't reproduce this message,
>
> This message is once-per-reboot.
>
> If you run something with exec stack after the message
> you shouldn't get it second time.
If you want to reset this flag, you can do:
# echo 1 > /sys/kernel/debug/clear_warn_once
--
Kees Cook
On Tue, 23 Jun 2020, Kees Cook wrote:
> > If you run something with exec stack after the message
> > you shouldn't get it second time.
>
> If you want to reset this flag, you can do:
> # echo 1 > /sys/kernel/debug/clear_warn_once
Thanks. Although, I tend to not mount /sys/kernel/{config,debug,tracing}
and other things, I always thought they are not needed and could maybe
lower the attack surface if not mounted. Or maybe my tinfoil hat needs
some adjustment...
Christian.
--
BOFH excuse #279:
The static electricity routing is acting up...
In Debian testing the initrd triggers the warning.
[ 34.529809] process '/usr/bin/fstype' started with executable stack
$ checksec --format=json --extended --file=/var/tmp/mkinitramfs_eTyMPQ/bin/fstype | jq
{
"file": {
"relro": "no",
"canary": "no",
"nx": "no",
"pie": "no",
"clangcfi": "no",
"safestack": "no",
"rpath": "no",
"runpath": "no",
"symbols": "no",
"fortify_source": "no",
"fortified": "0",
"fortify-able": "0"
}
}
regards,
dan carpenter
On Wed, Jun 24, 2020 at 07:51:48PM +0300, Dan Carpenter wrote:
> In Debian testing the initrd triggers the warning.
>
> [ 34.529809] process '/usr/bin/fstype' started with executable stack
Where does fstype come from there? I am going to guess it is either
busybox or linked against klibc?
klibc has known problems with executable stacks due to its trampoline
implementation:
https://wiki.ubuntu.com/SecurityTeam/Roadmap/ExecutableStacks
>
> $ checksec --format=json --extended --file=/var/tmp/mkinitramfs_eTyMPQ/bin/fstype | jq
> {
> "file": {
> "relro": "no",
> "canary": "no",
> "nx": "no",
> "pie": "no",
> "clangcfi": "no",
> "safestack": "no",
> "rpath": "no",
> "runpath": "no",
> "symbols": "no",
> "fortify_source": "no",
> "fortified": "0",
> "fortify-able": "0"
> }
> }
Wow. _nothing_ implemented. :(
--
Kees Cook
On Wed, Jun 24, 2020 at 12:39:24PM -0700, Kees Cook wrote:
> On Wed, Jun 24, 2020 at 07:51:48PM +0300, Dan Carpenter wrote:
> > In Debian testing the initrd triggers the warning.
> >
> > [ 34.529809] process '/usr/bin/fstype' started with executable stack
>
> Where does fstype come from there? I am going to guess it is either
> busybox or linked against klibc?
>
> klibc has known problems with executable stacks due to its trampoline
> implementation:
> https://wiki.ubuntu.com/SecurityTeam/Roadmap/ExecutableStacks
Yeah. It comes from klibc-utils.
regards,
dan carpenter
On Thu, Jun 25, 2020 at 01:04:29PM +0300, Dan Carpenter wrote:
> On Wed, Jun 24, 2020 at 12:39:24PM -0700, Kees Cook wrote:
> > On Wed, Jun 24, 2020 at 07:51:48PM +0300, Dan Carpenter wrote:
> > > In Debian testing the initrd triggers the warning.
> > >
> > > [ 34.529809] process '/usr/bin/fstype' started with executable stack
> >
> > Where does fstype come from there? I am going to guess it is either
> > busybox or linked against klibc?
> >
> > klibc has known problems with executable stacks due to its trampoline
> > implementation:
> > https://wiki.ubuntu.com/SecurityTeam/Roadmap/ExecutableStacks
>
> Yeah. It comes from klibc-utils.
This is exactly what I was worried about back in Feb:
https://lore.kernel.org/lkml/202002251341.48BC06E@keescook/
This warning, combined with klibc-based initrds, makes the whole thing
pointless because it will always warn once on boot for the klibc stack,
and then not warn about anything else after that.
It looks like upstream klibc hasn't been touched in about 4 years, and
it's been up to Ben to keep it alive in Debian.
A couple ideas, in order of my preference:
1) stop using klibc-utils[1]. initramfs-tools-core is the only thing with a
dependency on klibc-utils. Only a few things are missing from busybox.
2) make the warning rate-limited instead?
3) fix the use of trampolines in klibc
Thoughts?
-Kees
[1] Ben appears well aware of this idea, as he suggested it in 2018. :)
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=887159
--
Kees Cook
Kees Cook dixit:
>3) fix the use of trampolines in klibc
AIUI done in klibc, but post-2.0.7
bye,
//mirabilos
--
FWIW, I'm quite impressed with mksh interactively. I thought it was much
*much* more bare bones. But it turns out it beats the living hell out of
ksh93 in that respect. I'd even consider it for my daily use if I hadn't
wasted half my life on my zsh setup. :-) -- Frank Terbeck in #!/bin/mksh
On Thu, Jun 25, 2020 at 09:16:10PM +0000, Thorsten Glaser wrote:
> Kees Cook dixit:
>
> >3) fix the use of trampolines in klibc
>
> AIUI done in klibc, but post-2.0.7
Ah-ha, I see it. Ben got it fixed in Feb. :)
https://git.kernel.org/pub/scm/libs/klibc/klibc.git/commit/?id=9d8d648e604026b32cad00a84ed6c29cbd157641
I don't see it in Debian? I remain confused, but there's hope!
https://salsa.debian.org/kernel-team/klibc/-/commits/master
--
Kees Cook
Le 25/06/2020 à 22:20, Kees Cook a écrit :
> On Thu, Jun 25, 2020 at 01:04:29PM +0300, Dan Carpenter wrote:
>> On Wed, Jun 24, 2020 at 12:39:24PM -0700, Kees Cook wrote:
>>> On Wed, Jun 24, 2020 at 07:51:48PM +0300, Dan Carpenter wrote:
>>>> In Debian testing the initrd triggers the warning.
>>>>
>>>> [ 34.529809] process '/usr/bin/fstype' started with executable stack
>>>
>>> Where does fstype come from there? I am going to guess it is either
>>> busybox or linked against klibc?
>>>
>>> klibc has known problems with executable stacks due to its trampoline
>>> implementation:
>>> https://wiki.ubuntu.com/SecurityTeam/Roadmap/ExecutableStacks
>>
>> Yeah. It comes from klibc-utils.
>
> This is exactly what I was worried about back in Feb:
> https://lore.kernel.org/lkml/202002251341.48BC06E@keescook/
>
> This warning, combined with klibc-based initrds, makes the whole thing
> pointless because it will always warn once on boot for the klibc stack,
> and then not warn about anything else after that.
>
> It looks like upstream klibc hasn't been touched in about 4 years, and
> it's been up to Ben to keep it alive in Debian.
>
> A couple ideas, in order of my preference:
>
> 1) stop using klibc-utils[1]. initramfs-tools-core is the only thing with a
> dependency on klibc-utils. Only a few things are missing from busybox.
Does busybox cleanly build with klibc lib ?
If it does, is the result as small ?
>
> 2) make the warning rate-limited instead?
>
> 3) fix the use of trampolines in klibc
That's done, see
https://git.kernel.org/pub/scm/libs/klibc/klibc.git/commit/?id=9d8d648e604026b32cad00a84ed6c29cbd157641
Discussed here
https://lists.zytor.com/archives/klibc/2020-February/004271.html
Christophe
>
> Thoughts?
>
> -Kees
>
>
> [1] Ben appears well aware of this idea, as he suggested it in 2018. :)
> https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=887159
>
On Thu, 2020-06-25 at 13:20 -0700, Kees Cook wrote:
> On Thu, Jun 25, 2020 at 01:04:29PM +0300, Dan Carpenter wrote:
> > On Wed, Jun 24, 2020 at 12:39:24PM -0700, Kees Cook wrote:
> > > On Wed, Jun 24, 2020 at 07:51:48PM +0300, Dan Carpenter wrote:
> > > > In Debian testing the initrd triggers the warning.
> > > >
> > > > [ 34.529809] process '/usr/bin/fstype' started with executable stack
> > >
> > > Where does fstype come from there? I am going to guess it is either
> > > busybox or linked against klibc?
> > >
> > > klibc has known problems with executable stacks due to its trampoline
> > > implementation:
> > > https://wiki.ubuntu.com/SecurityTeam/Roadmap/ExecutableStacks
> >
> > Yeah. It comes from klibc-utils.
>
> This is exactly what I was worried about back in Feb:
> https://lore.kernel.org/lkml/202002251341.48BC06E@keescook/
>
> This warning, combined with klibc-based initrds, makes the whole thing
> pointless because it will always warn once on boot for the klibc stack,
> and then not warn about anything else after that.
>
> It looks like upstream klibc hasn't been touched in about 4 years, and
> it's been up to Ben to keep it alive in Debian.
>
> A couple ideas, in order of my preference:
>
> 1) stop using klibc-utils[1]. initramfs-tools-core is the only thing with a
> dependency on klibc-utils. Only a few things are missing from busybox.
>
> 2) make the warning rate-limited instead?
>
> 3) fix the use of trampolines in klibc
It only uses trampolines on alpha, m68k, parisc, s390, and sparc32. As
of today, the master branch should correctly enable executable stacks
on these and only these architecture.
I have a development branch that sets sa_restorer and disables
executable stacks on alpha, s390, and sparc32:
https://git.kernel.org/pub/scm/libs/klibc/klibc.git/log/?h=execstack-fixes
But I haven't yet tested those changes other than on qemu-user.
The m68k and parisc kernel ports still don't support any alternatives
to trampolines for signal return, or they didn't when I reviewed this
a few months ago.
Ben.
> Thoughts?
>
> -Kees
>
>
> [1] Ben appears well aware of this idea, as he suggested it in 2018. :)
> https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=887159
>
--
Ben Hutchings
Anthony's Law of Force: Don't force it, get a larger hammer.
On Sat, Jul 25, 2020 at 10:27:56PM +0100, Ben Hutchings wrote:
> On Thu, 2020-06-25 at 13:20 -0700, Kees Cook wrote:
> > On Thu, Jun 25, 2020 at 01:04:29PM +0300, Dan Carpenter wrote:
> > > On Wed, Jun 24, 2020 at 12:39:24PM -0700, Kees Cook wrote:
> > > > On Wed, Jun 24, 2020 at 07:51:48PM +0300, Dan Carpenter wrote:
> > > > > In Debian testing the initrd triggers the warning.
> > > > >
> > > > > [ 34.529809] process '/usr/bin/fstype' started with executable stack
> > > >
> > > > Where does fstype come from there? I am going to guess it is either
> > > > busybox or linked against klibc?
> > > >
> > > > klibc has known problems with executable stacks due to its trampoline
> > > > implementation:
> > > > https://wiki.ubuntu.com/SecurityTeam/Roadmap/ExecutableStacks
> > >
> > > Yeah. It comes from klibc-utils.
> >
> > This is exactly what I was worried about back in Feb:
> > https://lore.kernel.org/lkml/202002251341.48BC06E@keescook/
> >
> > This warning, combined with klibc-based initrds, makes the whole thing
> > pointless because it will always warn once on boot for the klibc stack,
> > and then not warn about anything else after that.
> >
> > It looks like upstream klibc hasn't been touched in about 4 years, and
> > it's been up to Ben to keep it alive in Debian.
> >
> > A couple ideas, in order of my preference:
> >
> > 1) stop using klibc-utils[1]. initramfs-tools-core is the only thing with a
> > dependency on klibc-utils. Only a few things are missing from busybox.
> >
> > 2) make the warning rate-limited instead?
> >
> > 3) fix the use of trampolines in klibc
>
> It only uses trampolines on alpha, m68k, parisc, s390, and sparc32. As
> of today, the master branch should correctly enable executable stacks
> on these and only these architecture.
>
> I have a development branch that sets sa_restorer and disables
> executable stacks on alpha, s390, and sparc32:
>
> https://git.kernel.org/pub/scm/libs/klibc/klibc.git/log/?h=execstack-fixes
Awesome! Thanks for tackling this. :)
--
Kees Cook