2019-01-28 08:35:15

by Alexander Sverdlin

[permalink] [raw]
Subject: Re: [PATCH] audit: always enable syscall auditing when supported and audit is enabled

Hello Paul,

On 08/12/2015 17:42, Paul Moore wrote:
> To the best of our knowledge, everyone who enables audit at compile
> time also enables syscall auditing; this patch simplifies the Kconfig
> menus by removing the option to disable syscall auditing when audit
> is selected and the target arch supports it.
>
> Signed-off-by: Paul Moore <[email protected]>

this patch is responsible for massive performance degradation for those
who used only CONFIG_SECURITY_APPARMOR.

And the numbers are, take the following test for instance:

dd if=/dev/zero of=/dev/null count=2M

ARM64: 500MB/s -> 350MB/s
ARM: 400MB/s -> 300MB/s

(with some particular CPU frequencies).

Should we revert this patch?

> ---
> init/Kconfig | 11 +++--------
> 1 file changed, 3 insertions(+), 8 deletions(-)
>
>
> --
> Linux-audit mailing list
> [email protected]
> https://www.redhat.com/mailman/listinfo/linux-audit
>
> diff --git a/init/Kconfig b/init/Kconfig
> index c24b6f7..d4663b1 100644
> --- a/init/Kconfig
> +++ b/init/Kconfig
> @@ -299,20 +299,15 @@ config AUDIT
> help
> Enable auditing infrastructure that can be used with another
> kernel subsystem, such as SELinux (which requires this for
> - logging of avc messages output). Does not do system-call
> - auditing without CONFIG_AUDITSYSCALL.
> + logging of avc messages output). System call auditing is included
> + on architectures which support it.
>
> config HAVE_ARCH_AUDITSYSCALL
> bool
>
> config AUDITSYSCALL
> - bool "Enable system-call auditing support"
> + def_bool y
> depends on AUDIT && HAVE_ARCH_AUDITSYSCALL
> - default y if SECURITY_SELINUX
> - help
> - Enable low-overhead system-call auditing infrastructure that
> - can be used independently or with another kernel subsystem,
> - such as SELinux.
>
> config AUDIT_WATCH
> def_bool y
>

--
Best regards,
Alexander Sverdlin.


2019-01-28 14:20:40

by Paul Moore

[permalink] [raw]
Subject: Re: [PATCH] audit: always enable syscall auditing when supported and audit is enabled

On Mon, Jan 28, 2019 at 4:04 AM Sverdlin, Alexander (Nokia - DE/Ulm)
<[email protected]> wrote:
> Hello Paul,
>
> On 08/12/2015 17:42, Paul Moore wrote:
> > To the best of our knowledge, everyone who enables audit at compile
> > time also enables syscall auditing; this patch simplifies the Kconfig
> > menus by removing the option to disable syscall auditing when audit
> > is selected and the target arch supports it.
> >
> > Signed-off-by: Paul Moore <[email protected]>
>
> this patch is responsible for massive performance degradation for those
> who used only CONFIG_SECURITY_APPARMOR.
>
> And the numbers are, take the following test for instance:
>
> dd if=/dev/zero of=/dev/null count=2M
>
> ARM64: 500MB/s -> 350MB/s
> ARM: 400MB/s -> 300MB/s

Hi there.

Out of curiosity, what kernel/distribution are you running, or is this
a custom kernel compile? Can you also share the output of 'auditctl
-l' from your system? The general approach taken by everyone to
turn-off the per-syscall audit overhead is to add the "-a never,task"
rule to their audit configuration:

# auditctl -a never,task

If you are using Fedora/CentOS/RHEL, or a similarly configured system,
you can find this configuration in the /etc/audit/audit.rules file (be
warned, that file is automatically generated based on
/etc/audit/rules.d).

--
paul moore
http://www.paul-moore.com

2019-01-28 14:38:15

by Alexander Sverdlin

[permalink] [raw]
Subject: Re: [PATCH] audit: always enable syscall auditing when supported and audit is enabled

Hello Paul,

On 28/01/2019 15:19, Paul Moore wrote:
>>> time also enables syscall auditing; this patch simplifies the Kconfig
>>> menus by removing the option to disable syscall auditing when audit
>>> is selected and the target arch supports it.
>>>
>>> Signed-off-by: Paul Moore <[email protected]>
>> this patch is responsible for massive performance degradation for those
>> who used only CONFIG_SECURITY_APPARMOR.
>>
>> And the numbers are, take the following test for instance:
>>
>> dd if=/dev/zero of=/dev/null count=2M
>>
>> ARM64: 500MB/s -> 350MB/s
>> ARM: 400MB/s -> 300MB/s
> Hi there.
>
> Out of curiosity, what kernel/distribution are you running, or is this
> a custom kernel compile? Can you also share the output of 'auditctl

This test was carried out with Linux 4.9. Custom built.

> -l' from your system? The general approach taken by everyone to
> turn-off the per-syscall audit overhead is to add the "-a never,task"
> rule to their audit configuration:
>
> # auditctl -a never,task
>
> If you are using Fedora/CentOS/RHEL, or a similarly configured system,

This is an embedded distribution. We are not using auditctl or any other
audit-related user-space packages.

> you can find this configuration in the /etc/audit/audit.rules file (be
> warned, that file is automatically generated based on
> /etc/audit/rules.d).

I suppose in this case rule list must be empty. Is there a way to check
this without extra user-space packages?

--
Best regards,
Alexander Sverdlin.

2019-01-28 14:46:53

by Alexander Sverdlin

[permalink] [raw]
Subject: Re: [PATCH] audit: always enable syscall auditing when supported and audit is enabled

Hello Paul,

On 28/01/2019 15:19, Paul Moore wrote:
>>> To the best of our knowledge, everyone who enables audit at compile
>>> time also enables syscall auditing; this patch simplifies the Kconfig
>>> menus by removing the option to disable syscall auditing when audit
>>> is selected and the target arch supports it.
>>>
>>> Signed-off-by: Paul Moore <[email protected]>
>> this patch is responsible for massive performance degradation for those
>> who used only CONFIG_SECURITY_APPARMOR.
>>
>> And the numbers are, take the following test for instance:
>>
>> dd if=/dev/zero of=/dev/null count=2M
>>
>> ARM64: 500MB/s -> 350MB/s
>> ARM: 400MB/s -> 300MB/s
> Hi there.
>
> Out of curiosity, what kernel/distribution are you running, or is this
> a custom kernel compile? Can you also share the output of 'auditctl
> -l' from your system? The general approach taken by everyone to
> turn-off the per-syscall audit overhead is to add the "-a never,task"
> rule to their audit configuration:
>
> # auditctl -a never,task
>
> If you are using Fedora/CentOS/RHEL, or a similarly configured system,
> you can find this configuration in the /etc/audit/audit.rules file (be
> warned, that file is automatically generated based on
> /etc/audit/rules.d).

here are some perf reports for your reference:

Linux fct-0a 4.9.144-xxx armv7l armv7l armv7l GNU/Linux
# Overhead Command Shared Object Symbol
# ........ ....... ................. ............................
#
14.32% dd [kernel.kallsyms] [k] vector_swi
13.70% dd libc-2.28.so [.] __libc_do_syscall
8.42% dd [kernel.kallsyms] [k] current_kernel_time64
5.65% dd [kernel.kallsyms] [k] arm_clear_user
5.65% dd [kernel.kallsyms] [k] __fget_light
5.54% dd [kernel.kallsyms] [k] __audit_syscall_entry
3.79% dd [kernel.kallsyms] [k] fsnotify
2.97% dd libc-2.28.so [.] memcpy
2.94% dd [kernel.kallsyms] [k] __audit_syscall_exit
2.67% dd [kernel.kallsyms] [k] __fsnotify_parent
2.43% dd [kernel.kallsyms] [k] vfs_read
2.24% dd [kernel.kallsyms] [k] __vfs_read
2.23% dd [kernel.kallsyms] [k] vfs_write
1.96% dd [kernel.kallsyms] [k] syscall_trace_exit
1.71% dd [kernel.kallsyms] [k] syscall_trace_enter
1.23% dd [kernel.kallsyms] [k] rw_verify_area
1.22% dd libc-2.28.so [.] read
1.14% dd [kernel.kallsyms] [k] security_file_permission
1.07% dd [kernel.kallsyms] [k] iov_iter_zero
1.01% dd [kernel.kallsyms] [k] __sys_trace_return
0.88% dd libc-2.28.so [.] __GI___libc_write
0.85% dd dd.coreutils [.] 0x00003c48
0.84% dd [kernel.kallsyms] [k] read_iter_zero
0.79% dd [kernel.kallsyms] [k] unroll_tree_refs
0.75% dd [kernel.kallsyms] [k] __vfs_write
0.68% dd [kernel.kallsyms] [k] __fdget_pos
0.64% dd dd.coreutils [.] 0x000013a4
0.59% dd [kernel.kallsyms] [k] dput
0.57% dd [kernel.kallsyms] [k] kfree
0.56% dd [kernel.kallsyms] [k] path_put
0.52% dd [kernel.kallsyms] [k] ret_to_user

Linux fctj-0a 4.4.167-g4220c09-xxx armv7l GNU/Linux
# Overhead Command Shared Object Symbol
# ........ ....... ................. ...............................
#
23.91% dd [kernel.kallsyms] [k] vector_swi
19.57% dd libc-2.28.so [.] __libc_do_syscall
5.67% dd [kernel.kallsyms] [k] arm_clear_user
4.25% dd libc-2.28.so [.] memcpy
4.22% dd [kernel.kallsyms] [k] fsnotify
3.93% dd [kernel.kallsyms] [k] __fget_light
3.10% dd [kernel.kallsyms] [k] read_iter_zero
2.51% dd [kernel.kallsyms] [k] vfs_write
2.06% dd [kernel.kallsyms] [k] ret_fast_syscall
1.97% dd [kernel.kallsyms] [k] __fsnotify_parent
1.66% dd libc-2.28.so [.] read
1.65% dd [kernel.kallsyms] [k] __vfs_read
1.63% dd [kernel.kallsyms] [k] mmioset
1.61% dd [kernel.kallsyms] [k] vfs_read
1.47% dd libc-2.28.so [.] __GI___libc_write
1.39% dd [kernel.kallsyms] [k] rw_verify_area
1.39% dd [kernel.kallsyms] [k] security_file_permission
1.34% dd [kernel.kallsyms] [k] iov_iter_init
1.34% dd [kernel.kallsyms] [k] iov_iter_zero
1.20% dd [kernel.kallsyms] [k] local_restart
0.89% dd [kernel.kallsyms] [k] sys_read
0.89% dd [kernel.kallsyms] [k] _cond_resched
0.86% dd [kernel.kallsyms] [k] __fdget_pos
0.85% dd [kernel.kallsyms] [k] sys_write
0.77% dd dd [.] 0x00003df0
0.72% dd [kernel.kallsyms] [k] __vfs_write
0.61% dd dd [.] 0x00003946
0.55% dd dd [.] 0x000038ee
0.51% dd dd [.] 0x00003ca8


--
Best regards,
Alexander Sverdlin.

2019-01-28 14:54:35

by Paul Moore

[permalink] [raw]
Subject: Re: [PATCH] audit: always enable syscall auditing when supported and audit is enabled

On Mon, Jan 28, 2019 at 9:36 AM Sverdlin, Alexander (Nokia - DE/Ulm)
<[email protected]> wrote:
> Hello Paul,
>
> On 28/01/2019 15:19, Paul Moore wrote:
> >>> time also enables syscall auditing; this patch simplifies the Kconfig
> >>> menus by removing the option to disable syscall auditing when audit
> >>> is selected and the target arch supports it.
> >>>
> >>> Signed-off-by: Paul Moore <[email protected]>
> >> this patch is responsible for massive performance degradation for those
> >> who used only CONFIG_SECURITY_APPARMOR.
> >>
> >> And the numbers are, take the following test for instance:
> >>
> >> dd if=/dev/zero of=/dev/null count=2M
> >>
> >> ARM64: 500MB/s -> 350MB/s
> >> ARM: 400MB/s -> 300MB/s
> > Hi there.
> >
> > Out of curiosity, what kernel/distribution are you running, or is this
> > a custom kernel compile? Can you also share the output of 'auditctl
>
> This test was carried out with Linux 4.9. Custom built.

I suspected that was the case, thanks.

> > -l' from your system? The general approach taken by everyone to
> > turn-off the per-syscall audit overhead is to add the "-a never,task"
> > rule to their audit configuration:
> >
> > # auditctl -a never,task
> >
> > If you are using Fedora/CentOS/RHEL, or a similarly configured system,
>
> This is an embedded distribution. We are not using auditctl or any other
> audit-related user-space packages.
>
> > you can find this configuration in the /etc/audit/audit.rules file (be
> > warned, that file is automatically generated based on
> > /etc/audit/rules.d).
>
> I suppose in this case rule list must be empty. Is there a way to check
> this without extra user-space packages?

Yes, unless you are loading rules through some other method I would
expect that your audit rule list is empty.

I'm not aware of any other tools besides auditctl to load audit rules
into the kernel, although I haven't ever had a need for another tool
so I haven't looked very hard. If you didn't want to bring auditctl
into your distribution, I expect it would be a rather trivial task to
create a small tool to load a single "-a never,task" into the kernel.

--
paul moore
http://www.paul-moore.com

2019-01-28 15:39:01

by Alexander Sverdlin

[permalink] [raw]
Subject: Re: [PATCH] audit: always enable syscall auditing when supported and audit is enabled

Hello Paul,

On 28/01/2019 15:52, Paul Moore wrote:
>>>>> time also enables syscall auditing; this patch simplifies the Kconfig
>>>>> menus by removing the option to disable syscall auditing when audit
>>>>> is selected and the target arch supports it.
>>>>>
>>>>> Signed-off-by: Paul Moore <[email protected]>
>>>> this patch is responsible for massive performance degradation for those
>>>> who used only CONFIG_SECURITY_APPARMOR.
>>>>
>>>> And the numbers are, take the following test for instance:
>>>>
>>>> dd if=/dev/zero of=/dev/null count=2M
>>>>
>>>> ARM64: 500MB/s -> 350MB/s
>>>> ARM: 400MB/s -> 300MB/s
>>> Hi there.
>>>
>>> Out of curiosity, what kernel/distribution are you running, or is this
>>> a custom kernel compile? Can you also share the output of 'auditctl
>> This test was carried out with Linux 4.9. Custom built.
> I suspected that was the case, thanks.
>
>>> -l' from your system? The general approach taken by everyone to
>>> turn-off the per-syscall audit overhead is to add the "-a never,task"
>>> rule to their audit configuration:
>>>
>>> # auditctl -a never,task
>>>
>>> If you are using Fedora/CentOS/RHEL, or a similarly configured system,
>> This is an embedded distribution. We are not using auditctl or any other
>> audit-related user-space packages.
>>
>>> you can find this configuration in the /etc/audit/audit.rules file (be
>>> warned, that file is automatically generated based on
>>> /etc/audit/rules.d).
>> I suppose in this case rule list must be empty. Is there a way to check
>> this without extra user-space packages?
> Yes, unless you are loading rules through some other method I would
> expect that your audit rule list is empty.
>
> I'm not aware of any other tools besides auditctl to load audit rules
> into the kernel, although I haven't ever had a need for another tool
> so I haven't looked very hard. If you didn't want to bring auditctl
> into your distribution, I expect it would be a rather trivial task to
> create a small tool to load a single "-a never,task" into the kernel.

I've done a quick test on my x86_64 PC and got the following results:

1. empty rules list:

perf record dd if=/dev/zero of=/dev/null count=2M
2097152+0 records in
2097152+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 1.69685 s, 633 MB/s

perf report:

# Overhead Command Shared Object Symbol
# ........ ....... ................. ..................................
#
14.26% dd [kernel.kallsyms] [k] entry_SYSCALL_64
11.33% dd [kernel.kallsyms] [k] __clear_user
5.00% dd [kernel.kallsyms] [k] fsnotify
4.92% dd libc-2.28.so [.] read
4.80% dd [kernel.kallsyms] [k] __audit_syscall_exit
4.60% dd [kernel.kallsyms] [k] syscall_return_via_sysret
4.24% dd libc-2.28.so [.] __GI___libc_write
3.84% dd [kernel.kallsyms] [k] __indirect_thunk_start
3.82% dd libc-2.28.so [.] __memcpy_ssse3_back
3.04% dd [kernel.kallsyms] [k] entry_SYSCALL_64_after_hwframe
2.98% dd [kernel.kallsyms] [k] __fget_light
2.97% dd [kernel.kallsyms] [k] do_syscall_64
2.33% dd [kernel.kallsyms] [k] vfs_write
2.32% dd [kernel.kallsyms] [k] __audit_syscall_entry
2.31% dd [kernel.kallsyms] [k] iov_iter_zero
2.22% dd [kernel.kallsyms] [k] syscall_trace_enter
1.89% dd [kernel.kallsyms] [k] syscall_slow_exit_work
1.56% dd [kernel.kallsyms] [k] __fsnotify_parent
1.52% dd [kernel.kallsyms] [k] __x64_sys_write
1.42% dd [kernel.kallsyms] [k] __vfs_read
1.34% dd [kernel.kallsyms] [k] __x64_sys_read
1.30% dd [kernel.kallsyms] [k] vfs_read
1.16% dd [kernel.kallsyms] [k] ksys_write
1.05% dd [kernel.kallsyms] [k] security_file_permission

2. auditctl -a never,task

perf record dd if=/dev/zero of=/dev/null count=2M
2097152+0 records in
2097152+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 1.29384 s, 830 MB/s

perf report:

# Overhead Command Shared Object Symbol
# ........ ....... ................. ...................................
#
16.90% dd [kernel.kallsyms] [k] entry_SYSCALL_64
14.24% dd [kernel.kallsyms] [k] __clear_user
6.00% dd [kernel.kallsyms] [k] fsnotify
5.35% dd [kernel.kallsyms] [k] syscall_return_via_sysret
5.26% dd [kernel.kallsyms] [k] __indirect_thunk_start
4.85% dd libc-2.28.so [.] read
4.81% dd libc-2.28.so [.] __GI___libc_write
4.09% dd libc-2.28.so [.] __memcpy_ssse3_back
3.92% dd [kernel.kallsyms] [k] __fget_light
3.43% dd [kernel.kallsyms] [k] entry_SYSCALL_64_after_hwframe
3.07% dd [kernel.kallsyms] [k] iov_iter_zero
2.93% dd [kernel.kallsyms] [k] do_syscall_64
2.45% dd [kernel.kallsyms] [k] vfs_write
2.07% dd [kernel.kallsyms] [k] __vfs_read
2.02% dd [kernel.kallsyms] [k] __fsnotify_parent
1.42% dd [kernel.kallsyms] [k] vfs_read
1.34% dd [kernel.kallsyms] [k] ksys_read
1.18% dd [kernel.kallsyms] [k] ksys_write
1.18% dd [kernel.kallsyms] [k] read_iter_zero
1.10% dd [kernel.kallsyms] [k] __vfs_write

Which brings me to an idea, that the subject patch should have been accompanied
by a default "never,task" rule inside the kernel, otherwise you require an
extra user-space package (audit) just to bring Linux 4.5+ to 4.4 performance
levels.

--
Best regards,
Alexander Sverdlin.

2019-01-28 16:28:02

by Paul Moore

[permalink] [raw]
Subject: Re: [PATCH] audit: always enable syscall auditing when supported and audit is enabled

On Mon, Jan 28, 2019 at 10:38 AM Sverdlin, Alexander (Nokia - DE/Ulm)
<[email protected]> wrote:
> Hello Paul,
>
> On 28/01/2019 15:52, Paul Moore wrote:
> >>>>> time also enables syscall auditing; this patch simplifies the Kconfig
> >>>>> menus by removing the option to disable syscall auditing when audit
> >>>>> is selected and the target arch supports it.
> >>>>>
> >>>>> Signed-off-by: Paul Moore <[email protected]>
> >>>> this patch is responsible for massive performance degradation for those
> >>>> who used only CONFIG_SECURITY_APPARMOR.
> >>>>
> >>>> And the numbers are, take the following test for instance:
> >>>>
> >>>> dd if=/dev/zero of=/dev/null count=2M
> >>>>
> >>>> ARM64: 500MB/s -> 350MB/s
> >>>> ARM: 400MB/s -> 300MB/s
> >>> Hi there.
> >>>
> >>> Out of curiosity, what kernel/distribution are you running, or is this
> >>> a custom kernel compile? Can you also share the output of 'auditctl
> >> This test was carried out with Linux 4.9. Custom built.
> > I suspected that was the case, thanks.
> >
> >>> -l' from your system? The general approach taken by everyone to
> >>> turn-off the per-syscall audit overhead is to add the "-a never,task"
> >>> rule to their audit configuration:
> >>>
> >>> # auditctl -a never,task
> >>>
> >>> If you are using Fedora/CentOS/RHEL, or a similarly configured system,
> >> This is an embedded distribution. We are not using auditctl or any other
> >> audit-related user-space packages.
> >>
> >>> you can find this configuration in the /etc/audit/audit.rules file (be
> >>> warned, that file is automatically generated based on
> >>> /etc/audit/rules.d).
> >> I suppose in this case rule list must be empty. Is there a way to check
> >> this without extra user-space packages?
> > Yes, unless you are loading rules through some other method I would
> > expect that your audit rule list is empty.
> >
> > I'm not aware of any other tools besides auditctl to load audit rules
> > into the kernel, although I haven't ever had a need for another tool
> > so I haven't looked very hard. If you didn't want to bring auditctl
> > into your distribution, I expect it would be a rather trivial task to
> > create a small tool to load a single "-a never,task" into the kernel.
>
> I've done a quick test on my x86_64 PC and got the following results:

...

> Which brings me to an idea, that the subject patch should have been accompanied
> by a default "never,task" rule inside the kernel, otherwise you require an
> extra user-space package (audit) just to bring Linux 4.5+ to 4.4 performance
> levels.

[NOTE: I dropped [email protected] from the To/CC line, I left Red Hat
for greener pastures several months ago.]

Well, it generally hasn't been an issue as 1) most people that enable
audit also enable syscall auditing and 2) most people that enable
audit have some sort of audit userspace tools to work with the audit
logs (and configure audit if necessary). I don't want to diminish
your report, but this change was made several years ago and we really
haven't heard of many issues surrounding the change. If we can ever
get all of the different arches to support syscall auditing, I'd love
to get rid of the syscall auditing Kconfig knob entirely.

If you wanted to put together a patch that added a single "-a
never,task" rule on boot I could get behind that, just make it default
to off.

--
paul moore
http://www.paul-moore.com

2019-01-28 20:05:26

by Steve Grubb

[permalink] [raw]
Subject: Re: [PATCH] audit: always enable syscall auditing when supported and audit is enabled

On Mon, 28 Jan 2019 11:26:51 -0500
Paul Moore <[email protected]> wrote:

> On Mon, Jan 28, 2019 at 10:38 AM Sverdlin, Alexander (Nokia - DE/Ulm)
> <[email protected]> wrote:
> > Hello Paul,
> >
> > On 28/01/2019 15:52, Paul Moore wrote:
> > >>>>> time also enables syscall auditing; this patch simplifies the
> > >>>>> Kconfig menus by removing the option to disable syscall
> > >>>>> auditing when audit is selected and the target arch supports
> > >>>>> it.
> > >>>>>
> > >>>>> Signed-off-by: Paul Moore <[email protected]>
> > >>>> this patch is responsible for massive performance degradation
> > >>>> for those who used only CONFIG_SECURITY_APPARMOR.
> > >>>>
> > >>>> And the numbers are, take the following test for instance:
> > >>>>
> > >>>> dd if=/dev/zero of=/dev/null count=2M
> > >>>>
> > >>>> ARM64: 500MB/s -> 350MB/s
> > >>>> ARM: 400MB/s -> 300MB/s
> > >>> Hi there.
> > >>>
> > >>> Out of curiosity, what kernel/distribution are you running, or
> > >>> is this a custom kernel compile? Can you also share the output
> > >>> of 'auditctl
> > >> This test was carried out with Linux 4.9. Custom built.
> > > I suspected that was the case, thanks.
> > >
> > >>> -l' from your system? The general approach taken by everyone to
> > >>> turn-off the per-syscall audit overhead is to add the "-a
> > >>> never,task" rule to their audit configuration:
> > >>>
> > >>> # auditctl -a never,task
> > >>>
> > >>> If you are using Fedora/CentOS/RHEL, or a similarly configured
> > >>> system,
> > >> This is an embedded distribution. We are not using auditctl or
> > >> any other audit-related user-space packages.
> > >>
> > >>> you can find this configuration in the /etc/audit/audit.rules
> > >>> file (be warned, that file is automatically generated based on
> > >>> /etc/audit/rules.d).
> > >> I suppose in this case rule list must be empty. Is there a way
> > >> to check this without extra user-space packages?
> > > Yes, unless you are loading rules through some other method I
> > > would expect that your audit rule list is empty.
> > >
> > > I'm not aware of any other tools besides auditctl to load audit
> > > rules into the kernel, although I haven't ever had a need for
> > > another tool so I haven't looked very hard. If you didn't want
> > > to bring auditctl into your distribution, I expect it would be a
> > > rather trivial task to create a small tool to load a single "-a
> > > never,task" into the kernel.
> >
> > I've done a quick test on my x86_64 PC and got the following
> > results:
>
> ...
>
> > Which brings me to an idea, that the subject patch should have been
> > accompanied by a default "never,task" rule inside the kernel,
> > otherwise you require an extra user-space package (audit) just to
> > bring Linux 4.5+ to 4.4 performance levels.
>
> [NOTE: I dropped [email protected] from the To/CC line, I left Red Hat
> for greener pastures several months ago.]
>
> Well, it generally hasn't been an issue as 1) most people that enable
> audit also enable syscall auditing and 2) most people that enable
> audit have some sort of audit userspace tools to work with the audit
> logs (and configure audit if necessary). I don't want to diminish
> your report, but this change was made several years ago and we really
> haven't heard of many issues surrounding the change. If we can ever
> get all of the different arches to support syscall auditing, I'd love
> to get rid of the syscall auditing Kconfig knob entirely.
>
> If you wanted to put together a patch that added a single "-a
> never,task" rule on boot I could get behind that, just make it default
> to off.

That will make processes unauditable for everyone. That's how it gets a
speedup is not entering into the audit machinery. I suppose its
possible that people may want MAC hardwired events but no syscall
events. I don't know if there are other kconfig audit options. but
maybe getting it down to audit enabled and syscall auditing as the only
choices is probably the most performant.

-Steve

2019-01-28 20:10:01

by Paul Moore

[permalink] [raw]
Subject: Re: [PATCH] audit: always enable syscall auditing when supported and audit is enabled

On Mon, Jan 28, 2019 at 3:03 PM Steve Grubb <[email protected]> wrote:
> On Mon, 28 Jan 2019 11:26:51 -0500
> Paul Moore <[email protected]> wrote:
>
> > On Mon, Jan 28, 2019 at 10:38 AM Sverdlin, Alexander (Nokia - DE/Ulm)
> > <[email protected]> wrote:
> > > Hello Paul,
> > >
> > > On 28/01/2019 15:52, Paul Moore wrote:
> > > >>>>> time also enables syscall auditing; this patch simplifies the
> > > >>>>> Kconfig menus by removing the option to disable syscall
> > > >>>>> auditing when audit is selected and the target arch supports
> > > >>>>> it.
> > > >>>>>
> > > >>>>> Signed-off-by: Paul Moore <[email protected]>
> > > >>>> this patch is responsible for massive performance degradation
> > > >>>> for those who used only CONFIG_SECURITY_APPARMOR.
> > > >>>>
> > > >>>> And the numbers are, take the following test for instance:
> > > >>>>
> > > >>>> dd if=/dev/zero of=/dev/null count=2M
> > > >>>>
> > > >>>> ARM64: 500MB/s -> 350MB/s
> > > >>>> ARM: 400MB/s -> 300MB/s
> > > >>> Hi there.
> > > >>>
> > > >>> Out of curiosity, what kernel/distribution are you running, or
> > > >>> is this a custom kernel compile? Can you also share the output
> > > >>> of 'auditctl
> > > >> This test was carried out with Linux 4.9. Custom built.
> > > > I suspected that was the case, thanks.
> > > >
> > > >>> -l' from your system? The general approach taken by everyone to
> > > >>> turn-off the per-syscall audit overhead is to add the "-a
> > > >>> never,task" rule to their audit configuration:
> > > >>>
> > > >>> # auditctl -a never,task
> > > >>>
> > > >>> If you are using Fedora/CentOS/RHEL, or a similarly configured
> > > >>> system,
> > > >> This is an embedded distribution. We are not using auditctl or
> > > >> any other audit-related user-space packages.
> > > >>
> > > >>> you can find this configuration in the /etc/audit/audit.rules
> > > >>> file (be warned, that file is automatically generated based on
> > > >>> /etc/audit/rules.d).
> > > >> I suppose in this case rule list must be empty. Is there a way
> > > >> to check this without extra user-space packages?
> > > > Yes, unless you are loading rules through some other method I
> > > > would expect that your audit rule list is empty.
> > > >
> > > > I'm not aware of any other tools besides auditctl to load audit
> > > > rules into the kernel, although I haven't ever had a need for
> > > > another tool so I haven't looked very hard. If you didn't want
> > > > to bring auditctl into your distribution, I expect it would be a
> > > > rather trivial task to create a small tool to load a single "-a
> > > > never,task" into the kernel.
> > >
> > > I've done a quick test on my x86_64 PC and got the following
> > > results:
> >
> > ...
> >
> > > Which brings me to an idea, that the subject patch should have been
> > > accompanied by a default "never,task" rule inside the kernel,
> > > otherwise you require an extra user-space package (audit) just to
> > > bring Linux 4.5+ to 4.4 performance levels.
> >
> > [NOTE: I dropped [email protected] from the To/CC line, I left Red Hat
> > for greener pastures several months ago.]
> >
> > Well, it generally hasn't been an issue as 1) most people that enable
> > audit also enable syscall auditing and 2) most people that enable
> > audit have some sort of audit userspace tools to work with the audit
> > logs (and configure audit if necessary). I don't want to diminish
> > your report, but this change was made several years ago and we really
> > haven't heard of many issues surrounding the change. If we can ever
> > get all of the different arches to support syscall auditing, I'd love
> > to get rid of the syscall auditing Kconfig knob entirely.
> >
> > If you wanted to put together a patch that added a single "-a
> > never,task" rule on boot I could get behind that, just make it default
> > to off.
>
> That will make processes unauditable for everyone. That's how it gets a
> speedup is not entering into the audit machinery.

That is pretty much what is being asked for in this thread. It's
really no different from what Fedora/CentOS/RHEL (and who knows how
many others) ship with their default audit config. It's important to
note that you could always delete this rule at runtime; all that is
being proposed is to have the kernel populate the the audit ruleset
with the "-a never,task" rule *IF* the proposed kernel Kconfig option
is enabled (and it would default to being off, you would have to turn
it on during build).

> I suppose its possible that people may want MAC hardwired events but no syscall
> events. I don't know if there are other kconfig audit options. but
> maybe getting it down to audit enabled and syscall auditing as the only
> choices is probably the most performant.
>
> -Steve

--
paul moore
http://www.paul-moore.com

2019-01-28 21:20:30

by Steve Grubb

[permalink] [raw]
Subject: Re: [PATCH] audit: always enable syscall auditing when supported and audit is enabled

On Mon, 28 Jan 2019 15:08:56 -0500
Paul Moore <[email protected]> wrote:

> On Mon, Jan 28, 2019 at 3:03 PM Steve Grubb <[email protected]> wrote:
> > On Mon, 28 Jan 2019 11:26:51 -0500
> > Paul Moore <[email protected]> wrote:
> >
> > > On Mon, Jan 28, 2019 at 10:38 AM Sverdlin, Alexander (Nokia -
> > > DE/Ulm) <[email protected]> wrote:
> > > > Hello Paul,
> > > >
> > > > On 28/01/2019 15:52, Paul Moore wrote:
> > > > >>>>> time also enables syscall auditing; this patch simplifies
> > > > >>>>> the Kconfig menus by removing the option to disable
> > > > >>>>> syscall auditing when audit is selected and the target
> > > > >>>>> arch supports it.
> > > > >>>>>
> > > > >>>>> Signed-off-by: Paul Moore <[email protected]>
> > > > >>>> this patch is responsible for massive performance
> > > > >>>> degradation for those who used only
> > > > >>>> CONFIG_SECURITY_APPARMOR.
> > > > >>>>
> > > > >>>> And the numbers are, take the following test for instance:
> > > > >>>>
> > > > >>>> dd if=/dev/zero of=/dev/null count=2M
> > > > >>>>
> > > > >>>> ARM64: 500MB/s -> 350MB/s
> > > > >>>> ARM: 400MB/s -> 300MB/s
> > > > >>> Hi there.
> > > > >>>
> > > > >>> Out of curiosity, what kernel/distribution are you running,
> > > > >>> or is this a custom kernel compile? Can you also share the
> > > > >>> output of 'auditctl
> > > > >> This test was carried out with Linux 4.9. Custom built.
> > > > > I suspected that was the case, thanks.
> > > > >
> > > > >>> -l' from your system? The general approach taken by
> > > > >>> everyone to turn-off the per-syscall audit overhead is to
> > > > >>> add the "-a never,task" rule to their audit configuration:
> > > > >>>
> > > > >>> # auditctl -a never,task
> > > > >>>
> > > > >>> If you are using Fedora/CentOS/RHEL, or a similarly
> > > > >>> configured system,
> > > > >> This is an embedded distribution. We are not using auditctl
> > > > >> or any other audit-related user-space packages.
> > > > >>
> > > > >>> you can find this configuration in
> > > > >>> the /etc/audit/audit.rules file (be warned, that file is
> > > > >>> automatically generated based on /etc/audit/rules.d).
> > > > >> I suppose in this case rule list must be empty. Is there a
> > > > >> way to check this without extra user-space packages?
> > > > > Yes, unless you are loading rules through some other method I
> > > > > would expect that your audit rule list is empty.
> > > > >
> > > > > I'm not aware of any other tools besides auditctl to load
> > > > > audit rules into the kernel, although I haven't ever had a
> > > > > need for another tool so I haven't looked very hard. If you
> > > > > didn't want to bring auditctl into your distribution, I
> > > > > expect it would be a rather trivial task to create a small
> > > > > tool to load a single "-a never,task" into the kernel.
> > > >
> > > > I've done a quick test on my x86_64 PC and got the following
> > > > results:
> > >
> > > ...
> > >
> > > > Which brings me to an idea, that the subject patch should have
> > > > been accompanied by a default "never,task" rule inside the
> > > > kernel, otherwise you require an extra user-space package
> > > > (audit) just to bring Linux 4.5+ to 4.4 performance levels.
> > >
> > > [NOTE: I dropped [email protected] from the To/CC line, I left
> > > Red Hat for greener pastures several months ago.]
> > >
> > > Well, it generally hasn't been an issue as 1) most people that
> > > enable audit also enable syscall auditing and 2) most people that
> > > enable audit have some sort of audit userspace tools to work with
> > > the audit logs (and configure audit if necessary). I don't want
> > > to diminish your report, but this change was made several years
> > > ago and we really haven't heard of many issues surrounding the
> > > change. If we can ever get all of the different arches to
> > > support syscall auditing, I'd love to get rid of the syscall
> > > auditing Kconfig knob entirely.
> > >
> > > If you wanted to put together a patch that added a single "-a
> > > never,task" rule on boot I could get behind that, just make it
> > > default to off.
> >
> > That will make processes unauditable for everyone. That's how it
> > gets a speedup is not entering into the audit machinery.
>
> That is pretty much what is being asked for in this thread. It's
> really no different from what Fedora/CentOS/RHEL (and who knows how
> many others) ship with their default audit config. It's important to
> note that you could always delete this rule at runtime; all that is
> being proposed is to have the kernel populate the the audit ruleset
> with the "-a never,task" rule *IF* the proposed kernel Kconfig option
> is enabled (and it would default to being off, you would have to turn
> it on during build).

Yes, but you can't add the auditable flag back to the task struct
because syscalls never enter audit machinery where it can be added back.
Meaning that even if you wanted to audit, there will be processes you
never can audit. Whereas deciding via auditd, they become unauditable
only after auditd loads the rule. Prior to that they are and all
processes are auditable if you decide to audit. So, this is
fundamentally different than what fedora does.

-Steve

> > I suppose its possible that people may want MAC hardwired events
> > but no syscall events. I don't know if there are other kconfig
> > audit options. but maybe getting it down to audit enabled and
> > syscall auditing as the only choices is probably the most
> > performant.
> >
> > -Steve
>


2019-01-28 21:50:41

by Richard Guy Briggs

[permalink] [raw]
Subject: Re: [PATCH] audit: always enable syscall auditing when supported and audit is enabled

On 2019-01-28 22:19, Steve Grubb wrote:
> On Mon, 28 Jan 2019 15:08:56 -0500
> Paul Moore <[email protected]> wrote:
>
> > On Mon, Jan 28, 2019 at 3:03 PM Steve Grubb <[email protected]> wrote:
> > > On Mon, 28 Jan 2019 11:26:51 -0500
> > > Paul Moore <[email protected]> wrote:
> > >
> > > > On Mon, Jan 28, 2019 at 10:38 AM Sverdlin, Alexander (Nokia -
> > > > DE/Ulm) <[email protected]> wrote:
> > > > > Hello Paul,
> > > > >
> > > > > On 28/01/2019 15:52, Paul Moore wrote:
> > > > > >>>>> time also enables syscall auditing; this patch simplifies
> > > > > >>>>> the Kconfig menus by removing the option to disable
> > > > > >>>>> syscall auditing when audit is selected and the target
> > > > > >>>>> arch supports it.
> > > > > >>>>>
> > > > > >>>>> Signed-off-by: Paul Moore <[email protected]>
> > > > > >>>> this patch is responsible for massive performance
> > > > > >>>> degradation for those who used only
> > > > > >>>> CONFIG_SECURITY_APPARMOR.
> > > > > >>>>
> > > > > >>>> And the numbers are, take the following test for instance:
> > > > > >>>>
> > > > > >>>> dd if=/dev/zero of=/dev/null count=2M
> > > > > >>>>
> > > > > >>>> ARM64: 500MB/s -> 350MB/s
> > > > > >>>> ARM: 400MB/s -> 300MB/s
> > > > > >>> Hi there.
> > > > > >>>
> > > > > >>> Out of curiosity, what kernel/distribution are you running,
> > > > > >>> or is this a custom kernel compile? Can you also share the
> > > > > >>> output of 'auditctl
> > > > > >> This test was carried out with Linux 4.9. Custom built.
> > > > > > I suspected that was the case, thanks.
> > > > > >
> > > > > >>> -l' from your system? The general approach taken by
> > > > > >>> everyone to turn-off the per-syscall audit overhead is to
> > > > > >>> add the "-a never,task" rule to their audit configuration:
> > > > > >>>
> > > > > >>> # auditctl -a never,task
> > > > > >>>
> > > > > >>> If you are using Fedora/CentOS/RHEL, or a similarly
> > > > > >>> configured system,
> > > > > >> This is an embedded distribution. We are not using auditctl
> > > > > >> or any other audit-related user-space packages.
> > > > > >>
> > > > > >>> you can find this configuration in
> > > > > >>> the /etc/audit/audit.rules file (be warned, that file is
> > > > > >>> automatically generated based on /etc/audit/rules.d).
> > > > > >> I suppose in this case rule list must be empty. Is there a
> > > > > >> way to check this without extra user-space packages?
> > > > > > Yes, unless you are loading rules through some other method I
> > > > > > would expect that your audit rule list is empty.
> > > > > >
> > > > > > I'm not aware of any other tools besides auditctl to load
> > > > > > audit rules into the kernel, although I haven't ever had a
> > > > > > need for another tool so I haven't looked very hard. If you
> > > > > > didn't want to bring auditctl into your distribution, I
> > > > > > expect it would be a rather trivial task to create a small
> > > > > > tool to load a single "-a never,task" into the kernel.
> > > > >
> > > > > I've done a quick test on my x86_64 PC and got the following
> > > > > results:
> > > >
> > > > ...
> > > >
> > > > > Which brings me to an idea, that the subject patch should have
> > > > > been accompanied by a default "never,task" rule inside the
> > > > > kernel, otherwise you require an extra user-space package
> > > > > (audit) just to bring Linux 4.5+ to 4.4 performance levels.
> > > >
> > > > [NOTE: I dropped [email protected] from the To/CC line, I left
> > > > Red Hat for greener pastures several months ago.]
> > > >
> > > > Well, it generally hasn't been an issue as 1) most people that
> > > > enable audit also enable syscall auditing and 2) most people that
> > > > enable audit have some sort of audit userspace tools to work with
> > > > the audit logs (and configure audit if necessary). I don't want
> > > > to diminish your report, but this change was made several years
> > > > ago and we really haven't heard of many issues surrounding the
> > > > change. If we can ever get all of the different arches to
> > > > support syscall auditing, I'd love to get rid of the syscall
> > > > auditing Kconfig knob entirely.
> > > >
> > > > If you wanted to put together a patch that added a single "-a
> > > > never,task" rule on boot I could get behind that, just make it
> > > > default to off.
> > >
> > > That will make processes unauditable for everyone. That's how it
> > > gets a speedup is not entering into the audit machinery.
> >
> > That is pretty much what is being asked for in this thread. It's
> > really no different from what Fedora/CentOS/RHEL (and who knows how
> > many others) ship with their default audit config. It's important to
> > note that you could always delete this rule at runtime; all that is
> > being proposed is to have the kernel populate the the audit ruleset
> > with the "-a never,task" rule *IF* the proposed kernel Kconfig option
> > is enabled (and it would default to being off, you would have to turn
> > it on during build).
>
> Yes, but you can't add the auditable flag back to the task struct
> because syscalls never enter audit machinery where it can be added back.
> Meaning that even if you wanted to audit, there will be processes you
> never can audit. Whereas deciding via auditd, they become unauditable
> only after auditd loads the rule. Prior to that they are and all
> processes are auditable if you decide to audit. So, this is
> fundamentally different than what fedora does.

I did see a patch proposed at one point that actually went through and
checked/flipped the TIF_SYSCALL_AUDIT bit when auditing was switched
off.

> -Steve
>
> > > I suppose its possible that people may want MAC hardwired events
> > > but no syscall events. I don't know if there are other kconfig
> > > audit options. but maybe getting it down to audit enabled and
> > > syscall auditing as the only choices is probably the most
> > > performant.
> > >
> > > -Steve

- RGB

--
Richard Guy Briggs <[email protected]>
Sr. S/W Engineer, Kernel Security, Base Operating Systems
Remote, Ottawa, Red Hat Canada
IRC: rgb, SunRaycer
Voice: +1.647.777.2635, Internal: (81) 32635

2019-01-28 22:02:07

by Paul Moore

[permalink] [raw]
Subject: Re: [PATCH] audit: always enable syscall auditing when supported and audit is enabled

On Mon, Jan 28, 2019 at 4:20 PM Steve Grubb <[email protected]> wrote:
> On Mon, 28 Jan 2019 15:08:56 -0500
> Paul Moore <[email protected]> wrote:
> > On Mon, Jan 28, 2019 at 3:03 PM Steve Grubb <[email protected]> wrote:
> > > On Mon, 28 Jan 2019 11:26:51 -0500
> > > Paul Moore <[email protected]> wrote:
> > >
> > > > On Mon, Jan 28, 2019 at 10:38 AM Sverdlin, Alexander (Nokia -
> > > > DE/Ulm) <[email protected]> wrote:
> > > > > Hello Paul,
> > > > >
> > > > > On 28/01/2019 15:52, Paul Moore wrote:
> > > > > >>>>> time also enables syscall auditing; this patch simplifies
> > > > > >>>>> the Kconfig menus by removing the option to disable
> > > > > >>>>> syscall auditing when audit is selected and the target
> > > > > >>>>> arch supports it.
> > > > > >>>>>
> > > > > >>>>> Signed-off-by: Paul Moore <[email protected]>
> > > > > >>>> this patch is responsible for massive performance
> > > > > >>>> degradation for those who used only
> > > > > >>>> CONFIG_SECURITY_APPARMOR.
> > > > > >>>>
> > > > > >>>> And the numbers are, take the following test for instance:
> > > > > >>>>
> > > > > >>>> dd if=/dev/zero of=/dev/null count=2M
> > > > > >>>>
> > > > > >>>> ARM64: 500MB/s -> 350MB/s
> > > > > >>>> ARM: 400MB/s -> 300MB/s
> > > > > >>> Hi there.
> > > > > >>>
> > > > > >>> Out of curiosity, what kernel/distribution are you running,
> > > > > >>> or is this a custom kernel compile? Can you also share the
> > > > > >>> output of 'auditctl
> > > > > >> This test was carried out with Linux 4.9. Custom built.
> > > > > > I suspected that was the case, thanks.
> > > > > >
> > > > > >>> -l' from your system? The general approach taken by
> > > > > >>> everyone to turn-off the per-syscall audit overhead is to
> > > > > >>> add the "-a never,task" rule to their audit configuration:
> > > > > >>>
> > > > > >>> # auditctl -a never,task
> > > > > >>>
> > > > > >>> If you are using Fedora/CentOS/RHEL, or a similarly
> > > > > >>> configured system,
> > > > > >> This is an embedded distribution. We are not using auditctl
> > > > > >> or any other audit-related user-space packages.
> > > > > >>
> > > > > >>> you can find this configuration in
> > > > > >>> the /etc/audit/audit.rules file (be warned, that file is
> > > > > >>> automatically generated based on /etc/audit/rules.d).
> > > > > >> I suppose in this case rule list must be empty. Is there a
> > > > > >> way to check this without extra user-space packages?
> > > > > > Yes, unless you are loading rules through some other method I
> > > > > > would expect that your audit rule list is empty.
> > > > > >
> > > > > > I'm not aware of any other tools besides auditctl to load
> > > > > > audit rules into the kernel, although I haven't ever had a
> > > > > > need for another tool so I haven't looked very hard. If you
> > > > > > didn't want to bring auditctl into your distribution, I
> > > > > > expect it would be a rather trivial task to create a small
> > > > > > tool to load a single "-a never,task" into the kernel.
> > > > >
> > > > > I've done a quick test on my x86_64 PC and got the following
> > > > > results:
> > > >
> > > > ...
> > > >
> > > > > Which brings me to an idea, that the subject patch should have
> > > > > been accompanied by a default "never,task" rule inside the
> > > > > kernel, otherwise you require an extra user-space package
> > > > > (audit) just to bring Linux 4.5+ to 4.4 performance levels.
> > > >
> > > > [NOTE: I dropped [email protected] from the To/CC line, I left
> > > > Red Hat for greener pastures several months ago.]
> > > >
> > > > Well, it generally hasn't been an issue as 1) most people that
> > > > enable audit also enable syscall auditing and 2) most people that
> > > > enable audit have some sort of audit userspace tools to work with
> > > > the audit logs (and configure audit if necessary). I don't want
> > > > to diminish your report, but this change was made several years
> > > > ago and we really haven't heard of many issues surrounding the
> > > > change. If we can ever get all of the different arches to
> > > > support syscall auditing, I'd love to get rid of the syscall
> > > > auditing Kconfig knob entirely.
> > > >
> > > > If you wanted to put together a patch that added a single "-a
> > > > never,task" rule on boot I could get behind that, just make it
> > > > default to off.
> > >
> > > That will make processes unauditable for everyone. That's how it
> > > gets a speedup is not entering into the audit machinery.
> >
> > That is pretty much what is being asked for in this thread. It's
> > really no different from what Fedora/CentOS/RHEL (and who knows how
> > many others) ship with their default audit config. It's important to
> > note that you could always delete this rule at runtime; all that is
> > being proposed is to have the kernel populate the the audit ruleset
> > with the "-a never,task" rule *IF* the proposed kernel Kconfig option
> > is enabled (and it would default to being off, you would have to turn
> > it on during build).
>
> Yes, but you can't add the auditable flag back to the task struct
> because syscalls never enter audit machinery where it can be added back.
> Meaning that even if you wanted to audit, there will be processes you
> never can audit. Whereas deciding via auditd, they become unauditable
> only after auditd loads the rule. Prior to that they are and all
> processes are auditable if you decide to audit. So, this is
> fundamentally different than what fedora does.

What I'm suggesting is adding an audit rule, just like Fedora, the
only difference is that it is going to be populated at boot by the
kernel. Sure, it is going to happen earlier than if you use auditctl
during boot, but arguably if you want to disable all the syscall
auditing, this is what you want. Calling it fundamentally different
from what Fedora does is not correct.

For the record, this is something I would only recommend for a limited
number of use cases (e.g. limited embedded environments with no need
for syscall auditing), as for most users it is better to install a
proper audit userspace and just configure it with auditctl during
boot.

--
paul moore
http://www.paul-moore.com