This note describes a patch against 2.6.4-rc1-bk2 that provides a
low-overhead system-call auditing framework for Linux that is usable by
LSM components (e.g., SELinux). Comments will be appreciated.
The main goals were to provide system call auditing with 1) as low
overhead as possible, and 2) without duplicating functionality that is
already provided by SELinux (and/or other security infrastructures).
This framework will work "stand-alone", but is not designed to provide,
e.g., CAPP functionality without another security component in place.
There are two main parts, one that is always on (generic logging in
audit.c) and one that you can disable at boot- or run-time
(per-system-call auditing in auditsc.c). The patch includes changes to
security/selinux/avc.c as an example of how system-call auditing can be
integrated with other code that identifies auditable events.
Logging:
1) Uses a netlink socket for communication with user-space. All
messages are logged via the netlink socket if a user-space daemon
is listening. If not, the messages are logged via printk to the
syslog daemon (by default).
2) Messages can be dropped (optionally) based on message rate or
memory use (this isn't fully integrated into the selinux/avc.c
part of the patch: the avc.c code that currently does this can be
eliminated).
3) When some part of the kernel generates part of an audit record,
the partial record is sent immediately to user-space, AND the
system call "auditable" flag is automatically set for that call
-- thereby producing extra information at syscall exit (if
syscall auditing is enabled).
System-call auditing:
1) At task-creation time, an audit context is allocated and linked
off the task structure.
2) At syscall entry time, if the audit context exists, information
is filled in (syscall number, timestamp; but not arguments).
3) During the system call, calls to getname() and path_lookup() are
intercepted. These routines are called when the kernel is
actually looking up information that will be used to make the
decision about whether the syscall will succeed or fail. An
effort has been made to avoid copying the information that
getname generates, since getname is already making a
kernel-private copy of the information. [Note that storing
copies of all syscall arguments requires complexity and overhead
that arguably isn't needed. With this patch, for example, if
chroot("foo") fails because you are not root, "foo" will not
appear in the audit record because the kernel determined the
syscall cannot proceed before it ever needed to look up "foo".
This approach avoids storing user-supplied information that could
be misleading or unreliable (e.g., due to a cooperative
shared-memory attack) in favor of reporting information actually
used by the kernel.]
4) At syscall exit time, if the "auditable" flag has been set (e.g.,
because SELinux generated an avc record; or some other part of
the kernel detected an auditable event), the syscall-part of the
audit record is generated, including file names and inode numbers
(if available). Some of this information is currently
complementary to the information that selinux/avc.c generates
(e.g., file names and some inode numbers), but some is less
complete (e.g., getname doesn't return a fully-qualified path,
and this patch does not add the overhead of determining one).
[Note that the complete audit record comes to userspace in
pieces, which eliminates the need to store messages for
arbitrarily long periods inside the kernel.]
5) At task-exit time, the audit context is destroyed.
At steps 1, 2, and 4, simple filtering can be done (e.g., a database
role uid might have syscall auditing disabled for performance
reasons). The filtering is simple and could be made more complex.
However, I tried to implement as much filtering as possible without
adding significant overhead (e.g., d_path()). In general, the audit
framework should rely on some other kernel component (e.g., SELinux)
to make the majority of the decisions about what is and is not
auditable.
A few (obvious) things are not yet implemented:
1) Filtering on syscall personality (i.e., i386 calls on x86_64).
2) Multi-platform syscall support (currently syscall auditing has
only been implemented on i386, x86_64, and ppc64).
3) Documentation.
I have also written very crude user-space tools that can be used to test
the features provided (auditd-0.3.tar.gz). This demo tarball, a
readme.txt (see for how to compile), and the patch (and more recent
patches?) are available from: http://people.redhat.com/faith/audit/
arch/i386/kernel/entry.S | 6
arch/i386/kernel/ptrace.c | 8
arch/ppc64/kernel/entry.S | 15
arch/ppc64/kernel/ptrace.c | 27 -
arch/x86_64/ia32/ia32entry.S | 12
arch/x86_64/kernel/entry.S | 21
arch/x86_64/kernel/ptrace.c | 23 -
fs/namei.c | 15
include/asm-i386/thread_info.h | 5
include/asm-ppc64/thread_info.h | 3
include/asm-x86_64/thread_info.h | 5
include/linux/audit.h | 210 +++++++++
include/linux/fs.h | 10
include/linux/netlink.h | 1
include/linux/sched.h | 5
init/Kconfig | 20
kernel/Makefile | 2
kernel/audit.c | 771 ++++++++++++++++++++++++++++++++++
kernel/auditsc.c | 872 +++++++++++++++++++++++++++++++++++++++
kernel/fork.c | 9
security/selinux/avc.c | 146 ++----
security/selinux/include/avc.h | 7
security/selinux/ss/services.c | 2
23 files changed, 2065 insertions(+), 130 deletions(-)
Due to size, the patch is available from;
http://people.redhat.com/faith/audit/audit-20040226.1411.patch
On Mon, Mar 01, 2004 at 11:28:45AM -0500, Rik Faith wrote:
> This note describes a patch against 2.6.4-rc1-bk2 that provides a
> low-overhead system-call auditing framework for Linux that is usable by
> LSM components (e.g., SELinux). Comments will be appreciated.
I haven't actually looked at the code, but why don't you use Olaf Kirch's
auditing framework that's used in production and already has gotten the
wizzbang certification you seem to be aiming at.
Whether we want syscall auditing in mainline is a completely different
question..
* Rik Faith ([email protected]) wrote:
> This note describes a patch against 2.6.4-rc1-bk2 that provides a
> low-overhead system-call auditing framework for Linux that is usable by
> LSM components (e.g., SELinux). Comments will be appreciated.
Do you have an perf numbers for simply enabling the auditint framework?
Seems to be a lot of use of likely/unlikely. Are these actually useful
optimizations? Doesn't seem like CONFIG_AUDIT=n disables all the code.
Not sure if the rest of the patch lot is mainlineable or not, but here are
some simple nits/questions.
Some basic CodingStyle bits:
+static int audit_initialized = 0;
no need to zero initialized static data.
+ if (in_irq()) BUG();
+ if (!context) BUG();
convert to BUG_ON(in_irq());
BUG_ON(!context);
+#ifdef CONFIG_AUDIT
+/***********************************************************************/
+/** audit.c **/
+/***********************************************************************/
Noisy comments.
+ audit_log_format(ab, " dev=%02x:%02x",
+ MAJOR(context->names[i].rdev),
+ MINOR(context->names[i].rdev));
format_dev_t() ?
+static int __init audit_enable(char *str)
+{
+ if (!strncmp(str, "on", 2) || !strncmp(str, "yes", 3))
+ audit_default = 1;
+ else if (!strncmp(str, "off", 3) || !strncmp(str, "no", 2))
+ audit_default = 0;
+ else
+ audit_default = simple_strtol(str, NULL, 0);
just pick one method, strings or int (int is so simple), and document the
method for enable/disable.
+void audit_log_vformat(struct audit_buffer *ab, const char *fmt, va_list args)
as of now, this could be static, and not exported, no?
+ ab = kmalloc(sizeof(*ab), GFP_ATOMIC);
need atomic allocations here? also, i wasn't clear on what makes sure the
audit_buffers can't be overflown, the bits in vformat talking about 1024 bytes?
+ audit_log_format(ab, "<toolong>");
is it useful to at least get the final d_name?
thanks,
-chris
--
Linux Security Modules http://lsm.immunix.org http://lsm.bkbits.net
On Mon 1 Mar 2004 19:45:01 +0000,
Christoph Hellwig <[email protected]> wrote:
> On Mon, Mar 01, 2004 at 11:28:45AM -0500, Rik Faith wrote:
> > This note describes a patch against 2.6.4-rc1-bk2 that provides a
> > low-overhead system-call auditing framework for Linux that is usable by
> > LSM components (e.g., SELinux). Comments will be appreciated.
>
> I haven't actually looked at the code, but why don't you use Olaf Kirch's
> auditing framework that's used in production and already has gotten the
> wizzbang certification you seem to be aiming at.
Different goals. My goals are to provide a generic very-low-overhead
auditing framework that can be used as a service by more complex systems
(e.g., SELinux). In contrast to Olaf's work, for example, my patch does
not have intimate knowledge of system call parameters and semantics.
This decreases the invasiveness of the patch and the work required for
long-term maintainability.
The price for this simplicity is that the "language" for describing
which system calls to audit is also very simple (and is, therefore, not
independently sufficient for certifications). However, I assume that a
system undergoing certification will be using SELinux or another
security infrastructure that will make auditing _and_ other decisions
-- adding sophistication to the auditing infrastructure only duplicates
the work that the security module will provide.
> Whether we want syscall auditing in mainline is a completely different
> question..
I believe we need a light-weight, maintainable framework that is
versatile enough to be used for non-security purposes (e.g., debugging).
In general, my patch meets these requirements since it provides very
little that helps specifically with security (failure modes, loginuid,
and small helper functions).
Hi Rik,
On Mon, Mar 01, 2004 at 03:28:45PM -0500, Rik Faith wrote:
> Different goals. My goals are to provide a generic very-low-overhead
> auditing framework that can be used as a service by more complex systems
> (e.g., SELinux). In contrast to Olaf's work, for example, my patch does
> not have intimate knowledge of system call parameters and semantics.
> This decreases the invasiveness of the patch and the work required for
> long-term maintainability.
I've looked at your patch. The idea to intercept path name arguments etc
inside getname and path_lookup is interesting. I had to jump through a lot
of hoops to prevent the caller from messing with the syscall arguments
inbetween the syscall and the audit intercept, and that made the code
a lot uglier than I had wanted.
This type of race condition is the major reason why I now believe
intercepting syscall entry/exit is not a good choice for auditing.
For debugging yes, but not for auditing.
However, I wonder your approach is sufficient to obtain certification.
The first thing that jumped at me was your comment that "netlink may
lose packets". This is an absolute show-stopper for CAPP compliance.
In general, during testing we noticed that passing audit from kernel to
a user land daemon can become a serious bottleneck. A better approach may
be to have the kernel write audit records directly to the audit trail;
this solves both the record loss issue and improves performance.
Another issue is that you have system calls such as link or rename
that consume two path name arguments, and your system call record
needs to identify which is which.
In general, I wonder how your frame work will audit other, non-pathname
system call arguments. Is that entirely a decision of the security
module? If so, then why create a generic audit framework at all?
In general, there is a question of whether the security framework
could be sufficient for auditing purposes. As it stands now, I can see
two shortcomings. The obvious one is that most security operations are
performed _before_ the operation, while audit records need to include the
outcome of the system call, so they are performed _after_ the operation.
That is something that could be solved the way your patch seems to be
doing it, i.e. flag the ongoing system call "interesting" and intercept
the syscall return.
The second issue with security operations though is that it is far from
clear if we cover all "relevant" system calls. CAPP is not very clear
regarding what is "relevant". For instance it's far from obivous to me
whether a system that fails with EFAULT or EINVAL needs to be audited
or not.
> I believe we need a light-weight, maintainable framework that is
> versatile enough to be used for non-security purposes (e.g., debugging).
I'd be happy if you could prove me wrong, but I believe that debugging and
(certifiable) system call auditing don't mix very well. They might be able
to share a minimal framework (e.g. for delivering an audit trail to user
land and/or a file).
The motivation behind these two objectives is quite different: audit
wants to capture security relevant operations, and deliver the events
reliably. Debugging doesn't care about reliability, but OTOH wants to
capture as wide a selection of system call events as possible.
Olaf
--
Olaf Kirch | Stop wasting entropy - start using predictable
[email protected] | tempfile names today!
---------------+
On Tue 2 Mar 2004 10:44:38 +0100,
Olaf Kirch <[email protected]> wrote:
> I've looked at your patch.
Hi Olaf. Thanks for your comments.
> However, I wonder your approach is sufficient to obtain certification.
> The first thing that jumped at me was your comment that "netlink may
> lose packets". This is an absolute show-stopper for CAPP compliance.
I didn't mean to imply that netlink was unreliable -- the drops aren't
arbitrary: you can detect them and you can decide if and when to drop.
Any logging system will have the problem of consuming kernel memory for
messages that are on their way to user-space. For those not interested
in CAPP compliance, the drops are tunable based on rate and size
("backlog" in the code). For those who are interested in CAPP, these
limits would not be in place, and total system memory would be the
limiting factor.
> A better approach may be to have the kernel write audit records
> directly to the audit trail; this solves both the record loss issue
> and improves performance.
This seems like it would solve several problems. I'll have to think
about it more. (You still have to worry about what happens when the
drive is too slow for the message rate, how to detect this, and,
perhaps, how to limit its impact on systems where reliable delivery of
messages is not absolutely critical. With netlink, these worries are
easy to deal with.)
> Another issue is that you have system calls such as link or rename
> that consume two path name arguments, and your system call record
> needs to identify which is which.
In order to keep the kernel-space piece tiny, I think the user-space
daemon should do as much of this work as possible. For example,
sys_rename always calls getname(oldname) before getname(newname), so it
is possible to tell which is which in user space.
> In general, I wonder how your frame work will audit other, non-pathname
> system call arguments. Is that entirely a decision of the security
> module? If so, then why create a generic audit framework at all?
Yes, that is a decision of the security module. The security module is
already looking at the arguments it believes to be important, so adding
additional overhead for auditing those arguments seems redundant. The
generic audit framework is providing more general services, such as the
netlink transport and the syscall entry/exit hooks services that can be
used to audit the syscall that was running when an auditable event
occurred. I.e., we probably don't want every security module using a
netlink number and patching entry.S.
> In general, there is a question of whether the security framework
> could be sufficient for auditing purposes. As it stands now, I can see
> two shortcomings. The obvious one is that most security operations are
> performed _before_ the operation, while audit records need to include the
> outcome of the system call, so they are performed _after_ the operation.
> That is something that could be solved the way your patch seems to be
> doing it, i.e. flag the ongoing system call "interesting" and intercept
> the syscall return.
Yes, the patch handles this. The sequence is:
1) syscall entry: build audit context
2) security operation detects auditable event and sets flag
3) syscall exit: if flag is set, syscall-specific record emitted
The audit framework doesn't know enough to do #2 -- that's the role of
the security module.
> The second issue with security operations though is that it is far from
> clear if we cover all "relevant" system calls. CAPP is not very clear
> regarding what is "relevant". For instance it's far from obivous to me
> whether a system that fails with EFAULT or EINVAL needs to be audited
> or not.
I don't have an interpretation of CAPP either, but I will explore ways
to make these two points easier to deal with.
> The motivation behind these two objectives is quite different: audit
> wants to capture security relevant operations, and deliver the events
> reliably. Debugging doesn't care about reliability, but OTOH wants to
> capture as wide a selection of system call events as possible.
In addition to delivering the audit trail to user-space, both objectives
may also require adding hooks to entry.S. If delivery and hooking is
useful for both, we should provide it only once inside the kernel. I've
tried to provide a generic framework that can be used as a service for
different objectives, with tuning that can favor one objective over the
other. It can grow to have more features, but I'm not sure if much can
be taken away (auditsc.c can be eliminated, but then you lose the
entry.S hooks).
On Tue 2 Mar 2004 06:09:09 -0500, Rik Faith <[email protected]> wrote:
> On Tue 2 Mar 2004 10:44:38 +0100, Olaf Kirch <[email protected]> wrote:
> > A better approach may be to have the kernel write audit records
> > directly to the audit trail; this solves both the record loss issue
> > and improves performance.
>
> This seems like it would solve several problems. I'll have to think
> about it more. (You still have to worry about what happens when the
> drive is too slow for the message rate, how to detect this, and,
> perhaps, how to limit its impact on systems where reliable delivery of
> messages is not absolutely critical. With netlink, these worries are
> easy to deal with.)
OK, I've thought about this more. I can appreciate the argument that
writing directly to disk avoids the performance impact of having a
user-space daemon write to disk (e.g., scheduling and, possibly, extra
copies). However, I do not believe this addresses the fundamental
problem and, because of the nature of the problem, I do not believe the
performance improvement will be observed in all cases. Let me explain.
CPUs can generate audit records magnitudes of order faster than they can
be written to disk. Further, the production is non-uniform. Because
this problem involves a bursty producer and a slow consumer, I view it
as a queuing problem. Writing directly to disk makes the consumer
marginally faster, but it does not change the fundamental relationships
in the problem -- the consumer is still magnitudes of order slower than
the producer. This means that, even though the direct-write consumer is
faster, the speed of the whole queue hasn't changed much, if at all.
Further, I am concerned that using direct writes instead of netlink
moves hard problems from one (simple, flexibly) layer to another (more
complex, less flexible) layer. For example, the file system may have
already made some decisions for you that you could have made differently
if you had access to information about the netlink-managed queue.
On Mon 1 Mar 2004 12:26:18 -0800,
Chris Wright <[email protected]> wrote:
> * Rik Faith ([email protected]) wrote:
> > This note describes a patch against 2.6.4-rc1-bk2 that provides a
> > low-overhead system-call auditing framework for Linux that is usable by
> > LSM components (e.g., SELinux). Comments will be appreciated.
>
> Do you have an perf numbers for simply enabling the auditint framework?
Comparing no patch with the patch applied and booted with audit=1 (so
the audit context is allocated for each task and filled in for every
syscall, but not sent to user-space), using lmbench 2.0.4:
Processor, Processes - times in microseconds - smaller is better
----------------------------------------------------------------
Host OS Mhz null null open selct sig sig fork exec sh
call I/O stat clos TCP inst hndl proc proc proc
--------- ------------- ---- ---- ---- ---- ---- ----- ---- ---- ---- ---- ----
1bk8stock Linux 2.6.3-b 2004 0.08 0.26 14.7 16.4 5.797 0.23 2.00 143. 443. 1804
4aon-nos Linux 2.6.3-b 2004 0.21 0.40 14.5 16.7 12.0 0.40 2.19 136. 441. 1717
At a higher level, I compiled the kernel three times with:
export CCACHE_DISABLE=1
make mrproper >& /dev/null
cp arch/x86_64/defconfig .config
yes '' | make oldconfig >& /dev/null
time make > /dev/null
Situation Best time (of 3) Factor
No output to user-space:
audit=0 (no audit context) 3:23.04u 34.560s 4:00.76 98% 1.000
audit=1 (audit context built) 3:23.47u 33.810s 4:01.02 98% 0.999
audit=1 (10 rules, no records) 3:28.39u 35.670s 4:07.48 98% 0.97
Output to user-space for _every_ syscall:
audit=1 (* all to /dev/null) 3:34.96u 1:16.98s 5:06.40 95% 0.79
audit=1 (** all to file) 4:22.82u 1:45.67s 7:11.51 85% 0.56
* Every syscall for the whole compile was audited and logged to
user-space. The user-space daemon sent all messages to /dev/null.
** Every syscall for the whole compile was audited and logged to
user-space. The user-space daemon sent all messages to a file (which
grew to 1.4GB on an IDE drive). This is the worst case scenario.
> Seems to be a lot of use of likely/unlikely. Are these actually useful
> optimizations?
The goal is for the non-auditing case to be as fast as possible, so all
of the likely/unlikely notations favor that case.
> Doesn't seem like CONFIG_AUDIT=n disables all the code.
The bit tests in entry.S are still there, but those are the same tests
that are used for ptrace, and there is nothing that sets the bits. So,
aside from that test, all of the code should be disabled.
> Some basic CodingStyle bits:
Except where noted below, I have either incorporated all your
suggestions or made notes in the code to do so later. The new patch is
at: http://people.redhat.com/faith/audit/audit-20040302.1632.patch
> + ab = kmalloc(sizeof(*ab), GFP_ATOMIC);
>
> need atomic allocations here?
These functions can be called in any context, so sleeping is not an
option.
> also, i wasn't clear on what makes sure the audit_buffers can't be
> overflown, the bits in vformat talking about 1024 bytes?
Yes, and the vsnprintf in audit_log_vformat.
* Rik Faith ([email protected]) wrote:
> > Doesn't seem like CONFIG_AUDIT=n disables all the code.
>
> The bit tests in entry.S are still there, but those are the same tests
> that are used for ptrace, and there is nothing that sets the bits. So,
> aside from that test, all of the code should be disabled.
I think, e.g. the code that calls audit_get/putname is still there.
> Except where noted below, I have either incorporated all your
> suggestions or made notes in the code to do so later. The new patch is
> at: http://people.redhat.com/faith/audit/audit-20040302.1632.patch
Oops, I wasn't clear re: the static initialized data...I just meant to
give a couple examples, there were more:
+static int audit_default = 0;
+static int audit_pid = 0;
+static int audit_rate_limit = 0;
+static int audit_freelist_count = 0;
etc...
thanks,
-chris
--
Linux Security Modules http://lsm.immunix.org http://lsm.bkbits.net
On Mon, Mar 01, 2004 at 03:28:45PM -0500, Rik Faith wrote:
> > Whether we want syscall auditing in mainline is a completely different
> > question..
>
> I believe we need a light-weight, maintainable framework that is
> versatile enough to be used for non-security purposes (e.g., debugging).
> In general, my patch meets these requirements since it provides very
> little that helps specifically with security (failure modes, loginuid,
> and small helper functions).
I'd like to second the sentiment - for syscalltrack, we would love to
have a framework that can be used for debugging system call events.
Cheers,
Muli
--
Muli Ben-Yehuda
http://www.mulix.org | http://mulix.livejournal.com/
On Tue 2 Mar 2004 16:49:51 -0800,
Chris Wright <[email protected]> wrote:
> * Rik Faith ([email protected]) wrote:
> > > Doesn't seem like CONFIG_AUDIT=n disables all the code.
> >
> > The bit tests in entry.S are still there, but those are the same tests
> > that are used for ptrace, and there is nothing that sets the bits. So,
> > aside from that test, all of the code should be disabled.
>
> I think, e.g. the code that calls audit_get/putname is still there.
When syscall auditing is disabled, the body of the if will become a nop
because of a #define, so the compiler will remove the whole if. I don't
want to move the if into a macro, since this would make it look like the
function was called all the time. I don't want the if in the function
because I'm trying not to call the function except when necessary.
I could put #ifdef CONFIG_AUDITSYSCALL around these statements, but I
find that often makes code harder to read. However, in this case, that
might avoid some confusion.
> > Except where noted below, I have either incorporated all your
> > suggestions or made notes in the code to do so later. The new patch is
> > at: http://people.redhat.com/faith/audit/audit-20040302.1632.patch
>
> Oops, I wasn't clear re: the static initialized data...
Yes, sorry, please see:
http://people.redhat.com/faith/audit/audit-20040303.0544.patch
On Wed 3 Mar 2004 10:55:01 +0200, Muli Ben-Yehuda <[email protected]> wrote:
> I'd like to second the sentiment - for syscalltrack, we would love to
> have a framework that can be used for debugging system call events.
I should clarify to avoid any confusion. The audit framework I am
proposing does _not_ allow for generic hooking of system calls.
However, the ability to audit the call is generic in the sense that
another part of the kernel can request that an audit record be generated
for the current in-progress system call (or filtering can be used to
request that audit records be generated). Although this should be
sufficient for SELinux and other security infrastructures, as well as
many non-security uses, it would only support a subset of what
syscalltrack can do currently (i.e., audit can provide information about
the syscall, but can't be used to change the behavior of the syscall).