On Tue, Feb 28, 2017 at 10:37 PM, Richard Guy Briggs <[email protected]> wrote:
> Sorry, I forgot to include Cc: in this cover letter for context to the 4
> alt patches.
>
> On 2017-02-28 22:15, Richard Guy Briggs wrote:
>> The background to this is:
>> https://github.com/linux-audit/audit-kernel/issues/8
>>
>> In short, audit SYSCALL records for *init_module were occasionally
>> accompanied by hundreds to thousands of null PATH records.
>>
>> I chatted with Al Viro and Eric Paris about this Friday afternoon and
>> they seemed to vaguely recall this issue and didn't have any solid
>> recommendations as to what was the right thing to do (other than the
>> same suggestion from both that I won't print here).
>>
>> It was reproducible on a number of vintages of distributions with
>> default kernels, but triggering on very few of the many modules loaded
>> at boot time. It was reproduced with fs-nfs4 and nfsv4 modules on
>> tracefs, but there are reports of it also happening with debugfs. It
>> was triggering only in __audit_inode_child with a parent that was not
>> found in the task context's audit names_list.
I'm no expert on the tracing system, but my understanding is that it
used to use debugfs but now prefers tracefs so perhaps depending on
the vintage of the kernel/userspace you will see it on either debugfs
or tracefs. I'm also guessing that module load order may have an
effect, maybe not.
>> I have four potential solutions listed in my order of preference and I'd
>> like to get some feedback about which one would be the most acceptable.
>From an audit perspective, I'm generally not a fan of throwing away
information, especially since solution #4 seems to provide some basic
PATH information. Although I guess the issue is do we care about
tracefs/debugfs PATH records?
>> 1 - In __audit_inode_child, return immedialy upon detecting TRACEFS and
>> DEBUGFS (and potentially other filesystems identified, via s_magic).
If we decide we want to ignore debugfs/tracefs this may be the best solution.
>> 2 - In __audit_inode_child, return after not finding the parent in that
>> task context's audit names_list.
This doesn't seem like the right answer.
>> 3 - In __audit_inode_child, mark the parent and its child as "hidden"
>> when the parent isn't found in that task context's audit names_list.
>> This will still result in an "items=" count that does not match the
>> number of accompanying PATH records for that SYSCALL record, which
>> may upset userspace tools but would still indicate suppressed
>> records.
Similar to door #2, this doesn't seem right to me.
>> 4 - In __audit_inode_child, when the parent isn't found, store the
>> child's dentry in the child's (new or not) audit_names structure
>> (properly refcounted with dget) and store the parent's dentry in its
>> newly created audit_names structure (via dget_parent), then if the
>> name isn't available at PATH record generation time, use that stored
>> value (with dentry_path_raw and released with dput)
This seems most in keeping with the spirit of audit.
>> Is there another more elegant solution that I've missed that catches
>> things before they get anywhere near audit_inode_child (called from
>> tracefs' notifiers)?
>>
>> I'll thread onto this message tested patches for all four solutions.
>>
>> - RGB
>>
>> --
>> Richard Guy Briggs <[email protected]>
>> Kernel Security Engineering, Base Operating Systems, Red Hat
>> Remote, Ottawa, Canada
>> Voice: +1.647.777.2635, Internal: (81) 32635
--
paul moore
http://www.paul-moore.com
On 2017-03-03 19:19, Paul Moore wrote:
> On Tue, Feb 28, 2017 at 10:37 PM, Richard Guy Briggs <[email protected]> wrote:
> > Sorry, I forgot to include Cc: in this cover letter for context to the 4
> > alt patches.
> >
> > On 2017-02-28 22:15, Richard Guy Briggs wrote:
> >> The background to this is:
> >> https://github.com/linux-audit/audit-kernel/issues/8
> >>
> >> In short, audit SYSCALL records for *init_module were occasionally
> >> accompanied by hundreds to thousands of null PATH records.
> >>
> >> I chatted with Al Viro and Eric Paris about this Friday afternoon and
> >> they seemed to vaguely recall this issue and didn't have any solid
> >> recommendations as to what was the right thing to do (other than the
> >> same suggestion from both that I won't print here).
> >>
> >> It was reproducible on a number of vintages of distributions with
> >> default kernels, but triggering on very few of the many modules loaded
> >> at boot time. It was reproduced with fs-nfs4 and nfsv4 modules on
> >> tracefs, but there are reports of it also happening with debugfs. It
> >> was triggering only in __audit_inode_child with a parent that was not
> >> found in the task context's audit names_list.
>
> I'm no expert on the tracing system, but my understanding is that it
> used to use debugfs but now prefers tracefs so perhaps depending on
> the vintage of the kernel/userspace you will see it on either debugfs
> or tracefs. I'm also guessing that module load order may have an
> effect, maybe not.
I'm at the same level of understanding.
> >> I have four potential solutions listed in my order of preference and I'd
> >> like to get some feedback about which one would be the most acceptable.
>
> From an audit perspective, I'm generally not a fan of throwing away
> information, especially since solution #4 seems to provide some basic
> PATH information. Although I guess the issue is do we care about
> tracefs/debugfs PATH records?
>From the output I've seen, it doesn't look particularly useful, but it
was useful to finally see the source of those huge numbers of PATH
records. Here's an fpaste:
https://paste.fedoraproject.org/paste/UpZoYuokojR0es1ayNdx5l5M1UNdIGYhyRLivL9gydE=/
> >> 1 - In __audit_inode_child, return immedialy upon detecting TRACEFS and
> >> DEBUGFS (and potentially other filesystems identified, via s_magic).
>
> If we decide we want to ignore debugfs/tracefs this may be the best solution.
Glad we agree there.
> >> 2 - In __audit_inode_child, return after not finding the parent in that
> >> task context's audit names_list.
>
> This doesn't seem like the right answer.
I have another patch that tried to reuse existing entries even if no
struct filename was supplied to __audit_inode, but it didn't seem to
make a difference. Everything that was working continued to do so and
everything that was broken remained so.
> >> 3 - In __audit_inode_child, mark the parent and its child as "hidden"
> >> when the parent isn't found in that task context's audit names_list.
> >> This will still result in an "items=" count that does not match the
> >> number of accompanying PATH records for that SYSCALL record, which
> >> may upset userspace tools but would still indicate suppressed
> >> records.
>
> Similar to door #2, this doesn't seem right to me.
I did think of supplementing that information with a general
characterization that all the items came from a filesystem that was of
no concern so that there was evidence of what was happenning but that it
wouldn't overwhelm the logs.
> >> 4 - In __audit_inode_child, when the parent isn't found, store the
> >> child's dentry in the child's (new or not) audit_names structure
> >> (properly refcounted with dget) and store the parent's dentry in its
> >> newly created audit_names structure (via dget_parent), then if the
> >> name isn't available at PATH record generation time, use that stored
> >> value (with dentry_path_raw and released with dput)
>
> This seems most in keeping with the spirit of audit.
Agreed, but looks like too much useless information.
> >> Is there another more elegant solution that I've missed that catches
> >> things before they get anywhere near audit_inode_child (called from
> >> tracefs' notifiers)?
> >>
> >> I'll thread onto this message tested patches for all four solutions.
> >>
> >> - RGB
>
> paul moore
- RGB
--
Richard Guy Briggs <[email protected]>
Kernel Security Engineering, Base Operating Systems, Red Hat
Remote, Ottawa, Canada
Voice: +1.647.777.2635, Internal: (81) 32635
On Fri, 3 Mar 2017 19:19:47 -0500
Paul Moore <[email protected]> wrote:
> On Tue, Feb 28, 2017 at 10:37 PM, Richard Guy Briggs <[email protected]> wrote:
> > Sorry, I forgot to include Cc: in this cover letter for context to the 4
> > alt patches.
> >
> > On 2017-02-28 22:15, Richard Guy Briggs wrote:
> >> The background to this is:
> >> https://github.com/linux-audit/audit-kernel/issues/8
> >>
> >> In short, audit SYSCALL records for *init_module were occasionally
> >> accompanied by hundreds to thousands of null PATH records.
> >>
> >> I chatted with Al Viro and Eric Paris about this Friday afternoon and
> >> they seemed to vaguely recall this issue and didn't have any solid
> >> recommendations as to what was the right thing to do (other than the
> >> same suggestion from both that I won't print here).
> >>
> >> It was reproducible on a number of vintages of distributions with
> >> default kernels, but triggering on very few of the many modules loaded
> >> at boot time. It was reproduced with fs-nfs4 and nfsv4 modules on
> >> tracefs, but there are reports of it also happening with debugfs. It
> >> was triggering only in __audit_inode_child with a parent that was not
> >> found in the task context's audit names_list.
>
> I'm no expert on the tracing system, but my understanding is that it
> used to use debugfs but now prefers tracefs so perhaps depending on
> the vintage of the kernel/userspace you will see it on either debugfs
> or tracefs. I'm also guessing that module load order may have an
> effect, maybe not.
Note, when you mount debugfs, it automounts tracefs in debugfs/tracing.
Userspace can also mount tracefs without debugfs. But tracing does not
use debugfs anymore, even though it appears in the debugfs directory.
>
> >> I have four potential solutions listed in my order of preference and I'd
> >> like to get some feedback about which one would be the most acceptable.
>
> >From an audit perspective, I'm generally not a fan of throwing away
> information, especially since solution #4 seems to provide some basic
> PATH information. Although I guess the issue is do we care about
> tracefs/debugfs PATH records?
I don't have enough context here to really understand what the issue
is. Is there a problem when modules have trace events and when they are
loaded, these trace events create files and directories in the tracefs
file system?
-- Steve
On Mon, 6 Mar 2017 22:39:54 -0500
Richard Guy Briggs <[email protected]> wrote:
> >From the output I've seen, it doesn't look particularly useful, but it
> was useful to finally see the source of those huge numbers of PATH
> records. Here's an fpaste:
> https://paste.fedoraproject.org/paste/UpZoYuokojR0es1ayNdx5l5M1UNdIGYhyRLivL9gydE=/
Those are the files for the module's trace events that are created.
I'm still confused about what the issue is.
-- Steve
On Tue, 7 Mar 2017 11:00:27 -0500
Richard Guy Briggs <[email protected]> wrote:
> On 2017-03-07 10:41, Steven Rostedt wrote:
> > On Mon, 6 Mar 2017 22:39:54 -0500
> > Richard Guy Briggs <[email protected]> wrote:
> >
> > > >From the output I've seen, it doesn't look particularly useful, but it
> > > was useful to finally see the source of those huge numbers of PATH
> > > records. Here's an fpaste:
> > > https://paste.fedoraproject.org/paste/UpZoYuokojR0es1ayNdx5l5M1UNdIGYhyRLivL9gydE=/
> >
> > Those are the files for the module's trace events that are created.
> >
> > I'm still confused about what the issue is.
>
> The issue is the audit subsystem being overwhelmed by potentially
> useless information.
>
> The initial report was "there's a bunch of null PATH records, please
> make them go away", which was anywhere from 500 to 6000 records.
I don't know the audit system and exactly what it is looking for. How
does it deal with other virtual filesystems like procfs? Why is tracefs
different?
-- Steve
On 2017-03-07 10:41, Steven Rostedt wrote:
> On Mon, 6 Mar 2017 22:39:54 -0500
> Richard Guy Briggs <[email protected]> wrote:
>
> > >From the output I've seen, it doesn't look particularly useful, but it
> > was useful to finally see the source of those huge numbers of PATH
> > records. Here's an fpaste:
> > https://paste.fedoraproject.org/paste/UpZoYuokojR0es1ayNdx5l5M1UNdIGYhyRLivL9gydE=/
>
> Those are the files for the module's trace events that are created.
>
> I'm still confused about what the issue is.
The issue is the audit subsystem being overwhelmed by potentially
useless information.
The initial report was "there's a bunch of null PATH records, please
make them go away", which was anywhere from 500 to 6000 records.
Once I found out what they generally were and a way to enumerate the
contents, we're reviewing that assessment to see if they really should
be tossed, or listed out in full.
> -- Steve
- RGB
--
Richard Guy Briggs <[email protected]>
Kernel Security Engineering, Base Operating Systems, Red Hat
Remote, Ottawa, Canada
Voice: +1.647.777.2635, Internal: (81) 32635
On 2017-03-07 11:20, Steven Rostedt wrote:
> On Tue, 7 Mar 2017 11:00:27 -0500
> Richard Guy Briggs <[email protected]> wrote:
>
> > On 2017-03-07 10:41, Steven Rostedt wrote:
> > > On Mon, 6 Mar 2017 22:39:54 -0500
> > > Richard Guy Briggs <[email protected]> wrote:
> > >
> > > > >From the output I've seen, it doesn't look particularly useful, but it
> > > > was useful to finally see the source of those huge numbers of PATH
> > > > records. Here's an fpaste:
> > > > https://paste.fedoraproject.org/paste/UpZoYuokojR0es1ayNdx5l5M1UNdIGYhyRLivL9gydE=/
> > >
> > > Those are the files for the module's trace events that are created.
> > >
> > > I'm still confused about what the issue is.
> >
> > The issue is the audit subsystem being overwhelmed by potentially
> > useless information.
> >
> > The initial report was "there's a bunch of null PATH records, please
> > make them go away", which was anywhere from 500 to 6000 records.
>
> I don't know the audit system and exactly what it is looking for. How
> does it deal with other virtual filesystems like procfs? Why is tracefs
> different?
The audit subsystem is looking, via sysadmin-crafted rules to notice
situations of interest, for details that could affect the integrity of
the system
This situation is specifically for syscall auditing. Various syscalls
have expected numbers of accompanying PATH records, for example open(2)
would have directory and file PATH records, while rename(2) would have 2
directory and 2 file PATH records. An open call on a file in /proc
could trigger a rule that was formulated to catch that type of activity
that lists its directory and its file PATH records, which would be fine.
We normally don't expect the init_module syscall to have any PATH
records associated with it, so when a few of them had hundreds or more
this was surprising.
If there is a way that debugfs or tracefs could be abused during an
init_module call (or any other syscall for that matter), we want to be
aware of it. This is why simply ignoring those PATH records is making
two of us nervous.
> -- Steve
- RGB
--
Richard Guy Briggs <[email protected]>
Kernel Security Engineering, Base Operating Systems, Red Hat
Remote, Ottawa, Canada
Voice: +1.647.777.2635, Internal: (81) 32635
On 2017-03-07 13:04, Steven Rostedt wrote:
> On Tue, 7 Mar 2017 12:39:55 -0500
> Richard Guy Briggs <[email protected]> wrote:
>
> > We normally don't expect the init_module syscall to have any PATH
> > records associated with it, so when a few of them had hundreds or more
> > this was surprising.
>
> Hmm, how does the syscall get a path associated to it? Just by its
> creation? That is, by calling init_module() which would load a module,
> would indeed create a path. Some modules do create their own debugfs
> files, which would explain why debugfs is shown too.
My understanding is that a module binary blob is already acquired from
some source and then handed to the init_module (or finit_module) syscall
to add to the running kernel. Some add functionality in /proc or /sys,
but these would not be exercised until they are called by name from
another syscall (such as open).
Syscall auditing is interested in the resources/details of *one* syscall
event at a time (from audit_syscall_entry to audit_syscall_exit),
logging the subject attributes of a process (who) doing what (which
syscall) to what (frequently a file). Depending on the syscall, there
could be any number of auxilliary records to that event that help fill
in the whole picture that interests us.
So which file are you talking about that "would indeed create a path"?
> > If there is a way that debugfs or tracefs could be abused during an
> > init_module call (or any other syscall for that matter), we want to be
> > aware of it. This is why simply ignoring those PATH records is making
> > two of us nervous.
>
> If there's a bug in the kernel code, then I'm sure there's probably a
> way to abuse it.
>
> I also don't believe it should be ignored, which is why I'm asking
> these questions. I want to know what exactly is being looked at, and
> what is considered "OK" and what isn't.
That one is harder to answer and depends on the syscall and its
potential to influence system behaviour, or to exfiltrate information.
> Now loading modules can indeed create files and directories. Is this
> something that the audit system needs to understand?
Does it create them immediately in that syscall? Or does it make that
path available for other operations later?
> -- Steve
- RGB
--
Richard Guy Briggs <[email protected]>
Kernel Security Engineering, Base Operating Systems, Red Hat
Remote, Ottawa, Canada
Voice: +1.647.777.2635, Internal: (81) 32635
On Tue, 7 Mar 2017 13:34:47 -0500
Richard Guy Briggs <[email protected]> wrote:
> On 2017-03-07 13:04, Steven Rostedt wrote:
> > On Tue, 7 Mar 2017 12:39:55 -0500
> > Richard Guy Briggs <[email protected]> wrote:
> >
> > > We normally don't expect the init_module syscall to have any PATH
> > > records associated with it, so when a few of them had hundreds or more
> > > this was surprising.
> >
> > Hmm, how does the syscall get a path associated to it? Just by its
> > creation? That is, by calling init_module() which would load a module,
> > would indeed create a path. Some modules do create their own debugfs
> > files, which would explain why debugfs is shown too.
>
> My understanding is that a module binary blob is already acquired from
> some source and then handed to the init_module (or finit_module) syscall
> to add to the running kernel. Some add functionality in /proc or /sys,
> but these would not be exercised until they are called by name from
> another syscall (such as open).
>
> Syscall auditing is interested in the resources/details of *one* syscall
> event at a time (from audit_syscall_entry to audit_syscall_exit),
> logging the subject attributes of a process (who) doing what (which
> syscall) to what (frequently a file). Depending on the syscall, there
> could be any number of auxilliary records to that event that help fill
> in the whole picture that interests us.
>
> So which file are you talking about that "would indeed create a path"?
The files in /sys/kernel{/debug}/tracing/events/*
A module may connect "trace events" to parts of its code. When a module
gets loaded, virtual directories and files are created with respect to
the events within the module. When the module is unloaded, those files
are removed.
>
> > > If there is a way that debugfs or tracefs could be abused during an
> > > init_module call (or any other syscall for that matter), we want to be
> > > aware of it. This is why simply ignoring those PATH records is making
> > > two of us nervous.
> >
> > If there's a bug in the kernel code, then I'm sure there's probably a
> > way to abuse it.
> >
> > I also don't believe it should be ignored, which is why I'm asking
> > these questions. I want to know what exactly is being looked at, and
> > what is considered "OK" and what isn't.
>
> That one is harder to answer and depends on the syscall and its
> potential to influence system behaviour, or to exfiltrate information.
>
> > Now loading modules can indeed create files and directories. Is this
> > something that the audit system needs to understand?
>
> Does it create them immediately in that syscall? Or does it make that
> path available for other operations later?
Not sure what you mean here. The files are created, but to use them,
another process needs to do an open and write to them.
The inodes and dentrys are created. But the process should not have any
file descriptors created associated with them.
-- Steve
On Tue, 7 Mar 2017 12:39:55 -0500
Richard Guy Briggs <[email protected]> wrote:
> We normally don't expect the init_module syscall to have any PATH
> records associated with it, so when a few of them had hundreds or more
> this was surprising.
Hmm, how does the syscall get a path associated to it? Just by its
creation? That is, by calling init_module() which would load a module,
would indeed create a path. Some modules do create their own debugfs
files, which would explain why debugfs is shown too.
>
> If there is a way that debugfs or tracefs could be abused during an
> init_module call (or any other syscall for that matter), we want to be
> aware of it. This is why simply ignoring those PATH records is making
> two of us nervous.
If there's a bug in the kernel code, then I'm sure there's probably a
way to abuse it.
I also don't believe it should be ignored, which is why I'm asking
these questions. I want to know what exactly is being looked at, and
what is considered "OK" and what isn't.
Now loading modules can indeed create files and directories. Is this
something that the audit system needs to understand?
-- Steve
On 2017-03-07 14:09, Steven Rostedt wrote:
> On Tue, 7 Mar 2017 13:34:47 -0500
> Richard Guy Briggs <[email protected]> wrote:
>
> > On 2017-03-07 13:04, Steven Rostedt wrote:
> > > On Tue, 7 Mar 2017 12:39:55 -0500
> > > Richard Guy Briggs <[email protected]> wrote:
> > >
> > > > We normally don't expect the init_module syscall to have any PATH
> > > > records associated with it, so when a few of them had hundreds or more
> > > > this was surprising.
> > >
> > > Hmm, how does the syscall get a path associated to it? Just by its
> > > creation? That is, by calling init_module() which would load a module,
> > > would indeed create a path. Some modules do create their own debugfs
> > > files, which would explain why debugfs is shown too.
> >
> > My understanding is that a module binary blob is already acquired from
> > some source and then handed to the init_module (or finit_module) syscall
> > to add to the running kernel. Some add functionality in /proc or /sys,
> > but these would not be exercised until they are called by name from
> > another syscall (such as open).
> >
> > Syscall auditing is interested in the resources/details of *one* syscall
> > event at a time (from audit_syscall_entry to audit_syscall_exit),
> > logging the subject attributes of a process (who) doing what (which
> > syscall) to what (frequently a file). Depending on the syscall, there
> > could be any number of auxilliary records to that event that help fill
> > in the whole picture that interests us.
> >
> > So which file are you talking about that "would indeed create a path"?
>
> The files in /sys/kernel{/debug}/tracing/events/*
These appear to be the null PATHs I'm looking for.
So these appear to be the PATH records that are being seen, mounted on
/sys/kernel{/debug}/tracing/, but showing up as anonymous because the
path to the mount point is unknown or unavailable in the audit_names
list at the time of the syscall.
Could that tracefs instance have been populated but not yet mounted at
the time the module was loaded on boot? (fs-nfs4, nfsv4)
> A module may connect "trace events" to parts of its code. When a module
> gets loaded, virtual directories and files are created with respect to
> the events within the module. When the module is unloaded, those files
> are removed.
Can the availability of debug or trace points in a module be forced off
by a user to avoid or limit the problem?
Is the availability of these trace points a build time or run time switch?
Is there any security liability to having those trace points available
in the filesystem in terms of control or information leakage? (Sounds
like yes.)
> > > > If there is a way that debugfs or tracefs could be abused during an
> > > > init_module call (or any other syscall for that matter), we want to be
> > > > aware of it. This is why simply ignoring those PATH records is making
> > > > two of us nervous.
> > >
> > > If there's a bug in the kernel code, then I'm sure there's probably a
> > > way to abuse it.
> > >
> > > I also don't believe it should be ignored, which is why I'm asking
> > > these questions. I want to know what exactly is being looked at, and
> > > what is considered "OK" and what isn't.
> >
> > That one is harder to answer and depends on the syscall and its
> > potential to influence system behaviour, or to exfiltrate information.
> >
> > > Now loading modules can indeed create files and directories. Is this
> > > something that the audit system needs to understand?
> >
> > Does it create them immediately in that syscall? Or does it make that
> > path available for other operations later?
>
> Not sure what you mean here. The files are created, but to use them,
> another process needs to do an open and write to them.
Ok, so I think I've concluded that these are the same files.
So the system is working as intended. The next question is how do we
address the issue, perhaps by answering the three questions above.
> The inodes and dentrys are created. But the process should not have any
> file descriptors created associated with them.
It doesn't need to. I assume a module could open a file from within the
kernel to read it or write it and then close it, all within the module's
init routine and be done by the time the syscall finishes.
> -- Steve
- RGB
--
Richard Guy Briggs <[email protected]>
Kernel Security Engineering, Base Operating Systems, Red Hat
Remote, Ottawa, Canada
Voice: +1.647.777.2635, Internal: (81) 32635
On Tuesday, March 7, 2017 11:00:27 AM EST Richard Guy Briggs wrote:
> On 2017-03-07 10:41, Steven Rostedt wrote:
> > On Mon, 6 Mar 2017 22:39:54 -0500
> >
> > Richard Guy Briggs <[email protected]> wrote:
> > > >From the output I've seen, it doesn't look particularly useful, but it
> > >
> > > was useful to finally see the source of those huge numbers of PATH
> > >
> > > records. Here's an fpaste:
> > > https://paste.fedoraproject.org/paste/
UpZoYuokojR0es1ayNdx5l5M1UNdIGYhy
> > > RLivL9gydE=/>
> > Those are the files for the module's trace events that are created.
> >
> > I'm still confused about what the issue is.
>
> The issue is the audit subsystem being overwhelmed by potentially
> useless information.
>
> The initial report was "there's a bunch of null PATH records, please
> make them go away", which was anywhere from 500 to 6000 records.
>
> Once I found out what they generally were and a way to enumerate the
> contents, we're reviewing that assessment to see if they really should
> be tossed, or listed out in full.
They should be tossed. They do not help in any way. What we need for an
auxiliary record is simply the module's name. That's all.
-Steve