So after about a week of trying to squeeze information out of
anti-malware companies I'm starting to feel like I can better speak for
their needs (although they probably don't like what I have to say). I
would like to point out that many enterprises are going to run this
stuff on their machines. Period. End of story. Personally I'd rather
support a clean interface than have to try to support support problems
my customers have when their hacked fragile systems have trouble.
So, what is it that anti-malware companies do? They scan files. That's
it. Many of the arguments they make and marketing speak we hear or
think we hear claim more grandious things. But from what I've seen they
aren't the real deal. Most of the security model descriptions that
people on list want actually are framed under the belief that these
products need to or attempt to completely block some class of attacks.
They don't. If you think they do you need to fix your frame of
reference. The only class of attacks this interface is supposed to
address is those that can be found by scanning files. This is NOT an
LSM. This is NOT an IDS. This is a file scanner. There may be all
sorts of marketing material or general beliefs that they provide
security against all sorts of wide and varied threats (and they do),
but in all reality the only threats they provide any help against are
those that can be found by scanning files. Simple as that. Some may
argue this isn't "good" security and I'm not going to make a strong
argument to the contrary, I can stand here for days and show cases where
this is highly useful but no one can provide a threat model more than to
say, "we attempt to locate files which may be harmful somewhere in the
digital ecosystem and try to deny access to that data."
It can be easily shown (and I can come up with LOTS of example) that
this is of value. Lets consider a simple Linux worm:
http://www.ca.com/us/securityadvisor/virusinfo/virus.aspx?id=47869
This worm needs 4 vulnerabilities for it to work and a default RHEL
SELinux policy would likely, although I haven't verified this fact, also
stop the propagation. I believe that the right way to "fix" this is to
patch your software and to run an LSM. But notice that 'scanning files'
is also enough to break the chain. When the download is completed
nothing would be able to execute the file in question since the
anti-malware software would prevent it. Assuming some new vulnerability
was found to replace one of the patched problems its possible the
anti-malware code could still recognize enough of the binary to stop it.
I'm told by one AV company that they look for 95 Linux-specific threats
composed of 42 trojans and 53 viruses or worms. At least 2 other
companies specifically stated that they look for malware which targets
Linux systems although didn't give hard numbers. All three vendors who
have spoken with me also admit that most of their threats are found on
Windows.
I also gave a previous example how file scanning can help with subverted
distro repos. That one is purely theoretical as opposed to the real
attack shown above, but clearly value can be easily demonstrated.
The value of a file scanning interface is not in stopping active
attacks. Its in making the Linux platform a more difficult location for
the storage and propagation of bad data. I think its reasonable to
think that we all agree we don't want to be the preferred hosting
platforms for trojan binaries intended to attack other non-Linux
systems. Why would one want consciencely choose to leave Linux as a
safe haven for the existence of malware? Even though the malware is not
attacking the Linux platform do we as the Linux community really want to
be a breeding and hosting ground as long as the costs are not too high?
(yes Ted, I'm talking directly to you here)
Assuming the costs are not too high, we should add this file scanning
subsystem to help prevent the storage and dissemination of malware
across our systems. If I'm running a hosting company I wouldn't want
people to be able to upload their trojans and host them on my Linux
server even if those trojans will only ever be downloaded by another OS.
We are part of the larger computing ecosystem. If we can make our
systems inhospitable to malware like data we help ourselves a little,
but the greater computing community a lot.
The real goal is to get files into to some userspace scanner and let
them do whatever they want. Remember, this isn't a new LSM. The goal
isn't to provide perfect security. The goal isn't to stop already
running malicious programs. The one and only goal is to scan files. We
should not be considering timing attacks, we should not be considering
processes actively trying to get around the system for small periods of
time. We should certainly not be considering root processes being able
to sneak things by. The idea is that a file exists on disk and we want
some userspace program to give a best effort at scanning it. Yes, we
might be able to prevent a root process from doing bad things, but a bad
root process already won.
Now costs:
1) Kernel or userspace. If we actually want a relatively robust
ecosystem of malware storage hardening it needs to be in kernel. knfsd
makes it plainly obvious that userspace scanners will leave big gaping
holes. Not to mention there cannot be any checks for files opened by
suid apps in userspace. And while no specific claims are being made
about intentionally malicious code, putting this in kernel makes
programs which call syscalls directly much more difficult to be used to
circumvent the scanning.
2) I think the "best" time to do scanning is at read and write. Any
disagreements there? But what's the cost? The cost is lots of scanning
and a huge performance hit on concurrent file access. I'm willing to
put hard numbers behind this, but it does seem quite obvious that open
and close happen a lot less than read and write. Sure that means
between open and read data may have become bad. We aren't an LSM. We
aren't and IDS. We aren't making solid claims about local system
security. We are claiming to harden the machine's storage and
propagation of 'bad' files.
3) code maintenance. I'm here. Sophos appears to be willing to help
and one of the other AV vendors said they may be willing to commit to
helping with support. If nothing else we know the AV vendors will hop
to get stuff fixed every time an enterprise distro is about to release
*smile*
I've yet to hear a technical argument against my patch set (although I
do plan some changes like using the audit system and changing the
kernel->userspace interface) Andi pointed out that I completely didn't
consider rawio on the block device but I have decided to just accept
that as a "you've already lost" condition. I've explained my mmap
support about 10 times, no one has said I'm wrong yet (I assume people
who can't read) keep talking about mmap.
No, this is not a new security system. This is not some great wonderful
panacea that's going to stop every exploit. Its a file scanner. It's
not bad. It clearly can help. This isn't just a pipe dream, I've got
code. Where's the problem?
-Eric
> So, what is it that anti-malware companies do? They scan files. That's
> it.
Good so lets instead have a discussion about making the file event
notification more scalable. That is the same thing I want for content
indexing. It is the same thing you want for certain kinds of smart
archiving, for on-line asynchronous backup and other stuff.
It ought to be a simple clean syscall interface.
Alan
On Wed, 2008-08-13 at 17:24 +0100, Alan Cox wrote:
> > So, what is it that anti-malware companies do? They scan files. That's
> > it.
>
> Good so lets instead have a discussion about making the file event
> notification more scalable. That is the same thing I want for content
> indexing. It is the same thing you want for certain kinds of smart
> archiving, for on-line asynchronous backup and other stuff.
>
> It ought to be a simple clean syscall interface.
Are you willing to make it blocking? I'm not sure how to make what we
have capable of assuring that the object you got a notification about is
actually the object you are acting on. Thoughts on how to accomplish
that? I'm here to code and I'm willing to throw all my work in the
garbage if someone can show me how to actually do it better.
-Eric
On Wed, 13 Aug 2008 12:47:45 -0400
Eric Paris <[email protected]> wrote:
> On Wed, 2008-08-13 at 17:24 +0100, Alan Cox wrote:
> > > So, what is it that anti-malware companies do? They scan files. That's
> > > it.
> >
> > Good so lets instead have a discussion about making the file event
> > notification more scalable. That is the same thing I want for content
> > indexing. It is the same thing you want for certain kinds of smart
> > archiving, for on-line asynchronous backup and other stuff.
> >
> > It ought to be a simple clean syscall interface.
>
> Are you willing to make it blocking? I'm not sure how to make what we
> have capable of assuring that the object you got a notification about is
> actually the object you are acting on. Thoughts on how to accomplish
> that? I'm here to code and I'm willing to throw all my work in the
> garbage if someone can show me how to actually do it better.
I don't think you need to be blocking if you passed up a file handle ?
fd = fileeventmumble(somestuff);
do_stuff
close(fd);
[taking care not to end up recursing as a result]
On Wed, Aug 13, 2008 at 12:36:15PM -0400, Eric Paris wrote:
> No, this is not a new security system. This is not some great wonderful
> panacea that's going to stop every exploit. Its a file scanner. It's
> not bad. It clearly can help. This isn't just a pipe dream, I've got
> code. Where's the problem?
Just repost your patches, with a description of what is happening with
them, taking into account the previously mentioned comments and we can
review it again :)
No problem at all, just a normal patch lifecycle...
thanks,
greg k-h
On Wed, 2008-08-13 at 17:37 +0100, Alan Cox wrote:
> On Wed, 13 Aug 2008 12:47:45 -0400
> Eric Paris <[email protected]> wrote:
>
> > On Wed, 2008-08-13 at 17:24 +0100, Alan Cox wrote:
> > > > So, what is it that anti-malware companies do? They scan files. That's
> > > > it.
> > >
> > > Good so lets instead have a discussion about making the file event
> > > notification more scalable. That is the same thing I want for content
> > > indexing. It is the same thing you want for certain kinds of smart
> > > archiving, for on-line asynchronous backup and other stuff.
> > >
> > > It ought to be a simple clean syscall interface.
> >
> > Are you willing to make it blocking? I'm not sure how to make what we
> > have capable of assuring that the object you got a notification about is
> > actually the object you are acting on. Thoughts on how to accomplish
> > that? I'm here to code and I'm willing to throw all my work in the
> > garbage if someone can show me how to actually do it better.
>
> I don't think you need to be blocking if you passed up a file handle ?
Without blocking and waiting how do you deny access? Maybe I needed
another thing they do. "They do file scanning and deny access to bad
files."
async scanning on close/write is great. but you need blocking/access
control on open/read.....
> fd = fileeventmumble(somestuff);
> do_stuff
> close(fd);
>
> [taking care not to end up recursing as a result]
[you pointed out the whole point of process exclusions in the original
work]
On Wed, Aug 13, 2008 at 05:24:37PM +0100, Alan Cox wrote:
> > So, what is it that anti-malware companies do? They scan files. That's
> > it.
>
> Good so lets instead have a discussion about making the file event
> notification more scalable. That is the same thing I want for content
> indexing. It is the same thing you want for certain kinds of smart
> archiving, for on-line asynchronous backup and other stuff.
Also for hierachial storage management, which also shares they other
requirement with the AV crowd that it want to be able to block the
calling process until the notification is ACKed (for recalling data
from offline media).
> It ought to be a simple clean syscall interface.
I was wondering whether to piggy-back on the audit code was the best
idea here..
On Wed, 13 Aug 2008 12:36:15 -0400
Eric Paris <[email protected]> wrote:
[finally good description snipped]
>
> 1) Kernel or userspace.
this is kind of the last question to ask, not the first ;)
> Not to mention there cannot be any checks for files
> opened by suid apps in userspace.
setuid apps do not use glibc? news to me.
> And while no specific claims are
> being made about intentionally malicious code, putting this in kernel
> makes programs which call syscalls directly much more difficult to be
> used to circumvent the scanning.
[so lets ignore this one]
> 2) I think the "best" time to do scanning is at read and write. Any
> disagreements there?
I agree about the "read" case.
"write" is tricky... because there's many vectors to write, from
write() system call to mmap() system call to truncate() system call.
(yes you can write a stream of bytes that passes virus scan, and then
truncate to be suddenly a live virus)
I would propose to use the word "dirty" instead, we all know what we
mean by that (it's all of them) and doesn't tie our thinking to the
write() system call.
I would like to introduce a concept in your discussion you did not
mention yet in this email: the difference between synchronous scanning
and asynchronous scanning.
It's clear from the protection model that you described that on 'read'
you want to wait until the scan is done before you give the data to the
process asking for it... and that's totally reasonable: "Do not give
out bad data" is a very clear line in terms of security.
for the "dirty" case it gets muddy. You clearly want to scan "some
time" after the write, from the principle of getting rid of malware
that's on the disk, but it's unclear if this HAS to be synchronous.
(obviously, synchronous behavior hurts performance bigtime so lets do
as little as we can of that without hurting the protection).
One advantage of doing the dirty case async (and a little time delayed)
is that repeated writes will get lumped up into one scan in practice,
saving a ton of performance.
(scan-on-close is just another way of implementing "delay the dirty
scan").
Based on Alans comments, to me this sounds like we should have an
efficient mechanism to notify userspace of "dirty events"; this is not
virus scan specific in any way or form. And this mechanism likely will
need to allow multiple subscribers.
for the open() case, I would argue that you don't need synchronous
behavior as long as the read() case is synchronous. I can imagine that
open() kicks off an async scan, and if it's done by the time the first
read() happens, no blocking at all happens.
For efficiency the kernel ought to keep track of which files have been
declared clean, and it needs to track of a 'generation' of the scan
with which it has been found clean (so that if you update your virus
definitions, you can invalidate all previous scanning just by bumping
the 'generation' number in whatever format we use).
Clearly the kernel needs to wipe this clean generation number on any
modification to the file (inode) of any kind. And clearly this needs to
be done inside the kernel because tracking dirty operations in any
other way is just insane.
Now this to me we have a few basic building blocks:
1) We need an efficient mechanism to notify userspace of files that get
dirtied. Virus scanners will subscribe to this for the async dirty
scanning; indexing agents also will subscribe to this.
2) We very likely should have a mechanism for a userspace app to
request a scan on a file, both sync or async (O_SYNC flag?). This is
useful regardless because it allows the source of many things to do the
right thing.
3) we need a mechanism in the kernel to track "scanned with generation
X of signatures" that invalidates on any dirty operation. The syscall
from 2) will use this as a cache to be quick.
I think few people will disagree about this.
Open questions now are
4) do we have the kernel kick off an async scan in open() or do we have
glibc do this
5) do we have the kernel do the sync scan on read/mmap/.. or do we have
glibc do this
I think this is where the whole debate is about now.
And a few hard ones
6) how do we deal with multiple scanning agents in parallel
7) how do we prevent malware from pretending to be a virus scanner
--
If you want to reach me at my work email, use [email protected]
For development, discussion and tips for power savings,
visit http://www.lesswatts.org
On Wed, Aug 13, 2008 at 12:36:15PM -0400, Eric Paris wrote:
I miss a clear answer to the question: is this
supposed to protect against malware injected as root or not?
Assuming it wants to protect against root:
> think we hear claim more grandious things. But from what I've seen they
> aren't the real deal. Most of the security model descriptions that
> people on list want actually are framed under the belief that these
> products need to or attempt to completely block some class of attacks.
> They don't. If you think they do you need to fix your frame of
> reference. The only class of attacks this interface is supposed to
> address is those that can be found by scanning files. This is NOT an
> LSM.
But you need some LSM like protections to be able to protect the file
scanner? Like the block device or kernel memory protection.
> The real goal is to get files into to some userspace scanner and let
> them do whatever they want. Remember, this isn't a new LSM. The goal
> isn't to provide perfect security. The goal isn't to stop already
> running malicious programs. The one and only goal is to scan files. We
> should not be considering timing attacks, we should not be considering
> processes actively trying to get around the system for small periods of
> time. We should certainly not be considering root processes being able
> to sneak things by.
This means you need significant LSM components simply to protect
the integrity of the file scanner against root. It's even
unclear it's possible in the general case (e.g. X server doing
arbitary DMA and no IOMMU -- how do you protect the file scanner?)
> The idea is that a file exists on disk and we want
> some userspace program to give a best effort at scanning it. Yes, we
Ok so you're implying it's ok to not protect against root?
In the later case that means that you don't have to scan anything
that only root can touch and you can trust file permissions,
which makes a lot of things easier.
I would suggest again to clarify this important point first. It has
significant impact on the whole design.
Personally I would think not protecting against root would be quite
limiting (e.g. it would mean that e.g. if some worm trojans rpms
people download then they wouldn't be caught because rpms are
installed as root), but on the other hand if you protect against
root you need most of selinux/aa/other lsm functionality simply
to guarantee the integrity of the scanner. Also it has impact
on some apps, e.g. X server running as root would be usually out due to
the arbitary DMA issue. Also protect block devices could theoretically
have significant impact on some sysadmin tasks.
-Andi
On Wed, Aug 13, 2008 at 10:39:51AM -0700, Arjan van de Ven wrote:
> for the "dirty" case it gets muddy. You clearly want to scan "some
> time" after the write, from the principle of getting rid of malware
> that's on the disk, but it's unclear if this HAS to be synchronous.
> (obviously, synchronous behavior hurts performance bigtime so lets do
> as little as we can of that without hurting the protection).
Something else to think about is what happens if the file is naturally
written in pieces. For example, I've been playing with bittorrent
recently, and it appears that trackerd will do something... not very
intelligent in that it will repeatedly try to index a file which is
being written in pieces, and in some cases, it will do things like
call pdftext that aren't exactly cheap. A timeout *can* help (i.e.,
don't try to scan/index this file until 15 minutes after the last
write), but it won't help if the torrent is very large, or the
download bitrate is very slow. One very simple workaround is to
disable trackerd altogether while you are downloading the file, but
that's not very pleasant solution; it's horribly manual.
Most of this may end up being outside of the kernel (i.e.,some kind of
interface where a bittorrent client can say, "look this file is still
being downloaded, so it's don't bother scanning it unless some process
*other* than the bittorrent client tries to access the file". And
maybe there should be some other more complex policies, such as the
bittorrent client explicitly telling the indexer/scanner that the file
is has been completely downloaded, so it's safe to index it now.
But what this points out is that if you want a good solution, (a) it
probably shouldn't all be in the kernel, since trying to get all of
this complexity into the kernel will be painful, and (b) the policy
about whether or not a bittorrent client should be allowed to say,
"it's OK not to check the file until it's completely downloaded, even
if I am handing out pieces to other people over the network --- after
all the entire file has its own SHA checksum for data integrity
verification --- is very much a policy question where different system
administrators will come down on different sides about what should and
shouldn't be allowed --- and therefore this kind of policy decision
should ****NOT**** be in the kernel.
> For efficiency the kernel ought to keep track of which files have been
> declared clean, and it needs to track of a 'generation' of the scan
> with which it has been found clean (so that if you update your virus
> definitions, you can invalidate all previous scanning just by bumping
> the 'generation' number in whatever format we use).
We have an i_version support for NFSv4, so we have that already as far
as the version of the file. We can have a single bit which means
"block on open" that is stored on a file, and some kind of policy
which dictates whether or not any modification to the file contens
should automatically set the bit.
However, questions of which version of virus database was used to scan
a particular file should be stored outside of the filesystem, since
each product will have its own version namespace, and the questions of
what happens if a user switches from one version checker to another is
going to be messy. So better that this be done in userspace, and that
this information be stored in some on-disk database.
- Ted
On Wed, 13 Aug 2008 14:15:49 -0400
Theodore Tso <[email protected]> wrote:
> On Wed, Aug 13, 2008 at 10:39:51AM -0700, Arjan van de Ven wrote:
> > for the "dirty" case it gets muddy. You clearly want to scan "some
> > time" after the write, from the principle of getting rid of malware
> > that's on the disk, but it's unclear if this HAS to be synchronous.
> > (obviously, synchronous behavior hurts performance bigtime so lets
> > do as little as we can of that without hurting the protection).
>
> Something else to think about is what happens if the file is naturally
> written in pieces. For example, I've been playing with bittorrent
> recently, and it appears that trackerd will do something... not very
> intelligent in that it will repeatedly try to index a file which is
> being written in pieces, and in some cases, it will do things like
> call pdftext that aren't exactly cheap. A timeout *can* help (i.e.,
> don't try to scan/index this file until 15 minutes after the last
> write), but it won't help if the torrent is very large, or the
> download bitrate is very slow. One very simple workaround is to
> disable trackerd altogether while you are downloading the file, but
> that's not very pleasant solution; it's horribly manual.
>
> Most of this may end up being outside of the kernel (i.e.,some kind of
> interface where a bittorrent client can say, "look this file is still
> being downloaded, so it's don't bother scanning it unless some process
> *other* than the bittorrent client tries to access the file". And
> maybe there should be some other more complex policies, such as the
> bittorrent client explicitly telling the indexer/scanner that the file
> is has been completely downloaded, so it's safe to index it now.
>
> verification --- is very much a policy question where different system
> administrators will come down on different sides about what should and
> shouldn't be allowed --- and therefore this kind of policy decision
> should ****NOT**** be in the kernel.
exactly. Even more, since this is async work, the scheduling of the
order of work also is a policy.. and userland is again the right place
for that.
>
> > For efficiency the kernel ought to keep track of which files have
> > been declared clean, and it needs to track of a 'generation' of the
> > scan with which it has been found clean (so that if you update your
> > virus definitions, you can invalidate all previous scanning just by
> > bumping the 'generation' number in whatever format we use).
>
> We have an i_version support for NFSv4, so we have that already as far
> as the version of the file. We can have a single bit which means
> "block on open" that is stored on a file, and some kind of policy
> which dictates whether or not any modification to the file contens
> should automatically set the bit.
>
> However, questions of which version of virus database was used to scan
> a particular file should be stored outside of the filesystem, since
well I was assuming we only store this in memory (say in the inode) and
just rescan the file if we destroy the in memory inode.
I don't see the need for this to be persistent data; in fact I assume
(Eric, please confirm) that this data is not *supposed* to be
persistent.
> each product will have its own version namespace, and the questions of
> what happens if a user switches from one version checker to another is
yes that's a hard question; what if you have 2 virus scanners active.
(they could register a version of the database with the kernel, and the
in kernel version-cookie could be a hash of all registered versions I
suppose.. if anything changes ever we just rehash and scan as if we
have to do that)
--
If you want to reach me at my work email, use [email protected]
For development, discussion and tips for power savings,
visit http://www.lesswatts.org
On Wed, 13 Aug 2008 20:17:14 +0200
Andi Kleen <[email protected]> wrote:
>
> I would suggest again to clarify this important point first. It has
> significant impact on the whole design.
agreed.
>
> Personally I would think not protecting against root would be quite
> limiting (e.g. it would mean that e.g. if some worm trojans rpms
> people download then they wouldn't be caught because rpms are
> installed as root),
on argument could be that root apps like that could/should do explicit
scanning regardless.
(if we have an explicit interface to scan a file that's not too hard)
--
If you want to reach me at my work email, use [email protected]
For development, discussion and tips for power savings,
visit http://www.lesswatts.org
Andi Kleen wrote:
>
> This means you need significant LSM components simply to protect
> the integrity of the file scanner against root. It's even
> unclear it's possible in the general case (e.g. X server doing
> arbitary DMA and no IOMMU -- how do you protect the file scanner?)
>
Without Treacherous Computing, it isn't possible, even in kernel space.
In Treacherous Computing you can put it in the hypervisor, which of
course just means the hypervisor is now much bigger and likely to
contain security holes.
-hpa
On Wed, 2008-08-13 at 10:39 -0700, Arjan van de Ven wrote:
> On Wed, 13 Aug 2008 12:36:15 -0400
> Eric Paris <[email protected]> wrote:
>
> [finally good description snipped]
>
> >
> > 1) Kernel or userspace.
>
> this is kind of the last question to ask, not the first ;)
>
> > Not to mention there cannot be any checks for files
> > opened by suid apps in userspace.
>
> setuid apps do not use glibc? news to me.
I was thinking about LD_PRELOAD. you're right for most things it could
be glibc. I still have knfsd on my side though.
> > And while no specific claims are
> > being made about intentionally malicious code, putting this in kernel
> > makes programs which call syscalls directly much more difficult to be
> > used to circumvent the scanning.
>
> [so lets ignore this one]
/me doesn't want to ignore things that help his argument, even if they
are weak. :)
> > 2) I think the "best" time to do scanning is at read and write. Any
> > disagreements there?
>
> I agree about the "read" case.
> "write" is tricky... because there's many vectors to write, from
> write() system call to mmap() system call to truncate() system call.
> (yes you can write a stream of bytes that passes virus scan, and then
> truncate to be suddenly a live virus)
>
> I would propose to use the word "dirty" instead, we all know what we
> mean by that (it's all of them) and doesn't tie our thinking to the
> write() system call.
>
> I would like to introduce a concept in your discussion you did not
> mention yet in this email: the difference between synchronous scanning
> and asynchronous scanning.
I already have that concept! my open time scanning is sync and can
return an EACCESS. my close time scanning is async and only updates the
in inode clean/dirty mark. It makes no access decisions. That way on
the next open we are good to go! And yes yes, mmap write after close
invalides the async clean mark.
> It's clear from the protection model that you described that on 'read'
> you want to wait until the scan is done before you give the data to the
> process asking for it... and that's totally reasonable: "Do not give
> out bad data" is a very clear line in terms of security.
>
> for the "dirty" case it gets muddy. You clearly want to scan "some
> time" after the write, from the principle of getting rid of malware
> that's on the disk, but it's unclear if this HAS to be synchronous.
> (obviously, synchronous behavior hurts performance bigtime so lets do
> as little as we can of that without hurting the protection).
> One advantage of doing the dirty case async (and a little time delayed)
> is that repeated writes will get lumped up into one scan in practice,
> saving a ton of performance.
> (scan-on-close is just another way of implementing "delay the dirty
> scan").
> Based on Alans comments, to me this sounds like we should have an
> efficient mechanism to notify userspace of "dirty events"; this is not
> virus scan specific in any way or form. And this mechanism likely will
> need to allow multiple subscribers.
I'm certainly willing to go down the inotify'ish path for async
notification of 'dirty' inodes instead of implement my own async
mechanism if I can find a way to do it.
> for the open() case, I would argue that you don't need synchronous
> behavior as long as the read() case is synchronous. I can imagine that
> open() kicks off an async scan, and if it's done by the time the first
> read() happens, no blocking at all happens.
An interesting addition. Trying to keep these queues of events gets
much more complex, but if people really think the open to read race is
that important I've always said it wasn't impossible to close.
> For efficiency the kernel ought to keep track of which files have been
> declared clean, and it needs to track of a 'generation' of the scan
> with which it has been found clean (so that if you update your virus
> definitions, you can invalidate all previous scanning just by bumping
> the 'generation' number in whatever format we use).
> Clearly the kernel needs to wipe this clean generation number on any
> modification to the file (inode) of any kind. And clearly this needs to
> be done inside the kernel because tracking dirty operations in any
> other way is just insane.
I agree! i_talpa_seq_no I added to struct inode is exactly that!
> Now this to me we have a few basic building blocks:
> 1) We need an efficient mechanism to notify userspace of files that get
> dirtied. Virus scanners will subscribe to this for the async dirty
> scanning; indexing agents also will subscribe to this.
> 2) We very likely should have a mechanism for a userspace app to
> request a scan on a file, both sync or async (O_SYNC flag?). This is
> useful regardless because it allows the source of many things to do the
> right thing.
This means what exactly? Some new syscall? You want some program to
say "notify userspace about this inode" and then every userspace thing
that wants the notification gets it? Not sure how much brains I want to
build into generic userspace apps. I certainly wouldn't want to make
any userspace app easily be able to avoid AV scanning even if I make no
promises about malicious apps. Just because I don't make gurantees
doesn't mean I should make it easy. Just because I can't be 100% that a
fence won't keep the kids from falling in the pool doesn't mean I
shouldn't build it. Differentiating between don't AV scan and don't
index is going to be hard in a single interface, but not unsolvable.
> 3) we need a mechanism in the kernel to track "scanned with generation
> X of signatures" that invalidates on any dirty operation. The syscall
> from 2) will use this as a cache to be quick.
i_talpa_seq_no
> I think few people will disagree about this.
>
> Open questions now are
> 4) do we have the kernel kick off an async scan in open() or do we have
> glibc do this
> 5) do we have the kernel do the sync scan on read/mmap/.. or do we have
> glibc do this
scan on mmap read? How do I implement this? Do we update atime on mmap
read? can I do it there like I do for writes with mtime?
> I think this is where the whole debate is about now.
>
>
> And a few hard ones
> 6) how do we deal with multiple scanning agents in parallel
My first suggestion is we don't, and worry about it as soon as we have
the first parallel users.
> 7) how do we prevent malware from pretending to be a virus scanner
We don't. Let the LSM do its job?
On Wed, 2008-08-13 at 20:17 +0200, Andi Kleen wrote:
> On Wed, Aug 13, 2008 at 12:36:15PM -0400, Eric Paris wrote:
>
> I miss a clear answer to the question: is this
> supposed to protect against malware injected as root or not?
I thought I had summed that up. We are not interested in providing
protections against maliciously attacking programs be they root or not.
We are interested in scanning files read and written by root. We are
especially interested in programs run by root.
yum (as root) downloads trojan.rpm from youareanidiot.repo. We aren't
worried about yum maliciously attacking the system. What's going to
happen is that the scanner is going to scan the trojan.rpm when yum
calls rpm and rpm is going to be denied access to that file. Doesn't
matter how you download it, its going to get scanned when you try to
exec it.
Stop thinking this is an LSM or as a new security model. It's a file
scanner and "it ain't perfect security." But its very useful and
practical. If some malicious root application wants to turn it off it
can and I make no claims otherwise. This isn't supposed to help once
root has been subverted, you've already lost as you admit, its supposed
to help keep root form getting subverted.
-Eric
>
> > think we hear claim more grandious things. But from what I've seen they
> > aren't the real deal. Most of the security model descriptions that
> > people on list want actually are framed under the belief that these
> > products need to or attempt to completely block some class of attacks.
> > They don't. If you think they do you need to fix your frame of
> > reference. The only class of attacks this interface is supposed to
> > address is those that can be found by scanning files. This is NOT an
> > LSM.
>
> But you need some LSM like protections to be able to protect the file
> scanner? Like the block device or kernel memory protection.
>
> > The real goal is to get files into to some userspace scanner and let
> > them do whatever they want. Remember, this isn't a new LSM. The goal
> > isn't to provide perfect security. The goal isn't to stop already
> > running malicious programs. The one and only goal is to scan files. We
> > should not be considering timing attacks, we should not be considering
> > processes actively trying to get around the system for small periods of
> > time. We should certainly not be considering root processes being able
> > to sneak things by.
>
> This means you need significant LSM components simply to protect
> the integrity of the file scanner against root. It's even
> unclear it's possible in the general case (e.g. X server doing
> arbitary DMA and no IOMMU -- how do you protect the file scanner?)
>
> > The idea is that a file exists on disk and we want
> > some userspace program to give a best effort at scanning it. Yes, we
>
> Ok so you're implying it's ok to not protect against root?
>
> In the later case that means that you don't have to scan anything
> that only root can touch and you can trust file permissions,
> which makes a lot of things easier.
>
> I would suggest again to clarify this important point first. It has
> significant impact on the whole design.
>
> Personally I would think not protecting against root would be quite
> limiting (e.g. it would mean that e.g. if some worm trojans rpms
> people download then they wouldn't be caught because rpms are
> installed as root), but on the other hand if you protect against
> root you need most of selinux/aa/other lsm functionality simply
> to guarantee the integrity of the scanner. Also it has impact
> on some apps, e.g. X server running as root would be usually out due to
> the arbitary DMA issue. Also protect block devices could theoretically
> have significant impact on some sysadmin tasks.
>
> -Andi
On Wed, 2008-08-13 at 14:15 -0400, Theodore Tso wrote:
> On Wed, Aug 13, 2008 at 10:39:51AM -0700, Arjan van de Ven wrote:
> > for the "dirty" case it gets muddy. You clearly want to scan "some
> > time" after the write, from the principle of getting rid of malware
> > that's on the disk, but it's unclear if this HAS to be synchronous.
> > (obviously, synchronous behavior hurts performance bigtime so lets do
> > as little as we can of that without hurting the protection).
>
> Something else to think about is what happens if the file is naturally
> written in pieces. For example, I've been playing with bittorrent
> recently, and it appears that trackerd will do something... not very
> intelligent in that it will repeatedly try to index a file which is
> being written in pieces, and in some cases, it will do things like
> call pdftext that aren't exactly cheap. A timeout *can* help (i.e.,
> don't try to scan/index this file until 15 minutes after the last
> write), but it won't help if the torrent is very large, or the
> download bitrate is very slow. One very simple workaround is to
> disable trackerd altogether while you are downloading the file, but
> that's not very pleasant solution; it's horribly manual.
>
> Most of this may end up being outside of the kernel (i.e.,some kind of
> interface where a bittorrent client can say, "look this file is still
> being downloaded, so it's don't bother scanning it unless some process
> *other* than the bittorrent client tries to access the file". And
> maybe there should be some other more complex policies, such as the
> bittorrent client explicitly telling the indexer/scanner that the file
> is has been completely downloaded, so it's safe to index it now.
>
> But what this points out is that if you want a good solution, (a) it
> probably shouldn't all be in the kernel, since trying to get all of
> this complexity into the kernel will be painful, and (b) the policy
> about whether or not a bittorrent client should be allowed to say,
> "it's OK not to check the file until it's completely downloaded, even
> if I am handing out pieces to other people over the network --- after
> all the entire file has its own SHA checksum for data integrity
> verification --- is very much a policy question where different system
> administrators will come down on different sides about what should and
> shouldn't be allowed --- and therefore this kind of policy decision
> should ****NOT**** be in the kernel.
I never suggested putting a scanner in kernel. Sound like you want the
"allow don't cache" response from your userspace scanner while this is
going on. The kernel doesn't need to be making decisions about when to
send events, nor should userspace tell the kernel not to send events.
Its up to whatever the scanner is to agree not to actually do any
scanning...
> However, questions of which version of virus database was used to scan
> a particular file should be stored outside of the filesystem, since
> each product will have its own version namespace, and the questions of
> what happens if a user switches from one version checker to another is
> going to be messy. So better that this be done in userspace, and that
> this information be stored in some on-disk database.
No. How in the heck can some out of kernel database store information
about what inodes have been scanned in any even slightly sane way? And
people think the race between open and read is too large and you suggest
moving clean/dirty marking to a userspace database? I MUCH prefer my
(and it sounds like arjan agrees) clean/dirty versioned flag in inode.
-Eris
On Wed, Aug 13, 2008 at 03:02:48PM -0400, Eric Paris wrote:
> I never suggested putting a scanner in kernel. Sound like you want the
> "allow don't cache" response from your userspace scanner while this is
> going on. The kernel doesn't need to be making decisions about when to
> send events, nor should userspace tell the kernel not to send events.
> Its up to whatever the scanner is to agree not to actually do any
> scanning...
And if the system isn't running a virus checker, but just a file
indexer (ala tracker), it shouldn't go to userspace at all. In that
case all that is necessary is an asynchronous notification.
Also something else that is needed is support for multiple clients.
(i.e., what happens if the user runs two virus checkers, or a virus
checker plus a hierarchical storage manager driving a tape robot, or
all of the above plus trackerd --- where some clients need to block
open(2) access, and some do not need block open(2) --- and in the case
of HSM, ordering becomes important; you want to retrieve the file from
the tape robot first, *then* scan it using the virus checker. :-)
> No. How in the heck can some out of kernel database store information
> about what inodes have been scanned in any even slightly sane way? And
> people think the race between open and read is too large and you suggest
> moving clean/dirty marking to a userspace database? I MUCH prefer my
> (and it sounds like arjan agrees) clean/dirty versioned flag in inode.
Don't ask me; I think most AV checkers for linux are security theater
and not very much use (other than making money for the AV company's
shareholders) anyway. I thought you were the one who wanted to record
information about which version of the virus db a particular file had
been scanned against. The place where I can see this being useful is
what happens you get a new virus DB, and so you need to start scanning
all of the files in your 5TB enterprise file server --- and then the
system crashes or it needs to be taken down for scheduled maintenance.
You want to have *some* off-line database for storing this
information, since it would be silly to want to have the first thing
that happens after a new virus DB gets downloaded is to interate over
the entire filesystem, clearing a persistent the "clean" bit --- that
would take *forever* on a 5TB filerserver; and what happens if you
crash in the middle of clearing the "clean" bit.. And if the system
gets shutdown in the middle of the scan, you need some way of
remembering which inodes have been scanned using the "new" db, and
which ones haven't yet been scanned via the new virus db. All of this
should be kept in userpsace, and is strictly speaking Not Our Problem.
I'm just arguing that there should be absolutely *no* support in the
kernel for solving this particular problem, since the question of
whether a file has been scanned with a particular version of the virus
DB is purely a userspace problem.
- Ted
> > I don't think you need to be blocking if you passed up a file handle ?
>
> Without blocking and waiting how do you deny access? Maybe I needed
> another thing they do. "They do file scanning and deny access to bad
> files."
Denying access is easy enough - chmod it or set an SELinux label on it.
There are a limited number of useful arrangements I can see here
'opened for write deny [some set of] access until verified'
Thats an selinux state transition on open for write, and one from the
user space end on the other
'run around after things'
Which is simply a notifier and a relabel as 'contaminated' or similar (or
a chmod or mv ...) by the daemon end of things.
Now I can see why you might want a 'has been open for a write and dirty
for a while' notifier - again for indexing as well as virus scanners and
the like.
What other semantic do you have in mind given that you have to deal with
open by me, open by writer, content modified.. after I have it open
anyway ? It seems the rest is a discussion about time intervals ?
> -----Original Message-----
> From: [email protected] [mailto:malware-list-
> [email protected]] On Behalf Of Theodore Tso
> Sent: Wednesday, August 13, 2008 3:29 PM
> To: Eric Paris
> Cc: [email protected]; [email protected]; malware-
> [email protected]; [email protected]; [email protected];
> [email protected]; [email protected]; Arjan van de Ven
> Subject: Re: [malware-list] TALPA - a threat model? well sorta.
>
> Also something else that is needed is support for multiple clients.
> (i.e., what happens if the user runs two virus checkers, or a virus
> checker plus a hierarchical storage manager driving a tape robot, or
> all of the above plus trackerd --- where some clients need to block
> open(2) access, and some do not need block open(2) --- and in the case
> of HSM, ordering becomes important; you want to retrieve the file from
> the tape robot first, *then* scan it using the virus checker. :-)
The issue of multiple clients does need to be accounted for. However, I
will mention that it is unusual (at least in my experience) to actually
run two AV products at the same time in "realtime" mode. We strongly
recommend that anyone who installs our product should remove any other
AV product on the system -- for technical reasons, not financial --
since they've already paid for ours by the time they get to this point.
I am not aware of anyone objecting to that idea.
> I'm just arguing that there should be absolutely *no* support in the
> kernel for solving this particular problem, since the question of
> whether a file has been scanned with a particular version of the virus
> DB is purely a userspace problem.
Caching of previous results can be done in either user space or kernel
space. We have seen it both ways. Wherever it is done, I would say
that rather than record AV signature file version numbers, there should
be a mechanism for the application to invalidate or flush the cache
whenever a signature update is done. There are other circumstances
where that would also be useful -- such as if the user changes a
scanning option in a way that increases the strictness of the scan. In
other words, a file that was marked as clean based on one level of
strictness may not be clean under a stricter scan. You wouldn't want
the cache to prevent it from being scanned under such circumstances.
Jon Press
> -----Original Message-----
> From: [email protected] [mailto:malware-list-
> [email protected]] On Behalf Of Alan Cox
> Sent: Wednesday, August 13, 2008 3:59 PM
> To: Eric Paris
> Cc: [email protected]; [email protected]; malware-
> [email protected]; [email protected]; [email protected];
> [email protected]; [email protected]
> Subject: Re: [malware-list] TALPA - a threat model? well sorta.
>
> > > I don't think you need to be blocking if you passed up a file
handle ?
> >
> > Without blocking and waiting how do you deny access? Maybe I needed
> > another thing they do. "They do file scanning and deny access to
bad
> > files."
>
> Denying access is easy enough - chmod it or set an SELinux label on
it.
I may be missing something about your suggestion, but I don't see how
this would work. Who does the chmod?
Here's a sequence:
- Application opens file
- AV scanner notified in some way without blocking
- Application reads file into memory
- AV scanner determines file is infected.
- AV scanner chmod's file -- oops, too late.
- Application sends file over the wire to another machine with a more
vulnerable OS
How would this be prevented?
Jon Press
> I may be missing something about your suggestion, but I don't see how
> this would work. Who does the chmod?
>
> Here's a sequence:
>
> - Application opens file
> - AV scanner notified in some way without blocking
> - Application reads file into memory
The discussion has been about scanning on write.
> - AV scanner determines file is infected.
> - AV scanner chmod's file -- oops, too late.
> - Application sends file over the wire to another machine with a more
> vulnerable OS
>
> How would this be prevented?
I don't think you can. In your case how does your AV scanner deal with
the case where the application opens the file while another user has it
open and the other user (or even other task with the same handle) changes
the content possibly via mmap.
Content may also directly be shared between users of a file using the
mmap interfaces so your scan on read model is rather dysfunctional.
On Wed, 13 Aug 2008 17:24:28 -0400
"Press, Jonathan" <[email protected]> wrote:
> I may be missing something about your suggestion, but I don't see how
> this would work. Who does the chmod?
Chmod is also not a solution to the hierarchical storage (or incremental
restore from backup) problem.
I believe we really do need the block-on-open.
--
All Rights Reversed
On Wed, 13 Aug 2008 14:57:44 -0400
Eric Paris <[email protected]> wrote:
> > for the open() case, I would argue that you don't need synchronous
> > behavior as long as the read() case is synchronous. I can imagine
> > that open() kicks off an async scan, and if it's done by the time
> > the first read() happens, no blocking at all happens.
>
> An interesting addition. Trying to keep these queues of events gets
> much more complex, but if people really think the open to read race is
> that important I've always said it wasn't impossible to close.
it's not "just" about open-to-read race.
it's about open being non-blocking, and if read is not immediate, never
hitting the latency at all.
The real point is that "read" is the actual point you want to
intercept, not "open" (you even wrote that in your description).. so
why not just do that ?
The open case then is just a performance optimization.
> > Open questions now are
> > 4) do we have the kernel kick off an async scan in open() or do we
> > have glibc do this
> > 5) do we have the kernel do the sync scan on read/mmap/.. or do we
> > have glibc do this
>
> scan on mmap read? How do I implement this?
on calling mmap(); not at fault time.
--
If you want to reach me at my work email, use [email protected]
For development, discussion and tips for power savings,
visit http://www.lesswatts.org
On Wed, 13 Aug 2008 17:35:29 -0400
Rik van Riel <[email protected]> wrote:
> On Wed, 13 Aug 2008 17:24:28 -0400
> "Press, Jonathan" <[email protected]> wrote:
>
> > I may be missing something about your suggestion, but I don't see how
> > this would work. Who does the chmod?
>
> Chmod is also not a solution to the hierarchical storage (or incremental
> restore from backup) problem.
>
> I believe we really do need the block-on-open.
The block on open is orthogonal really. Useful for HSM, useful for
certain very primitive scanning but not much else that I can see.
And its a minor mod to the security hooks to allow it as far as I can see
On Wednesday 13 August 2008, Andi Kleen wrote:
> On Wed, Aug 13, 2008 at 12:36:15PM -0400, Eric Paris wrote:
>
> I miss a clear answer to the question: is this
> supposed to protect against malware injected as root or not?
I honestly don't think we should worry about root. Sure, if the AV scanner
happens to catch something (as a consequence of it's implementation), then
very well. But designing an antimalware solution which assumes the root is
compromised will throw us into security talks for years and I don't think
we'll live to hear the end of them.
We should focus on the regular users and fix (if needed) the current userland
apps (ie. the ones that need root access to do their job). For anymore than
that we'll need a super user that supervises root. And then another one.
> Assuming it wants to protect against root:
> > think we hear claim more grandious things. But from what I've seen they
> > aren't the real deal. Most of the security model descriptions that
> > people on list want actually are framed under the belief that these
> > products need to or attempt to completely block some class of attacks.
> > They don't. If you think they do you need to fix your frame of
> > reference. The only class of attacks this interface is supposed to
> > address is those that can be found by scanning files. This is NOT an
> > LSM.
>
> But you need some LSM like protections to be able to protect the file
> scanner? Like the block device or kernel memory protection.
>
> > The real goal is to get files into to some userspace scanner and let
> > them do whatever they want. Remember, this isn't a new LSM. The goal
> > isn't to provide perfect security. The goal isn't to stop already
> > running malicious programs. The one and only goal is to scan files. We
> > should not be considering timing attacks, we should not be considering
> > processes actively trying to get around the system for small periods of
> > time. We should certainly not be considering root processes being able
> > to sneak things by.
>
> This means you need significant LSM components simply to protect
> the integrity of the file scanner against root. It's even
> unclear it's possible in the general case (e.g. X server doing
> arbitary DMA and no IOMMU -- how do you protect the file scanner?)
>
> > The idea is that a file exists on disk and we want
> > some userspace program to give a best effort at scanning it. Yes, we
>
> Ok so you're implying it's ok to not protect against root?
>
> In the later case that means that you don't have to scan anything
> that only root can touch and you can trust file permissions,
> which makes a lot of things easier.
>
> I would suggest again to clarify this important point first. It has
> significant impact on the whole design.
>
> Personally I would think not protecting against root would be quite
> limiting (e.g. it would mean that e.g. if some worm trojans rpms
> people download then they wouldn't be caught because rpms are
> installed as root)
If GPG signatures don't work, then please fix the rpm design and if the user
willingly installs a .rpm which is not signed (not from a known trusted host)
and somehow doges the basic antimalware scanner, then too bad. We've done all
we could.
> , but on the other hand if you protect against
> root you need most of selinux/aa/other lsm functionality simply
> to guarantee the integrity of the scanner. Also it has impact
> on some apps, e.g. X server running as root would be usually out due to
> the arbitary DMA issue. Also protect block devices could theoretically
> have significant impact on some sysadmin tasks.
I think we need to define the 'desktop user' and provide a decent protection
mechanism for his common activities (edit documents, listen music, navigate
the web, see movies, run scripts which change the IM status etc). For the
rest, there are two possibilities:
1. education (_extremely_ important);
2. SELinux (or similar);
I don't think there will ever be an AV product using the marketing line: "it
allows you to run your favorite rootkit and enjoy the pretty text it shows,
with no worries".
In conclusion: everything AV related should stop at the user root. Popular
distro-s already provide a way to do your daily office tasks without super
user rights, which _is_ the correct thing to do.
--
Mihai Donțu
(this was posted in linux.kernel, before I realized there was a
linux.kernel.malware. Hope it helps your discussion)
(FYI. Dazuko may have trailblazed some of the issues now under
discussion re: libmalware.so. It has worked well for me. It used to be
an LKM, it is now a source patch. It is used in a number of commercial
products)
<http://dazuko.dnsalias.org/wiki/index.php/Main_Page>
"A Virtual Device Driver to Allow Online File Access Control
A common interface is needed, which allows userland applications to
perform online file access control. Dazuko aims to provide that interface."
FWIW, I'm not associated with Dazuko or Antivir; I've been happily using
Dazuko with AntiVir for a year or so.
1. AntiVir includes numerous Linux signatures as well as Windows. So I
scan both 'ix downloads, as well as the process of compiling new software.
2. Other AntiMalwares are using Dazuko, though many are scanning for
Windows malware only.
3. The AntiVir/Dazuko combination with full heuristics has blocked
access to clearly dangerous JS scripts in my browser cache.
4. IMHO, what is needed is a Dazuko or libmalware/Integrity database
link. If an md5 of an executable or script is new or has changed, access
is blocked 'til a response to a popup is given. Access can be blocked;
one-time allowed; or permanently allowed, in which case the md5 is updated.
Hope This Helps.
<next msg>
Andi Kleen wrote:
> 7v5w7go9ub0o <[email protected]> writes:
>
>> (FYI. Dazuko may have trailblazed some of the issues now under
>> discussion re: libmalware.so. It has worked well for me.
>
> Against what exactly did it protect you? Please give a concrete example.
>
> -Andi
>
1. This came in a few minutes ago:
Aug 13 14:56:31 tux antivir[6381]: AntiVir ALERT: [EML/FakeLink.F]
/jail/tbird/root/.thunderbird/0r2957kg.default/Mail/L
ocal Folders/Junk.XXX <<< Contains detection pattern of EML/FakeLink.F
in EML form
2. I have not retained the logs of "suspicious scripts" in my browser,
but have come across perhaps 4 blocked scripts within the last month.
Admittedly at dodgy sites.
XSS attacks are platform independent, and are a significant concern.
Please note that when I say it has worked well for me, I am not saying
that it has saved my bacon! :-)
1. I am referring to the mechanics of having the Kernel/userland app
stop processing when it finds a malware signature or heuristic detection.
2. Am also referring to the totally manageable (IMHO) overhead.
I've mentioned my experience with Dazuko/antivir only because it may be
useful to the ongoing discussion about the nature of libmalware.so.
3. I am frankly waiting for a bug to get into my upstream distribution
chain - through a hijacking or some wonderful DNS prank - at which point
I ..hope.. a signature or heuristic will block my root-enabled make install.
4. Again, my hope for libmalware.so/dazuko is a realtime
integrity-management link.
<end posts>
HTH
p.s. The question has developed, should this monitor root activities.
IMHO, the answer is a definite YES! We are most vulnerable during
software updating; AntiMailware signatures may stop the compilation or
installation of a Trojan - by root.
7v5w7go9ub0o wrote:
>
> 4. Again, my hope for libmalware.so/dazuko is a realtime
> integrity-management link.
>
> <end posts>
>
> HTH
>
> p.s. The question has developed, should this monitor root activities.
> IMHO, the answer is a definite YES! We are most vulnerable during
> software updating; AntiMailware signatures may stop the compilation or
> installation of a Trojan - by root.
>
I just noticed a separate discussion about integrity-checking LKMs and LSMs.
Obviously, a libmalware.so or Dazuko based integrity-checker would block
a kernel from loading in a Trojaned LKM - noting that the MD5 had
changed, and asking you to block, temporarily allow, or permanently
allow the changed module.
Another security benefit of your pursuit.
HTH
Arjan van de Ven wrote on 13/08/2008 19:21:49:
> > However, questions of which version of virus database was used to scan
> > a particular file should be stored outside of the filesystem, since
>
> well I was assuming we only store this in memory (say in the inode) and
> just rescan the file if we destroy the in memory inode.
> I don't see the need for this to be persistent data; in fact I assume
> (Eric, please confirm) that this data is not *supposed* to be
> persistent.
Correct.
> > each product will have its own version namespace, and the questions of
> > what happens if a user switches from one version checker to another is
>
> yes that's a hard question; what if you have 2 virus scanners active.
>
> (they could register a version of the database with the kernel, and the
> in kernel version-cookie could be a hash of all registered versions I
> suppose.. if anything changes ever we just rehash and scan as if we
> have to do that)
It is in fact really simple with the proposed inode versioning approach.
Any client which wants to invalidate the cache just needs to bump the
global version number which will force a rescan on next access.
Tvrtko
Sophos Plc, The Pentagon, Abingdon Science Park, Abingdon,
OX14 3YP, United Kingdom.
Company Reg No 2096520. VAT Reg No GB 348 3873 20.
Theodore Tso wrote on 13/08/2008 20:29:22:
> On Wed, Aug 13, 2008 at 03:02:48PM -0400, Eric Paris wrote:
> > I never suggested putting a scanner in kernel. Sound like you want
the
> > "allow don't cache" response from your userspace scanner while this is
> > going on. The kernel doesn't need to be making decisions about when
to
> > send events, nor should userspace tell the kernel not to send events.
> > Its up to whatever the scanner is to agree not to actually do any
> > scanning...
>
> And if the system isn't running a virus checker, but just a file
> indexer (ala tracker), it shouldn't go to userspace at all. In that
> case all that is necessary is an asynchronous notification.
>
> Also something else that is needed is support for multiple clients.
> (i.e., what happens if the user runs two virus checkers, or a virus
> checker plus a hierarchical storage manager driving a tape robot, or
> all of the above plus trackerd --- where some clients need to block
> open(2) access, and some do not need block open(2) --- and in the case
> of HSM, ordering becomes important; you want to retrieve the file from
> the tape robot first, *then* scan it using the virus checker. :-)
Hm, maybe by implementing a facility with which a client can register it's
interface usage intent? Something like:
register(I_HAVE_NO_INTEREST_IN_CONTENT);
register(I_WANT_TO_EXAMINE_CONTENT);
All former ones would run first because they only want to have the
opportunity to block and do something unrelated to file content (like
HSMs), and later group would be ran last since they want to examine the
content.
Ordering inside those two groups is not important because I don't see how
a model other than restrictive can make sense with content security
scanning.
> > No. How in the heck can some out of kernel database store information
> > about what inodes have been scanned in any even slightly sane way? And
> > people think the race between open and read is too large and you
suggest
> > moving clean/dirty marking to a userspace database? I MUCH prefer my
> > (and it sounds like arjan agrees) clean/dirty versioned flag in inode.
>
> Don't ask me; I think most AV checkers for linux are security theater
> and not very much use (other than making money for the AV company's
> shareholders) anyway. I thought you were the one who wanted to record
> information about which version of the virus db a particular file had
> been scanned against. The place where I can see this being useful is
> what happens you get a new virus DB, and so you need to start scanning
> all of the files in your 5TB enterprise file server --- and then the
> system crashes or it needs to be taken down for scheduled maintenance.
>
> You want to have *some* off-line database for storing this
> information, since it would be silly to want to have the first thing
> that happens after a new virus DB gets downloaded is to interate over
> the entire filesystem, clearing a persistent the "clean" bit --- that
> would take *forever* on a 5TB filerserver; and what happens if you
> crash in the middle of clearing the "clean" bit.. And if the system
> gets shutdown in the middle of the scan, you need some way of
> remembering which inodes have been scanned using the "new" db, and
> which ones haven't yet been scanned via the new virus db. All of this
> should be kept in userpsace, and is strictly speaking Not Our Problem.
>
> I'm just arguing that there should be absolutely *no* support in the
> kernel for solving this particular problem, since the question of
> whether a file has been scanned with a particular version of the virus
> DB is purely a userspace problem.
The thing is the idea never was for clean-dirty "database" to be
persistent but to have the same lifetime as the inode (in memory one). And
once the cache/database gets invalidated re-scanning happens on-demand so
the 5TB problem does not exist. Concerns about multiple clients where
every has a different versioning scheme are also not relevant with the
proposed versioning scheme (see my reply to Arjan).
So this problem can be solved in kernel in a much more performant and
secure way than it would be possible in userspace and all that with just a
handfull of lines of code.
--
Tvrtko A. Ursulin
Senior Software Engineer, Sophos
"Views and opinions expressed in this email are strictly those of the
author.
The contents has not been reviewed or approved by Sophos."
Sophos Plc, The Pentagon, Abingdon Science Park, Abingdon,
OX14 3YP, United Kingdom.
Company Reg No 2096520. VAT Reg No GB 348 3873 20.
Eric Paris wrote on 13/08/2008 19:57:44:
> > It's clear from the protection model that you described that on 'read'
> > you want to wait until the scan is done before you give the data to
the
> > process asking for it... and that's totally reasonable: "Do not give
> > out bad data" is a very clear line in terms of security.
> >
> > for the "dirty" case it gets muddy. You clearly want to scan "some
> > time" after the write, from the principle of getting rid of malware
> > that's on the disk, but it's unclear if this HAS to be synchronous.
> > (obviously, synchronous behavior hurts performance bigtime so lets do
> > as little as we can of that without hurting the protection).
> > One advantage of doing the dirty case async (and a little time
delayed)
> > is that repeated writes will get lumped up into one scan in practice,
> > saving a ton of performance.
> > (scan-on-close is just another way of implementing "delay the dirty
> > scan").
> > Based on Alans comments, to me this sounds like we should have an
> > efficient mechanism to notify userspace of "dirty events"; this is not
> > virus scan specific in any way or form. And this mechanism likely will
> > need to allow multiple subscribers.
>
> I'm certainly willing to go down the inotify'ish path for async
> notification of 'dirty' inodes instead of implement my own async
> mechanism if I can find a way to do it.
Do I understand correctly that everyone agrees scanning whenever an inode
gets dirty would be a terrible thing for performance?
Another thing we have here is that malware could not be neccessariliy
identified until the very last write (one example where it will always be
the case are PDF files (I think)).
So the whole question is at which point should be performing an async
scan. Close seems like a natural point which should be ideal for majority
of applications, I don't see how any time-based lumping/delaying scheme
can be better than close?
> > for the open() case, I would argue that you don't need synchronous
> > behavior as long as the read() case is synchronous. I can imagine that
> > open() kicks off an async scan, and if it's done by the time the first
> > read() happens, no blocking at all happens.
>
> An interesting addition. Trying to keep these queues of events gets
> much more complex, but if people really think the open to read race is
> that important I've always said it wasn't impossible to close.
This really sounds pretty interesting. Not necessariliy so much as a
performance optimisation, because I am not sure there are so many programs
where first read comes long after the first open, but as closing the
open-read race.
Could the implementation be not so complicated after all? If we generated
the same (roughly) event on reads and pass it for scanning if cache has
been invalidated in the mean time? The only thing is this could be a big
performance hit so some benchmarking might be in order depending on which
the read hook could be made run-time optional.
--
Tvrtko A. Ursulin
Senior Software Engineer, Sophos
"Views and opinions expressed in this email are strictly those of the
author.
The contents has not been reviewed or approved by Sophos."
Sophos Plc, The Pentagon, Abingdon Science Park, Abingdon,
OX14 3YP, United Kingdom.
Company Reg No 2096520. VAT Reg No GB 348 3873 20.
> -----Original Message-----
> From: [email protected] [mailto:malware-list-
> [email protected]] On Behalf Of Mihai Don?u
> Sent: Wednesday, August 13, 2008 8:18 PM
> To: Andi Kleen
> Cc: [email protected]; [email protected]; malware-
> [email protected]; [email protected]; [email protected];
> [email protected]; [email protected]
> Subject: Re: [malware-list] TALPA - a threat model? well sorta.
>
> On Wednesday 13 August 2008, Andi Kleen wrote:
> > On Wed, Aug 13, 2008 at 12:36:15PM -0400, Eric Paris wrote:
> >
> > I miss a clear answer to the question: is this
> > supposed to protect against malware injected as root or not?
>
> I honestly don't think we should worry about root. Sure, if the AV scanner
> happens to catch something (as a consequence of it's implementation), then
> very well. But designing an antimalware solution which assumes the root is
> compromised will throw us into security talks for years and I don't think
> we'll live to hear the end of them.
>
> We should focus on the regular users and fix (if needed) the current userland
> apps (ie. the ones that need root access to do their job). For anymore than
> that we'll need a super user that supervises root. And then another one.
I think that some people are missing the important point of Eric's recent original statement of the "threat model". Whether we move further in the direction of other security protections or not, we are currently talking about providing a mechanism for basic AV product to do their job, and the job we are talking about is scanning files when they are about to be used and might cause harm, or have just been created and we want to make sure they are OK. That is, the AV products that we are talking about in this context don't do anything else other than scan files.
With that in mind, there is no difference between scanning files being accessed/executed/created by root and the same for any other users. And in fact, to the extent that we claim at all to have a somewhat complete protection in that realm, excluding root will completely blow that protect out of the water and make it essentially useless.
> I think we need to define the 'desktop user' and provide a decent protection
> mechanism for his common activities (edit documents, listen music, navigate
> the web, see movies, run scripts which change the IM status etc). For the
> rest, there are two possibilities:
> 1. education (_extremely_ important);
It's like abstinence education...it sounds good, at least to some, but it doesn't work. In a way, that's the whole point. There are millions of users. It doesn't take many who missed the class to create an outbreak that does real damage. It goes back to the medical analogy. Do you spray the swamps for the mosquitoes that carry Eastern Equine Encephalitis, or do you knock on everyone's door and tell them not to go near the swamps, and hope that everyone's home when you're in their neighborhood?
> I don't think there will ever be an AV product using the marketing line: "it
> allows you to run your favorite rootkit and enjoy the pretty text it shows,
> with no worries".
You are right... Complete rootkit protection is a whole other area not fundamentally addressed by a scan. So let's not create a straw man about the things we don't claim to do and then knock the products because we don't do them.
????{.n?+???????+%?????ݶ??w??{.n?+????{??G?????{ay? ʇڙ?,j??f???h?????????z_??(?階?ݢj"???m??????G????????????&???~???iO???z??v?^?m???? ????????I?
> -----Original Message-----
> From: [email protected] [mailto:malware-list-
> [email protected]] On Behalf Of [email protected]
> Sent: Thursday, August 14, 2008 5:31 AM
> To: Theodore Tso
> Cc: [email protected]; [email protected]; malware-
> [email protected]; [email protected]; [email protected];
> [email protected]; [email protected];
> [email protected]; Arjan van de Ven
> Subject: Re: [malware-list] TALPA - a threat model? well sorta.
> Hm, maybe by implementing a facility with which a client can register
it's
> interface usage intent? Something like:
>
> register(I_HAVE_NO_INTEREST_IN_CONTENT);
> register(I_WANT_TO_EXAMINE_CONTENT);
>
> All former ones would run first because they only want to have the
> opportunity to block and do something unrelated to file content (like
> HSMs), and later group would be ran last since they want to examine
the
> content.
>
> Ordering inside those two groups is not important because I don't see
how
> a model other than restrictive can make sense with content security
> scanning.
I'm not sure I understand why "interest in content" means not blocking,
and vice versa. However, I think this is a good idea if made more
explicit, i.e.:
> register(BLOCK);
> register(DON'T_BLOCK);
Jon Press
Jonathan Press wrote on 14/08/2008 13:03:40:
> > Hm, maybe by implementing a facility with which a client can register
> it's
> > interface usage intent? Something like:
> >
> > register(I_HAVE_NO_INTEREST_IN_CONTENT);
> > register(I_WANT_TO_EXAMINE_CONTENT);
> >
> > All former ones would run first because they only want to have the
> > opportunity to block and do something unrelated to file content (like
> > HSMs), and later group would be ran last since they want to examine
> the
> > content.
> >
> > Ordering inside those two groups is not important because I don't see
> how
> > a model other than restrictive can make sense with content security
> > scanning.
>
> I'm not sure I understand why "interest in content" means not blocking,
> and vice versa. However, I think this is a good idea if made more
> explicit, i.e.:
Small misunderstanding because both would block. If you go back to Ted's
original post I was replying to, he was worried about how would
anti-malware scanning interact with HSM since both may end up using the
same interface. HSM, as far as I understand it, needs to block on open and
"plant" the right file in place, while anti-malware also needs to block
and examine the right content. That is why ordering matters, anti-malware
needs to run after the content is put in place. And that is what my idea
solves (slight overstatement since I spent only seconds on it) by
separating them in two groups of clients. First which has no interest in
content and second which does.
--
Tvrtko A. Ursulin
Senior Software Engineer, Sophos
"Views and opinions expressed in this email are strictly those of the
author.
The contents has not been reviewed or approved by Sophos."
Sophos Plc, The Pentagon, Abingdon Science Park, Abingdon,
OX14 3YP, United Kingdom.
Company Reg No 2096520. VAT Reg No GB 348 3873 20.
On Thursday 14 August 2008, Press, Jonathan wrote:
> > On Wednesday 13 August 2008, Andi Kleen wrote:
> > > On Wed, Aug 13, 2008 at 12:36:15PM -0400, Eric Paris wrote:
> > >
> > > I miss a clear answer to the question: is this
> > > supposed to protect against malware injected as root or not?
> >
> > I honestly don't think we should worry about root. Sure, if the AV
> > scanner happens to catch something (as a consequence of it's
> > implementation), then very well. But designing an antimalware solution
> > which assumes the root is compromised will throw us into security talks
> > for years and I don't think we'll live to hear the end of them.
> >
> > We should focus on the regular users and fix (if needed) the current
> > userland apps (ie. the ones that need root access to do their job). For
> > anymore than that we'll need a super user that supervises root. And then
> > another one.
>
> I think that some people are missing the important point of Eric's recent
> original statement of the "threat model". Whether we move further in the
> direction of other security protections or not, we are currently talking
> about providing a mechanism for basic AV product to do their job, and the
> job we are talking about is scanning files when they are about to be used
> and might cause harm, or have just been created and we want to make sure
> they are OK. That is, the AV products that we are talking about in this
> context don't do anything else other than scan files.
I see. Well, as long as everyone sticks to _just_ the file scan. To be honest,
the only immediate use of the patch that is/was in question, is a "natural"
scanner for file servers (Samba, NFS etc). 7v5w7go9ub0o, however, might have
some more ideas. :)
I admit and I apologize, I got pretty worked up when people started asking
questions like: "how do we protect the file scanner", when the answer should
have been obvious: the way we protect any other daemon (service) today, by
means of chmod/chown.
> With that in mind, there is no difference between scanning files being
> accessed/executed/created by root and the same for any other users. And in
> fact, to the extent that we claim at all to have a somewhat complete
> protection in that realm, excluding root will completely blow that protect
> out of the water and make it essentially useless.
>
> > I think we need to define the 'desktop user' and provide a decent
> > protection mechanism for his common activities (edit documents, listen
> > music, navigate the web, see movies, run scripts which change the IM
> > status etc). For the rest, there are two possibilities:
> >
> > 1. education (_extremely_ important);
>
> It's like abstinence education...it sounds good, at least to some, but it
> doesn't work. In a way, that's the whole point. There are millions of
> users. It doesn't take many who missed the class to create an outbreak
> that does real damage. It goes back to the medical analogy. Do you spray
> the swamps for the mosquitoes that carry Eastern Equine Encephalitis, or do
> you knock on everyone's door and tell them not to go near the swamps, and
> hope that everyone's home when you're in their neighborhood?
>
> > I don't think there will ever be an AV product using the marketing line:
> > "it allows you to run your favorite rootkit and enjoy the pretty text it
> > shows, with no worries".
>
> You are right... Complete rootkit protection is a whole other area not
> fundamentally addressed by a scan. So let's not create a straw man about
> the things we don't claim to do and then knock the products because we
> don't do them.
--
Mihai Donțu
Again, this mail == my own opinion
On Wednesday 13 August 2008, Alan Cox wrote:
> > So, what is it that anti-malware companies do? ?They scan files. ?That's
> > it.
>
> Good so lets instead have a discussion about making the file event
> notification more scalable. That is the same thing I want for content
> indexing. It is the same thing you want for certain kinds of smart
> archiving, for on-line asynchronous backup and other stuff.
>
> It ought to be a simple clean syscall interface.
In this case, it seems to be the same kind of requirement that lead to
dnotify and inotify, right?
So in order to get the third notification syscall interface right, the
obvious questions are:
1. How do you make it cleaner than inotify?
2. How do you make it more scalable than inotify?
3. Do you need it to do anything that inotify doesn't do, beyond 1. and 2.?
Arnd <><
On Thu, Aug 14, 2008 at 10:30:56AM +0100, [email protected] wrote:
> The thing is the idea never was for clean-dirty "database" to be
> persistent but to have the same lifetime as the inode (in memory one). And
> once the cache/database gets invalidated re-scanning happens on-demand so
> the 5TB problem does not exist. Concerns about multiple clients where
> every has a different versioning scheme are also not relevant with the
> proposed versioning scheme (see my reply to Arjan).
So in essence, what I hear you saying is that all AV products want to
work in a mode where the moment the inode falls out of the inode
cache, and we lose the "clean" bit, when the inode is brought back
into the cache, it will be scanned again. That is, the "clean" bit is
never persistent, and never needs to be stored in memory.
That seems fair; if it turns out there is an AV product that wants to
optimize this a bit further, as long as we provide a persistent inode
version/generation number, they can always do their own persistent
database in userspace.
- Ted
On Thu, 14 Aug 2008 10:46:55 +0100
[email protected] wrote:
> So the whole question is at which point should be performing an async
> scan. Close seems like a natural point which should be ideal for
> majority of applications, I don't see how any time-based
> lumping/delaying scheme can be better than close?
close isn't the answer just because you can write content to the file
after that (and that's not just theoretical, glibc stdio supports mmap
etc); "dirty" *has* to be the event anyway. It's not impossible to
solve; even say a 1 second rearming delay would avoid 99.9% of the
useless rescans while still making sure everything gets scanned at some
point. Anyway this kind of policy can be done in userspace (and you can
get really fancy there and offer the admin various policies)
>
> > > for the open() case, I would argue that you don't need synchronous
> > > behavior as long as the read() case is synchronous. I can imagine
> > > that open() kicks off an async scan, and if it's done by the time
> > > the first read() happens, no blocking at all happens.
> >
> > An interesting addition. Trying to keep these queues of events gets
> > much more complex, but if people really think the open to read race
> > is that important I've always said it wasn't impossible to close.
>
> This really sounds pretty interesting. Not necessariliy so much as a
> performance optimisation, because I am not sure there are so many
> programs where first read comes long after the first open, but as
> closing the open-read race.
there's quite a few programs that open() but never read.
open+fstat is not uncommon as programming pattern for example;
with async-in-open (and then wait or sync in read) we wouldn't have the
big hit caused by the latency for the sync scan.
(I realize this pattern is much more likely to happen on posixy systems
than it is on windows.. one of those differences ;-)
--
If you want to reach me at my work email, use [email protected]
For development, discussion and tips for power savings,
visit http://www.lesswatts.org
On Thu, 2008-08-14 at 09:24 -0400, Theodore Tso wrote:
> On Thu, Aug 14, 2008 at 10:30:56AM +0100, [email protected] wrote:
> > The thing is the idea never was for clean-dirty "database" to be
> > persistent but to have the same lifetime as the inode (in memory one). And
> > once the cache/database gets invalidated re-scanning happens on-demand so
> > the 5TB problem does not exist. Concerns about multiple clients where
> > every has a different versioning scheme are also not relevant with the
> > proposed versioning scheme (see my reply to Arjan).
>
> So in essence, what I hear you saying is that all AV products want to
> work in a mode where the moment the inode falls out of the inode
> cache, and we lose the "clean" bit, when the inode is brought back
> into the cache, it will be scanned again. That is, the "clean" bit is
> never persistent, and never needs to be stored in memory.
There needs to be a way to say that an inode in cache needs to be
rescanned. 3 states this flag can be. Clean, Dirty, Infected. The
current talpa solution involves a global monotomically increasing
counter every time you change virus defs or make some "interesting"
change. If global == inode flag we are clean. If global == negative
inode flag we are infected. if global > inode flag we are dirty and
need a scan.
> That seems fair; if it turns out there is an AV product that wants to
> optimize this a bit further, as long as we provide a persistent inode
> version/generation number, they can always do their own persistent
> database in userspace.
exporting i_version might be useful for better userspace caching,
although I've yet to see any reasonable description of how a userspace
database can map between data on disk and what they have in userspace.
How can a userspace process, given 2 file descriptors know they are
actually the same thing on disk?
On Wed, 2008-08-13 at 14:39 -0700, Arjan van de Ven wrote:
> On Wed, 13 Aug 2008 14:57:44 -0400
> Eric Paris <[email protected]> wrote:
>
> > > for the open() case, I would argue that you don't need synchronous
> > > behavior as long as the read() case is synchronous. I can imagine
> > > that open() kicks off an async scan, and if it's done by the time
> > > the first read() happens, no blocking at all happens.
> >
> > An interesting addition. Trying to keep these queues of events gets
> > much more complex, but if people really think the open to read race is
> > that important I've always said it wasn't impossible to close.
>
> it's not "just" about open-to-read race.
> it's about open being non-blocking, and if read is not immediate, never
> hitting the latency at all.
>
> The real point is that "read" is the actual point you want to
> intercept, not "open" (you even wrote that in your description).. so
> why not just do that ?
> The open case then is just a performance optimization.
I've been thinking about this more over night. I really like the idea
for perf reasons but I'm scared of programs not expecting and thus
poorly handling -EACCESS from read. Every program ever is going to
expect that back from open, but once you have the fd open its not
common.
The idea of multiple concurrent outstanding async notifications is going
to be much more difficult to code, but hey, who am I to complain. We
could have outstanding async scanning requests for any number of writes,
any number of closes, and any number of opens all at the same time. The
current interface believes that every request out requires a response
but this type of interface basically requires some sort of coalescing.
Would you be opposed to a first patch round that did blocking
enforcement on open like we have today and do sync scanning (blocking or
return EWOULDBLOCK) on read if needed due to concurrent writes?
On Thu, Aug 14, 2008 at 09:48:33AM -0400, Eric Paris wrote:
>
> There needs to be a way to say that an inode in cache needs to be
> rescanned. 3 states this flag can be. Clean, Dirty, Infected. The
> current talpa solution involves a global monotomically increasing
> counter every time you change virus defs or make some "interesting"
> change. If global == inode flag we are clean. If global == negative
> inode flag we are infected. if global > inode flag we are dirty and
> need a scan.
"Infected" just means to instantly return an error when the file is
opened or if an already opened file descriptor is read or mmap'ed,
right? If file is already mmaped(), what's the plan? Send a kill -9
to the process, even if it ends up kill off an emacs or openoffice
process?
> > That seems fair; if it turns out there is an AV product that wants to
> > optimize this a bit further, as long as we provide a persistent inode
> > version/generation number, they can always do their own persistent
> > database in userspace.
>
> exporting i_version might be useful for better userspace caching,
> although I've yet to see any reasonable description of how a userspace
> database can map between data on disk and what they have in userspace.
> How can a userspace process, given 2 file descriptors know they are
> actually the same thing on disk?
>
If a userspace database knows that inode X, i_version Y was checked a
day ago, and inode X still has i_version Y, even if that inode has
been evicted from memory, the contents will be the same absent root
messing about with direct access to the block device. If there was an
intervening boot, the someone could remove the disk, edit the disk
block directly -- but that person could also add a backdoor to the
kernel while they were at it.
If your threat model is, "we do file scanning; that's it", then having
an external database which uses the inode number and i_version as a
tuple makes a lot of sense --- for filesystems where i_version is
getting bumped on every disk write, which is needed to support NFSv4
cache support, anyway.
- Ted
On Thu, 14 Aug 2008 10:12:13 -0400
Eric Paris <[email protected]> wrote:
> On Wed, 2008-08-13 at 14:39 -0700, Arjan van de Ven wrote:
> > On Wed, 13 Aug 2008 14:57:44 -0400
> > Eric Paris <[email protected]> wrote:
> >
> > > > for the open() case, I would argue that you don't need
> > > > synchronous behavior as long as the read() case is synchronous.
> > > > I can imagine that open() kicks off an async scan, and if it's
> > > > done by the time the first read() happens, no blocking at all
> > > > happens.
> > >
> > > An interesting addition. Trying to keep these queues of events
> > > gets much more complex, but if people really think the open to
> > > read race is that important I've always said it wasn't impossible
> > > to close.
> >
> > it's not "just" about open-to-read race.
> > it's about open being non-blocking, and if read is not immediate,
> > never hitting the latency at all.
> >
> > The real point is that "read" is the actual point you want to
> > intercept, not "open" (you even wrote that in your description).. so
> > why not just do that ?
> > The open case then is just a performance optimization.
>
> I've been thinking about this more over night. I really like the idea
> for perf reasons but I'm scared of programs not expecting and thus
> poorly handling -EACCESS from read. Every program ever is going to
> expect that back from open, but once you have the fd open its not
> common.
you could stretch things and report -EIO.
>
> The idea of multiple concurrent outstanding async notifications is
> going to be much more difficult to code, but hey, who am I to
well... do you really need a response?
you could just write it to a netlink socket...
> complain. We could have outstanding async scanning requests for any
> number of writes, any number of closes, and any number of opens all
> at the same time. The current interface believes that every request
> out requires a response but this type of interface basically requires
> some sort of coalescing.
or fire-and-forget.
>
> Would you be opposed to a first patch round that did blocking
> enforcement on open like we have today and do sync scanning (blocking
> or return EWOULDBLOCK) on read if needed due to concurrent writes?
I think we need to sort the interface issue on this for sure, and
probably from the start...
>
--
If you want to reach me at my work email, use [email protected]
For development, discussion and tips for power savings,
visit http://www.lesswatts.org
On Thu, 2008-08-14 at 11:50 -0400, Theodore Tso wrote:
> On Thu, Aug 14, 2008 at 09:48:33AM -0400, Eric Paris wrote:
> >
> > There needs to be a way to say that an inode in cache needs to be
> > rescanned. 3 states this flag can be. Clean, Dirty, Infected. The
> > current talpa solution involves a global monotomically increasing
> > counter every time you change virus defs or make some "interesting"
> > change. If global == inode flag we are clean. If global == negative
> > inode flag we are infected. if global > inode flag we are dirty and
> > need a scan.
>
> "Infected" just means to instantly return an error when the file is
> opened or if an already opened file descriptor is read or mmap'ed,
> right? If file is already mmaped(), what's the plan? Send a kill -9
> to the process, even if it ends up kill off an emacs or openoffice
> process?
We don't have a revocation mechanism in linux and this isn't about
adding one. Your trying to turn this into something it isn't. If you
have it opened and mmap'd you've got that file for as long as you want.
I've already said that given Arjan's belief that we can move it
read/write instead of open/close we are moving the open->read race to a
mmap->fault race. It isn't perfect at stopping bad data flow, but its
darn sure a lot better than nothing.
>
> > > That seems fair; if it turns out there is an AV product that wants to
> > > optimize this a bit further, as long as we provide a persistent inode
> > > version/generation number, they can always do their own persistent
> > > database in userspace.
> >
> > exporting i_version might be useful for better userspace caching,
> > although I've yet to see any reasonable description of how a userspace
> > database can map between data on disk and what they have in userspace.
> > How can a userspace process, given 2 file descriptors know they are
> > actually the same thing on disk?
> >
>
> If a userspace database knows that inode X, i_version Y was checked a
> day ago, and inode X still has i_version Y, even if that inode has
> been evicted from memory, the contents will be the same absent root
> messing about with direct access to the block device. If there was an
> intervening boot, the someone could remove the disk, edit the disk
> block directly -- but that person could also add a backdoor to the
> kernel while they were at it.
is i_version an on disk think? didn't realize that and just assumed it
was in in core thing. I wouldn't have an issue sending i_version to the
userspace scanner for them to use as they like.
-Eric
On Thu, Aug 14, 2008 at 01:29:45PM -0400, Eric Paris wrote:
>
> is i_version an on disk think? didn't realize that and just assumed it
> was in in core thing. I wouldn't have an issue sending i_version to the
> userspace scanner for them to use as they like.
>
It's on-disk for some filesystems, in order to support NFsv4 advanced
caching semantics (which means i_version has to survive a reboot,
which means it has to be on disk). It is *not* on disk for ext3,
although it is for ext4.
- Ted
On Thu, 2008-08-14 at 15:17 -0400, Theodore Tso wrote:
> On Thu, Aug 14, 2008 at 01:29:45PM -0400, Eric Paris wrote:
> >
> > is i_version an on disk think? didn't realize that and just assumed it
> > was in in core thing. I wouldn't have an issue sending i_version to the
> > userspace scanner for them to use as they like.
> >
>
> It's on-disk for some filesystems, in order to support NFsv4 advanced
> caching semantics (which means i_version has to survive a reboot,
> which means it has to be on disk). It is *not* on disk for ext3,
> although it is for ext4.
Is there some pragmatic way in kernel to know if the inode I have
corresponds to a persistent or in core only inode? I'll poke around
myself and look.
-Eris
On Thu, Aug 14, 2008 at 03:34:08PM -0400, Christoph Hellwig wrote:
>
> It's not used at all on regular files except for ext4 with a non-default,
> undocumented mount option. XFS will grow it soon in a similar way as ext4,
> except that it will be documented or I might have even figured out by
> then how to just enabled it from nfsd.
We do need a standardized way of enabling it (since it does cost you
something in performance, so not everyone will want it on), and a
standardized way of reading i_version out to userspace. Maybe a mount
option is the right way to do it, maybe not.
We may want to take this to the linux-fs list and try to get
agreements on these points; the main reason why it's not enabled by
default in ext4 is because the NFSv4 advanced caching code is in
common use (is it even in mainline)?
- Ted
On Thu, Aug 14, 2008 at 03:17:26PM -0400, Theodore Tso wrote:
> On Thu, Aug 14, 2008 at 01:29:45PM -0400, Eric Paris wrote:
> >
> > is i_version an on disk think? didn't realize that and just assumed it
> > was in in core thing. I wouldn't have an issue sending i_version to the
> > userspace scanner for them to use as they like.
> >
>
> It's on-disk for some filesystems, in order to support NFsv4 advanced
> caching semantics (which means i_version has to survive a reboot,
> which means it has to be on disk). It is *not* on disk for ext3,
> although it is for ext4.
It's not used at all on regular files except for ext4 with a non-default,
undocumented mount option. XFS will grow it soon in a similar way as ext4,
except that it will be documented or I might have even figured out by
then how to just enabled it from nfsd.
On Thu, Aug 14, 2008 at 03:41:11PM -0400, Theodore Tso wrote:
> We do need a standardized way of enabling it (since it does cost you
> something in performance, so not everyone will want it on), and a
> standardized way of reading i_version out to userspace. Maybe a mount
> option is the right way to do it, maybe not.
>
> We may want to take this to the linux-fs list and try to get
> agreements on these points; the main reason why it's not enabled by
> default in ext4 is because the NFSv4 advanced caching code is in
> common use (is it even in mainline)?
I have not yet seen code actually using it at all, neither in mainline
nor on one of the many nfs development lists.
On Thu, Aug 14, 2008 at 04:20:39PM -0400, Christoph Hellwig wrote:
> On Thu, Aug 14, 2008 at 03:41:11PM -0400, Theodore Tso wrote:
> > We do need a standardized way of enabling it (since it does cost you
> > something in performance, so not everyone will want it on), and a
> > standardized way of reading i_version out to userspace. Maybe a mount
> > option is the right way to do it, maybe not.
> >
> > We may want to take this to the linux-fs list and try to get
> > agreements on these points; the main reason why it's not enabled by
> > default in ext4 is because the NFSv4 advanced caching code is in
> > common use (is it even in mainline)?
>
> I have not yet seen code actually using it at all, neither in mainline
> nor on one of the many nfs development lists.
Oops, I'd love to, and it should be very easy. How do I find out if
i_version is supported on a given superblock?
There's nothing particularly "advanced" about this, by the way--this is
a very minor variation on the caching model that nfs has always had, and
our nfsv4 server is currently pretty broken without it.
Actually, it's pretty broken even on nfsv2/v3 for filesystems with poor
time resolution.
--b.
On Thu, Aug 14, 2008 at 05:21:09PM -0400, J. Bruce Fields wrote:
> > I have not yet seen code actually using it at all, neither in mainline
> > nor on one of the many nfs development lists.
>
> Oops, I'd love to, and it should be very easy. How do I find out if
> i_version is supported on a given superblock?
We don't have a way of exporting this fact at the moment. I assume
the best way would be to add a flag in struct super.
> There's nothing particularly "advanced" about this, by the way--this is
> a very minor variation on the caching model that nfs has always had, and
> our nfsv4 server is currently pretty broken without it.
Well, if you're willing to try it out, as I've mentioned on my
blog[1][2], ext4 is working pretty well on my laptop --- I'm running
it as my primary filesystem. There are a few problems with ext3
filesystems converted to use ext4, as opposed to starting afresh via
"mke2fs -t ext4dev /dev/hdXX" that we've just found in the past week
(and fixed within a day or two, although they haven't been pushed to
Linus yet), but overall, it's been pretty stable.
So this would be a good time for someone who is familiar wiht NFSv4 to
try it out and let us know if the i_version support is as you would
like in ext4 --- we're in the bugfixing/stablization phase right now,
so this would be an ideal time to get that feedback. For more
information, on how to get started, please see:
http://ext4.wiki.kernel.org/index.php/Ext4_Howto
for instructions, and mount the filesystem with the "-o i_version"
mount option.
> Actually, it's pretty broken even on nfsv2/v3 for filesystems with poor
> time resolution.
And that's fixed in ext4 as well....
- Ted
[1] http://thunk.org/tytso/blog/2008/06/30/ext4-is-now-the-primary-filesystem-on-my-laptop/
[2] http://thunk.org/tytso/blog/2008/08/08/fast-ext4-fsck-times/
On Wed, 13 Aug 2008, Arjan van de Ven wrote:
> 2) We very likely should have a mechanism for a userspace app to
> request a scan on a file, both sync or async (O_SYNC flag?). This is
> useful regardless because it allows the source of many things to do the
> right thing.
> 3) we need a mechanism in the kernel to track "scanned with generation
> X of signatures" that invalidates on any dirty operation. The syscall
> from 2) will use this as a cache to be quick.
>
> I think few people will disagree about this.
>
> Open questions now are
> 4) do we have the kernel kick off an async scan in open() or do we have
> glibc do this
the kernel should not kick off a scan, instead it should check to see
an open/read should not kick off a scan, instead it should check to see if
the scan generation tag(s) are current should be enough (remember, you may
have more then one type of scanner running on the system)
> 5) do we have the kernel do the sync scan on read/mmap/.. or do we have
> glibc do this
definantly not the kernel. the intent of this is to keep linux from being
a storage repository for malware used by other systems. there is no need
to penalize linux-only apps by making them wait for a scan to take place.
If it lives in glibc there should be a way for linux apps that know that
they will not be exporting files to other systems to tell the library not
to do a scan.
for example, why should a log analysis program looking at apache logs be
forced to wait while multiple 'virii scanners' go through several gigs of
logs before it can start looking at them.
you are going to need some way to bypass the checks anyway so that you can
avoid the recursive case of the scanners triggering scans on files that
they open.
by keeping the scans all in userspace it also simplifies things greatly.
All the kernel should do is to maintain the tags with the file (posix
attributes??) and have a mechanism to clear them when the file is dirtied.
> I think this is where the whole debate is about now.
>
> And a few hard ones
> 6) how do we deal with multiple scanning agents in parallel
not a problem, in fact multiple agents scanning in parallel is a good
thing, it lets them all see the data with one pass through the disk.
they will all need to set different tags anyway (the fact that agent1
blessed the data doesn't mean that it's safe if agent2 hasn't done so)
> 7) how do we prevent malware from pretending to be a virus scanner
this is not part of the threat model.
David Lang
On Thu, 14 Aug 2008, [email protected] wrote:
> Eric Paris wrote on 13/08/2008 19:57:44:
>
>>> It's clear from the protection model that you described that on 'read'
>>> you want to wait until the scan is done before you give the data to
> the
>>> process asking for it... and that's totally reasonable: "Do not give
>>> out bad data" is a very clear line in terms of security.
>>>
>>> for the "dirty" case it gets muddy. You clearly want to scan "some
>>> time" after the write, from the principle of getting rid of malware
>>> that's on the disk, but it's unclear if this HAS to be synchronous.
>>> (obviously, synchronous behavior hurts performance bigtime so lets do
>>> as little as we can of that without hurting the protection).
>>> One advantage of doing the dirty case async (and a little time
> delayed)
>>> is that repeated writes will get lumped up into one scan in practice,
>>> saving a ton of performance.
>>> (scan-on-close is just another way of implementing "delay the dirty
>>> scan").
>>> Based on Alans comments, to me this sounds like we should have an
>>> efficient mechanism to notify userspace of "dirty events"; this is not
>>> virus scan specific in any way or form. And this mechanism likely will
>>> need to allow multiple subscribers.
>>
>> I'm certainly willing to go down the inotify'ish path for async
>> notification of 'dirty' inodes instead of implement my own async
>> mechanism if I can find a way to do it.
>
> Do I understand correctly that everyone agrees scanning whenever an inode
> gets dirty would be a terrible thing for performance?
>
> Another thing we have here is that malware could not be neccessariliy
> identified until the very last write (one example where it will always be
> the case are PDF files (I think)).
>
> So the whole question is at which point should be performing an async
> scan. Close seems like a natural point which should be ideal for majority
> of applications, I don't see how any time-based lumping/delaying scheme
> can be better than close?
all you need is the ability to mark a file as 'dirty', and some way for
programs that are interested in dirty files learning about it later and
decideing to do a scan. if the file gets dirtied again after they do they
scan they will need to do another one (this is a classic trade-off between
the 'security' of looking for things quickly or 'efficiancy' of only
looking when you don't think they will change again. in other words,
policy -> userspace)
David Lang
On Thu, 14 Aug 2008, Theodore Tso wrote:
> On Thu, Aug 14, 2008 at 03:34:08PM -0400, Christoph Hellwig wrote:
>>
>> It's not used at all on regular files except for ext4 with a non-default,
>> undocumented mount option. XFS will grow it soon in a similar way as ext4,
>> except that it will be documented or I might have even figured out by
>> then how to just enabled it from nfsd.
>
> We do need a standardized way of enabling it (since it does cost you
> something in performance, so not everyone will want it on), and a
> standardized way of reading i_version out to userspace. Maybe a mount
> option is the right way to do it, maybe not.
>
> We may want to take this to the linux-fs list and try to get
> agreements on these points; the main reason why it's not enabled by
> default in ext4 is because the NFSv4 advanced caching code is in
> common use (is it even in mainline)?
could you do something like defining a namespace inside posix attributes
and then setting up a mechanism in the kernel to alert if the attributes
change (with the entire namespace getting cleared if the file gets
dirtied)?
this would have the advantage of storing the clean/dirty state in a known
location on the filesystem (you would need to enable posix attributes, but
that should be an acceptable limit), and would allow multiple scanners
(virii, indexing, etc) to do their own thing at their own schedule when
things get dirtied.
the userspace library calls that open the files would be able to take care
of the issue of deciding which of the configured scanners on a system
should be called for each attribute that's not set on a file.
This would seem to require minimal kernel support
1. reserve a attribute namespace
2. clear that namespace if the file is dirtied
3. provide a notification mechanism for programs to subscribe to that
lists all files where the namespace was not already empty when the file
was dirtied.
everything else can be userspace.
David Lang
On Thu, Aug 14, 2008 at 06:44:33PM -0700, [email protected] wrote:
> could you do something like defining a namespace inside posix attributes
> and then setting up a mechanism in the kernel to alert if the attributes
> change (with the entire namespace getting cleared if the file gets
> dirtied)?
According to Eric Paris the clean/dirty state is only stored in
memory. We could use the extended attribute interface as a way of not
defining a new system call, or some other interface, but I'm not sure
it's such a great match given that the extended attributes interface
are designed for persistent data.
I agree that doesn't actually work very well for the tracker use case,
where you the clean/dirty bit to be persistent (in case the tracker is
disabled due to the fact you are running on battery, for example, and
then you reboot).
- Ted
On Wed, 2008-08-13 at 22:23 +0100, Alan Cox wrote:
> On Wed, 13 Aug 2008 17:35:29 -0400
> Rik van Riel <[email protected]> wrote:
>
> > On Wed, 13 Aug 2008 17:24:28 -0400
> > "Press, Jonathan" <[email protected]> wrote:
> >
> > > I may be missing something about your suggestion, but I don't see how
> > > this would work. Who does the chmod?
> >
> > Chmod is also not a solution to the hierarchical storage (or incremental
> > restore from backup) problem.
> >
> > I believe we really do need the block-on-open.
>
> The block on open is orthogonal really. Useful for HSM, useful for
> certain very primitive scanning but not much else that I can see.
>
> And its a minor mod to the security hooks to allow it as far as I can see
So here's where I run into trouble. Lets assume I want to be helpful
and engineer in a vacuum for this unknown HSM user as well. Clearly
file scanners need the file to be there which means the inode ("on disk"
kind) needs to be there and stuff like that. I assume that the HSM user
is going to need to hook long before these things even exist. Where
would they need to hook?
Should I just design for my own needs and include stacking and a
priority number and hopefully the HSM people can use it later? I don't
know the details of what might someday by needed for a project I know
nothing about *smile*
-Eric
On Thu, 14 Aug 2008 22:04:00 -0400
Theodore Tso <[email protected]> wrote:
> On Thu, Aug 14, 2008 at 06:44:33PM -0700, [email protected] wrote:
> > could you do something like defining a namespace inside posix
> > attributes and then setting up a mechanism in the kernel to alert
> > if the attributes change (with the entire namespace getting cleared
> > if the file gets dirtied)?
>
> According to Eric Paris the clean/dirty state is only stored in
> memory. We could use the extended attribute interface as a way of not
> defining a new system call, or some other interface, but I'm not sure
> it's such a great match given that the extended attributes interface
> are designed for persistent data.
>
> I agree that doesn't actually work very well for the tracker use case,
> where you the clean/dirty bit to be persistent (in case the tracker is
> disabled due to the fact you are running on battery, for example, and
> then you reboot).
>
but we need a "give me all dirty files" solution, not a "is this file
dirty" solution.
I do not want a virus scanner to constantly have to poll the whole fs
for dirty files ;-)
--
If you want to reach me at my work email, use [email protected]
For development, discussion and tips for power savings,
visit http://www.lesswatts.org
On Thu, 14 Aug 2008, Theodore Tso wrote:
> On Thu, Aug 14, 2008 at 06:44:33PM -0700, [email protected] wrote:
>> could you do something like defining a namespace inside posix attributes
>> and then setting up a mechanism in the kernel to alert if the attributes
>> change (with the entire namespace getting cleared if the file gets
>> dirtied)?
>
> According to Eric Paris the clean/dirty state is only stored in
> memory. We could use the extended attribute interface as a way of not
> defining a new system call, or some other interface, but I'm not sure
> it's such a great match given that the extended attributes interface
> are designed for persistent data.
>
> I agree that doesn't actually work very well for the tracker use case,
> where you the clean/dirty bit to be persistent (in case the tracker is
> disabled due to the fact you are running on battery, for example, and
> then you reboot).
why would you not want the results of the viris scan to be persistant?
assuming that you are not dual-booting( or otherwise taking explicit
action to go around the OS), the fact that you scanned the file with
signature set 123 is just as valid now as it was yesterday.
If you are not trying to work around the actions of root, why are you
trying to work around the actions of the person who has physical control
of the system? (who could easily become root if they wanted to)
David Lang
On Thu, 14 Aug 2008, Arjan van de Ven wrote:
> On Thu, 14 Aug 2008 22:04:00 -0400
> Theodore Tso <[email protected]> wrote:
>
>> On Thu, Aug 14, 2008 at 06:44:33PM -0700, [email protected] wrote:
>>> could you do something like defining a namespace inside posix
>>> attributes and then setting up a mechanism in the kernel to alert
>>> if the attributes change (with the entire namespace getting cleared
>>> if the file gets dirtied)?
>>
>> According to Eric Paris the clean/dirty state is only stored in
>> memory. We could use the extended attribute interface as a way of not
>> defining a new system call, or some other interface, but I'm not sure
>> it's such a great match given that the extended attributes interface
>> are designed for persistent data.
>>
>> I agree that doesn't actually work very well for the tracker use case,
>> where you the clean/dirty bit to be persistent (in case the tracker is
>> disabled due to the fact you are running on battery, for example, and
>> then you reboot).
>>
>
> but we need a "give me all dirty files" solution, not a "is this file
> dirty" solution.
>
> I do not want a virus scanner to constantly have to poll the whole fs
> for dirty files ;-)
I'm not sure.
there are two situations (with the transition between them)
1. unscanned system, we want to do everything. (this happens immediatly
after a new signature file is deployed)
here you do just want to filter out the files that have been scanned from
the list of everything, and you probably want to check at the time of
scanning the file in case it was opened (and scanned) in the meantime.
2. mostly scanned system, we only want to scan files that have been
dirtied.
here you don't need to scan everything, you only need to scan in two cases
2a. the file was dirtied (you learn about it and add it to the queue of
files to scan when you get around to it)
2b. an unscanned file is opened (the library detects that the file was not
marked approved by all the current scanners, so it invokes the scanners on
this file before completing the open, or copy for mmap, or whatever)
In the first case the attributes work "don't bother scanning me".
In the second case they also work (becouse you aren't trying to scan
everything)
the only time there is a headache is in the transition between them when
you have scanned a lot of the system, but not all of it, and the machine
was rebooted so you lost track of what you had scanned.
it shouldn't be too hard to deal with this. even if you never resume the
scan you are still safe (becouse of the scan-on-open), but it's also
possible to either do a find f(or equivalent) or files without the
attribute and store the results (similar to updatedb) and then updating
the file to mark the files as being scanned (update in place, change the
first character or something to be fairly crash safe). after the full list
of files has been scanned shift to the second mode.
the sweep scan should be a background task, possibly disabled when on
battery power.
why would this not satisfy the requirements?
David Lang
Hi,
[email protected] writes:
> On Thu, 14 Aug 2008, Arjan van de Ven wrote:
>
>> On Thu, 14 Aug 2008 22:04:00 -0400
>> Theodore Tso <[email protected]> wrote:
>>
>>> On Thu, Aug 14, 2008 at 06:44:33PM -0700, [email protected] wrote:
>>>> could you do something like defining a namespace inside posix
>>>> attributes and then setting up a mechanism in the kernel to alert
>>>> if the attributes change (with the entire namespace getting cleared
>>>> if the file gets dirtied)?
>>>
>>> According to Eric Paris the clean/dirty state is only stored in
>>> memory. We could use the extended attribute interface as a way of not
>>> defining a new system call, or some other interface, but I'm not sure
>>> it's such a great match given that the extended attributes interface
>>> are designed for persistent data.
>>>
>>> I agree that doesn't actually work very well for the tracker use case,
>>> where you the clean/dirty bit to be persistent (in case the tracker is
>>> disabled due to the fact you are running on battery, for example, and
>>> then you reboot).
>>>
>>
>> but we need a "give me all dirty files" solution, not a "is this file
>> dirty" solution.
>>
>> I do not want a virus scanner to constantly have to poll the whole fs
>> for dirty files ;-)
>
> I'm not sure.
>
> there are two situations (with the transition between them)
>
> 1. unscanned system, we want to do everything. (this happens
> immediatly after a new signature file is deployed)
>
> here you do just want to filter out the files that have been scanned
> from the list of everything, and you probably want to check at the
> time of scanning the file in case it was opened (and scanned) in the
> meantime.
>
> 2. mostly scanned system, we only want to scan files that have been
> dirtied.
>
> here you don't need to scan everything, you only need to scan in two cases
>
> 2a. the file was dirtied (you learn about it and add it to the queue
> of files to scan when you get around to it)
>
> 2b. an unscanned file is opened (the library detects that the file was
> not marked approved by all the current scanners, so it invokes the
> scanners on this file before completing the open, or copy for mmap, or
> whatever)
2b could also be used as a general lazy scanning mechanism, no? If
there is a new signature or the file is dirty or unknown, scan it.
Always.
So this should all just be one case, no?
Hannes
On Fri, 15 Aug 2008, Johannes Weiner wrote:
> Hi,
>
> [email protected] writes:
>
>> On Thu, 14 Aug 2008, Arjan van de Ven wrote:
>>
>>> On Thu, 14 Aug 2008 22:04:00 -0400
>>> Theodore Tso <[email protected]> wrote:
>>>
>>>> On Thu, Aug 14, 2008 at 06:44:33PM -0700, [email protected] wrote:
>>>>> could you do something like defining a namespace inside posix
>>>>> attributes and then setting up a mechanism in the kernel to alert
>>>>> if the attributes change (with the entire namespace getting cleared
>>>>> if the file gets dirtied)?
>>>>
>>>> According to Eric Paris the clean/dirty state is only stored in
>>>> memory. We could use the extended attribute interface as a way of not
>>>> defining a new system call, or some other interface, but I'm not sure
>>>> it's such a great match given that the extended attributes interface
>>>> are designed for persistent data.
>>>>
>>>> I agree that doesn't actually work very well for the tracker use case,
>>>> where you the clean/dirty bit to be persistent (in case the tracker is
>>>> disabled due to the fact you are running on battery, for example, and
>>>> then you reboot).
>>>>
>>>
>>> but we need a "give me all dirty files" solution, not a "is this file
>>> dirty" solution.
>>>
>>> I do not want a virus scanner to constantly have to poll the whole fs
>>> for dirty files ;-)
>>
>> I'm not sure.
>>
>> there are two situations (with the transition between them)
>>
>> 1. unscanned system, we want to do everything. (this happens
>> immediatly after a new signature file is deployed)
>>
>> here you do just want to filter out the files that have been scanned
>> from the list of everything, and you probably want to check at the
>> time of scanning the file in case it was opened (and scanned) in the
>> meantime.
>>
>> 2. mostly scanned system, we only want to scan files that have been
>> dirtied.
>>
>> here you don't need to scan everything, you only need to scan in two cases
>>
>> 2a. the file was dirtied (you learn about it and add it to the queue
>> of files to scan when you get around to it)
>>
>> 2b. an unscanned file is opened (the library detects that the file was
>> not marked approved by all the current scanners, so it invokes the
>> scanners on this file before completing the open, or copy for mmap, or
>> whatever)
>
> 2b could also be used as a general lazy scanning mechanism, no? If
> there is a new signature or the file is dirty or unknown, scan it.
> Always.
>
> So this should all just be one case, no?
this depends on the policy in userspace.
you could:
1. plan on doing all scanning as files are opened (this is going to be
noticable to users)
2. run the background scanner continuously as a low priority (as
soon as you finish one scan, do the next)
3. do one background scan when you get a new signature, and after you
complete that only scan files that change.
and probably others
#3 has the advantage that the only time a user will hit a delay when
opening a file is if the file was just recently modified and the scanner
daemon(s) haven't had a chance to scan it yet.
David Lang
On Thu, 14 Aug 2008, [email protected] wrote:
> On Thu, 14 Aug 2008, Arjan van de Ven wrote:
>
>> On Thu, 14 Aug 2008 22:04:00 -0400
>> Theodore Tso <[email protected]> wrote:
>>
>>> On Thu, Aug 14, 2008 at 06:44:33PM -0700, [email protected] wrote:
>>>> could you do something like defining a namespace inside posix
>>>> attributes and then setting up a mechanism in the kernel to alert
>>>> if the attributes change (with the entire namespace getting cleared
>>>> if the file gets dirtied)?
>>>
>>> According to Eric Paris the clean/dirty state is only stored in
>>> memory. We could use the extended attribute interface as a way of not
>>> defining a new system call, or some other interface, but I'm not sure
>>> it's such a great match given that the extended attributes interface
>>> are designed for persistent data.
>>>
>>> I agree that doesn't actually work very well for the tracker use case,
>>> where you the clean/dirty bit to be persistent (in case the tracker is
>>> disabled due to the fact you are running on battery, for example, and
>>> then you reboot).
>>>
>>
>> but we need a "give me all dirty files" solution, not a "is this file
>> dirty" solution.
>>
>> I do not want a virus scanner to constantly have to poll the whole fs
>> for dirty files ;-)
>
> I'm not sure.
>
> there are two situations (with the transition between them)
>
> 1. unscanned system, we want to do everything. (this happens immediatly after
> a new signature file is deployed)
>
> here you do just want to filter out the files that have been scanned from the
> list of everything, and you probably want to check at the time of scanning
> the file in case it was opened (and scanned) in the meantime.
>
> 2. mostly scanned system, we only want to scan files that have been dirtied.
>
> here you don't need to scan everything, you only need to scan in two cases
>
> 2a. the file was dirtied (you learn about it and add it to the queue of files
> to scan when you get around to it)
>
> 2b. an unscanned file is opened (the library detects that the file was not
> marked approved by all the current scanners, so it invokes the scanners on
> this file before completing the open, or copy for mmap, or whatever)
>
>
>
> In the first case the attributes work "don't bother scanning me".
>
> In the second case they also work (becouse you aren't trying to scan
> everything)
>
>
> the only time there is a headache is in the transition between them when you
> have scanned a lot of the system, but not all of it, and the machine was
> rebooted so you lost track of what you had scanned.
>
> it shouldn't be too hard to deal with this. even if you never resume the scan
> you are still safe (becouse of the scan-on-open), but it's also possible to
> either do a find f(or equivalent) or files without the attribute and store
> the results (similar to updatedb) and then updating the file to mark the
> files as being scanned (update in place, change the first character or
> something to be fairly crash safe). after the full list of files has been
> scanned shift to the second mode.
>
> the sweep scan should be a background task, possibly disabled when on battery
> power.
>
> why would this not satisfy the requirements?
>
> David Lang
one way this approach would be a pain is that as signature files got
updated the attributes would accumulate.
a couple ways of dealing with this.
1. scanners clean up after themselves (when they go to add a new one the
remove the old one)
2. background sweep through the system removing all tags that aren't in
the current 'blessed' set
David Lang
> According to Eric Paris the clean/dirty state is only stored in
> memory. We could use the extended attribute interface as a way of not
> defining a new system call, or some other interface, but I'm not sure
> it's such a great match given that the extended attributes interface
> are designed for persistent data.
This is another application layer matter. At the end of the day why does
the kernel care where this data is kept. For all we know someone might
want to centralise it or even distribute it between nodes on a clustered
file system.
Alan
It seems to me that this "scan on file open" business is the
wrong way to do things - because it reduces performance.
If you scan on file open, then your security sw is too late and
getting in the way.
It is better to scan in advance. Most machines has lots of idle time.
Use that time to scan in advance, and mark the files as "clean".
A "clean" file can be opened without further checking anytime - giving
normal high performance. A file that gets written to becomes "dirty"
until checked again. Some mechanism for making a clean copy of a
clean file might help avoid excessive "dirtying".
"Scan on open" might still be useful for cases when the system
haven't kept up with writing, but please don't aim to have
this be the _primary_ mode of scanning. A file server
where most of the stuff is pre-scanned will likely perform much better
than one scanning everything on open.
Helge Hafting
On Fri, 2008-08-15 at 12:07 +0200, Helge Hafting wrote:
> It seems to me that this "scan on file open" business is the
> wrong way to do things - because it reduces performance.
>
> If you scan on file open, then your security sw is too late and
> getting in the way.
>
> It is better to scan in advance. Most machines has lots of idle time.
> Use that time to scan in advance, and mark the files as "clean".
>
> A "clean" file can be opened without further checking anytime - giving
> normal high performance. A file that gets written to becomes "dirty"
> until checked again. Some mechanism for making a clean copy of a
> clean file might help avoid excessive "dirtying".
>
> "Scan on open" might still be useful for cases when the system
> haven't kept up with writing, but please don't aim to have
> this be the _primary_ mode of scanning. A file server
> where most of the stuff is pre-scanned will likely perform much better
> than one scanning everything on open.
This is the source of all that scan on write/close rambling I've seen go
past the last couple of days..
So I'd not worry about this particular aspect
Helge Hafting wrote on 15/08/2008 11:07:48:
> It seems to me that this "scan on file open" business is the
> wrong way to do things - because it reduces performance.
It can never be free - what ever you do it has to happen some time and
that can and will clash with something else. So your reason why it is
wrong is a bit to simplistic.
> If you scan on file open, then your security sw is too late and
> getting in the way.
>
> It is better to scan in advance. Most machines has lots of idle time.
> Use that time to scan in advance, and mark the files as "clean".
>
> A "clean" file can be opened without further checking anytime - giving
> normal high performance. A file that gets written to becomes "dirty"
> until checked again. Some mechanism for making a clean copy of a
> clean file might help avoid excessive "dirtying".
>
> "Scan on open" might still be useful for cases when the system
> haven't kept up with writing, but please don't aim to have
> this be the _primary_ mode of scanning. A file server
> where most of the stuff is pre-scanned will likely perform much better
> than one scanning everything on open.
The first thing you have to make clear is whether in your vision inode
clean-dirty-unknown status is persistent or not? But in any case I think
you are making a problem where there isn't one, think about it a bit.
But the idea about the ability to make a clean copy is interesting. So we
would need a copy done without userspace intervention and preserving the
inode cache status alongside it. Maybe splice or tee could easily do it?
Although I am not sure how often copying would happen and whether this
would be such a gain.
--
Tvrtko A. Ursulin
Senior Software Engineer, Sophos
"Views and opinions expressed in this email are strictly those of the
author.
The contents has not been reviewed or approved by Sophos."
Sophos Plc, The Pentagon, Abingdon Science Park, Abingdon,
OX14 3YP, United Kingdom.
Company Reg No 2096520. VAT Reg No GB 348 3873 20.
> -----Original Message-----
> From: [email protected] [mailto:malware-list-
> [email protected]] On Behalf Of Peter Zijlstra
> Sent: Friday, August 15, 2008 6:37 AM
> To: Helge Hafting
> Cc: [email protected]; [email protected];
[email protected];
> [email protected]; [email protected];
[email protected]; Arjan van
> de Ven
> Subject: Re: [malware-list] TALPA - a threat model? well sorta.
>
> On Fri, 2008-08-15 at 12:07 +0200, Helge Hafting wrote:
> > It seems to me that this "scan on file open" business is the
> > wrong way to do things - because it reduces performance.
> >
> > If you scan on file open, then your security sw is too late and
> > getting in the way.
The problem is that you have to account for the cases where the malware
made it onto the system even if you were trying to catch it ahead of
time. For example:
- Administrator turns off or reduces AV protection for some reason for
some period of time. It happens all the time.
- New infection makes it onto the machine before the signatures have
caught up with it. This also happens. There is an ongoing PR race
among AV vendors about who was faster on the draw to get out signatures
to detect some new malware. The fact that this race exists reflects
that reality that there is some window during which new malware will
make it onto some number of machines before the scanners catch up.
> > It is better to scan in advance. Most machines has lots of idle
time.
> > Use that time to scan in advance, and mark the files as "clean".
> >
> > A "clean" file can be opened without further checking anytime -
giving
> > normal high performance. A file that gets written to becomes
"dirty"
> > until checked again. Some mechanism for making a clean copy of a
> > clean file might help avoid excessive "dirtying".
> >
> > "Scan on open" might still be useful for cases when the system
> > haven't kept up with writing, but please don't aim to have
> > this be the _primary_ mode of scanning. A file server
> > where most of the stuff is pre-scanned will likely perform much
better
> > than one scanning everything on open.
It isn't the primary mode. It's the mode that catches things as they
arrive, and as they are about to be used. Most sites will also employ
the applications' features for running regular scans of the whole system
on a scheduled basis, to catch anything that may have slipped through.
Jon Press
Jon Press wrote on 2008-08-15 14:10:21:
> > -----Original Message-----
> > From: [email protected] [mailto:malware-list-
> > [email protected]] On Behalf Of Peter Zijlstra
> > Sent: Friday, August 15, 2008 6:37 AM
> > To: Helge Hafting
> > Cc: [email protected]; [email protected];
> [email protected];
> > [email protected]; [email protected];
> [email protected]; Arjan van
> > de Ven
> > Subject: Re: [malware-list] TALPA - a threat model? well sorta.
> >
> > On Fri, 2008-08-15 at 12:07 +0200, Helge Hafting wrote:
> > > It seems to me that this "scan on file open" business is the
> > > wrong way to do things - because it reduces performance.
> > >
> > > If you scan on file open, then your security sw is too late and
> > > getting in the way.
>
> The problem is that you have to account for the cases where the malware
> made it onto the system even if you were trying to catch it ahead of
> time. For example:
>
> - Administrator turns off or reduces AV protection for some reason for
> some period of time. It happens all the time.
>
> - New infection makes it onto the machine before the signatures have
> caught up with it. This also happens. There is an ongoing PR race
> among AV vendors about who was faster on the draw to get out signatures
> to detect some new malware. The fact that this race exists reflects
> that reality that there is some window during which new malware will
> make it onto some number of machines before the scanners catch up.
>
Not to mention removable media - it might be old hat, but infected/malware
files can come in on floppies, CDs or USB flash discs careless left on the
pavement outside an office.
--
Douglas Leeder
Sophos Plc, The Pentagon, Abingdon Science Park, Abingdon,
OX14 3YP, United Kingdom.
Company Reg No 2096520. VAT Reg No GB 348 3873 20.
On Fri, 15 Aug 2008, Press, Jonathan wrote:
>> -----Original Message-----
>> From: [email protected] [mailto:malware-list-
>> [email protected]] On Behalf Of Peter Zijlstra
>> Sent: Friday, August 15, 2008 6:37 AM
>> To: Helge Hafting
>> Cc: [email protected]; [email protected];
> [email protected];
>> [email protected]; [email protected];
> [email protected]; Arjan van
>> de Ven
>> Subject: Re: [malware-list] TALPA - a threat model? well sorta.
>>
>> On Fri, 2008-08-15 at 12:07 +0200, Helge Hafting wrote:
>>> It seems to me that this "scan on file open" business is the
>>> wrong way to do things - because it reduces performance.
>>>
>>> If you scan on file open, then your security sw is too late and
>>> getting in the way.
>
> The problem is that you have to account for the cases where the malware
> made it onto the system even if you were trying to catch it ahead of
> time. For example:
>
> - Administrator turns off or reduces AV protection for some reason for
> some period of time. It happens all the time.
according to the threat model actions of the administrator do not matter.
David Lang
> -----Original Message-----
> From: [email protected] [mailto:[email protected]]
> Sent: Friday, August 15, 2008 12:25 PM
> To: Press, Jonathan
> Cc: Peter Zijlstra; Helge Hafting; [email protected];
malware-
> [email protected]; [email protected]; [email protected];
> [email protected]; [email protected]; Arjan van de Ven
> Subject: RE: [malware-list] TALPA - a threat model? well sorta.
> > The problem is that you have to account for the cases where the
malware
> > made it onto the system even if you were trying to catch it ahead of
> > time. For example:
> >
> > - Administrator turns off or reduces AV protection for some reason
for
> > some period of time. It happens all the time.
>
> according to the threat model actions of the administrator do not
matter.
Sorry, I don't know what you mean.
Jon
On Fri, 15 Aug 2008, Press, Jonathan wrote:
>> -----Original Message-----
>> From: [email protected] [mailto:[email protected]]
>>> The problem is that you have to account for the cases where the
> malware
>>> made it onto the system even if you were trying to catch it ahead of
>>> time. For example:
>>>
>>> - Administrator turns off or reduces AV protection for some reason
> for
>>> some period of time. It happens all the time.
>>
>> according to the threat model actions of the administrator do not
> matter.
>
> Sorry, I don't know what you mean.
the threat model that was posted two days ago in the initial message of
this thread specificly stated that actions of root are not something that
this is trying to defend against.
David Lang
> -----Original Message-----
> From: [email protected] [mailto:[email protected]]
> Sent: Friday, August 15, 2008 1:33 PM
> To: Press, Jonathan
> Cc: Peter Zijlstra; Helge Hafting; [email protected];
malware-
> [email protected]; [email protected]; [email protected];
> [email protected]; [email protected]; Arjan van de Ven
> Subject: RE: [malware-list] TALPA - a threat model? well sorta.
>
> On Fri, 15 Aug 2008, Press, Jonathan wrote:
>
> >> -----Original Message-----
> >> From: [email protected] [mailto:[email protected]]
> >>> The problem is that you have to account for the cases where the
malware
> >>> made it onto the system even if you were trying to catch it ahead
of
> >>> time. For example:
> >>>
> >>> - Administrator turns off or reduces AV protection for some reason
for
> >>> some period of time. It happens all the time.
> >>
> >> according to the threat model actions of the administrator do not
matter.
> >
> > Sorry, I don't know what you mean.
>
> the threat model that was posted two days ago in the initial message
of
> this thread specificly stated that actions of root are not something
that
> this is trying to defend against.
I think you may have missed the point of any such statement.
Just to clarify...
The model does not exclude root-owned processes from the notification
and scanning sequence. If root attempts to execute a file, that file
would be scanned before the execution is allowed. If a root-owned
process attempts to open a file, that access would be blocked until the
file is scanned. If a root-owned process closes a file that has been
written to, that file would be scanned.
In addition, to generalize from the incorrect idea that the actions of
root are not being defended against to the idea that the possible
impacts of an administrator's actions in configuring an application
should not be accounted for at all in our thinking doesn't make sense to
me anyway.
Jon Press
On Fri, 15 Aug 2008, Press, Jonathan wrote:
>> -----Original Message-----
>> From: [email protected] [mailto:[email protected]]
>>
>> On Fri, 15 Aug 2008, Press, Jonathan wrote:
>>
>>>> -----Original Message-----
>>>> From: [email protected] [mailto:[email protected]]
>>>>> The problem is that you have to account for the cases where the
> malware
>>>>> made it onto the system even if you were trying to catch it ahead
> of
>>>>> time. For example:
>>>>>
>>>>> - Administrator turns off or reduces AV protection for some reason
> for
>>>>> some period of time. It happens all the time.
>>>>
>>>> according to the threat model actions of the administrator do not
> matter.
>>>
>>> Sorry, I don't know what you mean.
>>
>> the threat model that was posted two days ago in the initial message
> of
>> this thread specificly stated that actions of root are not something
> that
>> this is trying to defend against.
>
> I think you may have missed the point of any such statement.
>
> Just to clarify...
>
> The model does not exclude root-owned processes from the notification
> and scanning sequence. If root attempts to execute a file, that file
> would be scanned before the execution is allowed. If a root-owned
> process attempts to open a file, that access would be blocked until the
> file is scanned. If a root-owned process closes a file that has been
> written to, that file would be scanned.
correct so far.
> In addition, to generalize from the incorrect idea that the actions of
> root are not being defended against to the idea that the possible
> impacts of an administrator's actions in configuring an application
> should not be accounted for at all in our thinking doesn't make sense to
> me anyway.
questions had been raised about how this model could defend against all
the tricky things that root can do, the answer was that they are not
trying to defend against root doing tricky things.
turning off the scanner, letting things get infected, and turning it back
on would fall in the same catagory as marking a file that the scanner
marked as bad as sucessfully scanned.
in any case the vunerability is limited as the next time the signatures
are updated the files would get scanned again, so I don't think it's a big
problem in practice.
David Lang
On Fri, Aug 15, 2008 at 02:18:12PM +0100, [email protected] wrote:
> > - New infection makes it onto the machine before the signatures have
> > caught up with it. This also happens. There is an ongoing PR race
> > among AV vendors about who was faster on the draw to get out signatures
> > to detect some new malware. The fact that this race exists reflects
> > that reality that there is some window during which new malware will
> > make it onto some number of machines before the scanners catch up.
Let's go back to the threat model. The Threat Model which Eric Paris
has suggested is that we are only trying to solve the Scanning
Problem. Just Scanning.
That implies if the malware has been written to the disk, we will
catch it once AV catching is turned on and the user attempts to run or
otherwise access the file with the bad content. However, if the
malware starts running, then regardless of whether the malware is
running with user privileges, or manages to get root privileges via
some buffer overflow that wasn't caught via
LSM/SELinux/AppAmor/whatever, this is out of scope of Eric's proposal.
Are we agreed on that? There may be other components of the solution
such as LSM, SELinux, etc., that will very likely be useful in
protecting the system once the malware starts running. But I thought
Eric's proposal proposed excluding that from the Threat Model for the
purposes of the interface we are trying to solve. If that's not true,
let's deal with it now.
> Not to mention removable media - it might be old hat, but infected/malware
> files can come in on floppies, CDs or USB flash discs careless left on the
> pavement outside an office.
That's not a problem given the scanning model proposed by Eric; when
you insert removable media, it will get scanned when it is first
accessed.
- Ted
On Fri, 15 Aug 2008 10:47:15 PDT, [email protected] said:
> in any case the vunerability is limited as the next time the signatures
> are updated the files would get scanned again, so I don't think it's a big
> problem in practice.
This problem is actually identical to "new file scanned, but you don't have
the signature available yet so malware isn't detected".
Those of us who have seen large mail servers pile up queues in the 10s of
millions in the 45 minutes between when the worm went critical-mass and when
we got a signature might disagree on it not being a big problem in practice.
Of course, if that's considered "outside" the threat model, somebody better
start writing down exactly what small corner of threat model this is actually
helping against...
> -----Original Message-----
> From: Theodore Tso [mailto:[email protected]]
> Sent: Friday, August 15, 2008 1:05 PM
> To: [email protected]
> Cc: Press, Jonathan; [email protected]; [email protected];
Arjan van de
> Ven; [email protected]; Helge Hafting; [email protected];
malware-
> [email protected]; Peter Zijlstra; [email protected]
> Subject: Re: [malware-list] TALPA - a threat model? well sorta.
>
> > Not to mention removable media - it might be old hat, but
infected/malware
> > files can come in on floppies, CDs or USB flash discs careless left
on the
> > pavement outside an office.
>
> That's not a problem given the scanning model proposed by Eric; when
> you insert removable media, it will get scanned when it is first
> accessed.
That is exactly the idea. However, the context of this particular
thread was the following statement by Helge Hafting:
It seems to me that this "scan on file open" business is the
wrong
way to do things - because it reduces performance.
If you scan on file open, then your security sw is too late and
getting in the way.
We were just pointing out that this is not a good argument in practical
terms AGAINST scanning on open. In fact, your reply completely
reinforces that point.
Jon Press
> -----Original Message-----
> From: [email protected] [mailto:[email protected]]
> Sent: Friday, August 15, 2008 1:47 PM
> To: Press, Jonathan
> Cc: Peter Zijlstra; Helge Hafting; [email protected];
malware-
> [email protected]; [email protected]; [email protected];
> [email protected]; [email protected]; Arjan van de Ven
> Subject: RE: [malware-list] TALPA - a threat model? well sorta.
>
> On Fri, 15 Aug 2008, Press, Jonathan wrote:
> > In addition, to generalize from the incorrect idea that the actions
of
> > root are not being defended against to the idea that the possible
> > impacts of an administrator's actions in configuring an application
> > should not be accounted for at all in our thinking doesn't make
sense to
> > me anyway.
>
> questions had been raised about how this model could defend against
all
> the tricky things that root can do, the answer was that they are not
> trying to defend against root doing tricky things.
>
> turning off the scanner, letting things get infected, and turning it
back
> on would fall in the same catagory as marking a file that the scanner
> marked as bad as sucessfully scanned.
Well, I agree that there are things you can't prevent, that's for sure.
But the point is to build the "threat model" and application
functionality around the idea that IF they happen, you want to be able
to plug the resulting holes as well as you can. You can't simply close
your eyes to the possibility.
Jon Press
On Fri, 15 Aug 2008, [email protected] wrote:
> On Fri, 15 Aug 2008 10:47:15 PDT, [email protected] said:
>
>> in any case the vunerability is limited as the next time the signatures
>> are updated the files would get scanned again, so I don't think it's a big
>> problem in practice.
>
> This problem is actually identical to "new file scanned, but you don't have
> the signature available yet so malware isn't detected".
>
> Those of us who have seen large mail servers pile up queues in the 10s of
> millions in the 45 minutes between when the worm went critical-mass and when
> we got a signature might disagree on it not being a big problem in practice.
>
> Of course, if that's considered "outside" the threat model, somebody better
> start writing down exactly what small corner of threat model this is actually
> helping against...
go back to the beginning of this thread. that is defining the (very
limited) thread model that they are trying to defend against.
the rest of us are not trying to defend against this threat model. we are
trying to identify the appropriate infrastructure that could be used by
the TALPA folks for their work, that can also be reasonable for inclusion
in the kernel (which includes being suitable for other similar purposes,
such as filesystem indexing)
we agree that the threat they are trying to defend against is a very small
portion of the overall threat, but there are other components (including
SELinux) that are available to deal with other portions of the overall
threat.
if you want to say that the solution is too limited to be worth while,
then you need to write a new threat model that you think is what should be
defended against and then we can start discussing how to defend against
it.
David Lang
On Fri, 15 Aug 2008, Press, Jonathan wrote:
>> -----Original Message-----
>> From: [email protected] [mailto:[email protected]]
>> Sent: Friday, August 15, 2008 1:47 PM
>> To: Press, Jonathan
>> Cc: Peter Zijlstra; Helge Hafting; [email protected];
> malware-
>> [email protected]; [email protected]; [email protected];
>> [email protected]; [email protected]; Arjan van de Ven
>> Subject: RE: [malware-list] TALPA - a threat model? well sorta.
>>
>> On Fri, 15 Aug 2008, Press, Jonathan wrote:
>>> In addition, to generalize from the incorrect idea that the actions
> of
>>> root are not being defended against to the idea that the possible
>>> impacts of an administrator's actions in configuring an application
>>> should not be accounted for at all in our thinking doesn't make
> sense to
>>> me anyway.
>>
>> questions had been raised about how this model could defend against
> all
>> the tricky things that root can do, the answer was that they are not
>> trying to defend against root doing tricky things.
>>
>> turning off the scanner, letting things get infected, and turning it
> back
>> on would fall in the same catagory as marking a file that the scanner
>> marked as bad as sucessfully scanned.
>
> Well, I agree that there are things you can't prevent, that's for sure.
> But the point is to build the "threat model" and application
> functionality around the idea that IF they happen, you want to be able
> to plug the resulting holes as well as you can. You can't simply close
> your eyes to the possibility.
correct, and the threat model that was documented as what we are trying to
defend against explicity says that actions by root (or other running
programs, including running malware) are not part of the threat model they
are trying to address (it doesn't mean that they aren't important, just
that they are not trying to deal with them). All they are trying to do is
to have a way to scan files. Since there is use in having support for this
(for security theater, a little real security, and for other purposes like
indexing) we are figuing out how to provide tools that would allow this.
David Lang
On Wed, Aug 13, 2008 at 05:35:29PM -0400, Rik van Riel wrote:
> On Wed, 13 Aug 2008 17:24:28 -0400
> "Press, Jonathan" <[email protected]> wrote:
>
> > I may be missing something about your suggestion, but I don't see how
> > this would work. Who does the chmod?
>
> Chmod is also not a solution to the hierarchical storage (or incremental
> restore from backup) problem.
>
> I believe we really do need the block-on-open.
Or use either the Fuse or Coda kernel modules and handle such requests
in userspace. With FUSE you should even be able to block on a per-page
granularity, Coda only has session semantics so it will only notify
userspace of the open and close events, while read and write and mmap
are passed directly to the underlying file.
Jan
On Fri, Aug 15, 2008 at 02:06:47PM -0400, [email protected] wrote:
> This problem is actually identical to "new file scanned, but you don't have
> the signature available yet so malware isn't detected".
>
> Those of us who have seen large mail servers pile up queues in the 10s of
> millions in the 45 minutes between when the worm went critical-mass and when
> we got a signature might disagree on it not being a big problem in practice.
For a mail server, I really think something specialized like ClamAV is
a much better solution than something in userspace, which will
probably decide it has to rescan every single file that gets written,
including your mail server logs. :-)
A specialized solution for a mail server is *always* going to be able
to a more efficient, more practical, and be able to do
application-specialized things (such as refusing the e-mail while the
connection is still open, so you don't have to worry about being RFC
compliant about sending bounce mails when the SMTP return-path is most
likely bogus).
- Ted
On Fri, 15 Aug 2008 16:16:22 -0400
Jan Harkes <[email protected]> wrote:
> > I believe we really do need the block-on-open.
I believe more that we need block-on-read ;-)
(on open we can start an async scan to cut latency)
read() (or mmap etc) is where the actual use/transfer of
contaminated data happens, not in the open.
--
If you want to reach me at my work email, use [email protected]
For development, discussion and tips for power savings,
visit http://www.lesswatts.org
On Fri, 2008-08-15 at 15:05 -0700, Arjan van de Ven wrote:
> On Fri, 15 Aug 2008 16:16:22 -0400
> Jan Harkes <[email protected]> wrote:
>
> > > I believe we really do need the block-on-open.
>
> I believe more that we need block-on-read ;-)
> (on open we can start an async scan to cut latency)
>
> read() (or mmap etc) is where the actual use/transfer of
> contaminated data happens, not in the open.
I could probably buy that, but I don't know how an HSM would work.
Would we have everything we need at open for them to fire off?
/me is HSM clueless and trying to include their needs is proving a
challenge.
-Eric
On Sun, 17 Aug 2008 19:19:37 -0400
Eric Paris <[email protected]> wrote:
> On Fri, 2008-08-15 at 15:05 -0700, Arjan van de Ven wrote:
> > On Fri, 15 Aug 2008 16:16:22 -0400
> > Jan Harkes <[email protected]> wrote:
> >
> > > > I believe we really do need the block-on-open.
> >
> > I believe more that we need block-on-read ;-)
> > (on open we can start an async scan to cut latency)
> >
> > read() (or mmap etc) is where the actual use/transfer of
> > contaminated data happens, not in the open.
>
> I could probably buy that, but I don't know how an HSM would work.
> Would we have everything we need at open for them to fire off?
>
> /me is HSM clueless and trying to include their needs is proving a
> challenge.
I bet it's the same; until you read/mmap... no need to get the data.
you wouldn't want an "ls -l" or "find" to restore your entire system
from tape after all :)
--
If you want to reach me at my work email, use [email protected]
For development, discussion and tips for power savings,
visit http://www.lesswatts.org
Arjan van de Ven wrote:
> Eric Paris <[email protected]> wrote:
>>/me is HSM clueless and trying to include their needs is proving a
>>challenge.
>
>
> I bet it's the same; until you read/mmap... no need to get the data.
>
> you wouldn't want an "ls -l" or "find" to restore your entire system
> from tape after all :)
>
The HSMs I'm familiar with keep the inode and the first block of the file on-disk,
so that stat and file commands work. For Windows, you also need to keep the pointer
to the icon to use valid, or create a fake one.
--dave
--
David Collier-Brown | Always do right. This will gratify
Sun Microsystems, Toronto | some people and astonish the rest
[email protected] | -- Mark Twain
cell: (647) 833-9377, bridge: (877) 385-4099 code: 506 9191#
Press, Jonathan wrote:
>> -----Original Message-----
>> From: [email protected] [mailto:malware-list-
>> [email protected]] On Behalf Of Peter Zijlstra
>> Sent: Friday, August 15, 2008 6:37 AM
>> To: Helge Hafting
>> Cc: [email protected]; [email protected];
>>
> [email protected];
>
>> [email protected]; [email protected];
>>
> [email protected]; Arjan van
>
>> de Ven
>> Subject: Re: [malware-list] TALPA - a threat model? well sorta.
>>
>> On Fri, 2008-08-15 at 12:07 +0200, Helge Hafting wrote:
>>
>>> It seems to me that this "scan on file open" business is the
>>> wrong way to do things - because it reduces performance.
>>>
>>> If you scan on file open, then your security sw is too late and
>>> getting in the way.
>>>
>
> The problem is that you have to account for the cases where the malware
> made it onto the system even if you were trying to catch it ahead of
> time. For example:
>
> - Administrator turns off or reduces AV protection for some reason for
> some period of time. It happens all the time.
>
Everything is "dirty" again when AV sw restarts.
> - New infection makes it onto the machine before the signatures have
> caught up with it. This also happens. There is an ongoing PR race
> among AV vendors about who was faster on the draw to get out signatures
> to detect some new malware. The fact that this race exists reflects
> that reality that there is some window during which new malware will
> make it onto some number of machines before the scanners catch up.
>
And when new signatures arrive, everything is dirty again. Well, you can
optimize
and say that everything only need to be checked with the new signatures,
since they passed the old ones already. Maybe that helps performance a
little.
The "clean" concept works fine, just make sure dirtying happens in all these
extra cases.
Helge Hafting
Press, Jonathan wrote:
>> -----Original Message-----
>> From: Theodore Tso [mailto:[email protected]]
>> Sent: Friday, August 15, 2008 1:05 PM
>> To: [email protected]
>> Cc: Press, Jonathan; [email protected]; [email protected];
>>
> Arjan van de
>
>> Ven; [email protected]; Helge Hafting; [email protected];
>>
> malware-
>
>> [email protected]; Peter Zijlstra; [email protected]
>> Subject: Re: [malware-list] TALPA - a threat model? well sorta.
>>
>>
>>> Not to mention removable media - it might be old hat, but
>>>
> infected/malware
>
>>> files can come in on floppies, CDs or USB flash discs careless left
>>>
> on the
>
>>> pavement outside an office.
>>>
>> That's not a problem given the scanning model proposed by Eric; when
>> you insert removable media, it will get scanned when it is first
>> accessed.
>>
>
> That is exactly the idea. However, the context of this particular
> thread was the following statement by Helge Hafting:
>
> It seems to me that this "scan on file open" business is the
> wrong
> way to do things - because it reduces performance.
>
> If you scan on file open, then your security sw is too late and
> getting in the way.
>
>
> We were just pointing out that this is not a good argument in practical
> terms AGAINST scanning on open. In fact, your reply completely
> reinforces that point.
>
Scanning on open should be a last resort. Scan in advance when you can.
Of course, removable media cannot be scanned until it is inserted and
mounted,
that is obvious. The scanning can start as soon as the filesystem is
mounted though,
there is no reason to wait until users try to access something.
A CD inserted into a CD-server may not necessarily be needed immediately, so
scanning in advance will help here too. The user inserting a CD in a home
computer may start to use stuff right away, or perhaps he spends
some time reading the docs before a complicated install. Sill room for some
scanning in advance, which also may end up with the nice effect of
caching the CD.
Helge Hafting
On Mon, 2008-08-18 at 12:09 +0200, Helge Hafting wrote:
>
> Scanning on open should be a last resort. Scan in advance when you can.
> Of course, removable media cannot be scanned until it is inserted and
> mounted,
> that is obvious. The scanning can start as soon as the filesystem is
> mounted though,
> there is no reason to wait until users try to access something.
>
> A CD inserted into a CD-server may not necessarily be needed immediately, so
> scanning in advance will help here too. The user inserting a CD in a home
> computer may start to use stuff right away, or perhaps he spends
> some time reading the docs before a complicated install. Sill room for some
> scanning in advance, which also may end up with the nice effect of
> caching the CD.
Hmm, then there is the issue that an active scanner will avoid me from
removing the media again. Most annoysome when the machine won't promptly
give back the disk.
Imagine me browsing through a stack of unmarked dvd-rw media trying to
find the one with the right file on it, but having to wait every time
for the whole media to get scanned..
Not a nice picture.
Peter Zijlstra wrote on 18/08/2008 11:14:00:
> On Mon, 2008-08-18 at 12:09 +0200, Helge Hafting wrote:
> >
> > Scanning on open should be a last resort. Scan in advance when you
can.
> > Of course, removable media cannot be scanned until it is inserted and
> > mounted,
> > that is obvious. The scanning can start as soon as the filesystem is
> > mounted though,
> > there is no reason to wait until users try to access something.
> >
> > A CD inserted into a CD-server may not necessarily be needed
immediately, so
> > scanning in advance will help here too. The user inserting a CD in a
home
> > computer may start to use stuff right away, or perhaps he spends
> > some time reading the docs before a complicated install. Sill room for
some
> > scanning in advance, which also may end up with the nice effect of
> > caching the CD.
>
> Hmm, then there is the issue that an active scanner will avoid me from
> removing the media again. Most annoysome when the machine won't promptly
> give back the disk.
>
> Imagine me browsing through a stack of unmarked dvd-rw media trying to
> find the one with the right file on it, but having to wait every time
> for the whole media to get scanned..
>
> Not a nice picture.
I agree. And not only that but also the fact that most of the time I would
not want mount to trigger a background scan of everything because not all
might be accessed later. Therefore it would be just wasting resources and
ruining user experience. This argument is flawed in a way that is
unsolvable in the same way normal preload to page cache is unsolvable
because it is impossible to predict the usage pattern.
--
Tvrtko A. Ursulin
Senior Software Engineer, Sophos
"Views and opinions expressed in this email are strictly those of the
author.
The contents has not been reviewed or approved by Sophos."
Sophos Plc, The Pentagon, Abingdon Science Park, Abingdon,
OX14 3YP, United Kingdom.
Company Reg No 2096520. VAT Reg No GB 348 3873 20.
Helge Hafting <[email protected]> wrote on 2008-08-18 11:09:25:
> Scanning on open should be a last resort. Scan in advance when you can.
> Of course, removable media cannot be scanned until it is inserted and
> mounted,
> that is obvious. The scanning can start as soon as the filesystem is
> mounted though,
> there is no reason to wait until users try to access something.
>
> A CD inserted into a CD-server may not necessarily be needed
immediately, so
> scanning in advance will help here too. The user inserting a CD in a
home
> computer may start to use stuff right away, or perhaps he spends
> some time reading the docs before a complicated install. Sill room for
some
> scanning in advance, which also may end up with the nice effect of
> caching the CD.
It might be useful to scan in advance, on mount, or scanner start. But
that
can be simulated using a user-space program, and kernel-based on-open
blocking scans:
Just have a background program open the files, and they will be scanned
and
cached.
The background program could also do clever things like check if running
on battery, check
load level, keep hot-lists of files to scan/scan first.
All of which can be implemented on top of the Talpa interface already
specified.
In practice scan on-open with blocking is fast enough for general usage,
even using our existing
hacky kernel module approach.
--
Douglas Leeder
Sophos Plc, The Pentagon, Abingdon Science Park, Abingdon,
OX14 3YP, United Kingdom.
Company Reg No 2096520. VAT Reg No GB 348 3873 20.
On Thu 2008-08-14 11:50:28, Theodore Tso wrote:
> On Thu, Aug 14, 2008 at 09:48:33AM -0400, Eric Paris wrote:
> >
> > There needs to be a way to say that an inode in cache needs to be
> > rescanned. 3 states this flag can be. Clean, Dirty, Infected. The
> > current talpa solution involves a global monotomically increasing
> > counter every time you change virus defs or make some "interesting"
> > change. If global == inode flag we are clean. If global == negative
> > inode flag we are infected. if global > inode flag we are dirty and
> > need a scan.
>
> "Infected" just means to instantly return an error when the file is
> opened or if an already opened file descriptor is read or mmap'ed,
> right? If file is already mmaped(), what's the plan? Send a kill -9
Me thinks about naming my machine
host-inline-real-virus-or-at-least-identifiable-part-here.cz, then
having fun with people not able to access their apache logs.
Returning an error when you think you see a virus is a great
invitation to DoS attack, really.
Pavel
--
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
Hi!
> Now this to me we have a few basic building blocks:
> 1) We need an efficient mechanism to notify userspace of files that get
> dirtied. Virus scanners will subscribe to this for the async dirty
> scanning; indexing agents also will subscribe to this.
ACK.
> I think few people will disagree about this.
>
> Open questions now are
> 4) do we have the kernel kick off an async scan in open() or do we have
> glibc do this
> 5) do we have the kernel do the sync scan on read/mmap/.. or do we have
> glibc do this
How does it work? Memory can still change after mmap; scanning at the
mmap time is _NOT_ enough.
You could do 'when app attempts to dirty memory, synchronously unmap
it from all apps that have it mapped' and then do sync scan on
pagefault time; but that sounds impractical.
Pavel
--
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
Hi!
> > > a model other than restrictive can make sense with content security
> > > scanning.
> >
> > I'm not sure I understand why "interest in content" means not blocking,
> > and vice versa. However, I think this is a good idea if made more
> > explicit, i.e.:
>
> Small misunderstanding because both would block. If you go back to Ted's
> original post I was replying to, he was worried about how would
> anti-malware scanning interact with HSM since both may end up using the
> same interface. HSM, as far as I understand it, needs to block on open and
> "plant" the right file in place, while anti-malware also needs to block
> and examine the right content. That is why ordering matters, anti-malware
> needs to run after the content is put in place. And that is what my idea
> solves (slight overstatement since I spent only seconds on it) by
> separating them in two groups of clients. First which has no interest in
> content and second which does.
I don't see why HSMs are mixed into this discussion -- they appear
very different. For one, you probably have just one filesystem with
tape storage, so it makes sense to implement HSM at filesystem level;
probably with something like FUSE+unionfs.
Pavel
--
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
On Fri, 15 Aug 2008, Pavel Machek wrote:
>> Now this to me we have a few basic building blocks:
>> 1) We need an efficient mechanism to notify userspace of files that get
>> dirtied. Virus scanners will subscribe to this for the async dirty
>> scanning; indexing agents also will subscribe to this.
>
> ACK.
>
>> I think few people will disagree about this.
>>
>> Open questions now are
>> 4) do we have the kernel kick off an async scan in open() or do we have
>> glibc do this
>> 5) do we have the kernel do the sync scan on read/mmap/.. or do we have
>> glibc do this
>
>
> How does it work? Memory can still change after mmap; scanning at the
> mmap time is _NOT_ enough.
>
> You could do 'when app attempts to dirty memory, synchronously unmap
> it from all apps that have it mapped' and then do sync scan on
> pagefault time; but that sounds impractical.
what is the threat you are trying to defend against?
for some threats you are right, for others the scan at mmap time is
enough.
David Lang
Hi!
> >How does it work? Memory can still change after mmap;
> >scanning at the
> >mmap time is _NOT_ enough.
> >
> >You could do 'when app attempts to dirty memory,
> >synchronously unmap
> >it from all apps that have it mapped' and then do sync
> >scan on
> >pagefault time; but that sounds impractical.
>
> what is the threat you are trying to defend against?
>
> for some threats you are right, for others the scan at
> mmap time is enough.
I don't see any threats when check at mmap time is okay.
As soon as file servers use mmap, this race can bite you even in very
simple 'make sure Linux fileserver does not pass on windows malwar'
threat model.
Pavel
--
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
> I could probably buy that, but I don't know how an HSM would work.
> Would we have everything we need at open for them to fire off?
>
> /me is HSM clueless and trying to include their needs is proving a
> challenge.
So don't bother. Its a theoretical use for the most part so we can mangle
the interface later.
Alan
On Mon, 18 Aug 2008 16:33:13 +0100
Alan Cox <[email protected]> wrote:
> > I could probably buy that, but I don't know how an HSM would work.
> > Would we have everything we need at open for them to fire off?
> >
> > /me is HSM clueless and trying to include their needs is proving a
> > challenge.
>
> So don't bother. Its a theoretical use for the most part so we can
> mangle the interface later.
Think of a consumer HSM application: backup to rsync.net
or Amazon S3.
Instead of waiting for the whole backup to be restored,
you can start using the filesystem immediately. The
block-on-open hook can be used by the restore program
to fetch files from the remote backup site on an
as-needed basis, with a full restore going on in the
background.
If the block-on-open hook can be used for that (even
with additional magic, like creating empty HSM inodes
with a special attr to notify "the data lives elsewhere"),
HSM should be good.
The "data lives elsewhere" bit/xattr/whatever could also
be used on directories, so not even the whole directory
tree would have to be restored right on restore :)
--
All rights reversed.
On Mon, 18 Aug 2008, Pavel Machek wrote:
> Hi!
>
>>> How does it work? Memory can still change after mmap;
>>> scanning at the
>>> mmap time is _NOT_ enough.
>>>
>>> You could do 'when app attempts to dirty memory,
>>> synchronously unmap
>>> it from all apps that have it mapped' and then do sync
>>> scan on
>>> pagefault time; but that sounds impractical.
>>
>> what is the threat you are trying to defend against?
>>
>> for some threats you are right, for others the scan at
>> mmap time is enough.
>
> I don't see any threats when check at mmap time is okay.
>
> As soon as file servers use mmap, this race can bite you even in very
> simple 'make sure Linux fileserver does not pass on windows malwar'
> threat model.
if you check the file at the time you mmap it and don't expect other
programs to change the file under you while you are reading it, there's
nothing to worry about.
shared mmap with one program being able to write to it is a expecially
hard case of the general problem that if someone writes to a file as you
are serving it, you don't know what goes out, it may be part of the old
file and part of the new file. it's just easier to see this problem with
mmap and harder to catch it.
David Lang
On Thu, Aug 14, 2008 at 07:34:30PM -0400, Theodore Tso wrote:
> On Thu, Aug 14, 2008 at 05:21:09PM -0400, J. Bruce Fields wrote:
> > > I have not yet seen code actually using it at all, neither in mainline
> > > nor on one of the many nfs development lists.
> >
> > Oops, I'd love to, and it should be very easy. How do I find out if
> > i_version is supported on a given superblock?
>
> We don't have a way of exporting this fact at the moment. I assume
> the best way would be to add a flag in struct super.
>
> > There's nothing particularly "advanced" about this, by the way--this is
> > a very minor variation on the caching model that nfs has always had, and
> > our nfsv4 server is currently pretty broken without it.
>
> Well, if you're willing to try it out, as I've mentioned on my
> blog[1][2], ext4 is working pretty well on my laptop --- I'm running
> it as my primary filesystem. There are a few problems with ext3
> filesystems converted to use ext4, as opposed to starting afresh via
> "mke2fs -t ext4dev /dev/hdXX" that we've just found in the past week
> (and fixed within a day or two, although they haven't been pushed to
> Linus yet), but overall, it's been pretty stable.
>
> So this would be a good time for someone who is familiar wiht NFSv4 to
> try it out and let us know if the i_version support is as you would
> like in ext4 --- we're in the bugfixing/stablization phase right now,
> so this would be an ideal time to get that feedback. For more
> information, on how to get started, please see:
>
> http://ext4.wiki.kernel.org/index.php/Ext4_Howto
>
> for instructions, and mount the filesystem with the "-o i_version"
> mount option.
Neato, thanks. I set up a toy ext4 filesystem (just a 512 meg sparse
file loopback mounted) on one of my test machines--so I just need to
add the superblock flag and a bit of nfsd code and then I should be able
to try it out.
> > Actually, it's pretty broken even on nfsv2/v3 for filesystems with poor
> > time resolution.
>
> And that's fixed in ext4 as well....
That's an improvement, yes. Of course the time is still updated only
every jiffy, so there's still a race:
file updated
client checks ctime
file updated again within a jiffy
client checks ctime again, concludes file hasn't changed.
--b.