2001-12-06 16:07:37

by Greg Hennessy

[permalink] [raw]
Subject: horrible disk thorughput on itanium

I recently installed a both a Dell dual cpu 2500 server (dual 1.6 ghz
ia32 chips) and a dell 7150 (dual IA64 chips). My users complained
that the disk io speed on the itanium seemed very slow, even though
both servers have a megaraid controller with seagate cheetah
disks. Bonnie also shows the ia64 machine having worse throughput than
the ia32 machine.

[root@hydra bonnie]# cat bonnie.hydra bonnie.leo
-------Sequential Output-------- ---Sequential Input--
--Random--
-Per Char- --Block--- -Rewrite-- -Per Char- --Block---
--Seeks---
Machine MB K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU
/sec %CPU
100 1765 100.0 282891 100.1 377295 100.0 2058 100.0 592709
99.5 51920.4 196.5
-------Sequential Output-------- ---Sequential Input--
--Random--
-Per Char- --Block--- -Rewrite-- -Per Char- --Block---
--Seeks---
Machine MB K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU
/sec %CPU
100 17049 100.1 265197 101.0 197094 98.2 16631 100.4 675831
99.0 40400.0 191.9

Hydra is the itanium, leo is the 32 bit machine. The character io of
hydra is a factor of 10 slower than that of leo. Is this more likely a
kernel issue, or a glibc issue? Both machiness run standard redhat
7.1, and 2.4.9-12smp kernels.



2001-12-06 16:18:19

by Arjan van de Ven

[permalink] [raw]
Subject: Re: horrible disk thorughput on itanium

Greg Hennessy wrote:
>
> I recently installed a both a Dell dual cpu 2500 server (dual 1.6 ghz
> ia32 chips) and a dell 7150 (dual IA64 chips). My users complained
> that the disk io speed on the itanium seemed very slow, even though
> both servers have a megaraid controller with seagate cheetah
> disks. Bonnie also shows the ia64 machine having worse throughput than
> the ia32 machine.

How much RAM do you have ?

2001-12-06 20:50:16

by Andrew Morton

[permalink] [raw]
Subject: Re: horrible disk thorughput on itanium

Greg Hennessy wrote:
>
> ...
> Hydra is the itanium, leo is the 32 bit machine. The character io of
> hydra is a factor of 10 slower than that of leo. Is this more likely a
> kernel issue, or a glibc issue? Both machiness run standard redhat
> 7.1, and 2.4.9-12smp kernels.
>

The character I/O part of bonnie++ writes a single character at a time,
via stdio. It's more a test of your C library than of the kernel.

The fact that you get the same throughput on each platform with
the block I/O part of the test indicates that the hardware and
kernel are OK, but the C library is broken.

Not sure how to diagnose this. Probably you should write a
simple five-line stdio-based test program, see if that exhibits
the same behaviour, then fiddle with setvbuf().

-

2001-12-06 22:14:07

by Andi Kleen

[permalink] [raw]
Subject: Re: horrible disk thorughput on itanium

Andrew Morton <[email protected]> writes:
>
> The fact that you get the same throughput on each platform with
> the block I/O part of the test indicates that the hardware and
> kernel are OK, but the C library is broken.

The usual difference is if you have a pthreads capable C library
or not. For newer glibc bonnie++ should definitely use
putc_unlocked(); otherwise it'll eat lock overhead for each character
to take the FILE lock.

As far as I can see bonnie++ doesn't use putc_unlocked, but putc.

With libc5 it likely would magically get a lot faster @)

-Andi

2001-12-07 06:09:33

by Mike Galbraith

[permalink] [raw]
Subject: Re: horrible disk thorughput on itanium

On 6 Dec 2001, Andi Kleen wrote:

> Andrew Morton <[email protected]> writes:
> >
> > The fact that you get the same throughput on each platform with
> > the block I/O part of the test indicates that the hardware and
> > kernel are OK, but the C library is broken.
>
> The usual difference is if you have a pthreads capable C library
> or not. For newer glibc bonnie++ should definitely use
> putc_unlocked(); otherwise it'll eat lock overhead for each character
> to take the FILE lock.
>
> As far as I can see bonnie++ doesn't use putc_unlocked, but putc.

Plain old Bonnie suffered from the same thing. I long ago made it
use putc_unlocked() here because throughput was horrible otherwise.

-Mike

2001-12-07 06:21:58

by Linus Torvalds

[permalink] [raw]
Subject: Re: horrible disk thorughput on itanium

In article <[email protected]>,
Mike Galbraith <[email protected]> wrote:
>
>> Andrew Morton <[email protected]> writes:
>>
>> As far as I can see bonnie++ doesn't use putc_unlocked, but putc.
>
>Plain old Bonnie suffered from the same thing. I long ago made it
>use putc_unlocked() here because throughput was horrible otherwise.

Oh, yeah, blame it on bonnie.

"Our C library 'putc' is horribly sucky"

"Well, then, use something else then".

Isn't somebody ashamed of glibc and willing to try to fix it? It might
be as simple as just testing a static flag "have I used pthread_create"
or even a function pointer that gets switched around at pthread_create..

"putc()" is a standard function. If it sucks, let's get it fixed. And
instead of changing bonnie, how about pinging the _real_ people who
write sucky code?

Linus

2001-12-07 06:41:24

by Dan Kegel

[permalink] [raw]
Subject: Re: horrible disk thorughput on itanium

Linus wrote:
> >> As far as I can see bonnie++ doesn't use putc_unlocked, but putc.
> >
> >Plain old Bonnie suffered from the same thing. I long ago made it
> >use putc_unlocked() here because throughput was horrible otherwise.
>
> Oh, yeah, blame it on bonnie.
>
> "Our C library 'putc' is horribly sucky"
>
> "Well, then, use something else then".
>
> Isn't somebody ashamed of glibc and willing to try to fix it? It might
> be as simple as just testing a static flag "have I used pthread_create"
> or even a function pointer that gets switched around at pthread_create..

That sounds racy. Better to make the change at compile time, maybe?
Say,

#ifdef __USE_REENTRANT
#define putc(_ch, _fp) _IO_putc (_ch, _fp)
#else
#define putc(_ch, _fp) _IO_putc_unlocked (_ch, _fp)
#endif

That's pedantically safe, I think.

- Dan

2001-12-07 12:32:48

by Greg Hennessy

[permalink] [raw]
Subject: Re: horrible disk thorughput on itanium

In article <[email protected]>,
Linus Torvalds <[email protected]> wrote:
> Isn't somebody ashamed of glibc and willing to try to fix it? It might
> be as simple as just testing a static flag "have I used pthread_create"
> or even a function pointer that gets switched around at pthread_create..

As the person who started this thread, I'll say that I'm willing to
test new alternatives, Redhat engineers gave me a newer kernel to see
if it helped (it didn't) and if someone can give me (or point me to) a
glibc with better io I'm glad.

Right now I have to explain to my boss why my $4K pentium computers do
io faster than my $20K itanium computer. And since our major software
code is 3rd party, we can't rewrite the appilcation.




2001-12-07 13:55:13

by Andi Kleen

[permalink] [raw]
Subject: Re: horrible disk thorughput on itanium

[email protected] (Linus Torvalds) writes:
>
> "putc()" is a standard function. If it sucks, let's get it fixed. And
> instead of changing bonnie, how about pinging the _real_ people who
> write sucky code?

It is easy to fix. Just do #define putc putc_unlocked
There is just a slight problem: it'll fail if your application is threaded
and wants to use the same FILE from multiple threads.

It is a common problem on all OS that eventually got threadsafe stdio.
I bet putc sucks on Solaris too.

-Andi

2001-12-07 14:21:20

by Padraig Brady

[permalink] [raw]
Subject: Re: horrible disk thorughput on itanium

Andi Kleen wrote:

> [email protected] (Linus Torvalds) writes:
>
>>"putc()" is a standard function. If it sucks, let's get it fixed. And
>>instead of changing bonnie, how about pinging the _real_ people who
>>write sucky code?
>>
>
> It is easy to fix. Just do #define putc putc_unlocked
> There is just a slight problem: it'll fail if your application is threaded
> and wants to use the same FILE from multiple threads.
>
> It is a common problem on all OS that eventually got threadsafe stdio.
> I bet putc sucks on Solaris too.
>
> -Andi


Interesting thread on this:
http://sources.redhat.com/ml/bug-glibc/2001-11/msg00079.html

for glibc 2.2.4 with the following program with input file
of 354371 lines of text(/usr/share/doc/*), where the average line
length was 37 chars.

getc/putc
real
2.181s
user
2.150s
sys
0.030s
getc_unlocked/putc_unlocked
real
0.326s
user
0.280s
sys
0.040s

I.E. 669% faster!

Padraig.

-------------------
#include <stdio.h>

#ifndef _REENTRANT
# undef getc
# define getc(x) getc_unlocked(x)
# undef putc
# define putc(x,y) putc_unlocked(x,y)
#endif //_REENTRANT

void main(void)
{
int c;
while((c=getc(stdin))!=EOF) putc(c,stdout);
}


2001-12-07 16:14:46

by Richard Gooch

[permalink] [raw]
Subject: Re: horrible disk thorughput on itanium

Andi Kleen writes:
> [email protected] (Linus Torvalds) writes:
> >
> > "putc()" is a standard function. If it sucks, let's get it fixed. And
> > instead of changing bonnie, how about pinging the _real_ people who
> > write sucky code?
>
> It is easy to fix. Just do #define putc putc_unlocked
> There is just a slight problem: it'll fail if your application is threaded
> and wants to use the same FILE from multiple threads.
>
> It is a common problem on all OS that eventually got threadsafe stdio.
> I bet putc sucks on Solaris too.

This kind of thing should be covered by _REENTRANT. If you don't
compile with _REENTRANT (the default), then putc() should be the
unlocked version.

Regards,

Richard....
Permanent: [email protected]
Current: [email protected]

2001-12-07 17:19:38

by Robert Love

[permalink] [raw]
Subject: Re: horrible disk thorughput on itanium

On Fri, 2001-12-07 at 11:14, Richard Gooch wrote:

> This kind of thing should be covered by _REENTRANT. If you don't
> compile with _REENTRANT (the default), then putc() should be the
> unlocked version.

The link to the mailing list post from bug-glibc says otherwise, that is
the problem. Using the unlocked version isn't implied by not setting
__REENTRANT.

Robert Love

2001-12-07 17:41:10

by Richard Gooch

[permalink] [raw]
Subject: Re: horrible disk thorughput on itanium

Robert Love writes:
> On Fri, 2001-12-07 at 11:14, Richard Gooch wrote:
>
> > This kind of thing should be covered by _REENTRANT. If you don't
> > compile with _REENTRANT (the default), then putc() should be the
> > unlocked version.
>
> The link to the mailing list post from bug-glibc says otherwise,
> that is the problem. Using the unlocked version isn't implied by
> not setting __REENTRANT.

The bug is in glibc. An application shouldn't need to be changed to
work around that bug. putc() is a well-known interface, and people
shouldn't have to code around a change in that interface.

Regards,

Richard....
Permanent: [email protected]
Current: [email protected]

2001-12-07 17:50:58

by Linus Torvalds

[permalink] [raw]
Subject: Re: horrible disk thorughput on itanium


On 7 Dec 2001, Andi Kleen wrote:
> [email protected] (Linus Torvalds) writes:
> >
> > "putc()" is a standard function. If it sucks, let's get it fixed. And
> > instead of changing bonnie, how about pinging the _real_ people who
> > write sucky code?
>
> It is easy to fix. Just do #define putc putc_unlocked

Sure. And why don't you also do

#define sin(x) (1)
#define sqrt(x) (1)
#define strlen(x) (1)
...

to make other benchmarks happier?

bonnie is a _benchmark_. It's meant for finding bad performance. Changing
it to make it work better when performance is bad is _pointless_. You've
now made the whole point of bonnie go away.

> There is just a slight problem: it'll fail if your application is threaded
> and wants to use the same FILE from multiple threads.
>
> It is a common problem on all OS that eventually got threadsafe stdio.

It's a common problem with bad programming.

You can be thread-safe without sucking dead baby donkeys through a straw.
I already mentioned two possible ways to fix it so that you have locking
when you need to, and no locking when you don't.

Linus

2001-12-07 17:49:25

by Robert Love

[permalink] [raw]
Subject: Re: horrible disk thorughput on itanium

On Fri, 2001-12-07 at 12:40, Richard Gooch wrote:

> > The link to the mailing list post from bug-glibc says otherwise,
> > that is the problem. Using the unlocked version isn't implied by
> > not setting __REENTRANT.
>
> The bug is in glibc. An application shouldn't need to be changed to
> work around that bug. putc() is a well-known interface, and people
> shouldn't have to code around a change in that interface.

Right. That's why I referenced a post on bug-glibc and called the issue
a problem. I'm not defending the heaping mass known as glibc ... it
should be fixed.

Robert Love

2001-12-07 17:57:45

by Marco Colombo

[permalink] [raw]
Subject: Re: horrible disk thorughput on itanium

On Fri, 7 Dec 2001, Greg Hennessy wrote:

> In article <[email protected]>,
> Linus Torvalds <[email protected]> wrote:
> > Isn't somebody ashamed of glibc and willing to try to fix it? It might
> > be as simple as just testing a static flag "have I used pthread_create"
> > or even a function pointer that gets switched around at pthread_create..
>
> As the person who started this thread, I'll say that I'm willing to
> test new alternatives, Redhat engineers gave me a newer kernel to see
> if it helped (it didn't) and if someone can give me (or point me to) a
> glibc with better io I'm glad.
>
> Right now I have to explain to my boss why my $4K pentium computers do
> io faster than my $20K itanium computer. And since our major software
> code is 3rd party, we can't rewrite the appilcation.

As I understand it, they just told you that the benchmark you use gives
uncorrect data on the itanium box. IIRC, you said that users were
complaing about poor performance, so I think you should investigate on
the application(s) you need instead of bonnie. Try and collect more real
world data (if possible) or at least use some other benchmarks.

If it turns out that the (hi-end, I presume) application you're using
it's both disk I/O bound and it actually uses putc(), then maybe the
problem it's just using 3rd party applications without sources... all you
have to explain to your boss is that you know what the problem is, and how
to fix it, and that the "fix" is really easy (... if only you had the
source). Add those 20K to the TCO of your closed source application.

.TM.
--
____/ ____/ /
/ / / Marco Colombo
___/ ___ / / Technical Manager
/ / / ESI s.r.l.
_____/ _____/ _/ [email protected]

2001-12-07 17:59:15

by Andi Kleen

[permalink] [raw]
Subject: Re: horrible disk thorughput on itanium

> You can be thread-safe without sucking dead baby donkeys through a straw.
> I already mentioned two possible ways to fix it so that you have locking
> when you need to, and no locking when you don't.

Your proposals sound rather dangerous. They would silently break recompiled
threaded programs that need the locking and don't use -D__REENTRANT (most
people do not seem to use it). I doubt the possible pain from that is
worth it for speeding up an basically obsolete interface like putc.

i.e. if someone wants speed they definitely shouldn't use putc()

-Andi

2001-12-07 18:15:03

by Michael Poole

[permalink] [raw]
Subject: Re: horrible disk thorughput on itanium

Andi Kleen <[email protected]> writes:

> > You can be thread-safe without sucking dead baby donkeys through a straw.
> > I already mentioned two possible ways to fix it so that you have locking
> > when you need to, and no locking when you don't.
>
> Your proposals sound rather dangerous. They would silently break recompiled
> threaded programs that need the locking and don't use -D__REENTRANT (most
> people do not seem to use it). I doubt the possible pain from that is
> worth it for speeding up an basically obsolete interface like putc.
>
> i.e. if someone wants speed they definitely shouldn't use putc()

Threaded programs that need locking and don't define _THREAD_SAFE or
_REENTRANT or whatever is appropriate are already broken -- they just
don't know it yet.

FreeBSD #defines putc and getc to their unlocked versions unless
_THREAD_SAFE is defined, and people don't seem to think its libc is
broken. Many lightly threaded programs, in fact, wouldn't need or
even want the locked variants to be the default. One app I've worked
with only reads and writes any given FILE* from one thread, and I saw
an 4x speedup by switching to the unlocked variants.

It's generally a bad idea to make people pay for a feature they don't
ask for. FreeBSD's libc understands this; glibc apaprently doesn't.

-- Michael Poole

2001-12-07 18:21:50

by Linus Torvalds

[permalink] [raw]
Subject: Re: horrible disk thorughput on itanium


On Fri, 7 Dec 2001, Andi Kleen wrote:
>
> Your proposals sound rather dangerous. They would silently break recompiled
> threaded programs that need the locking and don't use -D__REENTRANT

No it wouldn't.

Once you do a pthread_create(), the locking is there.

Before you do a pthread_create(), it doesn't lock.

What's the problem? Before you do a pthread_create(), you don't _NEED_
locking, because there is only one thread that accesses the stdio data
structures.

And there are no races - if there is only one thread, then another thread
couldn't be suddenly doing a pthread_create() during a stdio operations.

Safe, and efficient. Yes, it adds a flag test or a indirect branch, but
considering that you avoid a serialized locking instruction, the
optimization sounds obvious.

Linus

2001-12-07 18:33:40

by Padraig Brady

[permalink] [raw]
Subject: Re: horrible disk thorughput on itanium

Andi Kleen wrote:

>>You can be thread-safe without sucking dead baby donkeys through a straw.
>>I already mentioned two possible ways to fix it so that you have locking
>>when you need to, and no locking when you don't.
>>
>
> Your proposals sound rather dangerous. They would silently break recompiled
> threaded programs that need the locking and don't use -D__REENTRANT (most
> people do not seem to use it).


I would worry about threaded progs that don't -D_REENTRANT as
they are broken.

> I doubt the possible pain from that is
> worth it for speeding up an basically obsolete interface like putc.
>
> i.e. if someone wants speed they definitely shouldn't use putc()

It's not just putc, it's all of stdio.

Padraig.

2001-12-07 18:36:32

by Padraig Brady

[permalink] [raw]
Subject: Re: horrible disk thorughput on itanium

Michael Poole wrote:

> Andi Kleen <[email protected]> writes:
>
>
>>>You can be thread-safe without sucking dead baby donkeys through a straw.
>>>I already mentioned two possible ways to fix it so that you have locking
>>>when you need to, and no locking when you don't.
>>>
>>Your proposals sound rather dangerous. They would silently break recompiled
>>threaded programs that need the locking and don't use -D__REENTRANT (most
>>people do not seem to use it). I doubt the possible pain from that is
>>worth it for speeding up an basically obsolete interface like putc.
>>
>>i.e. if someone wants speed they definitely shouldn't use putc()
>>
>
> Threaded programs that need locking and don't define _THREAD_SAFE or
> _REENTRANT or whatever is appropriate are already broken -- they just
> don't know it yet.
>
> FreeBSD #defines putc and getc to their unlocked versions unless
> _THREAD_SAFE is defined, and people don't seem to think its libc is
> broken. Many lightly threaded programs, in fact, wouldn't need or
> even want the locked variants to be the default. One app I've worked
> with only reads and writes any given FILE* from one thread, and I saw
> an 4x speedup by switching to the unlocked variants.

This breaks for the case discussed @
http://sources.redhat.com/ml/bug-glibc/2001-11/msg00079.html
I.E. if you have a multithreaded lib being linked by
single threaded apps (Note multithreaded lib, not just a
threadsafe lib (I.E. the lib calls pthread_create())).

Padraig.

2001-12-07 18:41:59

by Padraig Brady

[permalink] [raw]
Subject: Re: horrible disk thorughput on itanium

Linus Torvalds wrote:

> On Fri, 7 Dec 2001, Andi Kleen wrote:
>
>>Your proposals sound rather dangerous. They would silently break recompiled
>>threaded programs that need the locking and don't use -D__REENTRANT
>>
>
> No it wouldn't.
>
> Once you do a pthread_create(), the locking is there.
>
> Before you do a pthread_create(), it doesn't lock.
>
> What's the problem? Before you do a pthread_create(), you don't _NEED_
> locking, because there is only one thread that accesses the stdio data
> structures.
>
> And there are no races - if there is only one thread, then another thread
> couldn't be suddenly doing a pthread_create() during a stdio operations.
>
> Safe, and efficient. Yes, it adds a flag test or a indirect branch, but
> considering that you avoid a serialized locking instruction, the
> optimization sounds obvious.
>
> Linus



2001-12-07 20:42:34

by David Miller

[permalink] [raw]
Subject: Re: horrible disk thorughput on itanium

From: Andi Kleen <[email protected]>
Date: 07 Dec 2001 14:54:49 +0100

It is a common problem on all OS that eventually got threadsafe stdio.
I bet putc sucks on Solaris too.

Nope, they even inline the full implementation it in their header
files so they beat even our putc_unlocked() to the point it is
embarassing.

2001-12-07 20:44:24

by Greg Hennessy

[permalink] [raw]
Subject: Re: horrible disk thorughput on itanium

In article <[email protected]>,
Linus Torvalds <[email protected]> wrote:
> bonnie is a _benchmark_. It's meant for finding bad performance. Changing
> it to make it work better when performance is bad is _pointless_. You've
> now made the whole point of bonnie go away.

It isn't just bonnie showing bad performance. My application shows it,
bonnie shows it, and tiobench shows it. I think the focus on putc
may be too intense. It isn't suprizing to me that either the kernel
or glibc may not be optimised on ia64, I'm just trying to figure out
how to get better io rates out of my itanium machine.



2001-12-07 21:23:19

by Michael Poole

[permalink] [raw]
Subject: Re: horrible disk thorughput on itanium

Padraig Brady <[email protected]> writes:

> This breaks for the case discussed @
> http://sources.redhat.com/ml/bug-glibc/2001-11/msg00079.html
> I.E. if you have a multithreaded lib being linked by
> single threaded apps (Note multithreaded lib, not just a
> threadsafe lib (I.E. the lib calls pthread_create())).

That's an interesting, but very contrived, example. Can you find a
single multi-threaded lib that uses FILE*'s shared with the
application using it?

Linus's suggestion to add hooks to pthread_create() gets around that
problem, anyway. Alternatively, the multi-threaded library could
require any application linking to it to define _REENTRANT.

After all, it's silly to talk about a 'multi-threaded' library linked
to a 'single-threaded' application -- the application plus any
libraries, as a whole, are either multithreaded or not. They have to
be on the same page to deal with *any* locking issues.

-- Michael

2001-12-07 21:37:53

by Padraig Brady

[permalink] [raw]
Subject: Re: horrible disk thorughput on itanium

Michael Poole wrote:

> Padraig Brady <[email protected]> writes:
>
>
>>This breaks for the case discussed @
>>http://sources.redhat.com/ml/bug-glibc/2001-11/msg00079.html
>>I.E. if you have a multithreaded lib being linked by
>>single threaded apps (Note multithreaded lib, not just a
>>threadsafe lib (I.E. the lib calls pthread_create())).
>>
>
> That's an interesting, but very contrived, example. Can you find a
> single multi-threaded lib that uses FILE*'s shared with the
> application using it?


Well you have to deal with the general case. A single threaded

app linking against a multithreaded lib. It mightn't be just
shared FILE*'s that could cause problems.


> Linus's suggestion to add hooks to pthread_create() gets around that
> problem, anyway.


I don't think this will work as I said before current apps that
use _unlocked() functions directly manipulate the stdio structures,
hence a "new smarter locking stdio" would never get used by existing
compiled apps.

> Alternatively, the multi-threaded library could
> require any application linking to it to define _REENTRANT.


It could, but what if an existing interface (lib) is changed

from signle to multithreaded. You can't preclude this.


> After all, it's silly to talk about a 'multi-threaded' library linked
> to a 'single-threaded' application -- the application plus any
> libraries, as a whole, are either multithreaded or not. They have to
> be on the same page to deal with *any* locking issues.
>
> -- Michael

Padraig.

2001-12-07 21:38:09

by Marco Colombo

[permalink] [raw]
Subject: Re: horrible disk thorughput on itanium

On Fri, 7 Dec 2001, Greg Hennessy wrote:

> In article <[email protected]>,
> Linus Torvalds <[email protected]> wrote:
> > bonnie is a _benchmark_. It's meant for finding bad performance. Changing
> > it to make it work better when performance is bad is _pointless_. You've
> > now made the whole point of bonnie go away.
>
> It isn't just bonnie showing bad performance. My application shows it,
> bonnie shows it, and tiobench shows it. I think the focus on putc
> may be too intense. It isn't suprizing to me that either the kernel
> or glibc may not be optimised on ia64, I'm just trying to figure out
> how to get better io rates out of my itanium machine.

Does a simple 'dd' show the problem? I mean,

time dd if=/dev/zero of=/somelargefile count=somelargenumber bs=8k

is it much slower on the itanium, too? dd doesn't use putc(), I hope.

Just for comparison, I've run the following command here:

# time dd if=/dev/zero of=/u2/test count=250000 bs=8k
250000+0 records in
250000+0 records out
0.14user 12.95system 1:23.15elapsed 15%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (117major+18minor)pagefaults 0swaps

(Almost idle box, 256MB RAM, UDMA disk, 2.2.x kernel)

You may try with a bigger file, expecially if you have more RAM.

.TM.
--
____/ ____/ /
/ / / Marco Colombo
___/ ___ / / Technical Manager
/ / / ESI s.r.l.
_____/ _____/ _/ [email protected]

2001-12-07 22:26:53

by Michael Poole

[permalink] [raw]
Subject: Re: horrible disk thorughput on itanium

Padraig Brady <[email protected]> writes:

> Well you have to deal with the general case. A single threaded
> app linking against a multithreaded lib. It mightn't be just
> shared FILE*'s that could cause problems.

The question was about putc(), and presumably other stdio functions.
They deal with FILE*'s. They are also commonly used and small
operations, so forcing locking on an app that doesn't ask for it can
(and has in the past) cause a major performance degredation. I bet
you could find very similar results for memory allocation in other
applications.

> > Linus's suggestion to add hooks to pthread_create() gets around that
> > problem, anyway.
>
> I don't think this will work as I said before current apps that
> use _unlocked() functions directly manipulate the stdio structures,
> hence a "new smarter locking stdio" would never get used by existing
> compiled apps.

As Linus pointed out, you just need another test in those macros, so
they can be forced to call functions rather than using inline code.
When you call a function, it can use whatever new smarter locking
library you want.

> > Alternatively, the multi-threaded library could
> > require any application linking to it to define _REENTRANT.
>
> It could, but what if an existing interface (lib) is changed
> from signle to multithreaded. You can't preclude this.

I certainly can preclude that. I'd consider adding threads and their
locking considerations to the library a change in the API and ABI --
and quite a big one at that. If you change the ABI (rather than just
extending it), you must increase the major version number, which
prevents linking with those applications that expect the semantics of
earlier versions.

(BSD variants have in the past had both libc and libc_r, where only
libc_r makes you pay locking overhead. This is yet another way to
enforce the ABI separation between single- and multi-threaded
applications, with different tradeoffs than the others mentioned.)

-- Michael

2001-12-09 01:19:35

by Tom Vier

[permalink] [raw]
Subject: [OT] fputc vs putc Re: horrible disk thorughput on itanium

On Fri, Dec 07, 2001 at 01:03:16PM -0800, David S. Miller wrote:
> They do it also for putc() if you haven't defined the appropriate
> defines.

what exactly is the difference between putc() and fputc()? the man page is
very vague.

--
Tom Vier <[email protected]>
DSA Key id 0x27371A2C

2001-12-09 01:53:56

by Eric W. Biederman

[permalink] [raw]
Subject: Re: [OT] fputc vs putc Re: horrible disk thorughput on itanium

Tom Vier <[email protected]> writes:

> On Fri, Dec 07, 2001 at 01:03:16PM -0800, David S. Miller wrote:
> > They do it also for putc() if you haven't defined the appropriate
> > defines.
>
> what exactly is the difference between putc() and fputc()? the man page is
> very vague.

fputc is a function putc is a macro because the overhead of a function call
in writing a character has always been a problem...

Eric

2001-12-09 09:28:30

by H. Peter Anvin

[permalink] [raw]
Subject: Re: [OT] fputc vs putc Re: horrible disk thorughput on itanium

Followup to: <[email protected]>
By author: [email protected] (Eric W. Biederman)
In newsgroup: linux.dev.kernel
>
> fputc is a function putc is a macro because the overhead of a function call
> in writing a character has always been a problem...
>

putc() is frequently defined as

#define putc(__C) fputc((__C), stdout)

... or some equivalent; I think the best way to say it's that it's a
shorthand.

-hpa
--
<[email protected]> at work, <[email protected]> in private!
"Unix gives you enough rope to shoot yourself in the foot."
http://www.zytor.com/~hpa/puzzle.txt <[email protected]>

2001-12-09 19:14:31

by Tom Vier

[permalink] [raw]
Subject: Re: [OT] fputc vs putc Re: horrible disk thorughput on itanium

On Sun, Dec 09, 2001 at 01:27:51AM -0800, H. Peter Anvin wrote:
> putc() is frequently defined as
>
> #define putc(__C) fputc((__C), stdout)
>
> ... or some equivalent; I think the best way to say it's that it's a
> shorthand.

according to the putc man page in debian stable, it takes the same args as
fputc. maybe it varies by glibc version (mine is 2.1.3-19). i guess anyone
using putc better use autoconf. also, "unix system programming for SVr4"
says the only difference is that putc is an inlined macro version of fputc.

--
Tom Vier <[email protected]>
DSA Key id 0x27371A2C

2001-12-09 22:16:06

by H. Peter Anvin

[permalink] [raw]
Subject: Re: [OT] fputc vs putc Re: horrible disk thorughput on itanium

Followup to: <20011209141408.A17671@zero>
By author: Tom Vier <[email protected]>
In newsgroup: linux.dev.kernel
>
> according to the putc man page in debian stable, it takes the same args as
> fputc. maybe it varies by glibc version (mine is 2.1.3-19). i guess anyone
> using putc better use autoconf. also, "unix system programming for SVr4"
> says the only difference is that putc is an inlined macro version of fputc.
>

Nevermind. I was thinking about putc() and putchar(), additionally
confused by the annoying fact that fputs() is to putc() as puts() is
to putchar()... makes no fscking sense.

-hpa
--
<[email protected]> at work, <[email protected]> in private!
"Unix gives you enough rope to shoot yourself in the foot."
http://www.zytor.com/~hpa/puzzle.txt <[email protected]>

2001-12-09 23:14:29

by Kurt Garloff

[permalink] [raw]
Subject: Re: horrible disk thorughput on itanium

Hi,

On Thu, Dec 06, 2001 at 11:07:14AM -0500, Greg Hennessy wrote:
> [root@hydra bonnie]# cat bonnie.hydra bonnie.leo
[...]
> --Seeks---
> Machine MB K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU
> /sec %CPU
> 100 1765 100.0 282891 100.1 377295 100.0 2058 100.0 592709
> 99.5 51920.4 196.5

I bet you have more than 100MB RAM, so you measure memory performance
instead of disk performance in the block reads and writes. Same for seeks.
Don't do that. Always use at least twice you ram size if you're interested
in seeing your disk speed.
Newer bonnies warn you about it. The current release is 1.2.
http://www.garloff.de/kurt/linux/bonnie/

The char reads/writes seem to suffer glibc overhead.
You can use the _unlocked variants with option -u and see whether this makes
a difference.

Regards,
--
Kurt Garloff <[email protected]> Eindhoven, NL
GPG key: See mail header, key servers Linux kernel development
SuSE GmbH, Nuernberg, DE SCSI, Security


Attachments:
(No filename) (1.02 kB)
(No filename) (232.00 B)
Download all attachments