There's a lot of drivers in the tree that allow you to access the device
either via IO port space or IO mem space. Here's some examples:
config SCSI_SYM53C8XX_IOMAPPED
bool "use normal IO"
depends on SCSI_SYM53C8XX_2
help
If you say Y here, the driver will preferently use normal IO
rather than memory mapped IO.
config 8139TOO_PIO
bool "Use PIO instead of MMIO"
depends on 8139TOO
help
This instructs the driver to use programmed I/O ports (PIO) instead
of PCI shared memory (MMIO). This can possibly solve some problems
in case your mainboard has memory consistency issues. If unsure,
say N.
config TULIP_MMIO
bool "Use PCI shared mem for NIC registers"
depends on TULIP
help
Use PCI shared memory for the NIC registers, rather than going through
the Tulip's PIO (programmed I/O ports). Faster, but could produce
obscure bugs if your mainboard has memory controller timing issues.
If in doubt, say N.
As you can see, everybody asks the question differently, and sometimes
you should answer Y and sometimes N to get MMIO. There are other drivers
which don't ask the question at all and you have to edit the .c file to
turn it on or off. I'll also note the acronym `PIO' is ambiguous --
sometimes it refers to Port IO and sometimes to Programmed IO (which
can be a memory access).
Personally, when I'm configuring a new kernel for a machine, I don't
want to learn about the suboptions for each device -- I've got dozens of
other things to configure right. My favourite solution to this problem
wuld be to ask the question once only and have it apply to all devices:
config MMIO
bool "Prefer Memory-mapped I/O accesses over Port I/O"
help
Many devices are accessible via both memory mapped I/O and
port I/O. Say Y here to access them via the slightly faster
memory mapped I/O method. If you experience problems, you may
wish to say N here.
If that's not acceptable, can I suggest that we at least ask a standard
question?
config WWWW_MMIO
bool "Use Memory mapped I/O in preference to Port I/O"
depends on WWWW
help
This device's registers can be accessed by either memory
mapped I/O or port I/O. Memory mapped I/O is faster, so you
are advised to say Y here.
(yes, we could wordsmith these questions into the ground; the key point
of this mail is whether we ask one question at the start of configuration
or whether we ask one question per device.)
--
"It's not Hollywood. War is real, war is primarily not about defeat or
victory, it is about death. I've seen thousands and thousands of dead bodies.
Do you think I want to have an academic debate on this subject?" -- Robert Fisk
Matthew Wilcox <[email protected]> writes:
> other things to configure right. My favourite solution to this problem
> wuld be to ask the question once only and have it apply to all devices:
What if need different settings for different drivers? I'd go for a
standard question applied to all drivers that have the option.
--
M?ns Rullg?rd
[email protected]
Matthew Wilcox <[email protected]> writes:
> There's a lot of drivers in the tree that allow you to access the device
> either via IO port space or IO mem space. Here's some examples:
My gut feeling is to just fix the drivers to make this runtime switchable
and get rid of the compile time options.
This would help distributions (who normally want to build conservative
by default, but still allow the users easy tuning without recompilation)
For that it would be nice if a standard module parameter or maybe
sysfs option existed.
The overhead of checking for PIO vs mmio at runtime in the drivers
should be completely in the noise on any non ancient CPU (both MMIO
and PIO typically take hundreds or thousands of CPU cycles for the bus
access, having an dynamic function call or an if before that is makes
no difference at all)
-Andi
On Thu, Sep 11, 2003 at 05:01:16PM +0100, Matthew Wilcox wrote:
> config MMIO
> bool "Prefer Memory-mapped I/O accesses over Port I/O"
> help
> Many devices are accessible via both memory mapped I/O and
> port I/O. Say Y here to access them via the slightly faster
> memory mapped I/O method. If you experience problems, you may
> wish to say N here.
>
> If that's not acceptable, can I suggest that we at least ask a standard
> question?
>
> config WWWW_MMIO
> bool "Use Memory mapped I/O in preference to Port I/O"
> depends on WWWW
> help
> This device's registers can be accessed by either memory
> mapped I/O or port I/O. Memory mapped I/O is faster, so you
> are advised to say Y here.
>
> (yes, we could wordsmith these questions into the ground; the key point
> of this mail is whether we ask one question at the start of configuration
> or whether we ask one question per device.)
I like this idea, esp. as a global option. It would help e.g. visws
which can't reliably do MMIO. Most platforms could default MMIO to y
also, to make things easier.
Jesse
[email protected] (Jesse Barnes) writes:
>
> I like this idea, esp. as a global option. It would help e.g. visws
> which can't reliably do MMIO. Most platforms could default MMIO to y
> also, to make things easier.
The problem AFAIK is usually that there are some buggy older chipset
versions that do not work properly with MMIO, but others do.
That's especially an issue on widely cloned cards like Tulip.
It's not a platform specific thing, more a device specific setting.
Of course some platforms want to disable it completely too, but that's
rare.
-Andi
On Thu, 11 Sep 2003 17:25:04 +0100
Matthew Wilcox <[email protected]> wrote:
> On Thu, Sep 11, 2003 at 06:17:02PM +0200, Andi Kleen wrote:
> > The overhead of checking for PIO vs mmio at runtime in the drivers
> > should be completely in the noise on any non ancient CPU (both MMIO
> > and PIO typically take hundreds or thousands of CPU cycles for the bus
> > access, having an dynamic function call or an if before that is makes
> > no difference at all)
>
> That's not true for MMIO writes which are posted. They should take
> no longer than a memory write. For MMIO reads and PIO reads & writes,
> you are, of course, correct.
Even a memory write is tens to hundres of cycles.
-Andi
On Thu, Sep 11, 2003 at 06:17:02PM +0200, Andi Kleen wrote:
> The overhead of checking for PIO vs mmio at runtime in the drivers
> should be completely in the noise on any non ancient CPU (both MMIO
> and PIO typically take hundreds or thousands of CPU cycles for the bus
> access, having an dynamic function call or an if before that is makes
> no difference at all)
That's not true for MMIO writes which are posted. They should take
no longer than a memory write. For MMIO reads and PIO reads & writes,
you are, of course, correct.
--
"It's not Hollywood. War is real, war is primarily not about defeat or
victory, it is about death. I've seen thousands and thousands of dead bodies.
Do you think I want to have an academic debate on this subject?" -- Robert Fisk
On Thu, Sep 11, 2003 at 06:31:36PM +0200, Andi Kleen wrote:
> On Thu, 11 Sep 2003 17:25:04 +0100
> Matthew Wilcox <[email protected]> wrote:
> > That's not true for MMIO writes which are posted. They should take
> > no longer than a memory write. For MMIO reads and PIO reads & writes,
> > you are, of course, correct.
>
> Even a memory write is tens to hundres of cycles.
So are mispredicted branches.
--
"It's not Hollywood. War is real, war is primarily not about defeat or
victory, it is about death. I've seen thousands and thousands of dead bodies.
Do you think I want to have an academic debate on this subject?" -- Robert Fisk
Andi Kleen wrote:
> Even a memory write is tens to hundres of cycles.
Not from the CPU's perspective. It is done in parallel with other
instructions.
-- Jamie
Andi Kleen wrote:
> My gut feeling is to just fix the drivers to make this runtime switchable
> and get rid of the compile time options.
>
> This would help distributions (who normally want to build conservative
> by default, but still allow the users easy tuning without recompilation)
> For that it would be nice if a standard module parameter or maybe
> sysfs option existed.
Another way to help distributions is to compile those drivers twice,
once for each access type. There aren't all that many drivers that
need it.
> The overhead of checking for PIO vs mmio at runtime in the drivers
> should be completely in the noise on any non ancient CPU (both MMIO
> and PIO typically take hundreds or thousands of CPU cycles for the bus
> access, having an dynamic function call or an if before that is makes
> no difference at all)
Ah, but what about the ancient CPUs?
-- Jamie
On Thu, 11 Sep 2003 18:12:05 +0100
Jamie Lokier <[email protected]> wrote:
> Andi Kleen wrote:
> > Even a memory write is tens to hundres of cycles.
>
> Not from the CPU's perspective. It is done in parallel with other
> instructions.
Only when there are more instructions to execute. But device
driver code often does a following read e.g. to check if it can submit
another request to the hardware.
My claim is basically:
Change everybody who currently does
#ifdef CONFIG_MMIO
writel(... )
readl(...)
#else
outl( ... )
inl ( ...)
#endif
to
if (dev->mmio) {
writel();
real();
} else {
outl();
inl();
}
and you will have a hard time to benchmark the difference on any non ancient system
in actual driver operation.
-Andi
On Thu, Sep 11, 2003 at 07:25:50PM +0200, Andi Kleen wrote:
> On Thu, 11 Sep 2003 18:12:05 +0100
> Jamie Lokier <[email protected]> wrote:
>
> > Andi Kleen wrote:
> > > Even a memory write is tens to hundres of cycles.
> >
> > Not from the CPU's perspective. It is done in parallel with other
> > instructions.
>
> Only when there are more instructions to execute. But device
> driver code often does a following read e.g. to check if it can submit
> another request to the hardware.
>
> My claim is basically:
>
> Change everybody who currently does
>
> #ifdef CONFIG_MMIO
> writel(... )
> readl(...)
> #else
> outl( ... )
> inl ( ...)
> #endif
>
> to
> if (dev->mmio) {
> writel();
> real();
> } else {
> outl();
> inl();
> }
>
> and you will have a hard time to benchmark the difference on any non ancient system
> in actual driver operation.
Shouldn't that be
dev->dev_ops->writel(...);
dev->dev_ops->readl(...);
with no conditionals?
--
________________________________________________________________
J.W. Schultz Pegasystems Technologies
email address: [email protected]
Remember Cernan and Schmitt
On Thu, 2003-09-11 at 10:25, Andi Kleen wrote:
> On Thu, 11 Sep 2003 18:12:05 +0100
> Jamie Lokier <[email protected]> wrote:
>
> > Andi Kleen wrote:
> > > Even a memory write is tens to hundres of cycles.
> >
> > Not from the CPU's perspective. It is done in parallel with other
> > instructions.
>
> Only when there are more instructions to execute. But device
> driver code often does a following read e.g. to check if it can submit
> another request to the hardware.
>
> My claim is basically:
>
> Change everybody who currently does
>
> #ifdef CONFIG_MMIO
> writel(... )
> readl(...)
> #else
> outl( ... )
> inl ( ...)
> #endif
>
> to
> if (dev->mmio) {
> writel();
> real();
> } else {
> outl();
> inl();
> }
>
> and you will have a hard time to benchmark the difference on any non
> ancient system
> in actual driver operation.
>
> -Andi
>
Hello,
Wouldn't it be better if we set the IN and OUT function pointers to the
right functions during driver init. based on the setting of dev->mmio.
And throughout the driver, we just call the IN and OUT functions by
their pointers. Then we don't have to do if (dev->mmio) every time.
It's similar to the concept of virtual member function in C++.
ADT
Anthony Dominic Truong wrote:
> Wouldn't it be better if we set the IN and OUT function pointers to the
> right functions during driver init. based on the setting of dev->mmio.
> And throughout the driver, we just call the IN and OUT functions by
> their pointers. Then we don't have to do if (dev->mmio) every time.
> It's similar to the concept of virtual member function in C++.
I think it would be faster to hide the "if (mmio)" part into an
inline function which calls one or the other, given a resource cookie.
Branch prediction will remove most of the cost of a conditional test,
which is always the same in these drivers after all, but I don't think
it's quite so good at predicting indirect function calls.
-- Jamie
On Fri, Sep 12, 2003 at 09:10:25AM -0700, Anthony Dominic Truong wrote:
> Andi Kleen wrote:
> > #ifdef CONFIG_MMIO
> > writel(... )
> > readl(...)
> > #else
> > outl( ... )
> > inl ( ...)
> > #endif
> >
> > to
> > if (dev->mmio) {
> > writel();
> > real();
> > } else {
> > outl();
> > inl();
> > }
> >
> > and you will have a hard time to benchmark the difference on any non
> > ancient system
> > in actual driver operation.
> >
> > -Andi
> >
> Hello,
> Wouldn't it be better if we set the IN and OUT function pointers to the
> right functions during driver init. based on the setting of dev->mmio.
> And throughout the driver, we just call the IN and OUT functions by
> their pointers. Then we don't have to do if (dev->mmio) every time.
> It's similar to the concept of virtual member function in C++.
I'd rather not see any more pointer dereferences or even branches than
absolutely necessary. Right now, readX/writeX and inX/outX are usually
inlines, and outX and writeX can be very fast. So I'd prefer either a
global CONFIG_MMIO option or consistent driver specific options that
explain what the difference between MMIO and port I/O actually is.
Jesse
> > My claim is basically:
> >
> > Change everybody who currently does
> >
> > #ifdef CONFIG_MMIO
> > writel(... )
> > readl(...)
> > #else
> > outl( ... )
> > inl ( ...)
> > #endif
> >
> > to
> > if (dev->mmio) {
> > writel();
> > real();
> > } else {
> > outl();
> > inl();
> > }
> >
> > and you will have a hard time to benchmark the difference on any non
> > ancient system
> > in actual driver operation.
Or an embedded system, maybe?
At the end of the day, it still means loosing a tiny amount of
performance just to make it easier for distributions to have one
generic kernel. There is nothing wrong with that, but why force it on
systems that are not running a generic kernel?
What's wrong with:
#ifdef CONFIG_MMIO
writel(... );
readl(... );
#endif
#ifdef CONFIG_NO_MMIO
outl(... );
inl(...);
#endif
#ifdef CONFIG_DONT_CARE_MMIO
if (dev->mmio) {
writel(... );
readl(... );
} else {
outl(... );
inl(... );
}
#endif
Although I'm dubious of having a global switch for MMIO anyway.
> Wouldn't it be better if we set the IN and OUT function pointers to the
> right functions during driver init. based on the setting of dev->mmio.
> And throughout the driver, we just call the IN and OUT functions by
> their pointers. Then we don't have to do if (dev->mmio) every time.
> It's similar to the concept of virtual member function in C++.
It's neater, but still means bloat, however small.
I know we're talking a few bytes here and there, but embedded systems
really care about them.
John.
Ok, Andi asked for benchmarks, so I ran some. Let this should be a
lesson on why you shouldn't use port I/O :) I ran these on an SGI Altix
w/900 MHz McKinley processors.
Just straight calls to the routines (all of these are based on the
average of 100 iterations):
writeq(val, reg) time: 64 cycles
outl(val, reg) time: 2126 cycles
A simple branch:
if (use_mmio)
writeq(val, reg) time: 132 cycles
else
outl(val, reg) time: 1990 cycles
Using a wrapper to point to one of the routines:
wrapper->write(val, reg) time: 215 cycles
wrapper->out(val, reg) time: 3931 cycles
Thanks,
Jesse
On Fri, Sep 12, 2003 at 08:00:03PM +0200, Andi Kleen wrote:
> [email protected] (Jesse Barnes) writes:
>
> > Ok, Andi asked for benchmarks, so I ran some. Let this should be a
> > lesson on why you shouldn't use port I/O :) I ran these on an SGI Altix
> > w/900 MHz McKinley processors.
> >
> > Just straight calls to the routines (all of these are based on the
> > average of 100 iterations):
> > writeq(val, reg) time: 64 cycles
> > outl(val, reg) time: 2126 cycles
> ^^^^^
> >
> > A simple branch:
> > if (use_mmio)
> > writeq(val, reg) time: 132 cycles
> > else
> > outl(val, reg) time: 1990 cycles
> ^^^^^
> Something seems to be wrong in your numbers.
>
> Surely the outl in the if () cannot be faster than the pure outl() ?
Well, they're approximate at best... For outl the branch doesn't seem
to matter since it takes forever regardless.
Jesse
[email protected] (Jesse Barnes) writes:
> Ok, Andi asked for benchmarks, so I ran some. Let this should be a
> lesson on why you shouldn't use port I/O :) I ran these on an SGI Altix
> w/900 MHz McKinley processors.
>
> Just straight calls to the routines (all of these are based on the
> average of 100 iterations):
> writeq(val, reg) time: 64 cycles
> outl(val, reg) time: 2126 cycles
^^^^^
>
> A simple branch:
> if (use_mmio)
> writeq(val, reg) time: 132 cycles
> else
> outl(val, reg) time: 1990 cycles
^^^^^
Something seems to be wrong in your numbers.
Surely the outl in the if () cannot be faster than the pure outl() ?
-Andi
On Fri, 12 Sep 2003 11:04:29 -0700
[email protected] (Jesse Barnes) wrote:
> On Fri, Sep 12, 2003 at 08:00:03PM +0200, Andi Kleen wrote:
> > [email protected] (Jesse Barnes) writes:
> >
> > > Ok, Andi asked for benchmarks, so I ran some. Let this should be a
> > > lesson on why you shouldn't use port I/O :) I ran these on an SGI Altix
> > > w/900 MHz McKinley processors.
> > >
> > > Just straight calls to the routines (all of these are based on the
> > > average of 100 iterations):
> > > writeq(val, reg) time: 64 cycles
> > > outl(val, reg) time: 2126 cycles
> > ^^^^^
> > >
> > > A simple branch:
> > > if (use_mmio)
> > > writeq(val, reg) time: 132 cycles
> > > else
> > > outl(val, reg) time: 1990 cycles
> > ^^^^^
> > Something seems to be wrong in your numbers.
> >
> > Surely the outl in the if () cannot be faster than the pure outl() ?
>
> Well, they're approximate at best... For outl the branch doesn't seem
> to matter since it takes forever regardless.
True. Ok for PIO it doesn't matter. I'm surprised the hit is that big
for MMIO. What code does the IA64 compiler generate for this?
If it was a branch it should have been predicted.
I guess we could use dynamic patching like the alternative() code on
i386 does for prefetches etc., but that may be a bit of overkill...
Or fix the makefiles that they can easily compile two versions of
the driver.
-Andi
On Fri, Sep 12, 2003 at 08:00:03PM +0200, Andi Kleen wrote:
> [email protected] (Jesse Barnes) writes:
>
> > Ok, Andi asked for benchmarks, so I ran some. Let this should be a
> > lesson on why you shouldn't use port I/O :) I ran these on an SGI Altix
> > w/900 MHz McKinley processors.
> >
> > Just straight calls to the routines (all of these are based on the
> > average of 100 iterations):
> > writeq(val, reg) time: 64 cycles
> > outl(val, reg) time: 2126 cycles
> ^^^^^
> >
> > A simple branch:
> > if (use_mmio)
> > writeq(val, reg) time: 132 cycles
> > else
> > outl(val, reg) time: 1990 cycles
> ^^^^^
> Something seems to be wrong in your numbers.
>
> Surely the outl in the if () cannot be faster than the pure outl() ?
Also - a perhaps more useful test is a write followed by a read.
--
Notice that as computers are becoming easier and easier to use,
suddenly there's a big market for "Dummies" books. Cause and effect,
or merely an ironic juxtaposition of unrelated facts?
On Fri, Sep 12, 2003 at 11:11:48AM -0700, Tim Hockin wrote:
> On Fri, Sep 12, 2003 at 08:00:03PM +0200, Andi Kleen wrote:
> > [email protected] (Jesse Barnes) writes:
> >
> > > Ok, Andi asked for benchmarks, so I ran some. Let this should be a
> > > lesson on why you shouldn't use port I/O :) I ran these on an SGI Altix
> > > w/900 MHz McKinley processors.
> > >
> > > Just straight calls to the routines (all of these are based on the
> > > average of 100 iterations):
> > > writeq(val, reg) time: 64 cycles
> > > outl(val, reg) time: 2126 cycles
> > ^^^^^
> > >
> > > A simple branch:
> > > if (use_mmio)
> > > writeq(val, reg) time: 132 cycles
> > > else
> > > outl(val, reg) time: 1990 cycles
> > ^^^^^
> > Something seems to be wrong in your numbers.
> >
> > Surely the outl in the if () cannot be faster than the pure outl() ?
>
> Also - a perhaps more useful test is a write followed by a read.
Well, someone else will have to run that test. On Altix, a read() is
freakishly expensive, and I'm not really interested in showing everyone
how bad it is ;)
Jesse
On Fri, 12 Sep 2003 11:24:30 -0700
[email protected] (Jesse Barnes) wrote:
> On Fri, Sep 12, 2003 at 11:11:48AM -0700, Tim Hockin wrote:
> > On Fri, Sep 12, 2003 at 08:00:03PM +0200, Andi Kleen wrote:
> > > [email protected] (Jesse Barnes) writes:
> > >
> > > > Ok, Andi asked for benchmarks, so I ran some. Let this should be a
> > > > lesson on why you shouldn't use port I/O :) I ran these on an SGI Altix
> > > > w/900 MHz McKinley processors.
> > > >
> > > > Just straight calls to the routines (all of these are based on the
> > > > average of 100 iterations):
> > > > writeq(val, reg) time: 64 cycles
> > > > outl(val, reg) time: 2126 cycles
> > > ^^^^^
> > > >
> > > > A simple branch:
> > > > if (use_mmio)
> > > > writeq(val, reg) time: 132 cycles
> > > > else
> > > > outl(val, reg) time: 1990 cycles
> > > ^^^^^
> > > Something seems to be wrong in your numbers.
> > >
> > > Surely the outl in the if () cannot be faster than the pure outl() ?
> >
> > Also - a perhaps more useful test is a write followed by a read.
>
> Well, someone else will have to run that test. On Altix, a read() is
> freakishly expensive, and I'm not really interested in showing everyone
> how bad it is ;)
I guess the read will be very bad everywhere, and the PIO inl even worse.
-Andi
>> Also - a perhaps more useful test is a write followed by a read.
>
> Well, someone else will have to run that test. On Altix, a read() is
> freakishly expensive, and I'm not really interested in showing everyone
> how bad it is ;)
I find percentile comparisons useful for hiding the embarassement
of occasional hardware realities ;-) (ie what's the speed *ratio*
between the two types of read).
M.
On Fri, Sep 12, 2003 at 11:58:07AM -0700, Martin J. Bligh wrote:
> >> Also - a perhaps more useful test is a write followed by a read.
> >
> > Well, someone else will have to run that test. On Altix, a read() is
> > freakishly expensive, and I'm not really interested in showing everyone
> > how bad it is ;)
>
> I find percentile comparisons useful for hiding the embarassement
> of occasional hardware realities ;-) (ie what's the speed *ratio*
> between the two types of read).
Well, in our case, read() and in() will be pretty close,
percentage-wise. The read() in itself is so bad that I think the
additional badness of in() will be mostly hidden.
Jesse
On Fri, Sep 12, 2003 at 05:41:47PM +0100, John Bradford wrote:
> #ifdef CONFIG_MMIO
> writel(... );
> readl(... );
> #endif
> #ifdef CONFIG_NO_MMIO
> outl(... );
> inl(...);
> #endif
> #ifdef CONFIG_DONT_CARE_MMIO
> if (dev->mmio) {
> writel(... );
> readl(... );
> } else {
> outl(... );
> inl(... );
> }
> #endif
If there's already a config option that keeps code from being compiled when
it's not used, then it should stay.
I'm all for a bunch of config options hidden away in broader questions in
the config system...
On Fri, Sep 12, 2003 at 02:52:03PM -0700, Mike Fedyk wrote:
> If there's already a config option that keeps code from being compiled when
> it's not used, then it should stay.
Nevermind, after reading the whole thread I see how that doesn't make sense
in this case.