Folks,
Here's a simple SMI detector we've been playing around with internally.
In case anyone wants to play with it, and especially help clean it up :)
SMI events are particularly unhelpful on Real Time systems because we
effectively have the CPU stolen from under us. We can't prevent this
easily without BIOS vendors/system vendor co-operation, but we can help
to diagnose and log that these are occuring...which can be very helpful
too.
Cheers,
Jon.
On Fri, Jan 23, 2009 at 5:55 PM, Jon Masters <[email protected]> wrote:
> Here's a simple SMI detector we've been playing around with internally.
> In case anyone wants to play with it, and especially help clean it up :)
>
> SMI events are particularly unhelpful on Real Time systems because we
> effectively have the CPU stolen from under us. We can't prevent this
> easily without BIOS vendors/system vendor co-operation, but we can help
> to diagnose and log that these are occuring...which can be very helpful
> too.
>
FYI the RTAI project has a patch to allow disabling of SMI:
https://listas.upv.es/pipermail/rtlinuxgpl/2007-April/000609.html
Lee
On Fri, 23 Jan 2009, Lee Revell wrote:
>
> FYI the RTAI project has a patch to allow disabling of SMI:
>
> https://listas.upv.es/pipermail/rtlinuxgpl/2007-April/000609.html
FYI, this patch should be removed from the internet archives and the
author should be made liable for the damage which can happen when
innocent users try to use it for fixing things which simply can not be
fixed.
>> .... disabled SMIs with RTAI code the latencies of cyclictest were
>> good, but after about 20 minutes my system stopped working and now
>> it does not even boot anymore. :( Any hint?
>
> Yeah. Buy a new CPU. You deep-fried your P4 because you disabled the
> thermal protection.
Aside of this worst case scenario, disabling SMIs is problematic as
SMI is used to emulate devices, to fix chip bugs and necessary for
enhanced features.
The only reasonable thing you can do on a SMI plagued system is to
identify the device which makes use of SMIs. Legacy ISA devices and
USB are usually good candidates. If that does not help, don't use it
for real-time :)
Thanks,
tglx
On Sat, 2009-01-24 at 17:30 +0100, Thomas Gleixner wrote:
> On Fri, 23 Jan 2009, Lee Revell wrote:
> >
> > FYI the RTAI project has a patch to allow disabling of SMI:
> >
> > https://listas.upv.es/pipermail/rtlinuxgpl/2007-April/000609.html
>
> FYI, this patch should be removed from the internet archives and the
> author should be made liable for the damage which can happen when
> innocent users try to use it for fixing things which simply can not be
> fixed.
Well, we disagree on the remedy there (I personally thing "caveat
emptor", although European law might well differ in that regard), but I
think a warning is very much called for, especially where end users are
concerned. Let's be sure those following along the LKML archives
understand something:
1). You (the end user) don't want to disable SMIs on existing systems.
2). Doing so might prevent thermal management software, system
management software, BIOS implemented fixes, and other components from
operating.
3). BIOS vendors might need to be encouraged to reduce SMI overhead.
Once systems have been designed to rely on lots of SMI overhead, the
game has already been lost. No amount of user fiddling will adequately
fix that.
What we are interested in doing with things like smi_detector is
noticing when there is substantial overhead, so that we can politely ask
the vendors involved to implement alternative SMI handlers that have
less of a hit on RT performance - the only real solution involving no
SMIs is more expensive hardware with dedicated system management/thermal
management processors.
> >> .... disabled SMIs with RTAI code the latencies of cyclictest were
> >> good, but after about 20 minutes my system stopped working and now
> >> it does not even boot anymore. :( Any hint?
> >
> > Yeah. Buy a new CPU. You deep-fried your P4 because you disabled the
> > thermal protection.
>
> Aside of this worst case scenario, disabling SMIs is problematic as
> SMI is used to emulate devices, to fix chip bugs and necessary for
> enhanced features.
>
> The only reasonable thing you can do on a SMI plagued system is to
> identify the device which makes use of SMIs. Legacy ISA devices and
> USB are usually good candidates. If that does not help, don't use it
> for real-time :)
Indeed. This is why I wrote an smi_detector that sits in kernel space
and can be reasonably sure measured discrepancies are attributable to
SMI behavior. We want to log and detect such things before RT systems
are deployed, not have users actively trying to work around SMI overhead
after the fact.
Jon.
On Sat, 2009-01-24 at 19:57 -0500, Jon Masters wrote:
> On Sat, 2009-01-24 at 17:30 +0100, Thomas Gleixner wrote:
> > On Fri, 23 Jan 2009, Lee Revell wrote:
> > >
> > > FYI the RTAI project has a patch to allow disabling of SMI:
> > >
> > > https://listas.upv.es/pipermail/rtlinuxgpl/2007-April/000609.html
> >
> > FYI, this patch should be removed from the internet archives and the
> > author should be made liable for the damage which can happen when
> > innocent users try to use it for fixing things which simply can not be
> > fixed.
(given an arbitrary existing hardware platform)
Note however, that in many cases, it is possible - by system vendors -
to modify / update firmware and BIOS implementations.
>
> Well, we disagree on the remedy there (I personally thing "caveat
> emptor", although European law might well differ in that regard), but I
> think a warning is very much called for, especially where end users are
> concerned. Let's be sure those following along the LKML archives
> understand something:
>
> 1). You (the end user) don't want to disable SMIs on existing systems.
> 2). Doing so might prevent thermal management software, system
> management software, BIOS implemented fixes, and other components from
> operating.
> 3). BIOS vendors might need to be encouraged to reduce SMI overhead.
> Once systems have been designed to rely on lots of SMI overhead, the
> game has already been lost. No amount of user fiddling will adequately
> fix that.
>
4. Disabling SMIs and mucking with BIOS code may void your warranty,
cause your circuit breakers to trip, and could interfere with the
operation of your toaster.
> What we are interested in doing with things like smi_detector is
> noticing when there is substantial overhead, so that we can politely ask
> the vendors involved to implement alternative SMI handlers that have
> less of a hit on RT performance - the only real solution involving no
> SMIs is more expensive hardware with dedicated system management/thermal
> management processors.
Some SMI implementations involve dedicated, "embedded" auxiliary
processors and associated firmware,
others simply exist as BIOS-level implementations.
When the effect of SMI execution governs worst-case task-level or IRQ
latencies on a hardware platform, you may indeed be using the wrong
hardware.
Hardware is subject to the market-driven process, admittedly in some
cases costly, cumbersome and difficult for those constrained by existing
installations and supply-side arrangements.
Latency-friendly SMI implementations may be found in some systems, and
not so much in others.
It should be up to the promotional divisions of the particular hardware
suppliers to assess the market pressure, and to report the performance
advantages of their specific systems.
Worst-case cpu-offlining and other execution-inhibiting system-level
latencies should be required as part of RFP by customers of systems
targeted for real-time, low-latency, and HPC applications.
I envision documentation of these specs, in a similar fashion to the
manner in which CPU clock-cycles are documented for specific instruction
executions - for those systems eligible (certified?) for low-latency
operation.
Broader SMP arrays combined with leveling-off CPU speed, cycle
management seems to be moving back to the fore-front on systems
architecture and design.
>
> > >> .... disabled SMIs with RTAI code the latencies of cyclictest were
> > >> good, but after about 20 minutes my system stopped working and now
> > >> it does not even boot anymore. :( Any hint?
> > >
> > > Yeah. Buy a new CPU. You deep-fried your P4 because you disabled the
> > > thermal protection.
> >
> > Aside of this worst case scenario, disabling SMIs is problematic as
> > SMI is used to emulate devices, to fix chip bugs and necessary for
> > enhanced features.
> >
> > The only reasonable thing you can do on a SMI plagued system is to
> > identify the device which makes use of SMIs. Legacy ISA devices and
> > USB are usually good candidates. If that does not help, don't use it
> > for real-time :)
>
> Indeed. This is why I wrote an smi_detector that sits in kernel space
> and can be reasonably sure measured discrepancies are attributable to
> SMI behavior. We want to log and detect such things before RT systems
> are deployed, not have users actively trying to work around SMI overhead
> after the fact.
>
Enterprise Distros (IMO) bear the responsibility to raise awareness of
these issues, and stand firm to relentlessly call for open specs and
open implementation.
Providing benchmarking tools is definitely an excellent step in that
direction (imo) - thanks.
Regards,
Sven
> Jon.
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rt-users" in
> the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html
On Sun, Jan 25, 2009 at 01:12:45PM +1100, Sven-Thorsten Dietrich wrote:
> Latency-friendly SMI implementations may be found in some systems, and
> not so much in others.
>
> It should be up to the promotional divisions of the particular hardware
> suppliers to assess the market pressure, and to report the performance
> advantages of their specific systems.
They definitely exist; I spent quite a bit of time working with IBM
hardware engineers on what we called "SMI remediation". In some cases
it was necessary to have some custom kernel and/or user space code to
run (at background priority) maintenance work such as predictive
failure calculations that really didn't need to be done at SMI
stop-the-OS-on-all-cpus priority, but which hardware manufacturers
would do at the SMI level to avoid needing to create
motherboard-specific device drivers and/or daemons for each OS that
they might want to support.
> I envision documentation of these specs, in a similar fashion to the
> manner in which CPU clock-cycles are documented for specific instruction
> executions - for those systems eligible (certified?) for low-latency
> operation.
I'm not sure exactly how one would do this certification, but I agree
that some kind of "real-time ready" logo/certification program would
make a huge amount of sense, with some standardized metrics of maximum
time spent in an SMI routine, and under what circumstances (in some
cases it occurs every 30-60 minutes; on other cases, only when the CPU
is about to melt itself into slag, or when there are ECC errors, etc.)
There is a huge difference between a system which stops the OS on all
CPU's dead in its tracks for milliseconds once every 45 minutes,
versus one which only triggers an SMI in extreme situations when the
hardware is about to destroy itself.
I will also note that for some applications (i.e., military hardware
running under battle conditions), where it might be that running the
hardware beyond its thermal limits might actually be *desirable*.
After all, an extra 15 minutes of running beyond thermal limits that
eventually causes the CPU to get flakey might be worth it if the
alternative is the ship getting sunk because the BIOS decided that
shutting down the CPU to save it from thermal damage was more
important than say, running the anti-aircraft guns....
- Ted
On Sat, 24 Jan 2009, Jon Masters wrote:
> > The only reasonable thing you can do on a SMI plagued system is to
> > identify the device which makes use of SMIs. Legacy ISA devices and
> > USB are usually good candidates. If that does not help, don't use it
> > for real-time :)
>
> Indeed. This is why I wrote an smi_detector that sits in kernel space
> and can be reasonably sure measured discrepancies are attributable to
> SMI behavior. We want to log and detect such things before RT systems
> are deployed, not have users actively trying to work around SMI overhead
> after the fact.
Agreed. A tool to detect SMI disturbance is a good thing. It just
needs to be documented that users should take the results and talk to
their board vendor. I know that off the shelf hardware will not be
fixed, but industrial grade hardware vendors usually have an interest
to get such problems resolved.
Thanks,
tglx
On Sat, 24 Jan 2009, Theodore Tso wrote:
> On Sun, Jan 25, 2009 at 01:12:45PM +1100, Sven-Thorsten Dietrich wrote:
> I will also note that for some applications (i.e., military hardware
> running under battle conditions), where it might be that running the
> hardware beyond its thermal limits might actually be *desirable*.
> After all, an extra 15 minutes of running beyond thermal limits that
> eventually causes the CPU to get flakey might be worth it if the
> alternative is the ship getting sunk because the BIOS decided that
> shutting down the CPU to save it from thermal damage was more
> important than say, running the anti-aircraft guns....
In that case the system designer knows exactly what he is doing and he
is aware of the consequences.
My concern about the SMI disable module is that it can damage Joe
users hardware. I have at least two reports where the CPU got fried
and some others where people got confused because chips started
behaving weird and it took quite a time to figure out that they used
the SMI disabler. A big fat warning about this code is definitely
necessary.
Thanks,
tglx
On Sun, Jan 25, 2009 at 10:40 AM, Thomas Gleixner <[email protected]> wrote:
> On Sat, 24 Jan 2009, Theodore Tso wrote:
>> On Sun, Jan 25, 2009 at 01:12:45PM +1100, Sven-Thorsten Dietrich wrote:
>> I will also note that for some applications (i.e., military hardware
>> running under battle conditions), where it might be that running the
>> hardware beyond its thermal limits might actually be *desirable*.
>> After all, an extra 15 minutes of running beyond thermal limits that
>> eventually causes the CPU to get flakey might be worth it if the
>> alternative is the ship getting sunk because the BIOS decided that
>> shutting down the CPU to save it from thermal damage was more
>> important than say, running the anti-aircraft guns....
>
> In that case the system designer knows exactly what he is doing and he
> is aware of the consequences.
>
> My concern about the SMI disable module is that it can damage Joe
> users hardware. I have at least two reports where the CPU got fried
> and some others where people got confused because chips started
> behaving weird and it took quite a time to figure out that they used
> the SMI disabler. A big fat warning about this code is definitely
> necessary.
>
> Thanks,
I suppose the non joe user could flash their motherboard with
linuxcore and therefore do not distrub by SMI :)
Regards
Bastien
On Sun, 25 Jan 2009, Thomas Gleixner wrote:
> On Sat, 24 Jan 2009, Jon Masters wrote:
> > > The only reasonable thing you can do on a SMI plagued system is to
> > > identify the device which makes use of SMIs. Legacy ISA devices and
> > > USB are usually good candidates. If that does not help, don't use it
> > > for real-time :)
> >
> > Indeed. This is why I wrote an smi_detector that sits in kernel space
> > and can be reasonably sure measured discrepancies are attributable to
> > SMI behavior. We want to log and detect such things before RT systems
> > are deployed, not have users actively trying to work around SMI overhead
> > after the fact.
>
> Agreed. A tool to detect SMI disturbance is a good thing. It just
> needs to be documented that users should take the results and talk to
> their board vendor. I know that off the shelf hardware will not be
> fixed, but industrial grade hardware vendors usually have an interest
> to get such problems resolved.
"gamer enthusiast" hardware might also get fixed. You just need two or
three posts to the "enthusiasts" forums about how SMI steals CPU cycles and
slow down their framerates, and suddenly, benchmarks will start going on and
on about how MoBo x has a high number of SMIs per minute, where MoBo y
doesn't...
The people who want low-latency desktops for audio work will also pay
attention to such benchmarks and will vote with their wallet.
--
"One disk to rule them all, One disk to find them. One disk to bring
them all and in the darkness grind them. In the Land of Redmond
where the shadows lie." -- The Silicon Valley Tarot
Henrique Holschuh
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
On Sun, 25 Jan 2009 12:49:18 +0100
Bastien ROUCARIES <[email protected]> wrote:
> On Sun, Jan 25, 2009 at 10:40 AM, Thomas Gleixner <[email protected]> wrote:
> >
> > My concern about the SMI disable module is that it can damage Joe
> > users hardware. I have at least two reports where the CPU got fried
> > and some others where people got confused because chips started
> > behaving weird and it took quite a time to figure out that they used
> > the SMI disabler. A big fat warning about this code is definitely
> > necessary.
> >
> > Thanks,
>
>
> I suppose the non joe user could flash their motherboard with
> linuxcore and therefore do not distrub by SMI :)
>
Except for the fact that linuxcore is unaware of the specific
requirements a particular motherboard has for thermal management.
Turning off SMI's unconditionally is a *bad* idea. Better to start
telling hardware vendors that we can't use their platform because of
unexplained latencies (which we presume to be SMI's).
Clark
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.10 (GNU/Linux)
iEYEARECAAYFAkl8f5oACgkQHyuj/+TTEp0JuwCfQaaugU+MY8nWHKsvXuVNmE6X
IEYAoMFn70BK9NczzHI1XnnQwy7bx/Ja
=aRKl
-----END PGP SIGNATURE-----
????{.n?+???????+%?????ݶ??w??{.n?+????{??G?????{ay? ʇڙ?,j??f???h?????????z_??(?階?ݢj"???m??????G????????????&???~???iO???z??v?^?m???? ????????I?
On Sun, 2009-01-25 at 13:12 +1100, Sven-Thorsten Dietrich wrote:
> I envision documentation of these specs, in a similar fashion to the
> manner in which CPU clock-cycles are documented for specific instruction
> executions - for those systems eligible (certified?) for low-latency
> operation.
I'd love to see a generic API that OS's could use to interface with
BIOS-level SMI implementations. Something provided in ACPI data. For all
I know, there already is something generic, but I've not heard of it?
It'd be really nice to have generic functions like:
disable_random_vendor_extensions_crap()
disable_all_non_essential_smis()
or whatever else.
> Providing benchmarking tools is definitely an excellent step in that
> direction (imo) - thanks.
Yeah. The more of this stuff, the better I think. So if someone wants to
help me polish what I posted then cool - it's in our "MRG" internal tree
already in the current form since we're only using it for debugging now.
Jon.
On Sat, 2009-01-24 at 23:02 -0500, Theodore Tso wrote:
> I'm not sure exactly how one would do this certification, but I agree
> that some kind of "real-time ready" logo/certification program would
> make a huge amount of sense, with some standardized metrics of maximum
> time spent in an SMI routine, and under what circumstances (in some
> cases it occurs every 30-60 minutes; on other cases, only when the CPU
> is about to melt itself into slag, or when there are ECC errors, etc.)
> There is a huge difference between a system which stops the OS on all
> CPU's dead in its tracks for milliseconds once every 45 minutes,
> versus one which only triggers an SMI in extreme situations when the
> hardware is about to destroy itself.
I actually already talked to a certain industry group about a different
program but will mention this idea also - some kind of industry notion
of "real time platform" probably wouldn't be a bad idea. I'm not sure
what the vendors would say/think, but it's probably worth having the
discussion anyway. I think we're all right now having to do a lot of
legwork in testing/certifying that systems have acceptable latencies.
Jon.
On Fri, Jan 23, 2009 at 05:55:12PM -0500, Jon Masters wrote:
> This patch adds the module smi_detector under drivers/misc
>
> Code from Jon Masters with small changes from Luis Goncalves and
> documentation from Clark Williams.
Good work Jon. I think something like this will be helpful.
Any reason why you could not do SMI detection in user level code running
at the highest RT priority? If something is running at the highest RT
priority, it should not be preempted by anything. Right? In the past,
we discoved discovered periodic SMIs on some hardware platforms by running
a user level process in an infinite loop sampling the clock and reporting
unexpected delays. Of course, this does hog an entire CPU.
--
Mike
On Sun, 2009-01-25 at 14:52 -0800, Mike Kravetz wrote:
> Any reason why you could not do SMI detection in user level code running
> at the highest RT priority?
Yeah. I had some suggestions about that already...the problem with that
is that you can't know whether measured latencies are due to SMIs /or/
other kernel latencies - maybe there's a bug or other problem? I really
want to be able to say to vendors "nope, it's definitely your problem".
So I think what'll end up happening is a user process for ease of
deployment but something like smi_detector for advanced diagnostics. And
then I just hope others will write tools like the one IBM already has
that will turn off extraneous SMI generation :)
Jon.
On Mon, Jan 26, 2009 at 12:51 PM, Jon Masters <[email protected]> wrote:
> On Sun, 2009-01-25 at 14:52 -0800, Mike Kravetz wrote:
>
>> Any reason why you could not do SMI detection in user level code running
>> at the highest RT priority?
>
> Yeah. I had some suggestions about that already...the problem with that
> is that you can't know whether measured latencies are due to SMIs /or/
> other kernel latencies - maybe there's a bug or other problem? I really
> want to be able to say to vendors "nope, it's definitely your problem".
Can't you do this with a known good version of the -rt kernel and the
latency tracer in full RT preempt mode? SMI induced latency will show
up in the trace as long delays in random code paths where kernel
induced latency can be ruled out.
Lee
On Mon, Jan 26, 2009 at 9:51 AM, Jon Masters <[email protected]> wrote:
> On Sun, 2009-01-25 at 14:52 -0800, Mike Kravetz wrote:
>
>> Any reason why you could not do SMI detection in user level code running
>> at the highest RT priority?
>
> Yeah. I had some suggestions about that already...the problem with that
> is that you can't know whether measured latencies are due to SMIs /or/
> other kernel latencies - maybe there's a bug or other problem? I really
> want to be able to say to vendors "nope, it's definitely your problem".
While I agree it is tricky to sort out OS V. SMI issues it really
doesn't matter if
your SMI periods are less than your OS period. If you OS blocks as long as
your SMI then it is hard to say for sure.
One class of SMI is periodic. Looking though latency data and looking for
periodic events might be handy. Say if every X seconds you experience extra
latency you can pretty much bet that is an SMI.
Just had an interesting idea. Why not just boot Linux as far as early console
access and put the test loop in right then and there?
> So I think what'll end up happening is a user process for ease of
> deployment but something like smi_detector for advanced diagnostics. And
> then I just hope others will write tools like the one IBM already has
> that will turn off extraneous SMI generation :)
I will be sending out a RFC with our implementation. On a few platforms the
firmware supports turning off all "non-fatal" SMIs. There is a small driver
to interface with the BIOS and some userspace bits.
Thanks,
Keith Mannthey
LTC Real-Time
Hi!
> In case anyone wants to play with it, and especially help clean it up :)
>
> SMI events are particularly unhelpful on Real Time systems because we
> effectively have the CPU stolen from under us. We can't prevent this
> easily without BIOS vendors/system vendor co-operation, but we can help
> to diagnose and log that these are occuring...which can be very helpful
> too.
>
> Cheers,
>
> Jon.
>
> This patch adds the module smi_detector under drivers/misc
>
> Code from Jon Masters with small changes from Luis Goncalves and
> documentation from Clark Williams.
>
> ---
> diff -uNp a/drivers/misc/Makefile b/drivers/misc/Makefile
> --- a/drivers/misc/Makefile 2009-01-09 13:25:31.000000000 -0200
> +++ b/drivers/misc/Makefile 2009-01-15 10:55:58.000000000 -0200
> @@ -18,3 +18,5 @@ obj-$(CONFIG_SONY_LAPTOP) += sony-laptop
> obj-$(CONFIG_THINKPAD_ACPI) += thinkpad_acpi.o
> obj-$(CONFIG_FUJITSU_LAPTOP) += fujitsu-laptop.o
> obj-$(CONFIG_EEPROM_93CX6) += eeprom_93cx6.o
> +obj-m += smi_detector.o
> +
Please use Kconfig here.
--
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
On Tue, 2009-01-27 at 12:22 +0100, Pavel Machek wrote:
> Please use Kconfig here.
Oops. I didn't notice that...someone else did the build and config bits
after I posted the module. Thanks, I'll get that fixed and repost!
Jon.
Is it possible that if the detector is running,
it may prevent (delay) the thing it is trying to detect?
Eg. some random AML gets interpreted by the kernel,
it needs to touch an MSR, but MSRs are not accessible
from AML, so it triggeres an SMI by touching a magic IO address
with a magic value.
So in this case, the random AML may not run until after
the detector has stopped running, because the detector
prevented some random user-space distro value-add
from polling the battery or the temperature or something...
The other example that comes to mind is that dreaded
USB PS/2 emulation done via SMM.
FWIW I detest SMM as much as any OS person should.
But I also acknowledge that it is virtually impossible to prevent
BIOS developers from giving it up. We OS types have proposed an
"SMI has happened bit or counter" many times,
even though we'd rather have a "SMM disable" bit:-)
cheers,
--
Len Brown, Intel Open Source Technology Center