John,
Have you run into this. One of the USB disc controllers has the ability to boot the system,
however, it needs SMM code to do this. This SMM code, somehow, causes SMI interrupts (which are
higher priority than NMI interrutps and not maskable) which it needs to do its thing.
Problem is that if one of these occurs while calibrating the TSC or the delay code, it can cause a
wrong result. We have seen both a too long and a too short result (depending on where the interrut
happens).
They have found the root cause of TSC calibration problem.
Now they ask for the fix or workaround.
That is the BIOS is periodically interrupted by USB controller and the CPU
waits during the processing of these interrupts.
Their experiments say the interrupt interval is 260mSec and the BIOS needs
150uSec - 200uSec for processing.
It is proved that the problem doesn't reproduce by masking such SMI in BIOS.
They say SMI is for BIOS emulation for connecting legacy devices to USB.
Without such an emulation it's impossible to boot from USB-FD for instance,
they say too.
--
George Anzinger [email protected]
HRT (High-res-timers): http://sourceforge.net/projects/high-res-timers/
On Fri, 2005-11-11 at 13:37 -0800, George Anzinger wrote:
> John,
>
> Have you run into this. One of the USB disc controllers has the ability to boot the system,
> however, it needs SMM code to do this. This SMM code, somehow, causes SMI interrupts (which are
> higher priority than NMI interrutps and not maskable) which it needs to do its thing.
>
> Problem is that if one of these occurs while calibrating the TSC or the delay code, it can cause a
> wrong result. We have seen both a too long and a too short result (depending on where the interrut
> happens).
>
> They have found the root cause of TSC calibration problem.
> Now they ask for the fix or workaround.
>
> That is the BIOS is periodically interrupted by USB controller and the CPU
> waits during the processing of these interrupts.
> Their experiments say the interrupt interval is 260mSec and the BIOS needs
> 150uSec - 200uSec for processing.
> It is proved that the problem doesn't reproduce by masking such SMI in BIOS.
> They say SMI is for BIOS emulation for connecting legacy devices to USB.
> Without such an emulation it's impossible to boot from USB-FD for instance,
> they say too.
Hmmm. I haven't heard of this issue specifically, but yes, I'm quite
familiar with the pain BIOS SMIs can cause and I'm not surprised that it
would affect the TSC/delay calibration code.
Is this still an issue w/ 2.6.14? I know the new TSC based delay
calibration code is supposed to be SMI resilient, but I haven't really
played with it closely.
Not sure what the best method to move forward would be. I suspect
disabling the SMI code early in boot (I thought the usb legacy handoff
stuff already did this?) would help. Then the actual Linux USB drivers
can take over before we switch from the initrd to the root filesystem.
Greg, do you have a suggestion?
thanks
-john
On Fri, Nov 11, 2005 at 01:57:07PM -0800, john stultz wrote:
> On Fri, 2005-11-11 at 13:37 -0800, George Anzinger wrote:
> > John,
> >
> > Have you run into this. One of the USB disc controllers has the ability to boot the system,
> > however, it needs SMM code to do this. This SMM code, somehow, causes SMI interrupts (which are
> > higher priority than NMI interrutps and not maskable) which it needs to do its thing.
> >
> > Problem is that if one of these occurs while calibrating the TSC or the delay code, it can cause a
> > wrong result. We have seen both a too long and a too short result (depending on where the interrut
> > happens).
> >
> > They have found the root cause of TSC calibration problem.
> > Now they ask for the fix or workaround.
> >
> > That is the BIOS is periodically interrupted by USB controller and the CPU
> > waits during the processing of these interrupts.
> > Their experiments say the interrupt interval is 260mSec and the BIOS needs
> > 150uSec - 200uSec for processing.
> > It is proved that the problem doesn't reproduce by masking such SMI in BIOS.
> > They say SMI is for BIOS emulation for connecting legacy devices to USB.
> > Without such an emulation it's impossible to boot from USB-FD for instance,
> > they say too.
>
> Hmmm. I haven't heard of this issue specifically, but yes, I'm quite
> familiar with the pain BIOS SMIs can cause and I'm not surprised that it
> would affect the TSC/delay calibration code.
>
> Is this still an issue w/ 2.6.14? I know the new TSC based delay
> calibration code is supposed to be SMI resilient, but I haven't really
> played with it closely.
>
> Not sure what the best method to move forward would be. I suspect
> disabling the SMI code early in boot (I thought the usb legacy handoff
> stuff already did this?) would help. Then the actual Linux USB drivers
> can take over before we switch from the initrd to the root filesystem.
>
> Greg, do you have a suggestion?
I only ever saw this when people forgot to load the USB drivers. Once
the kernel took over USB support, there was no problem (if there was,
that's a BIOS bug.) The handoff code in 2.6.14 should help a lot with
this too.
thanks,
greg k-h
Greg KH wrote:
> On Fri, Nov 11, 2005 at 01:57:07PM -0800, john stultz wrote:
>
>>On Fri, 2005-11-11 at 13:37 -0800, George Anzinger wrote:
>>
>>>John,
>>>
>>>Have you run into this. One of the USB disc controllers has the ability to boot the system,
>>>however, it needs SMM code to do this. This SMM code, somehow, causes SMI interrupts (which are
>>>higher priority than NMI interrutps and not maskable) which it needs to do its thing.
>>>
>>>Problem is that if one of these occurs while calibrating the TSC or the delay code, it can cause a
>>>wrong result. We have seen both a too long and a too short result (depending on where the interrut
>>>happens).
>>>
>>>They have found the root cause of TSC calibration problem.
>>>Now they ask for the fix or workaround.
>>>
>>>That is the BIOS is periodically interrupted by USB controller and the CPU
>>>waits during the processing of these interrupts.
>>>Their experiments say the interrupt interval is 260mSec and the BIOS needs
>>>150uSec - 200uSec for processing.
>>>It is proved that the problem doesn't reproduce by masking such SMI in BIOS.
>>>They say SMI is for BIOS emulation for connecting legacy devices to USB.
>>>Without such an emulation it's impossible to boot from USB-FD for instance,
>>>they say too.
>>
>>Hmmm. I haven't heard of this issue specifically, but yes, I'm quite
>>familiar with the pain BIOS SMIs can cause and I'm not surprised that it
>>would affect the TSC/delay calibration code.
>>
>>Is this still an issue w/ 2.6.14? I know the new TSC based delay
>>calibration code is supposed to be SMI resilient, but I haven't really
>>played with it closely.
>>
>>Not sure what the best method to move forward would be. I suspect
>>disabling the SMI code early in boot (I thought the usb legacy handoff
>>stuff already did this?) would help. Then the actual Linux USB drivers
>>can take over before we switch from the initrd to the root filesystem.
>>
>>Greg, do you have a suggestion?
>
>
> I only ever saw this when people forgot to load the USB drivers. Once
> the kernel took over USB support, there was no problem (if there was,
> that's a BIOS bug.) The handoff code in 2.6.14 should help a lot with
> this too.
>
Ah... are you saying that the USB support code stops the SMM/SMI prior to the TSC & delay
calibration? Also, this problem was noted in a 2.4.20 kernel. Any help there?
--
George Anzinger [email protected]
HRT (High-res-timers): http://sourceforge.net/projects/high-res-timers/
On Sat, Nov 12, 2005 at 08:06:37AM -0800, George Anzinger wrote:
> Greg KH wrote:
> >On Fri, Nov 11, 2005 at 01:57:07PM -0800, john stultz wrote:
> >
> >>On Fri, 2005-11-11 at 13:37 -0800, George Anzinger wrote:
> >>
> >>>John,
> >>>
> >>>Have you run into this. One of the USB disc controllers has the ability
> >>>to boot the system, however, it needs SMM code to do this. This SMM
> >>>code, somehow, causes SMI interrupts (which are higher priority than NMI
> >>>interrutps and not maskable) which it needs to do its thing.
> >>>
> >>>Problem is that if one of these occurs while calibrating the TSC or the
> >>>delay code, it can cause a wrong result. We have seen both a too long
> >>>and a too short result (depending on where the interrut happens).
> >>>
> >>>They have found the root cause of TSC calibration problem.
> >>>Now they ask for the fix or workaround.
> >>>
> >>>That is the BIOS is periodically interrupted by USB controller and the
> >>>CPU
> >>>waits during the processing of these interrupts.
> >>>Their experiments say the interrupt interval is 260mSec and the BIOS
> >>>needs
> >>>150uSec - 200uSec for processing.
> >>>It is proved that the problem doesn't reproduce by masking such SMI in
> >>>BIOS.
> >>>They say SMI is for BIOS emulation for connecting legacy devices to USB.
> >>>Without such an emulation it's impossible to boot from USB-FD for
> >>>instance,
> >>>they say too.
> >>
> >>Hmmm. I haven't heard of this issue specifically, but yes, I'm quite
> >>familiar with the pain BIOS SMIs can cause and I'm not surprised that it
> >>would affect the TSC/delay calibration code.
> >>
> >>Is this still an issue w/ 2.6.14? I know the new TSC based delay
> >>calibration code is supposed to be SMI resilient, but I haven't really
> >>played with it closely.
> >>
> >>Not sure what the best method to move forward would be. I suspect
> >>disabling the SMI code early in boot (I thought the usb legacy handoff
> >>stuff already did this?) would help. Then the actual Linux USB drivers
> >>can take over before we switch from the initrd to the root filesystem.
> >>
> >>Greg, do you have a suggestion?
> >
> >
> >I only ever saw this when people forgot to load the USB drivers. Once
> >the kernel took over USB support, there was no problem (if there was,
> >that's a BIOS bug.) The handoff code in 2.6.14 should help a lot with
> >this too.
> >
> Ah... are you saying that the USB support code stops the SMM/SMI prior to
> the TSC & delay calibration? Also, this problem was noted in a 2.4.20
> kernel. Any help there?
I do not know where in the boot process the TSC and delay calibration is
done. If it happens before the PCI bus is probed, then no, it does not
happen before this.
On these boxes, I'd just recommend disabling USB legacy support
completly, if possible. And then complain loudly to the vendor to fix
their BIOS.
thanks,
greg k-h
Greg KH wrote:
> On Sat, Nov 12, 2005 at 08:06:37AM -0800, George Anzinger wrote:
>
>>Greg KH wrote:
>>
>>>On Fri, Nov 11, 2005 at 01:57:07PM -0800, john stultz wrote:
>>>
>>>
>>>>On Fri, 2005-11-11 at 13:37 -0800, George Anzinger wrote:
>>>>
>>>>
>>>>>John,
>>>>>
>>>>>Have you run into this. One of the USB disc controllers has the ability
>>>>>to boot the system, however, it needs SMM code to do this. This SMM
>>>>>code, somehow, causes SMI interrupts (which are higher priority than NMI
>>>>>interrutps and not maskable) which it needs to do its thing.
>>>>>
>>>>>Problem is that if one of these occurs while calibrating the TSC or the
>>>>>delay code, it can cause a wrong result. We have seen both a too long
>>>>>and a too short result (depending on where the interrut happens).
>>>>>
>>>>>They have found the root cause of TSC calibration problem.
>>>>>Now they ask for the fix or workaround.
>>>>>
>>>>>That is the BIOS is periodically interrupted by USB controller and the
>>>>>CPU
>>>>>waits during the processing of these interrupts.
>>>>>Their experiments say the interrupt interval is 260mSec and the BIOS
>>>>>needs
>>>>>150uSec - 200uSec for processing.
>>>>>It is proved that the problem doesn't reproduce by masking such SMI in
>>>>>BIOS.
>>>>>They say SMI is for BIOS emulation for connecting legacy devices to USB.
>>>>>Without such an emulation it's impossible to boot from USB-FD for
>>>>>instance,
>>>>>they say too.
>>>>
>>>>Hmmm. I haven't heard of this issue specifically, but yes, I'm quite
>>>>familiar with the pain BIOS SMIs can cause and I'm not surprised that it
>>>>would affect the TSC/delay calibration code.
>>>>
>>>>Is this still an issue w/ 2.6.14? I know the new TSC based delay
>>>>calibration code is supposed to be SMI resilient, but I haven't really
>>>>played with it closely.
>>>>
>>>>Not sure what the best method to move forward would be. I suspect
>>>>disabling the SMI code early in boot (I thought the usb legacy handoff
>>>>stuff already did this?) would help. Then the actual Linux USB drivers
>>>>can take over before we switch from the initrd to the root filesystem.
>>>>
>>>>Greg, do you have a suggestion?
>>>
>>>
>>>I only ever saw this when people forgot to load the USB drivers. Once
>>>the kernel took over USB support, there was no problem (if there was,
>>>that's a BIOS bug.) The handoff code in 2.6.14 should help a lot with
>>>this too.
>>>
>>
>>Ah... are you saying that the USB support code stops the SMM/SMI prior to
>>the TSC & delay calibration? Also, this problem was noted in a 2.4.20
>>kernel. Any help there?
>
>
> I do not know where in the boot process the TSC and delay calibration is
> done. If it happens before the PCI bus is probed, then no, it does not
> happen before this.
My guess is that the PCI bus code would like to use delay() so I rather think it is calibrated prior
to this :(
>
> On these boxes, I'd just recommend disabling USB legacy support
> completly, if possible. And then complain loudly to the vendor to fix
> their BIOS.
But if one is booting from that device...
--
George Anzinger [email protected]
HRT (High-res-timers): http://sourceforge.net/projects/high-res-timers/
On Mon, Nov 14, 2005 at 10:56:05AM -0800, George Anzinger wrote:
> Greg KH wrote:
> >On Sat, Nov 12, 2005 at 08:06:37AM -0800, George Anzinger wrote:
> >
> >>Greg KH wrote:
> >>
> >>>On Fri, Nov 11, 2005 at 01:57:07PM -0800, john stultz wrote:
> >>>
> >>>
> >>>>On Fri, 2005-11-11 at 13:37 -0800, George Anzinger wrote:
> >>>>
> >>>>
> >>>>>John,
> >>>>>
> >>>>>Have you run into this. One of the USB disc controllers has the
> >>>>>ability to boot the system, however, it needs SMM code to do this.
> >>>>>This SMM code, somehow, causes SMI interrupts (which are higher
> >>>>>priority than NMI interrutps and not maskable) which it needs to do
> >>>>>its thing.
> >>>>>
> >>>>>Problem is that if one of these occurs while calibrating the TSC or
> >>>>>the delay code, it can cause a wrong result. We have seen both a too
> >>>>>long and a too short result (depending on where the interrut happens).
> >>>>>
> >>>>>They have found the root cause of TSC calibration problem.
> >>>>>Now they ask for the fix or workaround.
> >>>>>
> >>>>>That is the BIOS is periodically interrupted by USB controller and the
> >>>>>CPU
> >>>>>waits during the processing of these interrupts.
> >>>>>Their experiments say the interrupt interval is 260mSec and the BIOS
> >>>>>needs
> >>>>>150uSec - 200uSec for processing.
> >>>>>It is proved that the problem doesn't reproduce by masking such SMI in
> >>>>>BIOS.
> >>>>>They say SMI is for BIOS emulation for connecting legacy devices to
> >>>>>USB.
> >>>>>Without such an emulation it's impossible to boot from USB-FD for
> >>>>>instance,
> >>>>>they say too.
> >>>>
> >>>>Hmmm. I haven't heard of this issue specifically, but yes, I'm quite
> >>>>familiar with the pain BIOS SMIs can cause and I'm not surprised that it
> >>>>would affect the TSC/delay calibration code.
> >>>>
> >>>>Is this still an issue w/ 2.6.14? I know the new TSC based delay
> >>>>calibration code is supposed to be SMI resilient, but I haven't really
> >>>>played with it closely.
> >>>>
> >>>>Not sure what the best method to move forward would be. I suspect
> >>>>disabling the SMI code early in boot (I thought the usb legacy handoff
> >>>>stuff already did this?) would help. Then the actual Linux USB drivers
> >>>>can take over before we switch from the initrd to the root filesystem.
> >>>>
> >>>>Greg, do you have a suggestion?
> >>>
> >>>
> >>>I only ever saw this when people forgot to load the USB drivers. Once
> >>>the kernel took over USB support, there was no problem (if there was,
> >>>that's a BIOS bug.) The handoff code in 2.6.14 should help a lot with
> >>>this too.
> >>>
> >>
> >>Ah... are you saying that the USB support code stops the SMM/SMI prior to
> >>the TSC & delay calibration? Also, this problem was noted in a 2.4.20
> >>kernel. Any help there?
> >
> >
> >I do not know where in the boot process the TSC and delay calibration is
> >done. If it happens before the PCI bus is probed, then no, it does not
> >happen before this.
>
> My guess is that the PCI bus code would like to use delay() so I rather
> think it is calibrated prior to this :(
> >
> >On these boxes, I'd just recommend disabling USB legacy support
> >completly, if possible. And then complain loudly to the vendor to fix
> >their BIOS.
>
> But if one is booting from that device...
Booting from a USB device? I can see this happening when installing a
distro, and you boot from the USB cdrom, but not for "normal"
operations.
Oh well, publicly mock the manufacturer for doing horrible things in
their BIOS and then no one will buy the boxes, and we will not have
problems :)
thanks,
greg k-h
Greg KH wrote:
>>> completly, if possible. And then complain loudly to the vendor to fix
>>> their BIOS.
>> But if one is booting from that device...
>
> Booting from a USB device? I can see this happening when installing a
> distro, and you boot from the USB cdrom, but not for "normal"
> operations.
>
Just as a point of reference I have a machine here that boots from a USB keystick and runs from an
NFS Root. It's a diskless machine that has no net-boot ability. I have used this on a number of
machines over recent history.
Brad
--
"Human beings, who are almost unique in having the ability
to learn from the experience of others, are also remarkable
for their apparent disinclination to do so." -- Douglas Adams
On Mon, Nov 14, 2005 at 11:46:48PM +0400, Brad Campbell wrote:
> Greg KH wrote:
> >>>completly, if possible. And then complain loudly to the vendor to fix
> >>>their BIOS.
> >>But if one is booting from that device...
> >
> >Booting from a USB device? I can see this happening when installing a
> >distro, and you boot from the USB cdrom, but not for "normal"
> >operations.
> >
>
> Just as a point of reference I have a machine here that boots from a USB
> keystick and runs from an NFS Root. It's a diskless machine that has no
> net-boot ability. I have used this on a number of machines over recent
> history.
Then I guess your BIOS didn't do the horrible things that the original
poster was reporting, which is good :)
thanks,
greg k-h
On Mon, 2005-11-14 at 10:49 -0800, Greg KH wrote:
> On Mon, Nov 14, 2005 at 10:56:05AM -0800, George Anzinger wrote:
> > Greg KH wrote:
> > >On Sat, Nov 12, 2005 at 08:06:37AM -0800, George Anzinger wrote:
> > >>Greg KH wrote:
> > >On these boxes, I'd just recommend disabling USB legacy support
> > >completly, if possible. And then complain loudly to the vendor to fix
> > >their BIOS.
> >
> > But if one is booting from that device...
>
> Booting from a USB device? I can see this happening when installing a
> distro, and you boot from the USB cdrom, but not for "normal"
> operations.
>
> Oh well, publicly mock the manufacturer for doing horrible things in
> their BIOS and then no one will buy the boxes, and we will not have
> problems :)
I suspect the right fix is in-between. We should try to push hardware
makers away from using SMIs recklessly, but we should also do our best
to work around those that don't. The same problems crop up w/
virtualization where time-based calibration may be interrupted.
George, again, there has been some SMI resistant delay calibration code
added recently. You mentioned this problem was seen on 2.4 kernel, so
you could verify that the new code in 2.6.14 works and if so, try
backporting it.
If not we need to see what else we can do about improving delay
calibration (its a similar tick-based problem to what I'm addressing
with the timeofday rework) or reducing the use of delay by using
something else.
thanks
-john
john stultz wrote:
> On Mon, 2005-11-14 at 10:49 -0800, Greg KH wrote:
>
>>On Mon, Nov 14, 2005 at 10:56:05AM -0800, George Anzinger wrote:
>>
>>>Greg KH wrote:
>>>
>>>>On Sat, Nov 12, 2005 at 08:06:37AM -0800, George Anzinger wrote:
>>>>
>>>>>Greg KH wrote:
>>>>
>>>>On these boxes, I'd just recommend disabling USB legacy support
>>>>completly, if possible. And then complain loudly to the vendor to fix
>>>>their BIOS.
>>>
>>>But if one is booting from that device...
>>
>>Booting from a USB device? I can see this happening when installing a
>>distro, and you boot from the USB cdrom, but not for "normal"
>>operations.
I think it was a hard drive they were trying to support.
>>
>>Oh well, publicly mock the manufacturer for doing horrible things in
>>their BIOS and then no one will buy the boxes, and we will not have
>>problems :)
Long term, maybe, but it will not close the bug report I have in hand...
>
>
> I suspect the right fix is in-between. We should try to push hardware
> makers away from using SMIs recklessly, but we should also do our best
> to work around those that don't. The same problems crop up w/
> virtualization where time-based calibration may be interrupted.
>
> George, again, there has been some SMI resistant delay calibration code
> added recently. You mentioned this problem was seen on 2.4 kernel, so
> you could verify that the new code in 2.6.14 works and if so, try
> backporting it.
>
> If not we need to see what else we can do about improving delay
> calibration (its a similar tick-based problem to what I'm addressing
> with the timeofday rework) or reducing the use of delay by using
> something else.
>
I will look at that code, but we also need to address the same problem in the TSC calibration area.
--
George Anzinger [email protected]
HRT (High-res-timers): http://sourceforge.net/projects/high-res-timers/
George Anzinger wrote:
>>> Oh well, publicly mock the manufacturer for doing horrible things in
>>> their BIOS and then no one will buy the boxes, and we will not have
>>> problems :)
>
> Long term, maybe, but it will not close the bug report I have in hand...
>>
>>
>> I suspect the right fix is in-between. We should try to push hardware
>> makers away from using SMIs recklessly, but we should also do our best
>> to work around those that don't. The same problems crop up w/
>> virtualization where time-based calibration may be interrupted.
>>
>> George, again, there has been some SMI resistant delay calibration code
>> added recently. You mentioned this problem was seen on 2.4 kernel, so
>> you could verify that the new code in 2.6.14 works and if so, try
>> backporting it.
>>
>> If not we need to see what else we can do about improving delay
>> calibration (its a similar tick-based problem to what I'm addressing
>> with the timeofday rework) or reducing the use of delay by using
>> something else.
>>
> I will look at that code, but we also need to address the same problem
> in the TSC calibration area.
I think in the short term your best bet is to globally disable SMI at early
boot stage (ie before TSC calibration).
Some people might argue that it's not the most graceful solution because it might
brake some BIOS features but it's a very common trick that is used by RT folks
(for example RTAI has configurable option to enable SMI workaround) because
on some chipset/BIOS combinations SMIs introduce horrible latencies. And you
cannot do much about that other than disabling SMI.
I have not seen any reports of negative side effects of disabling SMI yet. But
if you're worried about that you could re-enable it later when you're done with
TSC calibration and stuff.
Max
On Wed, 2005-11-16 at 16:03 -0800, Max Krasnyansky wrote:
> I think in the short term your best bet is to globally disable SMI at
> early
> boot stage (ie before TSC calibration).
> Some people might argue that it's not the most graceful solution
> because it might
> brake some BIOS features but it's a very common trick that is used by
> RT folks
> (for example RTAI has configurable option to enable SMI workaround)
> because
> on some chipset/BIOS combinations SMIs introduce horrible latencies.
> And you
> cannot do much about that other than disabling SMI.
> I have not seen any reports of negative side effects of disabling SMI
> yet. But
> if you're worried about that you could re-enable it later when you're
> done with
> TSC calibration and stuff.
>
Hmm, interesting, I had no idea this was possible. Is there a generic
way to disable SMI? It would be useful for the -rt tree as lots of low
latency audio users have problems with SMI induced underruns.
Lee
Lee Revell wrote:
> On Wed, 2005-11-16 at 16:03 -0800, Max Krasnyansky wrote:
>> I think in the short term your best bet is to globally disable SMI at early
>> boot stage (ie before TSC calibration).
>
> Hmm, interesting, I had no idea this was possible. Is there a generic
> way to disable SMI? It would be useful for the -rt tree as lots of low
> latency audio users have problems with SMI induced underruns.
There is no generic way it's chipset specific.
Take a look at the attached smi.c that handles several Intel chipsets.
It's from Xenomai (ex Fusion branch of RTAI) source tree.
Max