Performance degrades by about 60% when I run Wine under any of the
2.6.24.* kernels. Attached are the output of lspci -vv and the two
config files of each kernel.
I upgraded to 2.6.24, back to 2.6.23.(not sure), then tried 2.6.24.1 and
2.6.24.3 and the issue is present in both; it's severe enough that Wine
apps are virtually unusable for me with this version of the kernel; I'm
having to use 2.6.23 until this somehow is resolved.
I'd like some help figuring out why this performance regression exists,
and what can be done to mitigate it.
Please CC replies; I'm not on the list.
On Sat, Mar 22, 2008 at 7:49 PM, Kai <[email protected]> wrote:
> Performance degrades by about 60% when I run Wine under any of the
> 2.6.24.* kernels. Attached are the output of lspci -vv and the two
> config files of each kernel.
>
> I upgraded to 2.6.24, back to 2.6.23.(not sure), then tried 2.6.24.1 and
> 2.6.24.3 and the issue is present in both; it's severe enough that Wine
> apps are virtually unusable for me with this version of the kernel; I'm
> having to use 2.6.23 until this somehow is resolved.
>
> I'd like some help figuring out why this performance regression exists,
> and what can be done to mitigate it.
As wine has a 'wineserver' running in a separate process, it may be
related to scheduler changes.
Regardless, if you have the time, please retest using he latest git
head (or nightly snapshot), and see if the performance regression is
still there, and report back. (There have been a lot of changes
between 2.6.24 and current git head that impact the scheduler.)
Please ensure that the fair group scheduler is disabled in your tests
(just as you have in your 2.6.24 config you attached).
On Sat, 22 Mar 2008 21:47:52 -0700, "Ray Lee" <[email protected]>
said:
> On Sat, Mar 22, 2008 at 7:49 PM, Kai <[email protected]> wrote:
> > Performance degrades by about 60% when I run Wine under any of the
> > 2.6.24.* kernels. Attached are the output of lspci -vv and the two
> > config files of each kernel.
> >
> > I upgraded to 2.6.24, back to 2.6.23.(not sure), then tried 2.6.24.1 and
> > 2.6.24.3 and the issue is present in both; it's severe enough that Wine
> > apps are virtually unusable for me with this version of the kernel; I'm
> > having to use 2.6.23 until this somehow is resolved.
> >
> > I'd like some help figuring out why this performance regression exists,
> > and what can be done to mitigate it.
>
> As wine has a 'wineserver' running in a separate process, it may be
> related to scheduler changes.
>
> Regardless, if you have the time, please retest using he latest git
> head (or nightly snapshot), and see if the performance regression is
> still there, and report back. (There have been a lot of changes
> between 2.6.24 and current git head that impact the scheduler.)
>
> Please ensure that the fair group scheduler is disabled in your tests
> (just as you have in your 2.6.24 config you attached).
If anything, 2.6.25-rc6-git7 is even worse. Previously I've simply had
delayed input, this time I had to actually kill the process, because it
was unresponsive to keyboard input.
Here's the config file; I'm pretty sure I set it up correctly, (I just
copied it from .24), but if I need to try it with any differing
settings, let me know.
Kai wrote:
> Performance degrades by about 60% when I run Wine under any of the
> 2.6.24.* kernels. Attached are the output of lspci -vv and the two
> config files of each kernel.
>
> I upgraded to 2.6.24, back to 2.6.23.(not sure), then tried 2.6.24.1 and
> 2.6.24.3 and the issue is present in both; it's severe enough that Wine
> apps are virtually unusable for me with this version of the kernel; I'm
> having to use 2.6.23 until this somehow is resolved.
>
> I'd like some help figuring out why this performance regression exists,
> and what can be done to mitigate it.
>
> Please CC replies; I'm not on the list.
>
This is strange, as my (two) Windows apps ran (and run) just fine under
2.6.24 (and now under 2.6.25-rc6). Could you specify the applications
you're having problems with?
David
Originally, it was just about everything -- Wine itself, and most any
app I tried to run with it.
In my most recent tests, however, it only seems to affect Age of Wonders
II and Age of Wonders Shadow Magic, which use very similar engines.
On Sun, 23 Mar 2008 10:44:06 +0000, "David" <[email protected]>
said:
> Kai wrote:
> > Performance degrades by about 60% when I run Wine under any of the
> > 2.6.24.* kernels. Attached are the output of lspci -vv and the two
> > config files of each kernel.
> >
> > I upgraded to 2.6.24, back to 2.6.23.(not sure), then tried 2.6.24.1 and
> > 2.6.24.3 and the issue is present in both; it's severe enough that Wine
> > apps are virtually unusable for me with this version of the kernel; I'm
> > having to use 2.6.23 until this somehow is resolved.
> >
> > I'd like some help figuring out why this performance regression exists,
> > and what can be done to mitigate it.
> >
> > Please CC replies; I'm not on the list.
> >
> This is strange, as my (two) Windows apps ran (and run) just fine under
> 2.6.24 (and now under 2.6.25-rc6). Could you specify the applications
> you're having problems with?
>
> David
(Please don't top post. If we figure out who to CC: who knows more
about this than I do -- which wouldn't be hard -- then they're going
to want to see all this in order. Also, please always do a
reply-to-all on this list, it's preferred.)
On Sun, Mar 23, 2008 at 3:10 PM, Kai <[email protected]> wrote:
> > Kai wrote:
> > > Performance degrades by about 60% when I run Wine under any of the
> > > 2.6.24.* kernels. Attached are the output of lspci -vv and the two
> > > config files of each kernel.
> > >
> > > I upgraded to 2.6.24, back to 2.6.23.(not sure), then tried 2.6.24.1 and
> > > 2.6.24.3 and the issue is present in both; it's severe enough that Wine
> > > apps are virtually unusable for me with this version of the kernel; I'm
> > > having to use 2.6.23 until this somehow is resolved.
> > >
> > > I'd like some help figuring out why this performance regression exists,
> > > and what can be done to mitigate it.
> > >
> > > Please CC replies; I'm not on the list.
> > >
> > This is strange, as my (two) Windows apps ran (and run) just fine under
> > 2.6.24 (and now under 2.6.25-rc6). Could you specify the applications
> > you're having problems with?
>
> Originally, it was just about everything -- Wine itself, and most any
> app I tried to run with it.
>
> In my most recent tests, however, it only seems to affect Age of Wonders
> II and Age of Wonders Shadow Magic, which use very similar engines.
What version of X? What version of WINE? Which video driver are you
using? Does the speed issue show up with glxgears, or something else
that would be much easier for someone else to reproduce (tuxracer,
etc.)? Is your system using software or hardware rendering?
>
> This is strange, as my (two) Windows apps ran (and run) just fine under
> 2.6.24 (and now under 2.6.25-rc6). Could you specify the applications
> you're having problems with?
>
>
Kai wrote:
> Originally, it was just about everything -- Wine itself, and most any
> app I tried to run with it.
>
> In my most recent tests, however, it only seems to affect Age of Wonders
> II and Age of Wonders Shadow Magic, which use very similar engines.
>
> On Sun, 23 Mar 2008 10:44:06 +0000, "David" <[email protected]>
> said:
>
OK.. I don't run Windows games, so won't have come across these issues.
Hopefully someone with scheduler knowledge might be able to reproduce.
David
"Kai" <[email protected]> writes:
> Originally, it was just about everything -- Wine itself, and most any
> app I tried to run with it.
>
> In my most recent tests, however, it only seems to affect Age of Wonders
> II and Age of Wonders Shadow Magic, which use very similar engines.
A simple test if the scheduler is likely to blame would be to compare
context switch rates between the different kernel versions who show
differing performance. You can do that with running "vmstat 1" in
parallel and checking the "cs" column.
Another powerful but quite slow method if you can easily reproduce it
would be a full git bisect. That would identify exactly which change
introduced your problem.
-Andi
On 24 Mar 2008 04:55:36 +0100, "Andi Kleen" <[email protected]> said:
> "Kai" <[email protected]> writes:
>
> > Originally, it was just about everything -- Wine itself, and most any
> > app I tried to run with it.
> >
> > In my most recent tests, however, it only seems to affect Age of Wonders
> > II and Age of Wonders Shadow Magic, which use very similar engines.
>
> A simple test if the scheduler is likely to blame would be to compare
> context switch rates between the different kernel versions who show
> differing performance. You can do that with running "vmstat 1" in
> parallel and checking the "cs" column.
>
> Another powerful but quite slow method if you can easily reproduce it
> would be a full git bisect. That would identify exactly which change
> introduced your problem.
>
> -Andi
Can you explain more clearly what you mean with vmstat?
Also, I'd like to find a way to feasibly do the git bisect (I assume you
basically mean testing against every change in sequence since .23), but
I'd like some advice on how to do this, and how to do it reasonably
expeditiously.
> On Sun, Mar 23, 2008 at 3:10 PM, Kai <[email protected]> wrote:
> > > Kai wrote:
> > > > Performance degrades by about 60% when I run Wine under any of the
> > > > 2.6.24.* kernels. Attached are the output of lspci -vv and the two
> > > > config files of each kernel.
> > > >
> > > > I upgraded to 2.6.24, back to 2.6.23.(not sure), then tried 2.6.24.1 and
> > > > 2.6.24.3 and the issue is present in both; it's severe enough that Wine
> > > > apps are virtually unusable for me with this version of the kernel; I'm
> > > > having to use 2.6.23 until this somehow is resolved.
> > > >
> > > > I'd like some help figuring out why this performance regression exists,
> > > > and what can be done to mitigate it.
> > > >
> > > > Please CC replies; I'm not on the list.
> > > >
> > > This is strange, as my (two) Windows apps ran (and run) just fine under
> > > 2.6.24 (and now under 2.6.25-rc6). Could you specify the applications
> > > you're having problems with?
> >
> > Originally, it was just about everything -- Wine itself, and most any
> > app I tried to run with it.
> >
> > In my most recent tests, however, it only seems to affect Age of Wonders
> > II and Age of Wonders Shadow Magic, which use very similar engines.
>
> What version of X? What version of WINE? Which video driver are you
> using? Does the speed issue show up with glxgears, or something else
> that would be much easier for someone else to reproduce (tuxracer,
> etc.)? Is your system using software or hardware rendering?
The issue only shows with Age of Wonders II and AoW Shadow Magic.
Every version of wine from 0.9.55 to 0.9.58, including several
intermediate git builds.
Xorg version 1.4.0.90, and a couple other versions (the problems has
persisted across a couple updates of Xorg)
Hardware rendering; using the prop. nV driver; updating that also
doesn't affect the issue, nor does retaining the same version.
> Can you explain more clearly what you mean with vmstat?
Open shell window. Enter "vmstat 1". Run Wine application. Watch "cs"
column over time.
An possible improvement would be also to use a larger number than 1 to get
values over long time.
Compare that value to the old kernel.
>
> Also, I'd like to find a way to feasibly do the git bisect (I assume you
> basically mean testing against every change in sequence since .23), but
Yes, although it is just every 1+log2 changes
> I'd like some advice on how to do this, and how to do it reasonably
> expeditiously.
Type git bisect into google, there are lots of tutorials.
-Andi
>
On Sat, 22 Mar 2008 21:47:52 -0700, "Ray Lee" <[email protected]>
said:
> On Sat, Mar 22, 2008 at 7:49 PM, Kai <[email protected]> wrote:
> > Performance degrades by about 60% when I run Wine under any of the
> > 2.6.24.* kernels. Attached are the output of lspci -vv and the two
> > config files of each kernel.
> >
> > I upgraded to 2.6.24, back to 2.6.23.(not sure), then tried 2.6.24.1 and
> > 2.6.24.3 and the issue is present in both; it's severe enough that Wine
> > apps are virtually unusable for me with this version of the kernel; I'm
> > having to use 2.6.23 until this somehow is resolved.
> >
> > I'd like some help figuring out why this performance regression exists,
> > and what can be done to mitigate it.
>
> As wine has a 'wineserver' running in a separate process, it may be
> related to scheduler changes.
>
> Regardless, if you have the time, please retest using he latest git
> head (or nightly snapshot), and see if the performance regression is
> still there, and report back. (There have been a lot of changes
> between 2.6.24 and current git head that impact the scheduler.)
>
> Please ensure that the fair group scheduler is disabled in your tests
> (just as you have in your 2.6.24 config you attached).
As mentioned in another response, it was happening as recently as
2.6.25-rc6-git7; I'm currently performing a git bisect between 2.6.23
and 2.6.24, unless someone has a better idea; it seems my best option,
as I'm not really very experienced with kernel hacking or debugging.
On Tue, Mar 25, 2008 at 5:12 AM, Kai <[email protected]> wrote:
>
> On Sat, 22 Mar 2008 21:47:52 -0700, "Ray Lee" <[email protected]>
> said:
>
>
> > On Sat, Mar 22, 2008 at 7:49 PM, Kai <[email protected]> wrote:
> > > Performance degrades by about 60% when I run Wine under any of the
> > > 2.6.24.* kernels. Attached are the output of lspci -vv and the two
> > > config files of each kernel.
> > >
> > > I upgraded to 2.6.24, back to 2.6.23.(not sure), then tried 2.6.24.1 and
> > > 2.6.24.3 and the issue is present in both; it's severe enough that Wine
> > > apps are virtually unusable for me with this version of the kernel; I'm
> > > having to use 2.6.23 until this somehow is resolved.
> > >
> > > I'd like some help figuring out why this performance regression exists,
> > > and what can be done to mitigate it.
> >
> > As wine has a 'wineserver' running in a separate process, it may be
> > related to scheduler changes.
> >
> > Regardless, if you have the time, please retest using he latest git
> > head (or nightly snapshot), and see if the performance regression is
> > still there, and report back. (There have been a lot of changes
> > between 2.6.24 and current git head that impact the scheduler.)
> >
> > Please ensure that the fair group scheduler is disabled in your tests
> > (just as you have in your 2.6.24 config you attached).
>
> As mentioned in another response, it was happening as recently as
> 2.6.25-rc6-git7; I'm currently performing a git bisect between 2.6.23
> and 2.6.24, unless someone has a better idea; it seems my best option,
> as I'm not really very experienced with kernel hacking or debugging.
>
Andi's idea of looking for excessive context switches is good -- I
didn't see a response to that one. Other than that, if you're only
noticing the issue in 3d games, then it could be several things (not
just the scheduler). Even just a few bisects (or testings of nightly
snapshots) would help narrow it down.
Ray
* Kai <[email protected]> wrote:
> The issue only shows with Age of Wonders II and AoW Shadow Magic.
>
> Every version of wine from 0.9.55 to 0.9.58, including several
> intermediate git builds.
>
> Xorg version 1.4.0.90, and a couple other versions (the problems has
> persisted across a couple updates of Xorg)
>
> Hardware rendering; using the prop. nV driver; updating that also
> doesn't affect the issue, nor does retaining the same version.
could you run this script while such a slowdown is really prominent:
http://people.redhat.com/mingo/cfs-scheduler/tools/cfs-debug-info.sh
and send me the output it generates? The output is the most useful if
you do this on a kernel that has CONFIG_SCHED_DEBUG=y and
CONFIG_SCHEDSTATS=y enabled.
on the off chance that this issue has been fixed in the soon-to-be
2.6.25 kernel, you might also want to try x86.git/latest, which is based
on the latest Linus tree and has all relevant x86 fixes and improvements
added as well:
http://people.redhat.com/mingo/x86.git/README
several of the changes can affect performance.
a third (and most comprehensive) way to debug this would be to send me a
scheduler trace of such a slowdown, you can generate a scheduler trace
the following way:
http://people.redhat.com/mingo/sched-devel.git/readme-tracer.txt
but we can probably give a first estimation based on the cfs-debug-info
output already. Btw., you can combine the scheduler and the x86 git tree
into a temporary unified tree by doing these two commands:
git-checkout -b tmp x86/latest
git-merge sched-devel/latest
(run "make oldconfig" to pick up the new config options.)
Ingo