2013-03-01 14:02:30

by Hillf Danton

[permalink] [raw]
Subject: Re: kswapd craziness round 2

On Fri, Mar 1, 2013 at 1:02 AM, Jiri Slaby <[email protected]> wrote:
>
> Ok, no difference, kswap is still crazy. I'm attaching the output of
> "grep -vw '0' /proc/vmstat" if you see something there.
>
Thanks to you for test and data.

Lets try to restore the deleted nap, then.

Hillf
--- a/mm/vmscan.c Thu Feb 21 20:01:02 2013
+++ b/mm/vmscan.c Fri Mar 1 21:55:40 2013
@@ -2817,6 +2817,10 @@ loop_again:
*/
if (sc.nr_reclaimed >= SWAP_CLUSTER_MAX)
break;
+
+ if (sc.priority < DEF_PRIORITY - 2)
+ congestion_wait(BLK_RW_ASYNC, HZ/10);
+
} while (--sc.priority >= 0);

out:
--


2013-03-07 19:37:22

by Jiri Slaby

[permalink] [raw]
Subject: Re: kswapd craziness round 2

On 03/01/2013 03:02 PM, Hillf Danton wrote:
> On Fri, Mar 1, 2013 at 1:02 AM, Jiri Slaby <[email protected]> wrote:
>>
>> Ok, no difference, kswap is still crazy. I'm attaching the output of
>> "grep -vw '0' /proc/vmstat" if you see something there.
>>
> Thanks to you for test and data.
>
> Lets try to restore the deleted nap, then.

Oh, it seems to be nice now:
root 579 0.0 0.0 0 0 ? S Mar04 0:13 [kswapd0]

Thanks.

> Hillf
> --- a/mm/vmscan.c Thu Feb 21 20:01:02 2013
> +++ b/mm/vmscan.c Fri Mar 1 21:55:40 2013
> @@ -2817,6 +2817,10 @@ loop_again:
> */
> if (sc.nr_reclaimed >= SWAP_CLUSTER_MAX)
> break;
> +
> + if (sc.priority < DEF_PRIORITY - 2)
> + congestion_wait(BLK_RW_ASYNC, HZ/10);
> +
> } while (--sc.priority >= 0);
>
> out:
> --
>


--
js
suse labs

2013-03-08 06:42:33

by Hillf Danton

[permalink] [raw]
Subject: Re: kswapd craziness round 2

On Fri, Mar 8, 2013 at 3:37 AM, Jiri Slaby <[email protected]> wrote:
> On 03/01/2013 03:02 PM, Hillf Danton wrote:
>> On Fri, Mar 1, 2013 at 1:02 AM, Jiri Slaby <[email protected]> wrote:
>>>
>>> Ok, no difference, kswap is still crazy. I'm attaching the output of
>>> "grep -vw '0' /proc/vmstat" if you see something there.
>>>
>> Thanks to you for test and data.
>>
>> Lets try to restore the deleted nap, then.
>
> Oh, it seems to be nice now:
> root 579 0.0 0.0 0 0 ? S Mar04 0:13 [kswapd0]
>
Double thanks.

But Mel does not like it, probably.
Lets try nap in another way.

Hillf

--- a/mm/vmscan.c Thu Feb 21 20:01:02 2013
+++ b/mm/vmscan.c Fri Mar 8 14:36:10 2013
@@ -2793,6 +2793,10 @@ loop_again:
* speculatively avoid congestion waits
*/
zone_clear_flag(zone, ZONE_CONGESTED);
+
+ else if (sc.priority > 2 &&
+ sc.priority < DEF_PRIORITY - 2)
+ wait_iff_congested(zone, BLK_RW_ASYNC, HZ/10);
}

/*
--

>>
>> --- a/mm/vmscan.c Thu Feb 21 20:01:02 2013
>> +++ b/mm/vmscan.c Fri Mar 1 21:55:40 2013
>> @@ -2817,6 +2817,10 @@ loop_again:
>> */
>> if (sc.nr_reclaimed >= SWAP_CLUSTER_MAX)
>> break;
>> +
>> + if (sc.priority < DEF_PRIORITY - 2)
>> + congestion_wait(BLK_RW_ASYNC, HZ/10);
>> +
>> } while (--sc.priority >= 0);
>>
>> out:
>> --
>>
>
>
> --
> js
> suse labs

2013-03-08 07:40:26

by Zlatko Calusic

[permalink] [raw]
Subject: Re: kswapd craziness round 2

On 08.03.2013 07:42, Hillf Danton wrote:
> On Fri, Mar 8, 2013 at 3:37 AM, Jiri Slaby <[email protected]> wrote:
>> On 03/01/2013 03:02 PM, Hillf Danton wrote:
>>> On Fri, Mar 1, 2013 at 1:02 AM, Jiri Slaby <[email protected]> wrote:
>>>>
>>>> Ok, no difference, kswap is still crazy. I'm attaching the output of
>>>> "grep -vw '0' /proc/vmstat" if you see something there.
>>>>
>>> Thanks to you for test and data.
>>>
>>> Lets try to restore the deleted nap, then.
>>
>> Oh, it seems to be nice now:
>> root 579 0.0 0.0 0 0 ? S Mar04 0:13 [kswapd0]
>>
> Double thanks.
>
> But Mel does not like it, probably.
> Lets try nap in another way.
>
> Hillf
>
> --- a/mm/vmscan.c Thu Feb 21 20:01:02 2013
> +++ b/mm/vmscan.c Fri Mar 8 14:36:10 2013
> @@ -2793,6 +2793,10 @@ loop_again:
> * speculatively avoid congestion waits
> */
> zone_clear_flag(zone, ZONE_CONGESTED);
> +
> + else if (sc.priority > 2 &&
> + sc.priority < DEF_PRIORITY - 2)
> + wait_iff_congested(zone, BLK_RW_ASYNC, HZ/10);
> }
>
> /*
> --
>

There's another bug in there, which I'm still chasing. Artificial sleeps
like this just mask the real bug and introduce new problems (on my 4GB
server kswapd spends all the time in those congestion wait calls). The
problem is that the bug needs about 5 days of uptime to reveal it's ugly
head. So far I can only tell that it was introduced somewhere between
3.1 & 3.4.

Also, check shrink_inactive_list(), it already sleeps if really needed:

if (nr_writeback && nr_writeback >=
(nr_taken >> (DEF_PRIORITY - sc->priority)))
wait_iff_congested(zone, BLK_RW_ASYNC, HZ/10);

Regards,
--
Zlatko

2013-03-08 08:27:54

by Hillf Danton

[permalink] [raw]
Subject: Re: kswapd craziness round 2

On Fri, Mar 8, 2013 at 3:29 PM, Zlatko Calusic <[email protected]> wrote:
> There's another bug in there, which I'm still chasing.
>
I am busy in discovering an employer(a really hard work?) so
I dunno the hours I have for that bug.

Hmm, take a look at Mels thoughts?
http://marc.info/?l=linux-mm&m=136189593423501&w=2

BTW, he will be online next week.

Hillf

2013-03-08 23:21:13

by Jiri Slaby

[permalink] [raw]
Subject: Re: kswapd craziness round 2

On 03/08/2013 07:42 AM, Hillf Danton wrote:
> On Fri, Mar 8, 2013 at 3:37 AM, Jiri Slaby <[email protected]> wrote:
>> On 03/01/2013 03:02 PM, Hillf Danton wrote:
>>> On Fri, Mar 1, 2013 at 1:02 AM, Jiri Slaby <[email protected]> wrote:
>>>>
>>>> Ok, no difference, kswap is still crazy. I'm attaching the output of
>>>> "grep -vw '0' /proc/vmstat" if you see something there.
>>>>
>>> Thanks to you for test and data.
>>>
>>> Lets try to restore the deleted nap, then.
>>
>> Oh, it seems to be nice now:
>> root 579 0.0 0.0 0 0 ? S Mar04 0:13 [kswapd0]
>>
> Double thanks.

There is one downside. I'm not sure whether that patch was the culprit.
My Thunderbird is jerky when scrolling and lags while writing this
message. The letters sometimes appear later than typed and in groups. Like
I (kbd): My Thunder
TB: My Thunder
I (kbd): b-i-r-d
TB: is silent
I (kbd): still typing...
TB: bird is

Perhaps it's not only TB.

> But Mel does not like it, probably.
> Lets try nap in another way.

Will try next week.

> --- a/mm/vmscan.c Thu Feb 21 20:01:02 2013
> +++ b/mm/vmscan.c Fri Mar 8 14:36:10 2013
> @@ -2793,6 +2793,10 @@ loop_again:
> * speculatively avoid congestion waits
> */
> zone_clear_flag(zone, ZONE_CONGESTED);
> +
> + else if (sc.priority > 2 &&
> + sc.priority < DEF_PRIORITY - 2)
> + wait_iff_congested(zone, BLK_RW_ASYNC, HZ/10);
> }
>
> /*

--
js
suse labs

2013-03-19 17:01:21

by Pádraig Brady

[permalink] [raw]
Subject: Re: kswapd craziness round 2

On 03/08/2013 11:21 PM, Jiri Slaby wrote:
> On 03/08/2013 07:42 AM, Hillf Danton wrote:
>> On Fri, Mar 8, 2013 at 3:37 AM, Jiri Slaby <[email protected]> wrote:
>>> On 03/01/2013 03:02 PM, Hillf Danton wrote:
>>>> On Fri, Mar 1, 2013 at 1:02 AM, Jiri Slaby <[email protected]> wrote:
>>>>>
>>>>> Ok, no difference, kswap is still crazy. I'm attaching the output of
>>>>> "grep -vw '0' /proc/vmstat" if you see something there.
>>>>>
>>>> Thanks to you for test and data.
>>>>
>>>> Lets try to restore the deleted nap, then.
>>>
>>> Oh, it seems to be nice now:
>>> root 579 0.0 0.0 0 0 ? S Mar04 0:13 [kswapd0]
>>>
>> Double thanks.
>
> There is one downside. I'm not sure whether that patch was the culprit.
> My Thunderbird is jerky when scrolling and lags while writing this
> message. The letters sometimes appear later than typed and in groups. Like
> I (kbd): My Thunder
> TB: My Thunder
> I (kbd): b-i-r-d
> TB: is silent
> I (kbd): still typing...
> TB: bird is
>
> Perhaps it's not only TB.

I notice the same thunderbird issue on the much older 2.6.40.4-5.fc15.x86_64
which I'd hoped would be fixed on upgrade :(

My Thunderbird is using 1957m virt, 722m RSS on my 3G system.
What are your corresponding mem values?

For reference:
http://marc.info/?t=130865025500001&r=1&w=2
https://bugzilla.redhat.com/show_bug.cgi?id=712019

thanks,
Pádraig.

2013-03-20 04:12:34

by Hillf Danton

[permalink] [raw]
Subject: Re: kswapd craziness round 2

On Wed, Mar 20, 2013 at 12:59 AM, Pádraig Brady <[email protected]> wrote:
>
> I notice the same thunderbird issue on the much older 2.6.40.4-5.fc15.x86_64
> which I'd hoped would be fixed on upgrade :(
>
> My Thunderbird is using 1957m virt, 722m RSS on my 3G system.
> What are your corresponding mem values?
>
> For reference:
> http://marc.info/?t=130865025500001&r=1&w=2
> https://bugzilla.redhat.com/show_bug.cgi?id=712019
>
Hey, would you all please try Mels new work?
http://marc.info/?l=linux-mm&m=136352546814642&w=4

thanks
Hillf

2013-03-20 08:38:48

by Jiri Slaby

[permalink] [raw]
Subject: Re: kswapd craziness round 2

On 03/20/2013 05:12 AM, Hillf Danton wrote:
> Hey, would you all please try Mels new work?
> http://marc.info/?l=linux-mm&m=136352546814642&w=4

Yeah, I was in CC and also asked Mel if I should apply those. I will as
soon as I'm back home (next week).

thanks,
--
js
suse labs