Subject: Re: Quad core CPUs loaded at only 50% when running a CPU and mmap
	intensive multi-threaded task
From: Peter Zijlstra <peterz@infradead.org>
To: edwin <edwintorok@gmail.com>
Cc: Ingo Molnar <mingo@elte.hu>, rml@tech9.net,
       Linux Kernel <linux-kernel@vger.kernel.org>,
       "Thomas Gleixner mingo@redhat.com" <tglx@linutronix.de>,
       "H. Peter Anvin" <hpa@zytor.com>
In-Reply-To: <48B25988.8040302@gmail.com>
References: <48B1CC15.2040006@gmail.com> <1219643476.20732.1.camel@twins>
	 <48B25988.8040302@gmail.com>
Content-Type: text/plain; charset=utf-8
Date: Mon, 25 Aug 2008 11:23:10 +0200
Message-Id: <1219656190.8515.7.camel@twins>
Mime-Version: 1.0
Content-Transfer-Encoding: 8bit
Sender: linux-kernel-owner@vger.kernel.org
Content-Length: 2239
Lines: 57

On Mon, 2008-08-25 at 10:04 +0300, edwin wrote:
> Peter Zijlstra wrote:
> > On Mon, 2008-08-25 at 00:01 +0300, Török Edwin wrote:
> >   
> >> Hi Ingo,
> >>
> >> When I run clamd (www.clamav.net), I can only get to load my CPU 50% 
> >> (according to top), and disks at 30% (according to iostat -x 3), 
> >> regardless how many threads I set (I tried 4, 8, 16, 32).
> >>     
> >
> >
> > Can you share your .config, and prehaps tell what kernel version did
> > work for you?
> 
> Sorry, I forgot to include the .config, its at the end of this mail (the 
> cfs debug info output included the .config though)
> 
> Well, I just bought this new box, so there isn't a kernel version that I 
> know that worked on this hardware (but I am trying to boot some older 
> versions now).
> However on my previous box (Athlon64, non-SMP) I have never seen such a 
> problem (that the CPU is loaded only 50% with clamd) and I've been
> running 2.6.26 and 2.6.27-rc4 there too.
> 
> Details below, short summary here:
> 2.6.24: WORKS, clamd 400% CPU, testprogram runs in 27.4 seconds, 67% CPU 
> load; and 28.5 seconds w/o setting affinity
> 2.6.25+: DOES NOT WORK, clamd 200%-300% CPU, testprogram runs in 38-40 
> seconds, 48-48% CPU load, and 47-56 seconds w/o setting affinity
> 
> Debian has 2.6.18, 2.6.22, 2.6.24, 2.6.25, 2.6.26.
> 2.6.22 won't work with my lvm, so I can't boot that, so I tried 2.6.24:
> 
> 2.6.24 doesn't have sched_debug enabled in the stock kernel 
> unfortunately, but the output of cfs-debug-info.sh is available here, 
> maybe it contains some useful info:
> http://edwintorok.googlepages.com/testrun-1219645937.tar.gz
> 
> Is this enough info for you to reproduce the problem, or do you want me 
> to try and bisect?

No, I think I know what's going on..

mmap() and munmap() need to take the mmap_sem for writing (since they
modify the memory map) and you let each thread (one for each cpu) take
that process wide lock, twice, for a million times.

Guess what happens ;-)


--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/