I *think* this is the right place to ask this, and apologies if it's not
(is there a better place?).
We have checks which read /proc/slabinfo once a minute, and have noticed
that this causes the entire system to stall for a few milliseconds.
It's long enough that it causes noticeable delays in latency-sensitive
applications (between 10ms and 100ms).
Is this a known condition? Are there work arounds or other ways to get the
slab allocation data which don't cause stalls?
Thanks :-)
Hi Avleen,
On Tue, Oct 16, 2012 at 8:51 PM, Avleen Vig <[email protected]> wrote:
> I *think* this is the right place to ask this, and apologies if it's not
> (is there a better place?).
>
> We have checks which read /proc/slabinfo once a minute, and have noticed
> that this causes the entire system to stall for a few milliseconds.
> It's long enough that it causes noticeable delays in latency-sensitive
> applications (between 10ms and 100ms).
>
> Is this a known condition? Are there work arounds or other ways to get the
> slab allocation data which don't cause stalls?
What kernel version are you using? What does your .config look like?
By default we use the 2.6.32-279.5.2 as provided by CentOS.
However I also tried the 3.0.46 from elrepo:
http://elrepo.org/linux/kernel/el6/x86_64/RPMS/
The config for the 3.0 kernel is:
https://gist.github.com/3901068
They both seem to exhibit the same problem.
Thanks :)
On Tue, Oct 16, 2012 at 1:04 PM, Pekka Enberg <[email protected]> wrote:
> Hi Avleen,
>
> On Tue, Oct 16, 2012 at 8:51 PM, Avleen Vig <[email protected]> wrote:
>> I *think* this is the right place to ask this, and apologies if it's not
>> (is there a better place?).
>>
>> We have checks which read /proc/slabinfo once a minute, and have noticed
>> that this causes the entire system to stall for a few milliseconds.
>> It's long enough that it causes noticeable delays in latency-sensitive
>> applications (between 10ms and 100ms).
>>
>> Is this a known condition? Are there work arounds or other ways to get the
>> slab allocation data which don't cause stalls?
>
> What kernel version are you using? What does your .config look like?
On Tue, 16 Oct 2012, Pekka Enberg wrote:
> > Is this a known condition? Are there work arounds or other ways to get the
> > slab allocation data which don't cause stalls?
>
> What kernel version are you using? What does your .config look like?
slabinfo access requires a mutex which will stall certain slab operations
(shrinking caches, retuning caches, cleaning caches in SLAB).
If the allocator is SLAB then I would think that this is a stall of
slabinfo mutex vs. slab clean operations (gets more frequent the more
processors are in the box).
If this is the case then using the SLUB allocator will fix the issue since
SLUB does not need to perform cache cleaning.
On Tue, 16 Oct 2012, Christoph Lameter wrote:
> If the allocator is SLAB then I would think that this is a stall of
> slabinfo mutex vs. slab clean operations (gets more frequent the more
> processors are in the box).
>
> If this is the case then using the SLUB allocator will fix the issue since
> SLUB does not need to perform cache cleaning.
>
Or convert slab_mutex to be a rwlock?
On Tue, 16 Oct 2012, David Rientjes wrote:
> Or convert slab_mutex to be a rwlock?
The RT folks just converted it to a mutex.... ;-)
On Tue, Oct 16, 2012 at 1:55 PM, Christoph Lameter <[email protected]> wrote:
> On Tue, 16 Oct 2012, David Rientjes wrote:
>
>> Or convert slab_mutex to be a rwlock?
>
> The RT folks just converted it to a mutex.... ;-)
Damn :-)
I'll play around with SLUB.
There were some concerns that it doesn't perform as well as SLAB on
large hardware (we're almost always running >16G RAM, and regularly
>96G RAM on things like memcache servers).
If the performance pans out, SLUB would be great.
On Tue, 16 Oct 2012, Avleen Vig wrote:
> There were some concerns that it doesn't perform as well as SLAB on
> large hardware (we're almost always running >16G RAM, and regularly
> >96G RAM on things like memcache servers).
> If the performance pans out, SLUB would be great.
We have been using SLUB here for years with servers up to 256G RAM and
64 processors in both HPC and Enterprise environments.