Hi all,
slub.c sets the default value of ARCH_SLAB_MINALIGN to sizeof(unsigned
long long) if the architecture didn't already override it.
And as x86_32 doesn't set a value this means that slab objects get
aligned to 8 bytes, potentially wasting 4 bytes per object. Slub forces
objects to be aligned to sizeof(void *) anyway, but I don't see that
there is any need for it to be 8 on 32bits.
I'm working on a patch to pack more buffer_heads into each kmem_cache
slab page.
On 32 bits the structure size is 52 bytes and with the alignment applied
I end up with a slab of 73 x 56 byte objects. However, if the minimum
alignment was sizeof(void *) then I'd get 78 x 52 byte objects. So there
is quite a memory saving to be had in changing this.
Can I define a ARCH_SLAB_MINALIGN in x86_64 to sizeof(void *) ?
or would it be ok to change the default in slub.c to sizeof(void *) ?
Or am I missing something ?
regards
Richard
On Wed, 3 Feb 2010, Richard Kennedy wrote:
> slub.c sets the default value of ARCH_SLAB_MINALIGN to sizeof(unsigned
> long long) if the architecture didn't already override it.
>
> And as x86_32 doesn't set a value this means that slab objects get
> aligned to 8 bytes, potentially wasting 4 bytes per object. Slub forces
> objects to be aligned to sizeof(void *) anyway, but I don't see that
> there is any need for it to be 8 on 32bits.
Note that 64 bit entities may exist even under 32 bit (llong) that need
to be aligned properly.
struct buffer_head contains a sector_t which is 64 bit so you should align
to an 8 byte boundary.
> I'm working on a patch to pack more buffer_heads into each kmem_cache
> slab page.
> On 32 bits the structure size is 52 bytes and with the alignment applied
> I end up with a slab of 73 x 56 byte objects. However, if the minimum
> alignment was sizeof(void *) then I'd get 78 x 52 byte objects. So there
> is quite a memory saving to be had in changing this.
SLUB is not restricted to order 0 pages and can use order 1 or 2 pages as
long as this reduces the memory footprint (byte wastage in a slab page is
reduced) and as long as the kernel has contiguous memory available. It
will use order 0 when memory is fragmented.
> Can I define a ARCH_SLAB_MINALIGN in x86_64 to sizeof(void *) ?
> or would it be ok to change the default in slub.c to sizeof(void *) ?
>
> Or am I missing something ?
I'd say leave it alone.
On Wed, 2010-02-03 at 09:41 -0600, Christoph Lameter wrote:
> On Wed, 3 Feb 2010, Richard Kennedy wrote:
>
> > slub.c sets the default value of ARCH_SLAB_MINALIGN to sizeof(unsigned
> > long long) if the architecture didn't already override it.
> >
> > And as x86_32 doesn't set a value this means that slab objects get
> > aligned to 8 bytes, potentially wasting 4 bytes per object. Slub forces
> > objects to be aligned to sizeof(void *) anyway, but I don't see that
> > there is any need for it to be 8 on 32bits.
>
> Note that 64 bit entities may exist even under 32 bit (llong) that need
> to be aligned properly.
>
> struct buffer_head contains a sector_t which is 64 bit so you should align
> to an 8 byte boundary.
>
> > I'm working on a patch to pack more buffer_heads into each kmem_cache
> > slab page.
> > On 32 bits the structure size is 52 bytes and with the alignment applied
> > I end up with a slab of 73 x 56 byte objects. However, if the minimum
> > alignment was sizeof(void *) then I'd get 78 x 52 byte objects. So there
> > is quite a memory saving to be had in changing this.
>
> SLUB is not restricted to order 0 pages and can use order 1 or 2 pages as
> long as this reduces the memory footprint (byte wastage in a slab page is
> reduced) and as long as the kernel has contiguous memory available. It
> will use order 0 when memory is fragmented.
>
> > Can I define a ARCH_SLAB_MINALIGN in x86_64 to sizeof(void *) ?
> > or would it be ok to change the default in slub.c to sizeof(void *) ?
> >
> > Or am I missing something ?
>
> I'd say leave it alone.
I definitely don't want to break the alignment ;) but gcc aligns
unsigned long long on 4 byte boundaries on 32 bit.
Running this test code :-
#ifdef __compiler_offsetof
#define offsetof(TYPE,MEMBER) __compiler_offsetof(TYPE,MEMBER)
#else
#define offsetof(TYPE, MEMBER) ((size_t) &((TYPE *)0)->MEMBER)
#endif
struct test_align {
char c;
unsigned long long l;
};
void main() {
printf( "size = %d , offset of l = $d\n",
sizeof(struct test_align),
offsetof(struct test_align,l) );
}
gives me this output :-
32 bit : size = 12 , offset of l = 4
64 bit : size = 16 , offset of l = 8
Doesn't that suggest that it would be safe to use sizeof(void *) ?
(at least on x86 anyway).
We end up with a large number of buffer_heads and as they are pretty
small an extra 4 bytes does make a significant difference.
On my 64 bit machine I often see thousands of pages of buffer_heads, so
squeezing a few more per page could be a considerable saving.
regards
Richard
On Wed, Feb 03, 2010 at 02:22:26PM +0000, Richard Kennedy wrote:
> Can I define a ARCH_SLAB_MINALIGN in x86_64 to sizeof(void *) ?
> or would it be ok to change the default in slub.c to sizeof(void *) ?
>
Note that this is precisely what ARCH_SLAB_MINALIGN was introduced to
avoid (BYTES_PER_WORD alignment used to be the default for slab, before
ARCH_SLAB_MINALIGN was introduced). Consider the case of 64-bit platforms
using a 32-bit ABI, the native alignment remains 64-bit while sizeof(void
*) == 4. There are a number of (mainly embedded) architectures that
support these sorts of configurations in-tree.
On Wed, 3 Feb 2010, Richard Kennedy wrote:
> gives me this output :-
> 32 bit : size = 12 , offset of l = 4
> 64 bit : size = 16 , offset of l = 8
>
> Doesn't that suggest that it would be safe to use sizeof(void *) ?
> (at least on x86 anyway).
Maybe. But the rule of thumb is to align objects by their size which we
would be violating.
A 64 bit object may span multiple cachelines if aligned to a 32 bit
boundary. Which may result in nasty surprise because the object can no
longer be read and written from memory in an atomic way. If there is
a guarantee that no 64 bit operation ever occurs then it may be
fine.
Fetching a 64 bit object that straddles a cacheline boundary also requires
2 fetches instead of one to read the object which can increase the
cache footprint of functions accessing the structure.
Slab allocators (aside from SLOB which is rarely used) assume the minimal
alignment to be sizeof(unsigned long long).
> We end up with a large number of buffer_heads and as they are pretty
> small an extra 4 bytes does make a significant difference.
> On my 64 bit machine I often see thousands of pages of buffer_heads, so
> squeezing a few more per page could be a considerable saving.
On your 64 bit machine you wont be able to do the optimization that you
are talking about.
The buffer head structure is already fairly big so this wont make too much
of a difference.