Dipankar Sarma wrote:
>
> On Wed, Dec 04, 2002 at 11:34:35AM -0800, Andrew Morton wrote:
> > > +/* Use these with kmalloc_percpu */
> > > +#define get_cpu_ptr(ptr) per_cpu_ptr(ptr, get_cpu())
> > > +#define put_cpu_ptr(ptr) put_cpu()
> >
> > These names sound very much like get_cpu_var() and put_cpu_var(),
> > yet they are using a quite different subsystem. It would be good
> > to choose something more distinct. Not that I can think of anything
> > at present ;)
>
> Well, they are similar, aren't they ? get_cpu_ptr() can just be thought
> of as the dynamic twin of get_cpu_var(). So, in that sense it seems ok
> to me.
hm. spose so.
> >
> > If we were to remove the percpu_interlaced_alloc() leg here, we'd
> > have a nice, simple per-cpu kmalloc implementation.
> >
> > Could you please explain what all the other code is there for?
>
> The interlaced allocator allows you to save space when kmalloc_percpu()
> is used to allocate small objects. That is done by interlacing each
> CPU's copy of the objects just like the static per-cpu data area.
>
Where in the kernel is such a large number of 4-, 8- or 16-byte
objects being used?
The slab allocator will support caches right down to 1024 x 4-byte
objects per page. Why is that not appropriate?
If it is for locality-of-reference between individual objects then
where in the kernel is this required, and are performance measurements
available? It is very unusual to have objects which are so small,
and a better design would be to obtain the locality of reference
by aggregating the data into an array or structure.
Sorry, but you have what is basically a brand new allocator in
there, and we need a very good reason for including it. I'd like
to know what that reason is, please.
On Wed, Dec 04, 2002 at 08:32:58PM -0800, Andrew Morton wrote:
> Where in the kernel is such a large number of 4-, 8- or 16-byte
> objects being used?
> The slab allocator will support caches right down to 1024 x 4-byte
> objects per page. Why is that not appropriate?
> If it is for locality-of-reference between individual objects then
> where in the kernel is this required, and are performance measurements
> available? It is very unusual to have objects which are so small,
> and a better design would be to obtain the locality of reference
> by aggregating the data into an array or structure.
I will argue not on the frequency of calls but on the preciousness of
space; highmem feels very serious pain when internal fragmentation
of pinned pages occurs (which this is designed to prevent). I don't
have direct experience with this patch/API, but I can say that
fragmentation in ZONE_NORMAL is deadly (witness pagetable occupancy
vs. ZONE_NORMAL consumption, which motivated highpte, despite my
pagetable reclamation smoke blowing).
Bill
Hi Andrew,
On Wed, Dec 04, 2002 at 08:32:58PM -0800, Andrew Morton wrote:
> Where in the kernel is such a large number of 4-, 8- or 16-byte
> objects being used?
Well, kernel objects may not be that small, but one would expect
the per-cpu parts of the kernel objects to be sometimes small, often down to
a couple of counters counting statistics.
>
> The slab allocator will support caches right down to 1024 x 4-byte
> objects per page. Why is that not appropriate?
Well, if you allocated 4-byte objects directly from the slab allocator,
you aren't guranteed to *not* share a cache line with another object
modified by a different cpu.
>
> Sorry, but you have what is basically a brand new allocator in
> there, and we need a very good reason for including it. I'd like
> to know what that reason is, please.
The reason is concern about per-cpu allocation for small per-CPU
parts (typically counters) of objects. If a driver has two counters
counting reads and writes, you don't want to eat up a whole cacheline
for them for each CPU per instance of the device.
Thanks
--
Dipankar Sarma <[email protected]> http://lse.sourceforge.net
Linux Technology Center, IBM Software Lab, Bangalore, India.
On Wed, Dec 04, 2002 at 08:32:58PM -0800, Andrew Morton wrote:
>>> Where in the kernel is such a large number of 4-, 8- or 16-byte
>>> objects being used?
On Thu, Dec 05, 2002 at 04:23:29PM +0530, Dipankar Sarma wrote:
> > Well, kernel objects may not be that small, but one would expect
> > the per-cpu parts of the kernel objects to be sometimes small, often down to
> > a couple of counters counting statistics.
On Thu, Dec 05, 2002 at 04:23:12AM -0700, [email protected] wrote:
> Doesn't your allocator increase chances of cache conflict on the same
> cpu ?
This is so; I'm personally far more concerned about ZONE_NORMAL space
consumption in the cacheline aligned case.
Bill
On Thu, Dec 05, 2002 at 04:23:29PM +0530, Dipankar Sarma wrote:
> Hi Andrew,
>
> On Wed, Dec 04, 2002 at 08:32:58PM -0800, Andrew Morton wrote:
> > Where in the kernel is such a large number of 4-, 8- or 16-byte
> > objects being used?
>
> Well, kernel objects may not be that small, but one would expect
> the per-cpu parts of the kernel objects to be sometimes small, often down to
> a couple of counters counting statistics.
Doesn't your allocator increase chances of cache conflict on the same
cpu ?
On Thu, Dec 05, 2002 at 11:33:15AM +0000, [email protected] wrote:
>
> >
> > Well, kernel objects may not be that small, but one would expect
> > the per-cpu parts of the kernel objects to be sometimes small, often down to
> > a couple of counters counting statistics.
>
>
> Doesn't your allocator increase chances of cache conflict on the same
> cpu ?
>
You mean by increasing the footprint and the chance of eviction ? It
is a compromise. Or you would face NR_CPUS bloat and non-NUMA-node-local
accesses for all CPUs outside the NUMA node where your NR_CPUS array
is located.
Thanks
--
Dipankar Sarma <[email protected]> http://lse.sourceforge.net
Linux Technology Center, IBM Software Lab, Bangalore, India.
On Thu, Dec 05, 2002 at 06:11:53PM +0530, Dipankar Sarma wrote:
> > Doesn't your allocator increase chances of cache conflict on the same
> > cpu ?
> >
>
> You mean by increasing the footprint and the chance of eviction ? It
> is a compromise. Or you would face NR_CPUS bloat and non-NUMA-node-local
> accesses for all CPUs outside the NUMA node where your NR_CPUS array
> is located.
What do you base the trade-off decision on?
>
> Thanks
> --
> Dipankar Sarma <[email protected]> http://lse.sourceforge.net
> Linux Technology Center, IBM Software Lab, Bangalore, India.
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at http://www.tux.org/lkml/
--
---------------------------------------------------------
Victor Yodaiken
Finite State Machine Labs: The RTLinux Company.
http://www.fsmlabs.com http://www.rtlinux.com
1+ 505 838 9109