Hi,
A question/request came up during our cache to cache analysis. We were
wondering if give an unique mmap2 id (major, minor, inode, inode
generation), if it was possible to determine a cpu socket id that memory
was attached to at the time of the captured perf event?
We ran into a scenario awhile back where a dcache struct was allocated on
say node2 and then subsequently free'd. The memory was thought to be
re-allocated on node0 for another dcache entry. It turned out the memory
was still attached to node2 and was causing slowdowns.
Our cache-to-cache tool noticed the slowdown but we couldn't understand
why because we had falsely assumed the memory was allocated on the local
node but instead it was on the remote node.
I wasn't involved too much in this problem so the details provided above
are a little hazy.
Anyway, an idea popped up that having socket information might have helped
us solve/notice this problem sooner.
Thoughts? Not sure if that data is possible.
Cheers,
Don
On Wed, Oct 22, 2014 at 12:20:26PM -0400, Don Zickus wrote:
> Hi,
>
> A question/request came up during our cache to cache analysis. We were
> wondering if give an unique mmap2 id (major, minor, inode, inode
> generation), if it was possible to determine a cpu socket id that memory
> was attached to at the time of the captured perf event?
No, see below. Also socket is the wrong information, both AMD and Intel
have chips with two nodes in one socket :-)
> We ran into a scenario awhile back where a dcache struct was allocated on
> say node2 and then subsequently free'd. The memory was thought to be
> re-allocated on node0 for another dcache entry. It turned out the memory
> was still attached to node2 and was causing slowdowns.
Yes, kernel memory is directly addresses, you basically have a static
address->node mapping, it never changes.
For instance, on my ivb-ep I can find the following in my dmesg:
[ 0.000000] NUMA: Initialized distance table, cnt=2
[ 0.000000] NUMA: Node 0 [mem 0x00000000-0xbfffffff] + [mem 0x100000000-0x43fffffff] -> [mem 0x00000000-0x43fffffff]
[ 0.000000] NODE_DATA(0) allocated [mem 0x43fffc000-0x43fffffff]
[ 0.000000] NODE_DATA(1) allocated [mem 0x83fff9000-0x83fffcfff]
[ 0.000000] [ffffea0000000000-ffffea000edfffff] PMD -> [ffff88042fe00000-ffff88043ddfffff] on node 0
[ 0.000000] [ffffea000ee00000-ffffea001cdfffff] PMD -> [ffff88082f600000-ffff88083d5fffff] on node 1
[ 0.000000] Zone ranges:
[ 0.000000] DMA [mem 0x00001000-0x00ffffff]
[ 0.000000] DMA32 [mem 0x01000000-0xffffffff]
[ 0.000000] Normal [mem 0x100000000-0x83fffffff]
[ 0.000000] Movable zone start for each node
[ 0.000000] Early memory node ranges
[ 0.000000] node 0: [mem 0x00001000-0x0008dfff]
[ 0.000000] node 0: [mem 0x00100000-0xbad28fff]
[ 0.000000] node 0: [mem 0xbaf90000-0xbafc4fff]
[ 0.000000] node 0: [mem 0xbafda000-0xbb3d3fff]
[ 0.000000] node 0: [mem 0xbdfac000-0xbdffffff]
[ 0.000000] node 0: [mem 0x100000000-0x43fffffff]
[ 0.000000] node 1: [mem 0x440000000-0x83fffffff]
[ 0.000000] Initmem setup node 0 [mem 0x00001000-0x43fffffff]
[ 0.000000] On node 0 totalpages: 4174137
[ 0.000000] DMA zone: 56 pages used for memmap
[ 0.000000] DMA zone: 21 pages reserved
[ 0.000000] DMA zone: 3981 pages, LIFO batch:0
[ 0.000000] DMA32 zone: 10422 pages used for memmap
[ 0.000000] DMA32 zone: 762284 pages, LIFO batch:31
[ 0.000000] Normal zone: 46592 pages used for memmap
[ 0.000000] Normal zone: 3407872 pages, LIFO batch:31
[ 0.000000] Initmem setup node 1 [mem 0x440000000-0x83fffffff]
[ 0.000000] On node 1 totalpages: 4194304
[ 0.000000] Normal zone: 57344 pages used for memmap
[ 0.000000] Normal zone: 4194304 pages, LIFO batch:31
> Our cache-to-cache tool noticed the slowdown but we couldn't understand
> why because we had falsely assumed the memory was allocated on the local
> node but instead it was on the remote node.
But in general, you can never say for user memory, since that has the
process page table mapping in between, the user virtual address is
unrelated to backing (and can change frequently and without
notification).
Therefore the mmap(2) information is useless for this, it only concerns
user memory.
Em Wed, Oct 22, 2014 at 06:45:10PM +0200, Peter Zijlstra escreveu:
> On Wed, Oct 22, 2014 at 12:20:26PM -0400, Don Zickus wrote:
> > Our cache-to-cache tool noticed the slowdown but we couldn't understand
> > why because we had falsely assumed the memory was allocated on the local
> > node but instead it was on the remote node.
> But in general, you can never say for user memory, since that has the
> process page table mapping in between, the user virtual address is
> unrelated to backing (and can change frequently and without
> notification).
> Therefore the mmap(2) information is useless for this, it only concerns
> user memory.
So what you are saying is that it is difficult to have some sort of
mechanism that an mmap moved from one node to another, when that
happens, i.e. a new tracepoint for that?
- Arnaldo
On 10/22/2014 12:45 PM, Peter Zijlstra wrote:
> On Wed, Oct 22, 2014 at 12:20:26PM -0400, Don Zickus wrote:
>> Hi,
>>
>> A question/request came up during our cache to cache analysis. We were
>> wondering if give an unique mmap2 id (major, minor, inode, inode
>> generation), if it was possible to determine a cpu socket id that memory
>> was attached to at the time of the captured perf event?
>
> No, see below. Also socket is the wrong information, both AMD and Intel
> have chips with two nodes in one socket :-)
>
>> We ran into a scenario awhile back where a dcache struct was allocated on
>> say node2 and then subsequently free'd. The memory was thought to be
>> re-allocated on node0 for another dcache entry. It turned out the memory
>> was still attached to node2 and was causing slowdowns.
>
> Yes, kernel memory is directly addresses, you basically have a static
> address->node mapping, it never changes.
For kernel addresses, is there a reason not to have it available in perf,
especially when that knowledge is important to understanding a numa-related slowdown?
In our case, when we booted with one configuration, AIM ran fine. When we
booted another way, AIM's performance dropped 50%. It was all due to the dentry
lock being located on a different (now remote) numa node.
We used your dmesg approach to track down the home node in an attempt to understand
what was different between the two boots. But the problem would have been obvious
if perf simply listed the home node info.
>
> For instance, on my ivb-ep I can find the following in my dmesg:
>
> [ 0.000000] NUMA: Initialized distance table, cnt=2
> [ 0.000000] NUMA: Node 0 [mem 0x00000000-0xbfffffff] + [mem 0x100000000-0x43fffffff] -> [mem 0x00000000-0x43fffffff]
> [ 0.000000] NODE_DATA(0) allocated [mem 0x43fffc000-0x43fffffff]
> [ 0.000000] NODE_DATA(1) allocated [mem 0x83fff9000-0x83fffcfff]
> [ 0.000000] [ffffea0000000000-ffffea000edfffff] PMD -> [ffff88042fe00000-ffff88043ddfffff] on node 0
> [ 0.000000] [ffffea000ee00000-ffffea001cdfffff] PMD -> [ffff88082f600000-ffff88083d5fffff] on node 1
> [ 0.000000] Zone ranges:
> [ 0.000000] DMA [mem 0x00001000-0x00ffffff]
> [ 0.000000] DMA32 [mem 0x01000000-0xffffffff]
> [ 0.000000] Normal [mem 0x100000000-0x83fffffff]
> [ 0.000000] Movable zone start for each node
> [ 0.000000] Early memory node ranges
> [ 0.000000] node 0: [mem 0x00001000-0x0008dfff]
> [ 0.000000] node 0: [mem 0x00100000-0xbad28fff]
> [ 0.000000] node 0: [mem 0xbaf90000-0xbafc4fff]
> [ 0.000000] node 0: [mem 0xbafda000-0xbb3d3fff]
> [ 0.000000] node 0: [mem 0xbdfac000-0xbdffffff]
> [ 0.000000] node 0: [mem 0x100000000-0x43fffffff]
> [ 0.000000] node 1: [mem 0x440000000-0x83fffffff]
> [ 0.000000] Initmem setup node 0 [mem 0x00001000-0x43fffffff]
> [ 0.000000] On node 0 totalpages: 4174137
> [ 0.000000] DMA zone: 56 pages used for memmap
> [ 0.000000] DMA zone: 21 pages reserved
> [ 0.000000] DMA zone: 3981 pages, LIFO batch:0
> [ 0.000000] DMA32 zone: 10422 pages used for memmap
> [ 0.000000] DMA32 zone: 762284 pages, LIFO batch:31
> [ 0.000000] Normal zone: 46592 pages used for memmap
> [ 0.000000] Normal zone: 3407872 pages, LIFO batch:31
> [ 0.000000] Initmem setup node 1 [mem 0x440000000-0x83fffffff]
> [ 0.000000] On node 1 totalpages: 4194304
> [ 0.000000] Normal zone: 57344 pages used for memmap
> [ 0.000000] Normal zone: 4194304 pages, LIFO batch:31
>
>> Our cache-to-cache tool noticed the slowdown but we couldn't understand
>> why because we had falsely assumed the memory was allocated on the local
>> node but instead it was on the remote node.
>
> But in general, you can never say for user memory, since that has the
> process page table mapping in between, the user virtual address is
> unrelated to backing (and can change frequently and without
> notification).
>
> Therefore the mmap(2) information is useless for this, it only concerns
> user memory.
>
Em Wed, Oct 22, 2014 at 02:42:26PM -0300, Arnaldo Carvalho de Melo escreveu:
> Em Wed, Oct 22, 2014 at 06:45:10PM +0200, Peter Zijlstra escreveu:
> > On Wed, Oct 22, 2014 at 12:20:26PM -0400, Don Zickus wrote:
> > > Our cache-to-cache tool noticed the slowdown but we couldn't understand
> > > why because we had falsely assumed the memory was allocated on the local
> > > node but instead it was on the remote node.
> > But in general, you can never say for user memory, since that has the
> > process page table mapping in between, the user virtual address is
> > unrelated to backing (and can change frequently and without
> > notification).
> > Therefore the mmap(2) information is useless for this, it only concerns
> > user memory.
> So what you are saying is that it is difficult to have some sort of
> mechanism that an mmap moved from one node to another, when that
> happens, i.e. a new tracepoint for that?
Humm, so, after reading a bit more, I see that we can figure out which
ranges are in which NUMA nodes (like shown in dmesg), and we have the
phys addr, so is this just a matter of perf tooling complement the
information the kernel provides on mmap2 (and the hardware provides on
those events)?
- Arnaldo
On Wed, Oct 22, 2014 at 02:42:26PM -0300, Arnaldo Carvalho de Melo wrote:
> Em Wed, Oct 22, 2014 at 06:45:10PM +0200, Peter Zijlstra escreveu:
> > On Wed, Oct 22, 2014 at 12:20:26PM -0400, Don Zickus wrote:
> > > Our cache-to-cache tool noticed the slowdown but we couldn't understand
> > > why because we had falsely assumed the memory was allocated on the local
> > > node but instead it was on the remote node.
>
> > But in general, you can never say for user memory, since that has the
> > process page table mapping in between, the user virtual address is
> > unrelated to backing (and can change frequently and without
> > notification).
>
> > Therefore the mmap(2) information is useless for this, it only concerns
> > user memory.
>
> So what you are saying is that it is difficult to have some sort of
> mechanism that an mmap moved from one node to another, when that
> happens, i.e. a new tracepoint for that?
Node is per page, not per mapping. Every single page in a mmap can have
a different node (assuming enough nodes etc..), and it can change at
relatively high frequency.
On Wed, Oct 22, 2014 at 03:15:53PM -0300, Arnaldo Carvalho de Melo wrote:
> Em Wed, Oct 22, 2014 at 02:42:26PM -0300, Arnaldo Carvalho de Melo escreveu:
> > Em Wed, Oct 22, 2014 at 06:45:10PM +0200, Peter Zijlstra escreveu:
> > > On Wed, Oct 22, 2014 at 12:20:26PM -0400, Don Zickus wrote:
> > > > Our cache-to-cache tool noticed the slowdown but we couldn't understand
> > > > why because we had falsely assumed the memory was allocated on the local
> > > > node but instead it was on the remote node.
>
> > > But in general, you can never say for user memory, since that has the
> > > process page table mapping in between, the user virtual address is
> > > unrelated to backing (and can change frequently and without
> > > notification).
>
> > > Therefore the mmap(2) information is useless for this, it only concerns
> > > user memory.
>
> > So what you are saying is that it is difficult to have some sort of
> > mechanism that an mmap moved from one node to another, when that
> > happens, i.e. a new tracepoint for that?
>
> Humm, so, after reading a bit more, I see that we can figure out which
> ranges are in which NUMA nodes (like shown in dmesg), and we have the
> phys addr, so is this just a matter of perf tooling complement the
> information the kernel provides on mmap2 (and the hardware provides on
> those events)?
We explicitly do not expose the phys address of user space pages. So
nope.
On Wed, Oct 22, 2014 at 02:09:35PM -0400, Joe Mario wrote:
> >Yes, kernel memory is directly addresses, you basically have a static
> >address->node mapping, it never changes.
>
> For kernel addresses, is there a reason not to have it available in perf,
> especially when that knowledge is important to understanding a numa-related slowdown?
Dunno why that isn't exposed in sysfs.
> In our case, when we booted with one configuration, AIM ran fine. When we
> booted another way, AIM's performance dropped 50%. It was all due to the dentry
> lock being located on a different (now remote) numa node.
>
> We used your dmesg approach to track down the home node in an attempt to understand
> what was different between the two boots. But the problem would have been obvious
> if perf simply listed the home node info.
Or if you'd used more counters that track the node interconnect traffic
;-) There are a few simple ones that count local/remote type things
(offcore), but using the uncore counters you can track way more.
On Wed, Oct 22, 2014 at 10:02:19PM +0200, Peter Zijlstra wrote:
> On Wed, Oct 22, 2014 at 02:09:35PM -0400, Joe Mario wrote:
> > >Yes, kernel memory is directly addresses, you basically have a static
> > >address->node mapping, it never changes.
> >
> > For kernel addresses, is there a reason not to have it available in perf,
> > especially when that knowledge is important to understanding a numa-related slowdown?
>
> Dunno why that isn't exposed in sysfs.
>
> > In our case, when we booted with one configuration, AIM ran fine. When we
> > booted another way, AIM's performance dropped 50%. It was all due to the dentry
> > lock being located on a different (now remote) numa node.
> >
> > We used your dmesg approach to track down the home node in an attempt to understand
> > what was different between the two boots. But the problem would have been obvious
> > if perf simply listed the home node info.
>
> Or if you'd used more counters that track the node interconnect traffic
> ;-) There are a few simple ones that count local/remote type things
> (offcore), but using the uncore counters you can track way more.
Ha! I have been telling myself for a year I would try to learn more about
those offcore/uncore counters. Is there documentation for how to access
the uncore stuff? Do I have to long hand it with 'perf record -e
uncore_qpi_1/<stuff>/ foo'?
Cheers,
Don
On Wed, Oct 22, 2014 at 04:38:34PM -0400, Don Zickus wrote:
> On Wed, Oct 22, 2014 at 10:02:19PM +0200, Peter Zijlstra wrote:
> > Or if you'd used more counters that track the node interconnect traffic
> > ;-) There are a few simple ones that count local/remote type things
> > (offcore), but using the uncore counters you can track way more.
>
> Ha! I have been telling myself for a year I would try to learn more about
> those offcore/uncore counters. Is there documentation for how to access
> the uncore stuff? Do I have to long hand it with 'perf record -e
> uncore_qpi_1/<stuff>/ foo'?
So uncore is something unholy complicated and different for each cpu,
although the newer chips seems to have limited the variation somewhat.
I would suggest you start with the offcore bits, those are in the SDM.
We have some numa cache events, but if those don't work for you you
might want to look at the offcore code we have in the kernel to help
construct your own events.
I Cc'ed you on a patch series cleaning up some of the recent HSW code.
On Wed, Oct 22, 2014 at 04:38:34PM -0400, Don Zickus wrote:
> Ha! I have been telling myself for a year I would try to learn more about
> those offcore/uncore counters. Is there documentation for how to access
> the uncore stuff? Do I have to long hand it with 'perf record -e
> uncore_qpi_1/<stuff>/ foo'?
For uncore there is some in the SDM and there are these documents; there
might be more but who knows.
http://www.intel.com/Assets/PDF/designguide/323535.pdf
http://www.intel.com/content/dam/www/public/us/en/documents/design-guides/xeon-e5-2600-uncore-guide.pdf
On Thu, Oct 23, 2014 at 01:30:01PM +0200, Peter Zijlstra wrote:
> On Wed, Oct 22, 2014 at 04:38:34PM -0400, Don Zickus wrote:
> > Ha! I have been telling myself for a year I would try to learn more about
> > those offcore/uncore counters. Is there documentation for how to access
> > the uncore stuff? Do I have to long hand it with 'perf record -e
> > uncore_qpi_1/<stuff>/ foo'?
>
> For uncore there is some in the SDM and there are these documents; there
> might be more but who knows.
>
> http://www.intel.com/Assets/PDF/designguide/323535.pdf
> http://www.intel.com/content/dam/www/public/us/en/documents/design-guides/xeon-e5-2600-uncore-guide.pdf
Thanks!
Cheers,
Don