Heelo,
Here is another topic of the interface I'd like to see discussed on this list
because it was raised in the past and I am not necessarily satisfied with
the current solution.
Tools can program the PMU by passing structures to read/write the PMC/PMD
registers. Some of those structures contain bitmasks. For instance,
When sampling, one can load a PMD (counter) value and indicate which
other PMD registers must be included in the samples using the bitmask.
The interface supports the same fixed number of PMD registers on
all platforms. As such the number of bits in the bitmask is fixed
and we have arrange for it to be multiple of 4 and 8. The structure
looks as follows:
typedef struct {
u16 reg_num; /* which register */
u16 reg_set; /* event set for this register */
pfm_flags_t reg_flags; /* input: flags, return: reg error */
u64 reg_value; /* initial pmc/pmd value */
u64 reg_long_reset; /* value to reload after notification */
u64 reg_short_reset; /* reset after counter overflow */
u64 reg_last_reset_val; /* return: PMD last reset value */
u64 reg_ovfl_switch_cnt;/* #overflows before switch */
unsigned long reg_reset_pmds[PFM_PMD_BV]; /* reset on overflow */
unsigned long reg_smpl_pmds[PFM_PMD_BV]; /* record in sample */
u64 reg_smpl_eventid; /* opaque event identifier */
u64 reg_random_mask; /* bitmask used to limit random value */
u32 reg_random_seed; /* seed for randomization */
u32 reg_reserved2[7]; /* for future use */
} pfarg_pmd_t;
We use unsigned long for bitmask because we can leverage the
kernel bitmap.h interface which is nice. All the others fields
in the struct have fixed size. If it was not for the bitmask
the structure would be identical in 32-bit or 64-bit mode.
Why does it matter?
Many 64-bit Linux kernel do support running 32-bit native applications.
That is the case on PPC64, MIPS64K, X86-64, for instance. One could well
write a 32-bit monitoring tool on top of a 64-bit OS. If we want to avoid
the emulation trampoline in the kernel, we need to ensure that the 32-bit
applications use the same structure layout as the 64-bit OS.
In our particular case we rely on the fact that the number of bits is fixed.
We use 320 bits, which is either 10x32 bits or 5x64 bits. Either way,
the sizeof() is the same. As such the struct is identical. Internally,
the kernel will take the bitmask as u64. The only issue would be the
alignment of the bitmasks. The 32-bit version must be aligned on 64-bit
boundary. Structures are aligned on the size of their largest member,
here 64-bit. The layout is such that the bitmask are always aligned.
We have verified that this does indeed work on MIPS with Phil Mucci.
The alternative approach would be to hardcode bitmask to be u64.
But that would require extra casting in the kernel to get to the
bitmap interface and may cause extra overhead in the 32-bit user programs.
Yet I don't anticipate that programming those bitmask be in the critical
path of monitoring tools.
The second alternative looks cleaner and safer in a sense and I am certainly
willing to make the change but I would like to get everybody else's opinion
as well.
Note that there are similar issues with the remapped sampling buffer.
There, you need to explicitly compile your tool with a special option
to force certain types to be 64-bit (size_t, void *). Unless someone
tell me how to tell the compiler we're compiling 32-bit to execute
on an 64-bit OS ABI.
Thanks.
On Fri, 2006-02-10 at 07:36 -0800, Stephane Eranian wrote:
> Many 64-bit Linux kernel do support running 32-bit native applications.
> That is the case on PPC64, MIPS64K, X86-64, for instance.
And sparc64 and s390.
> One could well
> write a 32-bit monitoring tool on top of a 64-bit OS.
On some 64-bit arches (e.g. x86_64), most userspace code is 64-bit,
while on others (e.g. powerpc), most is 32-bit. Reducing the number of
things that a userspace tool or library writer can trip over seems like
a good thing here, even if it slightly complicates perfmon's internals.
> Note that there are similar issues with the remapped sampling buffer.
> There, you need to explicitly compile your tool with a special option
> to force certain types to be 64-bit (size_t, void *).
It's pretty normal to just use 64-bit quantities in these cases, and
cast appropriately.
<b
Bryan,
On Fri, Feb 10, 2006 at 10:27:02AM -0800, Bryan O'Sullivan wrote:
>
> On some 64-bit arches (e.g. x86_64), most userspace code is 64-bit,
> while on others (e.g. powerpc), most is 32-bit. Reducing the number of
> things that a userspace tool or library writer can trip over seems like
> a good thing here, even if it slightly complicates perfmon's internals.
>
> > Note that there are similar issues with the remapped sampling buffer.
> > There, you need to explicitly compile your tool with a special option
> > to force certain types to be 64-bit (size_t, void *).
>
> It's pretty normal to just use 64-bit quantities in these cases, and
> cast appropriately.
>
So if I understand you correctly, you are saying it is best to have bitmasks
hardcoded to u64 and have the kernel cast to match the bitmap_*() interface.
This would not cause any alignment problems on neither 32-bit nor 64-bit system.
--
-Stephane
Hello,
So I worked some more on these 32bit vs 64bit ABI issues. I came
to the conclusion that the only way to make this work cleanly
without any special compilation flags or runtime support is
to simply ensure that all data structures exchanged/shared between
the monitoring tool and the kernel use only fixed size types.
This way, the sizeof struct and fields offset remains constant
in ILP32 or LP64 mode. As a consequence I now have:
- all bitmasks use u64
- all size_t are replaced with u64
- the default format IP (instruction pointer) is u64
For the IP, the most significant bits contain all zeroes in
ILP32 mode. With the default sampling buffer format, you add
4 bytes to the sample header but then you have nothing special
to do with user level code. I think this is also inline with
the fact that 64-bit computing is becoming widely available
for desktop and laptop computers.
As an example, I can now compile a tool for P4 in 32-bit and
run it on my 32-bit Xeon. But I can also run the same *binary*
on my EM64T and it works as expected including for the sampling
buffer. I believe the same would be possible with Opteron, although
I have not tried.
I hope this closes another sticking point about the interface.
On Tue, Feb 14, 2006 at 12:57:55AM +0600, Philip Mucci wrote:
>
> I know this is ugly, but what about in the user code checking the arch?
> Is it not true that if a kernel is running 64 bit, it has the 64 tagged
> on the the end of the arch? i.e. mips64, ppc64, x86_64?
>
> The other solution would be to add am information call to the API, like
> perfctr has. This would export the processor type along with other
> feature bits, including the number of bits of the IP.
>
> Either of these combined with compile time ABI/bitness constants should
> cover the cases no?
>
> Phil
>
>
> > The problem is in the user level header file for the sampling buffer.
> > We would need a data type that is 64-bit for IP if the host OS is 64-bit
> > (regardless of the ABI used by the tool, i.e., the compiler). And a data
> > type that is 32-bit on 32-bit OS. The problem is that there is no compiler
> > flag or header flag somewhere that could guide the compiler. In the case
> > of MIPS, we have defined a libpfm compile flags that indicates we want
> > the 64-bit OS definition when compiling for a 32-bit application.
> >
>
>
>
> -------------------------------------------------------
> This SF.net email is sponsored by: Splunk Inc. Do you grep through log files
> for problems? Stop! Download the new AJAX search engine that makes
> searching your log files as easy as surfing the web. DOWNLOAD SPLUNK!
> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=103432&bid=230486&dat=121642
> _______________________________________________
> Perfctr-devel mailing list
> [email protected]
> https://lists.sourceforge.net/lists/listinfo/perfctr-devel
--
-Stephane