Hi Maciej,
On Fri, Sep 22, 2023 at 10:48:23AM +0200, Maciej Wieczor-Retman wrote:
> The setting for non-contiguous 1s support in Intel CAT is
> hardcoded to false. On these systems, writing non-contiguous
> 1s into the schemata file will fail before resctrl passes
> the value to the hardware.
>
> In Intel CAT CPUID.0x10.1:ECX[3] and CPUID.0x10.2:ECX[3] stopped
> being reserved and now carry information about non-contiguous 1s
> value support for L3 and L2 cache respectively. The CAT
> capacity bitmask (CBM) supports a non-contiguous 1s value if
> the bit is set.
How new of an SDM do I need? The June 2023 revision I downloaded today didn't
list it.
> diff --git a/arch/x86/kernel/cpu/resctrl/core.c b/arch/x86/kernel/cpu/resctrl/core.c
> index 030d3b409768..c783a873147c 100644
> --- a/arch/x86/kernel/cpu/resctrl/core.c
> +++ b/arch/x86/kernel/cpu/resctrl/core.c
> @@ -152,6 +152,7 @@ static inline void cache_alloc_hsw_probe(void)
> r->cache.cbm_len = 20;
> r->cache.shareable_bits = 0xc0000;
> r->cache.min_cbm_bits = 2;
> + r->cache.arch_has_sparse_bitmaps = false;
> r->alloc_capable = true;
>
> rdt_alloc_capable = true;
> @@ -267,15 +268,18 @@ static void rdt_get_cache_alloc_cfg(int idx, struct rdt_resource *r)
> {
> struct rdt_hw_resource *hw_res = resctrl_to_arch_res(r);
> union cpuid_0x10_1_eax eax;
> + union cpuid_0x10_x_ecx ecx;
> union cpuid_0x10_x_edx edx;
> - u32 ebx, ecx;
> + u32 ebx;
>
> - cpuid_count(0x00000010, idx, &eax.full, &ebx, &ecx, &edx.full);
> + cpuid_count(0x00000010, idx, &eax.full, &ebx, &ecx.full, &edx.full);
> hw_res->num_closid = edx.split.cos_max + 1;
> r->cache.cbm_len = eax.split.cbm_len + 1;
> r->default_ctrl = BIT_MASK(eax.split.cbm_len + 1) - 1;
> r->cache.shareable_bits = ebx & r->default_ctrl;
> r->data_width = (r->cache.cbm_len + 3) / 4;
> + if (boot_cpu_data.x86_vendor == X86_VENDOR_INTEL)
> + r->cache.arch_has_sparse_bitmaps = ecx.split.noncont;
This seems to be called after the clearing of arch_has_sparse_bitmaps in
cache_alloc_hsw_probe(). If we can't make use of the CPUID bit on Haswell,
is it safe to use its value here?
Thanks!
-Peter
Hi, and thanks for the review!
On 2023-09-22 at 16:14:41 +0200, Peter Newman wrote:
>Hi Maciej,
>
>On Fri, Sep 22, 2023 at 10:48:23AM +0200, Maciej Wieczor-Retman wrote:
>> The setting for non-contiguous 1s support in Intel CAT is
>> hardcoded to false. On these systems, writing non-contiguous
>> 1s into the schemata file will fail before resctrl passes
>> the value to the hardware.
>>
>> In Intel CAT CPUID.0x10.1:ECX[3] and CPUID.0x10.2:ECX[3] stopped
>> being reserved and now carry information about non-contiguous 1s
>> value support for L3 and L2 cache respectively. The CAT
>> capacity bitmask (CBM) supports a non-contiguous 1s value if
>> the bit is set.
>
>How new of an SDM do I need? The June 2023 revision I downloaded today didn't
>list it.
It's not currently in the SDM but in the Intel? Architecture
Instruction Set Extensions and Future Features (which I mentioned in the
second paragraph of the cover letter). My version of the ISA pdf was
from June 2023.
>> diff --git a/arch/x86/kernel/cpu/resctrl/core.c b/arch/x86/kernel/cpu/resctrl/core.c
>> index 030d3b409768..c783a873147c 100644
>> --- a/arch/x86/kernel/cpu/resctrl/core.c
>> +++ b/arch/x86/kernel/cpu/resctrl/core.c
>> @@ -152,6 +152,7 @@ static inline void cache_alloc_hsw_probe(void)
>> r->cache.cbm_len = 20;
>> r->cache.shareable_bits = 0xc0000;
>> r->cache.min_cbm_bits = 2;
>> + r->cache.arch_has_sparse_bitmaps = false;
>> r->alloc_capable = true;
>>
>> rdt_alloc_capable = true;
>> @@ -267,15 +268,18 @@ static void rdt_get_cache_alloc_cfg(int idx, struct rdt_resource *r)
>> {
>> struct rdt_hw_resource *hw_res = resctrl_to_arch_res(r);
>> union cpuid_0x10_1_eax eax;
>> + union cpuid_0x10_x_ecx ecx;
>> union cpuid_0x10_x_edx edx;
>> - u32 ebx, ecx;
>> + u32 ebx;
>>
>> - cpuid_count(0x00000010, idx, &eax.full, &ebx, &ecx, &edx.full);
>> + cpuid_count(0x00000010, idx, &eax.full, &ebx, &ecx.full, &edx.full);
>> hw_res->num_closid = edx.split.cos_max + 1;
>> r->cache.cbm_len = eax.split.cbm_len + 1;
>> r->default_ctrl = BIT_MASK(eax.split.cbm_len + 1) - 1;
>> r->cache.shareable_bits = ebx & r->default_ctrl;
>> r->data_width = (r->cache.cbm_len + 3) / 4;
>> + if (boot_cpu_data.x86_vendor == X86_VENDOR_INTEL)
>> + r->cache.arch_has_sparse_bitmaps = ecx.split.noncont;
>
>This seems to be called after the clearing of arch_has_sparse_bitmaps in
>cache_alloc_hsw_probe(). If we can't make use of the CPUID bit on Haswell,
>is it safe to use its value here?
I believe the calls go like this for a haswell system:
resctrl_late_init() -> check_quirks() -> __check_quirks_intel() ->
-> cache_alloc_hsw_probe()
There this line is executed:
rdt_alloc_capable = true;
where rdt_alloc_capable is global in the file scope.
Then later in:
resctrl_late_init() -> get_rdt_resources() -> get_rdt_alloc_resources()
this is executed at the function beginning:
if (rdt_alloc_capable)
return true;
So the rest of the get_rdt_alloc_resources() is skipped and calls to
rdt_get_cache_alloc_cfg() never get executed.
--
Kind regards
Maciej Wiecz?r-Retman
Hi Maciej,
On Wed, Sep 27, 2023 at 11:20 AM Maciej Wieczór-Retman
<[email protected]> wrote:
> On 2023-09-22 at 16:14:41 +0200, Peter Newman wrote:
> >On Fri, Sep 22, 2023 at 10:48:23AM +0200, Maciej Wieczor-Retman wrote:
> >> In Intel CAT CPUID.0x10.1:ECX[3] and CPUID.0x10.2:ECX[3] stopped
> >> being reserved and now carry information about non-contiguous 1s
> >> value support for L3 and L2 cache respectively. The CAT
> >> capacity bitmask (CBM) supports a non-contiguous 1s value if
> >> the bit is set.
> >
> >How new of an SDM do I need? The June 2023 revision I downloaded today didn't
> >list it.
>
> It's not currently in the SDM but in the Intel® Architecture
> Instruction Set Extensions and Future Features (which I mentioned in the
> second paragraph of the cover letter). My version of the ISA pdf was
> from June 2023.
>
I see it now, thanks!
> >> - cpuid_count(0x00000010, idx, &eax.full, &ebx, &ecx, &edx.full);
> >> + cpuid_count(0x00000010, idx, &eax.full, &ebx, &ecx.full, &edx.full);
> >> hw_res->num_closid = edx.split.cos_max + 1;
> >> r->cache.cbm_len = eax.split.cbm_len + 1;
> >> r->default_ctrl = BIT_MASK(eax.split.cbm_len + 1) - 1;
> >> r->cache.shareable_bits = ebx & r->default_ctrl;
> >> r->data_width = (r->cache.cbm_len + 3) / 4;
> >> + if (boot_cpu_data.x86_vendor == X86_VENDOR_INTEL)
> >> + r->cache.arch_has_sparse_bitmaps = ecx.split.noncont;
> >
> >This seems to be called after the clearing of arch_has_sparse_bitmaps in
> >cache_alloc_hsw_probe(). If we can't make use of the CPUID bit on Haswell,
> >is it safe to use its value here?
>
> I believe the calls go like this for a haswell system:
> resctrl_late_init() -> check_quirks() -> __check_quirks_intel() ->
> -> cache_alloc_hsw_probe()
>
> There this line is executed:
> rdt_alloc_capable = true;
> where rdt_alloc_capable is global in the file scope.
>
> Then later in:
> resctrl_late_init() -> get_rdt_resources() -> get_rdt_alloc_resources()
>
> this is executed at the function beginning:
> if (rdt_alloc_capable)
> return true;
>
> So the rest of the get_rdt_alloc_resources() is skipped and calls to
> rdt_get_cache_alloc_cfg() never get executed.
Yuck. But it works I guess.
The series looks fine to me.
Reviewed-by: Peter Newman <[email protected]>
I applied the series and was able to confirm the behavior was still
correct for contiguous-bitmap Intel hardware and that sprase_bitmaps
is true on AMD and continues to work as expected.
Tested-by: Peter Newman <[email protected]>
I'm not sure if I have access to any Intel hardware with
non-contiguous bitmaps right now. Are you able to say where that would
be implemented?
Thanks!
-Peter