Until recently Intel CPUs didn't support using non-contiguous 1s
in Cache Allocation Technology (CAT). Writing a bitmask with
non-contiguous 1s to the resctrl schemata file would fail.
Intel CPUs that support non-contiguous 1s can be identified through a
CPUID leaf mentioned in the "Intel® 64 and IA-32 Architectures
Software Developer’s Manual" document available at:
https://www.intel.com/content/www/us/en/developer/articles/technical/intel-sdm.html
Add kernel support for detecting if non-contiguous 1s in Cache
Allocation Technology (CAT) are supported by the hardware. Also add a
new resctrl FS file to output this information to the userspace.
Keep the hardcoded value for Haswell CPUs only since they do not have
CPUID enumeration support for Cache allocation.
Since the selftests/resctrl files are going through many rewrites and
cleanups the appropriate selftest is still a work in progress. For
basic selftesting capabilities use the bash script attached below this
paragraph. It checks whether various bitmasks written into resctrl FS
generate output consistent with reported feature support.
#!/bin/bash
# must be run as root, depends on a recent cpuid tool (20230406 or later)
# variables
RESCTRL_INFO="/sys/fs/resctrl/info"
L3_NON_CONT_VAL="${RESCTRL_INFO}/L3/sparse_masks"
L2_NON_CONT_VAL="${RESCTRL_INFO}/L2/sparse_masks"
L3_NON_CONT_CBM="${RESCTRL_INFO}/L3/cbm_mask"
L2_NON_CONT_CBM="${RESCTRL_INFO}/L2/cbm_mask"
L3_CPUID_CMD="cpuid -1 -l 0x10 -s 0x01"
L2_CPUID_CMD="cpuid -1 -l 0x10 -s 0x02"
PASSED_TESTS=0
L3_SUPPORT=0
L2_SUPPORT=0
TESTS=0
run_test() {
# L2 or L3
CACHE_LEVEL=$1
CACHE_LEVEL_SUPPORT="${CACHE_LEVEL}_SUPPORT"
echo "Checking ${RESCTRL_INFO}/${CACHE_LEVEL}..."
if [[ -d "${RESCTRL_INFO}/${CACHE_LEVEL}" ]]; then
eval "${CACHE_LEVEL_SUPPORT}=1"
echo "${CACHE_LEVEL} CAT Feature is supported"
else
echo "${CACHE_LEVEL} CAT Feature is not supported"
fi
if [[ ${!CACHE_LEVEL_SUPPORT} -eq 1 ]]; then
echo " --- Running tests for ${CACHE_LEVEL} CAT ---"
# read sysfs entries
# are non-contiguous cbm supported? (driver sysfs)
eval "NON_CONT_VAL=${CACHE_LEVEL}_NON_CONT_VAL"
eval "NON_CONT_FEAT=$( cat ${!NON_CONT_VAL} )"
# are non-contiguous cbm supported? (cpuid)
CACHE_CPUID_CMD="${CACHE_LEVEL}_CPUID_CMD"
NONCONT_CPUID=$(${!CACHE_CPUID_CMD} | grep non-contiguous | grep true)
NONCONT_CPUID_RET=$(( !$? ))
# what is the mask size?
eval "NON_CONT_CBM=${CACHE_LEVEL}_NON_CONT_CBM"
MAX_MASK=$(( 16#$( cat ${!NON_CONT_CBM} ) ))
# prepare contiguous and non-contiguous masks for tests
BC_STRING="l(${MAX_MASK})/l(2)"
MAX_MASK_BIT_COUNT=$(echo ${BC_STRING} | bc -l)
MAX_MASK_BIT_COUNT=$(printf "%.0f" "$MAX_MASK_BIT_COUNT")
BITSHIFT=$(( $MAX_MASK_BIT_COUNT/2 - ($MAX_MASK_BIT_COUNT/2 % 4) ))
CONT_MASK=$(( $MAX_MASK >> $BITSHIFT ))
NONCONT_MASK=$(( ~( $MAX_MASK & ( 15<<$BITSHIFT) ) ))
NONCONT_MASK=$(( $NONCONT_MASK & $MAX_MASK ))
# test if cpuid reported support matches the sysfs one
echo " * Testing if CPUID matches ${CACHE_LEVEL}/sparse_masks..."
TESTS=$((TESTS + 1))
if [[ $NONCONT_CPUID_RET -eq $NON_CONT_FEAT ]]; then
PASSED_TESTS=$((PASSED_TESTS + 1))
echo "There is a match!"
else
echo "Error - no match!"
fi
# test by writing CBMs to the schemata
printf " * Writing 0x%x mask to the schemata...\n" ${CONT_MASK}
TESTS=$((TESTS + 1))
SCHEMATA=$(printf "${CACHE_LEVEL}:0=%x" $CONT_MASK)
echo "$SCHEMATA" > /sys/fs/resctrl/schemata
if [[ $? -eq 0 ]]; then
PASSED_TESTS=$((PASSED_TESTS + 1))
echo "Contiguous ${CACHE_LEVEL} write correct!"
else
echo "Contiguous ${CACHE_LEVEL} write ERROR!"
fi
printf " * Writing 0x%x mask to the schemata...\n" ${NONCONT_MASK}
TESTS=$((TESTS + 1))
SCHEMATA=$(printf "${CACHE_LEVEL}:0=%x" $NONCONT_MASK)
echo "$SCHEMATA" > /sys/fs/resctrl/schemata
if [[ (($? -eq 0) && ($NON_CONT_FEAT -eq 1)) || \
(($? -ne 0) && ($NON_CONT_FEAT -eq 0)) ]]; then
PASSED_TESTS=$((PASSED_TESTS + 1))
echo "Non-contiguous ${CACHE_LEVEL} write correct!"
else
echo "Non-contiguous ${CACHE_LEVEL} write ERROR!"
fi
fi
}
# mount resctrl
mount -t resctrl resctrl /sys/fs/resctrl
run_test L3
run_test L2
echo "TESTS PASSED / ALL TESTS : ${PASSED_TESTS} / ${TESTS}"
# unmount resctrl
umount /sys/fs/resctrl
Changelog v4:
- Added Ilpo's reviewed-by tags.
- Added Reinette's reviewed-by tags.
- Reordered tags in alignment with maintainer-tip.rst.
Changelog v3:
- Add Peter's tested-by and reviewed-by tags.
- Change patch order to make 4th one the 1st.
- Add error checking to schema_len variable.
- Update cover letter since now the feature has moved from the SDM.
Changelog v2:
- Change git signature from Wieczor-Retman Maciej to Maciej
Wieczor-Retman.
- Change bitmap naming convention to bit mask.
- Add patch to change arch_has_sparse_bitmaps name to match bitmask
naming convention.
Fenghua Yu (2):
x86/resctrl: Add sparse_masks file in info
Documentation/x86: Document resctrl's new sparse_masks
Maciej Wieczor-Retman (2):
x86/resctrl: Rename arch_has_sparse_bitmaps
x86/resctrl: Enable non-contiguous CBMs in Intel CAT
Documentation/arch/x86/resctrl.rst | 16 ++++++++++++----
arch/x86/kernel/cpu/resctrl/core.c | 11 +++++++----
arch/x86/kernel/cpu/resctrl/ctrlmondata.c | 14 ++++++++------
arch/x86/kernel/cpu/resctrl/internal.h | 9 +++++++++
arch/x86/kernel/cpu/resctrl/rdtgroup.c | 18 ++++++++++++++++++
include/linux/resctrl.h | 4 ++--
6 files changed, 56 insertions(+), 16 deletions(-)
base-commit: 27bbf45eae9ca98877a2d52a92a188147cd61b07
--
2.42.0
Hi Maciej,
On 10/5/2023 1:14 AM, Maciej Wieczor-Retman wrote:
> Until recently Intel CPUs didn't support using non-contiguous 1s
> in Cache Allocation Technology (CAT). Writing a bitmask with
> non-contiguous 1s to the resctrl schemata file would fail.
>
> Intel CPUs that support non-contiguous 1s can be identified through a
> CPUID leaf mentioned in the "Intel® 64 and IA-32 Architectures
> Software Developer’s Manual" document available at:
> https://www.intel.com/content/www/us/en/developer/articles/technical/intel-sdm.html
>
> Add kernel support for detecting if non-contiguous 1s in Cache
> Allocation Technology (CAT) are supported by the hardware. Also add a
> new resctrl FS file to output this information to the userspace.
> Keep the hardcoded value for Haswell CPUs only since they do not have
> CPUID enumeration support for Cache allocation.
>
> Since the selftests/resctrl files are going through many rewrites and
> cleanups the appropriate selftest is still a work in progress. For
> basic selftesting capabilities use the bash script attached below this
> paragraph. It checks whether various bitmasks written into resctrl FS
> generate output consistent with reported feature support.
This work conflicts with Babu's series [1] that is also ready for inclusion.
We could wait for outcome of next level review to determine who will need
to rebase. It may help to provide a snippet of the conflict resolution
in anticipation of Babu's series being merged first (I will propose exactly
the same to Babu for the scenario of this work merged first).
Reinette
[1] https://lore.kernel.org/lkml/[email protected]/
On 2023-10-06 at 10:53:52 -0700, Reinette Chatre wrote:
>Hi Maciej,
>
>On 10/5/2023 1:14 AM, Maciej Wieczor-Retman wrote:
>> Until recently Intel CPUs didn't support using non-contiguous 1s
>> in Cache Allocation Technology (CAT). Writing a bitmask with
>> non-contiguous 1s to the resctrl schemata file would fail.
>>
>> Intel CPUs that support non-contiguous 1s can be identified through a
>> CPUID leaf mentioned in the "Intel® 64 and IA-32 Architectures
>> Software Developer’s Manual" document available at:
>> https://www.intel.com/content/www/us/en/developer/articles/technical/intel-sdm.html
>>
>> Add kernel support for detecting if non-contiguous 1s in Cache
>> Allocation Technology (CAT) are supported by the hardware. Also add a
>> new resctrl FS file to output this information to the userspace.
>> Keep the hardcoded value for Haswell CPUs only since they do not have
>> CPUID enumeration support for Cache allocation.
>>
>> Since the selftests/resctrl files are going through many rewrites and
>> cleanups the appropriate selftest is still a work in progress. For
>> basic selftesting capabilities use the bash script attached below this
>> paragraph. It checks whether various bitmasks written into resctrl FS
>> generate output consistent with reported feature support.
>
>This work conflicts with Babu's series [1] that is also ready for inclusion.
>We could wait for outcome of next level review to determine who will need
>to rebase. It may help to provide a snippet of the conflict resolution
>in anticipation of Babu's series being merged first (I will propose exactly
>the same to Babu for the scenario of this work merged first).
>
>Reinette
>
>[1] https://lore.kernel.org/lkml/[email protected]/
Thanks for fiding this issue. I can see how to resolve the conflict but
where can I put the solution?
I'm guessing in the cover letter?
I'm going to resend the series very soon to apply Babu's comment and
tags. I'll then attach the snippet you mentioned wherever you think it
would fit best.
--
Kind regards
Maciej Wieczór-Retman
On Fri, Oct 06, 2023 at 10:53:52AM -0700, Reinette Chatre wrote:
> This work conflicts with Babu's series [1] that is also ready for inclusion.
> We could wait for outcome of next level review to determine who will need
Who is "next level review"?
> to rebase. It may help to provide a snippet of the conflict resolution
> in anticipation of Babu's series being merged first (I will propose exactly
> the same to Babu for the scenario of this work merged first).
Just lemme know which ones I should merge first and the others can be
rebased on top.
Thx.
--
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquette
Hi Boris,
On 10/9/2023 5:40 AM, Borislav Petkov wrote:
> On Fri, Oct 06, 2023 at 10:53:52AM -0700, Reinette Chatre wrote:
>> This work conflicts with Babu's series [1] that is also ready for inclusion.
>> We could wait for outcome of next level review to determine who will need
>
> Who is "next level review"?
This is just a term I made up for when a series is deemed ready to be
considered for inclusion by x86 maintainer team.
>> to rebase. It may help to provide a snippet of the conflict resolution
>> in anticipation of Babu's series being merged first (I will propose exactly
>> the same to Babu for the scenario of this work merged first).
>
> Just lemme know which ones I should merge first and the others can be
> rebased on top.
I believe that Babu's series is ready for inclusion and can be merged first.
It is at:
https://lore.kernel.org/lkml/[email protected]/
Maciej is already planning to send a new version of this series and can
rebase on top of Babu's work at that time.
Thank you very much.
Reinette
On Mon, Oct 09, 2023 at 08:32:07AM -0700, Reinette Chatre wrote:
> This is just a term I made up for when a series is deemed ready to be
> considered for inclusion by x86 maintainer team.
Aha. :-)
> I believe that Babu's series is ready for inclusion and can be merged first.
> It is at:
> https://lore.kernel.org/lkml/[email protected]/
Ok, lemme look at it.
> Maciej is already planning to send a new version of this series and can
> rebase on top of Babu's work at that time.
Ok, you'll get the tip-bot notifications and then you'll know when and
ontop of what branch to rebase.
> Thank you very much.
Thanks too!
--
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquette