Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
From:   Florian Weimer <fweimer@redhat.com>
To:     Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Cc:     linux-kernel@vger.kernel.org, Ingo Molnar <mingo@redhat.com>,
        Peter Zijlstra <peterz@infradead.org>,
        libc-alpha@sourceware.org, Waiman Long <longman@redhat.com>
Subject: Re: rseq CPU ID not correct on 6.0 kernels for pinned threads
References: <87lem9cnxr.fsf@oldenburg.str.redhat.com>
        <8b52b0d6-c973-f959-b44a-1b54fb808a04@efficios.com>
        <871qnzpv9l.fsf@oldenburg.str.redhat.com>
        <5c64b742-c41a-8c59-c0c8-8b4cdedaaba5@efficios.com>
Date:   Fri, 13 Jan 2023 17:06:48 +0100
In-Reply-To: <5c64b742-c41a-8c59-c0c8-8b4cdedaaba5@efficios.com> (Mathieu
        Desnoyers's message of "Thu, 12 Jan 2023 15:25:42 -0500")
Message-ID: <877cxq4dwn.fsf@oldenburg.str.redhat.com>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/28.2 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain
Precedence: bulk

* Mathieu Desnoyers:

> On 2023-01-12 11:33, Florian Weimer wrote:
>> * Mathieu Desnoyers:
>> 
>>> As you also point out, it can also be caused by some other task
>>> modifying the affinity of your task concurrently. You could print
>>> the result of sched_getaffinity on error to get a better idea of
>>> the expected vs actual mask.
>>>
>>> Lastly, it could be caused by CPU hotplug which would set all bits
>>> in the affinity mask as a fallback. As you mention it should not be
>>> the cause there.
>>>
>>> Can you share your kernel configuration ?
>> Attached.
>> cpupower frequency-info says:
>> analyzing CPU 0:
>>    driver: intel_cpufreq
>>    CPUs which run at the same hardware frequency: 0
>>    CPUs which need to have their frequency coordinated by software: 0
>>    maximum transition latency: 20.0 us
>>    hardware limits: 800 MHz - 4.60 GHz
>>    available cpufreq governors: conservative ondemand userspace powersave performance schedutil
>>    current policy: frequency should be within 800 MHz and 4.60 GHz.
>>                    The governor "schedutil" may decide which speed to use
>>                    within this range.
>>    current CPU frequency: Unable to call hardware
>>    current CPU frequency: 3.20 GHz (asserted by call to kernel)
>>    boost state support:
>>      Supported: yes
>>      Active: yes
>> And I have: kernel.sched_energy_aware = 1
>> 
>>> Is this on a physical machine or in a virtual machine ?
>> I think it happened on both.
>> I added additional error reporting to the test (running on kernel
>> 6.0.18-300.fc37.x86_64), and it seems that there is something that is
>> mucking with affinity masks:
>> info: Detected CPU set size (in bits): 64
>> info: Maximum test CPU: 19
>> error: Pinned thread 17 ran on impossible cpu 7
>> info: getcpu reported CPU 7, node 0
>> info: CPU affinity mask: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19
>> error: Pinned thread 3 ran on impossible cpu 13
>> info: getcpu reported CPU 13, node 0
>> info: CPU affinity mask: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19
>> info: Main thread ran on 2 CPU(s) of 20 available CPU(s)
>> info: Other threads ran on 20 CPU(s)
>> For each of these threads, the affinity mask should be a singleton
>> set.
>> Now I need to find out if there is a process that changes affinity
>> settings.
>
> If it's not cpu hotunplug, then perhaps something like systemd
> modifies the AllowedCPUs of your cpuset concurrently ?

It's probably just this kernel bug:

commit da019032819a1f09943d3af676892ec8c627668e
Author: Waiman Long <longman@redhat.com>
Date:   Thu Sep 22 14:00:39 2022 -0400

    sched: Enforce user requested affinity
    
    It was found that the user requested affinity via sched_setaffinity()
    can be easily overwritten by other kernel subsystems without an easy way
    to reset it back to what the user requested. For example, any change
    to the current cpuset hierarchy may reset the cpumask of the tasks in
    the affected cpusets to the default cpuset value even if those tasks
    have pre-existing user requested affinity. That is especially easy to
    trigger under a cgroup v2 environment where writing "+cpuset" to the
    root cgroup's cgroup.subtree_control file will reset the cpus affinity
    of all the processes in the system.
    
    That is problematic in a nohz_full environment where the tasks running
    in the nohz_full CPUs usually have their cpus affinity explicitly set
    and will behave incorrectly if cpus affinity changes.
    
    Fix this problem by looking at user_cpus_ptr in __set_cpus_allowed_ptr()
    and use it to restrcit the given cpumask unless there is no overlap. In
    that case, it will fallback to the given one. The SCA_USER flag is
    reused to indicate intent to set user_cpus_ptr and so user_cpus_ptr
    masking should be skipped. In addition, masking should also be skipped
    if any of the SCA_MIGRATE_* flag is set.
    
    All callers of set_cpus_allowed_ptr() will be affected by this change.
    A scratch cpumask is added to percpu runqueues structure for doing
    additional masking when user_cpus_ptr is set.
    
    Signed-off-by: Waiman Long <longman@redhat.com>
    Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
    Link: https://lkml.kernel.org/r/20220922180041.1768141-4-longman@redhat.com

I don't think it's been merged into any stable kernels yet?

Thanks,
Florian