Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:18 as permitted sender) client-ip=2620:137:e000::1:18;
Message-ID: <e282377b-8ee4-99fc-432f-2aa3441b2989@intel.com>
Date:   Fri, 4 Mar 2022 11:20:10 -0800
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:91.0) Gecko/20100101
 Thunderbird/91.6.1
Subject: Re: [RFC 07/10] platform/x86/intel/ifs: Create kthreads for online
 cpus for scan test
Content-Language: en-US
To:     "Williams, Dan J" <dan.j.williams@intel.com>,
        "hdegoede@redhat.com" <hdegoede@redhat.com>,
        "markgross@kernel.org" <markgross@kernel.org>
CC:     "corbet@lwn.net" <corbet@lwn.net>,
        "Raj, Ashok" <ashok.raj@intel.com>,
        "Luck, Tony" <tony.luck@intel.com>,
        "dave.hansen@linux.intel.com" <dave.hansen@linux.intel.com>,
        "patches@lists.linux.dev" <patches@lists.linux.dev>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        "mingo@redhat.com" <mingo@redhat.com>,
        "rostedt@goodmis.org" <rostedt@goodmis.org>,
        "Shankar, Ravi V" <ravi.v.shankar@intel.com>,
        "tglx@linutronix.de" <tglx@linutronix.de>,
        "platform-driver-x86@vger.kernel.org" 
        <platform-driver-x86@vger.kernel.org>,
        "linux-doc@vger.kernel.org" <linux-doc@vger.kernel.org>,
        "hpa@zytor.com" <hpa@zytor.com>, "bp@alien8.de" <bp@alien8.de>,
        "gregkh@linuxfoundation.org" <gregkh@linuxfoundation.org>,
        "andriy.shevchenko@linux.intel.com" 
        <andriy.shevchenko@linux.intel.com>,
        "x86@kernel.org" <x86@kernel.org>
References: <20220301195457.21152-1-jithu.joseph@intel.com>
 <20220301195457.21152-8-jithu.joseph@intel.com>
 <09b5b05018a8600ca8fab896790ab16827c80e4e.camel@intel.com>
From:   "Joseph, Jithu" <jithu.joseph@intel.com>
In-Reply-To: <09b5b05018a8600ca8fab896790ab16827c80e4e.camel@intel.com>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: 8bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?MGkxMExzSHViVlpJekRJL3FIaHA3TlFZbzdjVlZQTGpJdnpoazRrUG5RVWk1?=
 =?utf-8?B?ampHeG9IbVBGZkFNK1k0WVRrMnNMNWE2dS9SRkp3YjJtNDNuOUE4cjdiQVBU?=
 =?utf-8?B?YUo0SjhjNjRsTWN4MnlER2Z5aVRZRW5veHV5cm91Q0ZONmtTK3RlQlgrcitV?=
 =?utf-8?B?Ti9hT1JUT3ZXVEZiRk95b2xsWUVxejhRdTdxSUU1UzE1SXRGZGxTSmFsTkVW?=
 =?utf-8?B?bWlvYTlwckgxT0VHbEh5OUx3d2JVNXBlOTFGV0RJbG1wWTg5aURIejBTS2cw?=
 =?utf-8?B?M2NpZ04xZmE1ZzZOMEtmVkFzUGpiOGZSMGxaWkZEeXRob0JHQ1J4eTBEckZF?=
 =?utf-8?B?Ylo4Tjh6bHlhd1lsejFRc2xnbjAvdjhWMWQzQkFPRVBlNzNpenI0UjNTMUE0?=
 =?utf-8?B?djhZenZvWEVSNi9RRHNKU3RHQjVvaDMzVXh6ZTFYWnR0N0s5ZmtlbDhwWmpv?=
 =?utf-8?B?bmovSFQ5UC8vMEs5WmxyMWxJUm5rU0NqeGxTMG1WRzN3UjErU053M0kvNGtB?=
 =?utf-8?B?bTJWQkEyeVo5eWRObDZkMUtJbkh4Y2Q4WHR6SmhDVWxCemkweE4xU3Q5Y3N6?=
 =?utf-8?B?YmtZU3hWTlRhR2hmbXhHa2ZOTlg3dmNaSHZRUWZLQ3FEeTVmcVBudDJkQXZM?=
 =?utf-8?B?WHJvN2FFakVwRTRWZSt0bUllQklic0hMQlJGVUsySjhGVjhVa2N5OUVKRVNS?=
 =?utf-8?B?Rk02L0hRb0g0KzlqNUVZYTVQYytXMHRtc1hWcXJkWEUvN09sNTIvT3oyY1VC?=
 =?utf-8?B?UWJ4WTN0SVNxamNMYzNVTjVFM2J0MllKM1NNN2I5K3lPd0Q1SThmcVBoNTJq?=
 =?utf-8?B?NU9sdmpTZUNEYkhqUGtWWnFjY2MxQUtNNERWYjZENExKNGM0Vzg1WlNXRUk3?=
 =?utf-8?B?WE5aQ05MMFhZOUJYcDBrSER2bHErOHhNVkVGRDJzdTJ5MHFINk8wUU1pWXBX?=
 =?utf-8?B?NGU2OHBvNndCQVpURFEyUmlYc05xbTZKOWF1Mms5bnVNMUFoUG1wNGVWUjVa?=
 =?utf-8?B?SjZwVUw4UC83UUdyUVIrVXFKdysyOTM0SHZOTWNXSWJJUkYzT0M2cFU0ZDht?=
 =?utf-8?B?UWFXaDVYbEhaRVNkdUNua3dGdU1uSnpZSmFjanBhQnArNzQ3ZnZPeVRJanFK?=
 =?utf-8?B?a0tnMVZjSlNnbGNWNW1sUEFmc2p0cjZXM2did1hPaTZyNWpZdDUxU0FLeVRY?=
 =?utf-8?B?Z0tEeTdPTHVYSUFTMXd3OWl5UjRLZVZUZ3lvWnI3MlA4V3d6M2ZQMmdzSklB?=
 =?utf-8?B?UU9MMEcxZ3Fkd1B6c296R2ZEdTZDbFFCb1luRUhWc2l0cHJML1dqV3JVNE8r?=
 =?utf-8?B?ejhFWXJzQ0RJam9mUHZrRWg2dlQrbkt0R2xpdUE2N1dsb1cyb0RMV1RBclNk?=
 =?utf-8?B?MkNZSEhQWk4vaGI1NEdhbU9QZXQ0a1cyYUdnMk83clFLVWpJMEhENkx4VzFP?=
 =?utf-8?B?SzJoZ3Y0elNueWhQWXd2OXhoUHV1bnJyblJuZ3VrSXJvTEM0M1NBQUdaeUNK?=
 =?utf-8?B?cUozWkRhTDMvbS8vSXRNV1UrSjFlYnRwT09GQm13VWdsUE94VEdPck15QWp0?=
 =?utf-8?B?RUhFV2piSUdjNDN3cW1sQkNjWjRhQko1YmZJdDdPTEhuYWlidVFFSFd0R0o0?=
 =?utf-8?B?RFVNUW9NN1dWWXZsY0VKVUxjdzdvWG1oL2MxWDBoOTFnNE4wV0srbzlTd05D?=
 =?utf-8?B?b0JwOWRNVnoxR3kraGgvdWJncFNvd0dTNEVQN3VDV2dsL3dwbEkyOFBSN1g4?=
 =?utf-8?B?Mnh2UWZ2Tms4d0R1REF1SldPcnhxQlNHZ3BocGhzellWVWNJbytCdTA3eDZT?=
 =?utf-8?B?VUVwWU5XanFFb3dFcTlwME1LSzhXQWlTdGEzdmVPMGRzS1JWTTdIdlFySXUr?=
 =?utf-8?B?emt2YzcybUhhQStOK05tZ0JRQndDN3g2Y3Z5bEhTczJXeTZsMmJJeTVWYkxL?=
 =?utf-8?B?bXpUaWw1d2wzWVppc0VnMkFQNUNFdjJEQ01ZcTRqWkZUTWF5bHB6czd2YThy?=
 =?utf-8?B?dHlxbE5IeXdOaE1WYlZxYXltZ2lsVkpBd0NPdVB6dHovSm12SnM3b2hkZE51?=
 =?utf-8?B?Z2JtN0E0TkNkUEM2RVFJNGJOWEV3T0ZFSHNJd1FRK1g2UUcxZjFRUVRFRmFp?=
 =?utf-8?B?WXFaUTQxWkRjRjArNUkvZXk2NzJYSjNCNjlTclV1Si9kVkVVb0V5MjJsb2kv?=
 =?utf-8?Q?KDAN8JwasixBdRIlnww2/68=3D?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 60199803-f11d-40da-38d2-08d9fe1402d3
X-MS-Exchange-CrossTenant-AuthSource: BYAPR11MB3768.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 04 Mar 2022 19:20:15.2177
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: HtKcO8v+EWfNAB7HiN0hUSKGwcoXQWDXx053bhDnG0QUnEMdbHauvsKtyAsq5JCYDiZkgteKG6HeXeD8a7g9SQ==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: MW3PR11MB4554
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-2.3 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
	DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,
	MAILING_LIST_MULTI,NICE_REPLY_A,RDNS_NONE,SPF_HELO_NONE,
	T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no
	version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
	lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org


On 3/2/2022 8:17 PM, Williams, Dan J wrote:
> On Tue, 2022-03-01 at 11:54 -0800, Jithu Joseph wrote:
>> Create scan kthreads for online logical cpus. Once scan test is triggered,
>> it wakes up the corresponding thread and its sibling threads to execute
>> the test. Once the scan test is done, the threads go back to thread wait
>> for next signal to start a new scan.
>>
...
>> +
>> +static int retry = 5;
>> +module_param_cb(retry, &ifs_retry_ops, &retry, 0644);
>> +
>> +MODULE_PARM_DESC(retry, "Maximum retry count when the test is not executed");
> 
> Why does this retry need to be in the kernel? Can't the test runner
> retry the test if they want? If it stays in the kernel, why a module
> parameter and not a sysfs attribute?

A core is tested by writing 1 to "runtest" file.  When user writes  a 1 to run_test file it tests all the subtests (chunks) on a core. 
Distinct from this, retry parameter describes the autoretry driver would do at  "chunk" granularity (bit more on why, is available in the doc)

Why not a sysfs attribute: good qn,  Our earlier prototype had this as a percpu sysfs attribute, however this was removed to keep the sysfs entries simple/minimal and less confusing.
(and tunable options which are not strictly needed in the normal course of use were moved to module parameters)
In the percpu sysfs we now only have the essential stuff i.e  run_test , status , and details making it simpler for user who wants to test the core.


> 
>> +
>> +static bool noint = 1;
>> +module_param(noint, bool, 0644);
>> +MODULE_PARM_DESC(noint, "Option to enable/disable interrupt during test");
> 
> Same sysfs vs module parameter question...

Same as above

> 

>> +
>> +static const char * const scan_test_status[] = {
>> +       "SCAN no error",
>> +       "Other thread could not join.",
>> +       "Interrupt occurred prior to SCAN coordination.",
>> +       "Core Abort SCAN Response due to power management condition.",
>> +       "Non valid chunks in the range",
>> +       "Mismatch in arguments between threads T0/T1.",
>> +       "Core not capable of performing SCAN currently",
>> +       "Unassigned error code 0x7",
>> +       "Exceeded number of Logical Processors (LP) allowed to run Scan-At-Field concurrently",
>> +       "Interrupt occurred prior to SCAN start",
> 
> This looks unmaintainable...
> 
> /me finds large comment block around IFS_* error codes and suggests
> killing 2 birds with one stone, i.e. delete that comment and make this
> self documenting:
> 
> static const char * const scan_test_status[] = {
>     [IFS_NO_ERROR] = "SCAN no error",
>     [IFS_OTHER_THREAD_DID_NOT_JOIN] = "Other thread could not join.",
>     ...etc...
> };

Will use this format.

> 
> Btw, which is it "did not join" and "could not join"? If the symbol
> name is going to be that long might as well make it match the log
> message verbatim.

Will make them identical. Thanks for pointing this out.

> 
> That way you can add / delete error codes without wondering if you
> managed to match the code number to the right position in the array.
> 
> Even better would be to kick this out of the kernel and let the user
> tool translate the error codes to test result log messages.
> 
>> +};
>> +
>> +static void message_not_tested(int cpu, union ifs_status status)
>> +{
>> +       if (status.error_code < ARRAY_SIZE(scan_test_status))
>> +               pr_warn("CPU %d: SCAN operation did not start. %s\n", cpu,
>> +                       scan_test_status[status.error_code]);
>> +       else if (status.error_code == IFS_SW_TIMEOUT)
>> +               pr_warn("CPU %d: software timeout during scan\n", cpu);
>> +       else if (status.error_code == IFS_SW_PARTIAL_COMPLETION)
>> +               pr_warn("CPU %d: %s\n", cpu,
>> +                       "Not all scan chunks were executed. Maximum forward progress retries exceeded");
> 
> Why are these codes not in the scan_test_status set? I see that
> IFS_SW_PARTIAL_COMPLETION and IFS_SW_TIMEOUT are defined with larger
> values, but why?

These are software(driver) defined error codes. Rest of the error codes are supplied by
the hardware. Software defined error codes were kept at the other end to provide ample space
in case (future) hardware decides to provide extend error codes.

> 
>> +       else
>> +               pr_warn("CPU %d: SCAN unknown status %llx\n", cpu, status.data);
>> +}
>> +
>> +static void message_fail(int cpu, union ifs_status status)
>> +{
>> +       if (status.control_error) {
>> +               pr_err("CPU %d: scan failed. %s\n", cpu,
>> +                      "Suggest reload scan file: # echo 1 > /sys/devices/system/cpu/ifs/reload");
>> +       }
>> +       if (status.signature_error) {
>> +               pr_err("CPU %d: test signature incorrect. %s\n", cpu,
>> +                      "Suggest retry scan to check if problem is transient");
>> +       }
> 
> This looks and feels more like tools/testing/selftests/ style code
> printing information for a kernel developer to read. For a production
> capability I would expect these debug messages to be elided and have an
> ifs user tool that knows when to "Suggest reload scan file". Otherwise,
> it's not scalable to use the kernel log buffer like a utility's stdout.

The two pr_err here are for really really grave errors and warrants being displayed to console,
possibly indicating some fault with the particular core. They are never expected to come in normal
course of testing on a working core.

But I see your larger point. We will convert all the pr_warn preceeding this (in message_not_tested())
to pr_dbg so that they dont normally take up the kernel log buffer. (they are not as grave a scenario as the earlier one).

The same information is also available from percpu sysfs/cpu#/ifs/status for user spaces tools to operate on

Jithu