Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <fd5a27de-c8a9-892c-f413-66ea41221fdd@amd.com>
Date:   Fri, 9 Jun 2023 09:13:15 +0530
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:91.0) Gecko/20100101
 Thunderbird/91.13.1
Subject: Re: [PATCH 14/24] workqueue: Generalize unbound CPU pods
Content-Language: en-US
To:     Tejun Heo <tj@kernel.org>
Cc:     Sandeep Dhavale <dhavale@google.com>, jiangshanlai@gmail.com,
        torvalds@linux-foundation.org, peterz@infradead.org,
        linux-kernel@vger.kernel.org, kernel-team@meta.com,
        joshdon@google.com, brho@google.com, briannorris@chromium.org,
        nhuck@google.com, agk@redhat.com, snitzer@kernel.org,
        void@manifault.com, kernel-team@android.com,
        Swapnil Sapkal <swapnil.sapkal@amd.com>
References: <20230519001709.2563-1-tj@kernel.org>
 <20230519001709.2563-15-tj@kernel.org>
 <CAB=BE-S=cxewXu7nqJY1DC5w9Bapar_C0cTjpZOQ-Qd5GGwYyw@mail.gmail.com>
 <c307ba94-0d8c-3cbf-19da-44ee31751428@amd.com>
 <ZIEBB-A1arYKSK2P@slm.duckdns.org>
 <fb3461cd-3fc2-189a-a86b-c638816a2440@amd.com>
 <ZIJbMQOu_k07jkFf@slm.duckdns.org>
From:   K Prateek Nayak <kprateek.nayak@amd.com>
In-Reply-To: <ZIJbMQOu_k07jkFf@slm.duckdns.org>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?dzFPc3Q3STluOENaWlBDTnB2bFUwSXIvU2xNb3pldnBVWStXdTZ3YlFXQVlI?=
 =?utf-8?B?YVlROFJ1b05FKzc5R2t5TkIyVmJzK3V1TlUwR1NxWWVGZlJMRDdMUStncTdW?=
 =?utf-8?B?Z0cybGl0RDN1M0JVUmxnVkNPSHNGRkkzSkNlLzUzVzZUeEU1b3BHL05VRW1h?=
 =?utf-8?B?WUZjaGRTc2hwcnBBWjZock5LMzF5V0VQSEk0bEIxanVlSzM3RVRBczFmRWpw?=
 =?utf-8?B?OWpkQk9Jdm81WUl0ZXI0dFdpMTdKOHdLc1RzY2V3RkJVNlYxcFA0RndZWW1x?=
 =?utf-8?B?dE9hL2syckZ3amtNVlczbndZRklySE5FTVZpWXRyMm84eVc5OGJ2MlZGZFQ4?=
 =?utf-8?B?SEtUdjE0UktNdEhLeWN5UWlYSmdGaGpBQ1ZwQkVaeVdNQkVMR2tjVzBBNzR4?=
 =?utf-8?B?UXZpWnR2a1RFeWh6Vmx5WERRak1oRU9PNTNob2hOa1NITXJubkpnVHRKQ0My?=
 =?utf-8?B?bkVUemMxUHY3Qy9tUE5EZUVIUTlyditLYlNSUEt0RGNUSkNEQU84c1FJZzQv?=
 =?utf-8?B?RUtkRnJ3MHcwcW1jdjIwNmIwOFVzR2N2eTFmOFVTOEtVTDE3NzZjclNzcmxr?=
 =?utf-8?B?cUdNdGd6c1UrRE5sM3RJZDQ5aHlsNHRFUzFQZUNDQjJOOHFlbWNsVGlvd0w4?=
 =?utf-8?B?RU0reUpXS3VpVk5oOTBiN3FBRTErazQ4c1dvV1NaQ0hPZ3BkZWZKUEx4MU5E?=
 =?utf-8?B?NTBhM3I5cHBBT1RnTGhKQ09OK1RSZFhlcEJzcmIvYjFLVHl0QndyVnIrb3NX?=
 =?utf-8?B?U1Btc1pyZ3grQUFlRlArcUE4aFRFNWJSZlJlVXdPYW5XY1lnMkpZS1l1UGZh?=
 =?utf-8?B?V2pwS3pEamFpVlBUZFZDTW9zQlJMaVhGdU1iR1NGN2xWNElRSWJJK05tMkpj?=
 =?utf-8?B?dkdSelNnMVFYNFFMVUQ5UWc3M1lyeGxQRngrRWk5UHBneTk0ZTRFTTZ3dElS?=
 =?utf-8?B?L3hNUnBCUnNBWWhzL3NzL2RYanlUbmkwKzJYS3lFV1pHVGhDZ3FCdlVBbXRs?=
 =?utf-8?B?VEQ4THRBNndtTWZZMlFpb1dYKytPdDduOExuWDlpTkVGMXpRYnFvUExkNHZU?=
 =?utf-8?B?a3RFeVlvamlPUXk2UGVsZmlkYnJMVExRV2xGdTduRWsvK1ZEMjc0NE0xWjlt?=
 =?utf-8?B?SVZYTnI3NDJKSys1YUZyZVh2MndzMEpjalQ1SzFsVk5nUWhlbGNZSklKOE5l?=
 =?utf-8?B?RTQwZTRQSlBEYnM0R04yMWRFM1poTUlmVFFjZHNWQllWQUF1RWxjZlVJNDVs?=
 =?utf-8?B?N25aVzNwZTQxQ25PaG9YSHdWTVQ5SzRwOThMVVNIZkJxWDhjZGVlTk53bXZF?=
 =?utf-8?B?WFh2ZmtNT2ppbGszU1BXcHJCWngrRG4rakIrKzMxNHNkNWdMVEJXazdGdXpI?=
 =?utf-8?B?TTdHYkEzUkorUDNGVnpEcnBZYkZrKzJOanVYOXhrSHJvMHprMUZQdFkrZDVz?=
 =?utf-8?B?YlV3OWFsN2UzNzRmbW1ZbkZvT1dPcDd4RjNKVit5R2h0a00vdWRSd3FNVzg4?=
 =?utf-8?B?WlMzMFBtc0pBbkI5c3ExallWbGZqdmpneW85a0pYbytGa2RxWTFKQStJMnRE?=
 =?utf-8?B?cXhYOEhLcEhma0FEWjNSVHBuTnFSUDZ6N0JQSHpzUFROU3lweXp4bHkyNVlo?=
 =?utf-8?B?b1JYOG9vUkduVFFKVGQ0cDVUMG9uNWp5dXhISGgxeExTNWtiNGZxaCtmQURm?=
 =?utf-8?B?Z3p3UCtYWFRjYmJhSkdZQk9YQmFNSkgvS2xmWTJ4ZUZHcW1YZXM0bmxHbzN0?=
 =?utf-8?B?cnlFYWVsWWZtU1MxQjJyS0svZEtrcVZiV293dkw4bVIrc3R4Zk9rZ2E0U0o0?=
 =?utf-8?B?MXdWOGd4d0picTcwQ2dITUNGTERuNG5RSjdmMWtWaGk3U1VvSkJsd0ZmTlN4?=
 =?utf-8?B?WnVwcDU2QTNoeGZKd0M4VG40ZitqSld5Q0tyZUFTRS9rd2poOWhqWUlxQWtP?=
 =?utf-8?B?dktUUGUvTG81UjZLbFdTRW1EREduKzdZaXh2MTh2alMzNGYySFdXRllkcm04?=
 =?utf-8?B?R29Cc3hIU0hYS1llZEFjZDBCZVNUUlk5YUVGRVUxKzd6U0k4TnAvR1VuMnRa?=
 =?utf-8?B?eHlaU3lERERndDdrK0Z0bHBUQnllOXdyL0FQNEVXM2E2clRNbmpxSERlZ0cx?=
 =?utf-8?Q?100cjzvGl6xBYKin4UQMopLi5?=
X-OriginatorOrg: amd.com
X-MS-Exchange-CrossTenant-Network-Message-Id: fab74375-770b-406a-d2a2-08db689bb0b7
X-MS-Exchange-CrossTenant-AuthSource: MW2PR12MB2379.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 09 Jun 2023 03:43:30.1094
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: 4SaqQTljDjopJYHuz/E9fhRYto+snop2bsRt3mngd9WVuwu4Gsdv9Yk8/hQO/YIQVzqicOgTwZUc75D3rADSJQ==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH8PR12MB7205
X-Spam-Status: No, score=-1.2 required=5.0 tests=BAYES_00,DKIM_SIGNED,
        DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FORGED_SPF_HELO,NICE_REPLY_A,
        RCVD_IN_DNSWL_NONE,RCVD_IN_MSPIKE_H2,SPF_HELO_PASS,SPF_NONE,
        T_SCC_BODY_TEXT_LINE,URIBL_BLOCKED autolearn=no autolearn_force=no
        version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

Hello Tejun,

On 6/9/2023 4:20 AM, Tejun Heo wrote:
> Hello,
> 
> On Thu, Jun 08, 2023 at 08:31:34AM +0530, K Prateek Nayak wrote:
>> [..snip..]
>> o I consistently see a WARN_ON_ONCE() in kick_pool() being hit when I
>>   run "sudo ./stress-ng --iomix 96 --timeout 1m". I've seen few
>>   different stack traces so far. Including all below just in case:
> ...
>> This is the same WARN_ON_ONCE() you had added in the HEAD commit:
>>
>>     $ scripts/faddr2line vmlinux kick_pool+0xdb
>>     kick_pool+0xdb/0xe0:
>>     kick_pool at kernel/workqueue.c:1130 (discriminator 1)
>>
>>     $ sed -n 1130,1132p kernel/workqueue.c
>>     if (!WARN_ON_ONCE(wake_cpu >= nr_cpu_ids))
>>         p->wake_cpu = wake_cpu;
>>     get_work_pwq(work)->stats[PWQ_STAT_REPATRIATED]++;
>>
>> Let me know if you need any more data from my test setup.
>> P.S. The kernel is still up and running (~30min) despite hitting this
>> WARN_ON_ONCE() in my case :)
> 
> Okay, that was me being stupid and not initializing the new fields for
> per-cpu workqueues. Can you please test the following branch? It should have
> both bugs fixed properly.
> 
>  git://git.kernel.org/pub/scm/linux/kernel/git/tj/wq.git affinity-scopes-v2

I've not run into any panics or warnings with this one. Kernel has been
stable for ~30min while running stress-ng iomix. We'll resume the testing
with v2 :)

> 
> If that doesn't crash, I'd love to hear how it affects the perf regressions
> reported over that past few months.> 
> Thanks.
> 

--
Thanks and Regards,
Prateek