Received: by 2002:a05:6358:3188:b0:123:57c1:9b43 with SMTP id q8csp4495273rwd; Sun, 11 Jun 2023 08:37:11 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ79XhFAX2mfEcrnfI/iXV8secI3s4b2lM/Gq8N+UjqxrSSTBzgrb+eWdzCpmpEvuSJd9Tcn X-Received: by 2002:a17:902:d88c:b0:1b3:a41e:8dc5 with SMTP id b12-20020a170902d88c00b001b3a41e8dc5mr4016520plz.49.1686497830846; Sun, 11 Jun 2023 08:37:10 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1686497830; cv=none; d=google.com; s=arc-20160816; b=kWEray0BHiABZrgn8o3eYfNWyV+k95VktzV6jyd+lskCZng3E2RXy2l854MfwDjSpo 6pxoy0DegeaRV4N6HAKHrry8875TgPYzE6N3uJmTAiuzv40v2VXxDmBQt/SurxOwC/2Y Q6sopvv4YArW1kORfEFv8EAtG53EqUXu7s3pBj9MESRvXWRhJlMNKFWC/iwOZV0TJ386 b9ZNjrNbzG7wW0zosxNCKNcZeEzVpm01Njp0Ecv6fRF+4YVzhgRFkMaSPHyBd/gK/OEn GzgZoKXxCMNVutnwHUwD5xmhCKFRirv0HHgK4Xcqt/yPo2osZVFyqrk5AMWEUbRyICQ2 v+sQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:in-reply-to:subject :references:cc:to:from:content-language:user-agent:mime-version:date :message-id:sender:dkim-signature; bh=7WY33NERKlb17TukmGOgkhDaG5sdGDG/5hEoznoyZyc=; b=EbnIsg9On85u5ALgWBkyCG98YZjhszMglyqu45hHGhkpVBuMbLLuTfi8wdE3INIsr9 SydgzCZMPxGzXrqNLoxalfiz/pXpCfUzuGGeN4K3mE5bjOeeaXsFflUEwHqLgGVSEMw5 SoxjUX+3hUt2bsZcjDyWhjRSPyp2imJaSycbcSWflHOn6W1i7BQuNFLlKdm5KgIp3gXs vL5n1cYaKO48tB8PtqovYN2p13KKEdqkvRFvE2yWCt63rxrNVu3XO6ojtnaMnDJf9WPb bo9kkYVqQeT0VuOmbdE0d+RPGkKSrwxlhf9yqWZaq+5cZOihYE5bOvfmdGYWe8WAjbD8 4uVw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20221208 header.b=KwDvtnK+; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id b7-20020a656687000000b005074cff89b9si5787323pgw.250.2023.06.11.08.36.51; Sun, 11 Jun 2023 08:37:10 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20221208 header.b=KwDvtnK+; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229861AbjFKPO1 (ORCPT + 99 others); Sun, 11 Jun 2023 11:14:27 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49364 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229497AbjFKPOZ (ORCPT ); Sun, 11 Jun 2023 11:14:25 -0400 Received: from mail-pf1-x42b.google.com (mail-pf1-x42b.google.com [IPv6:2607:f8b0:4864:20::42b]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id DF7D419A; Sun, 11 Jun 2023 08:14:23 -0700 (PDT) Received: by mail-pf1-x42b.google.com with SMTP id d2e1a72fcca58-652dd220d67so3648197b3a.3; Sun, 11 Jun 2023 08:14:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1686496463; x=1689088463; h=content-transfer-encoding:in-reply-to:subject:references:cc:to:from :content-language:user-agent:mime-version:date:message-id:sender :from:to:cc:subject:date:message-id:reply-to; bh=7WY33NERKlb17TukmGOgkhDaG5sdGDG/5hEoznoyZyc=; b=KwDvtnK+ZKfFuIcSDBnGZGcuyVdYfWVKiDGusWxDKK7vtT4+rOpxM6PZSVLv8DxmQW vAbTmuE8KHbyXmldyGPGn7On0esT7A1R7FX2K+uU/4twbBbhxhPv9ICSBb6hnVnFOpTn 8S3Lot2Yi4PD6Nttp+N/Fu2Cv4Pe2uxpST2/uebP3QxHUyiou5Q+HjaZ1n4hBEPZjwkk iSEbv7JJoOHyyo/cVIxAUz8AgntHlytq3qj4iTD85j+wGkBTjMnCNm5XJc+HcfHTcWFm BPoGr9jY74VX3XBiE4X/QaPCHDsGWC5x0DpQhDWKcCEpkRkGGH37KX99qxBY/81jbnnm GuDA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1686496463; x=1689088463; h=content-transfer-encoding:in-reply-to:subject:references:cc:to:from :content-language:user-agent:mime-version:date:message-id:sender :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=7WY33NERKlb17TukmGOgkhDaG5sdGDG/5hEoznoyZyc=; b=Nn59Tdvg1ivoHUWHd+1d5SoKpOw0OZYY03hgyKT1vFrLHG7krnYO2RgmKcoud2ywTI pM9iKmWFjJ3wejtvDbG3NyaV7z5oFZJlhNTQa4Gk0a9GgTCvc4EGNDc3BIRwyHf2aPRV aOHTtiNIB3HGcQwo4WJPITbVMEendlwG0ymm7X819C7YvQjeMxMk+w2fGrO9l3j7EHoQ cSg5neuIW3LkYUOKoa3lpSLpspM4kH45wdW0lE/Cx4Hem6N6craMUuZcjELPKhw9p3D8 BbIHe5d6tWZ+u897M5HoQ+bkjBrgYxs+hGy+XD+bcEtYMgDPTSaF7L4gK29sTeacMbJl nvzw== X-Gm-Message-State: AC+VfDyQ4NBxuiT53S/VH3O0nbCRSj/ts5QQoUw+gKgqOCDiX8XK6P9r TvaDX0PT8m1tmg/HZk6Jzy0= X-Received: by 2002:a05:6a00:1aca:b0:658:8eae:a8de with SMTP id f10-20020a056a001aca00b006588eaea8demr8076103pfv.4.1686496463153; Sun, 11 Jun 2023 08:14:23 -0700 (PDT) Received: from ?IPV6:2600:1700:e321:62f0:329c:23ff:fee3:9d7c? ([2600:1700:e321:62f0:329c:23ff:fee3:9d7c]) by smtp.gmail.com with ESMTPSA id k67-20020a632446000000b005439aaf0301sm5910871pgk.64.2023.06.11.08.14.20 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Sun, 11 Jun 2023 08:14:22 -0700 (PDT) Sender: Guenter Roeck Message-ID: <1e9e0ce4-dc47-81d6-e3a5-681d4a9ba9b0@roeck-us.net> Date: Sun, 11 Jun 2023 08:14:19 -0700 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Thunderbird/102.11.0 Content-Language: en-US From: Guenter Roeck To: Pavel Machek Cc: Greg Kroah-Hartman , stable@vger.kernel.org, patches@lists.linux.dev, linux-kernel@vger.kernel.org, torvalds@linux-foundation.org, akpm@linux-foundation.org, shuah@kernel.org, patches@kernelci.org, lkft-triage@lists.linaro.org, jonathanh@nvidia.com, f.fainelli@gmail.com, sudipm.mukherjee@gmail.com, srw@sladewatkins.net, rwarsow@gmx.de, Thomas Gleixner , Ido Schimmel References: <20230607200903.652580797@linuxfoundation.org> <39b34446-88a7-eeb5-cf95-e4f1843100b3@roeck-us.net> Subject: Re: [PATCH 5.15 000/159] 5.15.116-rc1 review In-Reply-To: <39b34446-88a7-eeb5-cf95-e4f1843100b3@roeck-us.net> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Spam-Status: No, score=-1.4 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_EF,FREEMAIL_ENVFROM_END_DIGIT, FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS, NICE_REPLY_A,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS, T_SCC_BODY_TEXT_LINE autolearn=no autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 6/10/23 14:14, Guenter Roeck wrote: > Hi, > > On 6/10/23 12:23, Pavel Machek wrote: >> Hi! >> >>>> Build results: >>>>     total: 155 pass: 155 fail: 0 >>>> Qemu test results: >>>>     total: 499 pass: 498 fail: 1 >>>> Failed tests: >>>>     arm:kudo-bmc:multi_v7_defconfig:npcm:usb0.1:nuvoton-npcm730-kudo:rootfs >>>> >>>> The test failure is spurious and not new. I observe it randomly on >>>> multi_v7_defconfig builds, primarily on npcm platforms. There is no error >>>> message, just a stalled boot. I have been trying to bisect for a while, >>>> but I have not been successful so far. No immediate concern; I just wanted >>>> to mention it in case someone else hits the same or a similar problem. >>>> >>> >>> I managed to revise my bisect script sufficiently enough to get reliable >>> results. It looks like the culprit is commit 503e554782c9 (" debugobject: >>> Ensure pool refill (again)"); see bisect log below. Bisect on four >>> different systems all have the same result. After reverting this patch, >>> I do not see the problem anymore (again, confirmed on four different >>> systems). If anyone has an idea how to debug this, please let me know. >>> I'll be happy to give it a try. >> >> You may want to comment out debug_objects_fill_pool() in >> debug_object_activate or debug_object_assert_init to see which one is >> causing the failure... >> >> CONFIG_PREEMPT_RT is disabled for you, right? (Should 5.15 even have >> that option?) >> > > CONFIG_PREEMPT_RT is disabled (it depends on ARCH_SUPPORTS_RT which is not > enabled by any architecture in v5.15.y). > > The added call in debug_object_activate() triggers the problem. > Any idea what to do about it or how to debug it further ? > I did some more debugging. The call to debug_object_activate() from debug_hrtimer_activate() causes the immediate problem, and the call from debug_timer_activate() causes a second (less likely) problem, where the stall is seen during reboot. In other words, the problem is (only) seen if DEBUG_OBJECTS_TIMERS is enabled. Guenter