Received: by 2002:a05:6602:2086:0:0:0:0 with SMTP id a6csp4729145ioa; Wed, 27 Apr 2022 09:49:49 -0700 (PDT) X-Google-Smtp-Source: ABdhPJwSzFYBHyiuFFICetie7yyygNiqYAChXHL1jEdZAcTk4gpNZkQmesiG2qZWi2mtbvKU0EU4 X-Received: by 2002:a17:902:aa07:b0:158:6d3e:7c54 with SMTP id be7-20020a170902aa0700b001586d3e7c54mr28759595plb.91.1651078189709; Wed, 27 Apr 2022 09:49:49 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1651078189; cv=none; d=google.com; s=arc-20160816; b=hJEjtLBCZFywqImn5py41PJjFusYTfxxTE4sI9IRkMx4TbfQvEFxK+nIOy/mB0T1xI h4N0US45Q5DbD7DI1sh7dJLQ3hGzFI8Dz+WM/xiUdvVKtgFWuSf/xTqezfleSxKeZ67o 3+D3JqZ+t005mkLaTjywtcC87fBTfVQobrtO+mXOSpv2pXc7vKvWRVhFCVpfc14VLNp4 TbsmsVu8Nh21aE/sWPdm+sp5QBhzRceeLsoPzwqtavH76l3QWlgVfnRT0hZPkOWAlB44 3xxeBDrKnYYewC9U3Ztbk6W0rcRLe54UuHZfckyXZFIbAzwxTMGlfdqBw/9xdqqbtWP7 YVyw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:mime-version:message-id:date:references :in-reply-to:subject:cc:to:dkim-signature:dkim-signature:from; bh=hL0SFvjJMATe9c/xNNoB1N/3UB9xTPkwRzwHyvMkOEo=; b=FmMs7hwbdgBNf25mgLaRRpZNQz3TcEaYOFVPvYjenrrb5fSW+oDPHZGfjRM+69NZzH VH23cbYfaDNw9/ujOJwf0h7IyXqD4zf0nbnxzXEO39YgUV+i3cqmdX79y7tVA4w9lLci xSBnB5m79r0lSPV++nm/6addZG8lIv/sQWkRD+E10RHtTcWTzHqNtBRiD4gnE+xvrAaH 8Sr5ZnWHEkh5v3CoybA4XRYqyH+mZMsCBo06urL2X8jokeWAN4hIZoTbR5ru4NM1Yrlp ucQ4evXSBJnEoiP5oEMqH+Pvwb8lyoWcufhAHZL7vKipJCEowuupDR0SwDiWPR9niJIr wjjQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=AQglSgvg; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e; spf=softfail (google.com: domain of transitioning linux-kernel-owner@vger.kernel.org does not designate 23.128.96.19 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Return-Path: Received: from lindbergh.monkeyblade.net (lindbergh.monkeyblade.net. [23.128.96.19]) by mx.google.com with ESMTPS id n12-20020a170902f60c00b0015baf4aee8dsi2092346plg.151.2022.04.27.09.49.49 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 27 Apr 2022 09:49:49 -0700 (PDT) Received-SPF: softfail (google.com: domain of transitioning linux-kernel-owner@vger.kernel.org does not designate 23.128.96.19 as permitted sender) client-ip=23.128.96.19; Authentication-Results: mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=AQglSgvg; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e; spf=softfail (google.com: domain of transitioning linux-kernel-owner@vger.kernel.org does not designate 23.128.96.19 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id 5882D227915; Wed, 27 Apr 2022 09:20:00 -0700 (PDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S243076AbiD0QW1 (ORCPT + 99 others); Wed, 27 Apr 2022 12:22:27 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49828 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S243600AbiD0QTn (ORCPT ); Wed, 27 Apr 2022 12:19:43 -0400 Received: from galois.linutronix.de (Galois.linutronix.de [193.142.43.55]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1C1E051E5A for ; Wed, 27 Apr 2022 09:16:18 -0700 (PDT) From: John Ogness DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1651076116; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=hL0SFvjJMATe9c/xNNoB1N/3UB9xTPkwRzwHyvMkOEo=; b=AQglSgvgOxwUtydaGqvaRV8wKFYWNJ5YBFXokRswfDPGPvtqovA5fe7IEUlYQrfA8A/KK4 vhr+l3sx/bzG5dOHf+iPmNipYtStckHEv7JIIE5HNvBHm5T/Gn8/ipXhmdaHViSUtmKn+l /6ijh6YutZVGxD63L2Tj8820/NMLtvkUWUBO7PMpZjag8/RqsLxRM1RHOAmsr8CSBKitKv 6MNBpflkms+etpyCPbShCWd7nWAqTAiXU+P7NhzCw4DIxFS3rl+G3nqitELh67Q/Q2sgjI AwUEGcKf9JrZwYtTihJWP02Qx5qhgxwgnXOfJV9oJWhFBWHJ+DUogeol4RNguA== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1651076116; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=hL0SFvjJMATe9c/xNNoB1N/3UB9xTPkwRzwHyvMkOEo=; b=da0NgqfyPY/ECXVc5ePxpKHwiEQm5i/p0C8aGvfMy0pAy0eKvexqsUYEauVs/uDOkAC+tD wxFcKLSC1SDrVZAw== To: Marek Szyprowski , Petr Mladek Cc: Sergey Senozhatsky , Steven Rostedt , Thomas Gleixner , linux-kernel@vger.kernel.org, Greg Kroah-Hartman , linux-amlogic@lists.infradead.org Subject: Re: [PATCH printk v5 1/1] printk: extend console_lock for per-console locking In-Reply-To: References: <20220421212250.565456-1-john.ogness@linutronix.de> <20220421212250.565456-15-john.ogness@linutronix.de> <878rrs6ft7.fsf@jogness.linutronix.de> <2a82eae7-a256-f70c-fd82-4e510750906e@samsung.com> Date: Wed, 27 Apr 2022 18:21:16 +0206 Message-ID: <87fslyv6y3.fsf@jogness.linutronix.de> MIME-Version: 1.0 Content-Type: text/plain X-Spam-Status: No, score=-1.5 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INVALID_DATE_TZ_ABSURD,MAILING_LIST_MULTI,RDNS_NONE,SPF_HELO_NONE autolearn=no autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Hi Marek, On 2022-04-27, Marek Szyprowski wrote: > Here is the full serial console log: > > https://pastebin.com/E5CDH88L Here are a few ideas from me: 1. For next-20220427 the printk-threaded series was slightly changed. I do not expect it to work any different, but I would prefer we are debugging the current version. If possible, could you move to next-20220427? 2. I noticed you boot with the kernel boot arguments "earlycon" and "no_console_suspend". Could you try booting without this? I expect this will make no difference. 3. It looks like the problem happens quite late in the boot process. I expect it is due to some userspace process that is running that is interacting with printk (either /dev/kmsg or /proc/kmsg) and is causing problems. If you boot with init=/bin/sh then I expect the system is running fine. (You don't have much of a system running, but it should not hang.) We need to isolate which userspace process is triggering the issue. 4. Have you tried issuing magic sysrq commands on the serial line? (For example, sending a break signal and then the letter 't' or sending a break signal and then the letter 'c'?) That might trigger various dumps so that we can see the system state. 5. You are not running a VT console, so the graphics driver should not be affecting the printk subsystem at all. I expect your autologin is also starting various services and programs. If you disable the automatic login and instead manually login (perhaps as another user) can you manually start those services one at a time to see at what point the system hangs? Thanks for you help with this! John Ogness