Received: by 2002:a6b:500f:0:0:0:0:0 with SMTP id e15csp1051460iob; Thu, 28 Apr 2022 17:45:55 -0700 (PDT) X-Google-Smtp-Source: ABdhPJzAn2Wd5N+4GL9bMx7KAG6hqTY52Vg/AQDUeydR90ua5GgX8dyns/YtbbZw6G/9O4NK3abs X-Received: by 2002:a05:6512:13a7:b0:447:3dac:7a03 with SMTP id p39-20020a05651213a700b004473dac7a03mr26808386lfa.362.1651193155473; Thu, 28 Apr 2022 17:45:55 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1651193155; cv=none; d=google.com; s=arc-20160816; b=bJMU0wEoEGDk6I+AgEIt41d6a/rU6x8UjeeP+HNZAeM5huOZO8UCwcgW3kJ6agsSuo itEUSya/AWJbHvrtxMtSjxTMRlR7l4gCxQSojF96Ms/xbcQ2m70Bd4cmieCdZrC+XiRj pzPprXPSZtr4SAJ0E1BHM4VAvQf0Uggovp0gS9mJTb74GXY5pCTksuZN1XmwnAMiHTLI v+qsBppz2OLttwxI48+xIo7/iijRu0pObuLcrj0+sDtPJB8NqZ2Vx/Yz6xAjIKiE0EGf GCYZaRn/d2SpmJ0MPsTef40DchHPhYEACROT3Z7/GrD138XPBeyiID+ns4WjZu1VTcmt ao+A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:in-reply-to:content-disposition:mime-version :references:message-id:subject:cc:to:from:date:dkim-signature; bh=Oui+HQWDmS1z2iw8Iqn3J1xquHUwKC8gitMw4oJxH4g=; b=bnBfFPRsmcZi+GeAzbqxBBtHpgMTqUX4fpVtWJ2LrGEjQOWc8xp7+r5nbX2Oe0ed65 RjZJ6tgjerumW59UXO7KamgB1e/8OZ6KWM6B7JuXF9Y9AKBIaxo8TG3K41npawhJvzoA tybc1A7eUO7j0rAicy29jM4mJHGQ7yGrDth34eJ/DrLn2BI5MsCKNzEK02DG64/7aD2R yHMXhCNnVs+32nclMJ2VDxEVLa4Ng8rB+K0f7MP21eSIvuAva49ZNpmgECEyVnD0IcZM NhhGDhJKjvy0IudvHSwxArMIJ9qcU/hXwjmSAMRHsJUt6UKyzH9/wDes5TONqgjbs7YT T8fw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@suse.com header.s=susede1 header.b=bQNeM7h8; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=suse.com Return-Path: Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id r8-20020a2e94c8000000b0024998aa2d83si5680628ljh.179.2022.04.28.17.45.29; Thu, 28 Apr 2022 17:45:55 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@suse.com header.s=susede1 header.b=bQNeM7h8; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=suse.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235910AbiD1O6K (ORCPT + 99 others); Thu, 28 Apr 2022 10:58:10 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41978 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235854AbiD1O6J (ORCPT ); Thu, 28 Apr 2022 10:58:09 -0400 Received: from smtp-out2.suse.de (smtp-out2.suse.de [195.135.220.29]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6CCEE9E9FE for ; Thu, 28 Apr 2022 07:54:54 -0700 (PDT) Received: from relay2.suse.de (relay2.suse.de [149.44.160.134]) by smtp-out2.suse.de (Postfix) with ESMTP id 26AC51F88B; Thu, 28 Apr 2022 14:54:53 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.com; s=susede1; t=1651157693; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=Oui+HQWDmS1z2iw8Iqn3J1xquHUwKC8gitMw4oJxH4g=; b=bQNeM7h8q0WUlej4Jf2wj7R6YC8nlIOwlFGDsSxXyocm5+zlJszZyO+ov3XhD/O7Yj3Lbl MGCTQDDB6t8EBefFQsNbhgLuMcyExZz1xntLXDzzzpUx7fGHAI4tSgGTXofPB28Dvzd1IY Sd9SIM8HH2zP/p88mKC7H+EmeE6CXd0= Received: from suse.cz (unknown [10.100.224.162]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by relay2.suse.de (Postfix) with ESMTPS id 9D6CD2C141; Thu, 28 Apr 2022 14:54:52 +0000 (UTC) Date: Thu, 28 Apr 2022 16:54:49 +0200 From: Petr Mladek To: John Ogness Cc: Marek Szyprowski , Sergey Senozhatsky , Steven Rostedt , Thomas Gleixner , linux-kernel@vger.kernel.org, Greg Kroah-Hartman , linux-amlogic@lists.infradead.org Subject: Re: [PATCH printk v5 1/1] printk: extend console_lock for per-console locking Message-ID: References: <20220421212250.565456-1-john.ogness@linutronix.de> <20220421212250.565456-15-john.ogness@linutronix.de> <878rrs6ft7.fsf@jogness.linutronix.de> <2a82eae7-a256-f70c-fd82-4e510750906e@samsung.com> <87fslyv6y3.fsf@jogness.linutronix.de> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <87fslyv6y3.fsf@jogness.linutronix.de> X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,SPF_HELO_NONE, SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Wed 2022-04-27 18:21:16, John Ogness wrote: > Hi Marek, > > On 2022-04-27, Marek Szyprowski wrote: > > Here is the full serial console log: > > > > https://pastebin.com/E5CDH88L > > Here are a few ideas from me: > > 3. It looks like the problem happens quite late in the boot process. I > expect it is due to some userspace process that is running that is > interacting with printk (either /dev/kmsg or /proc/kmsg) and is causing > problems. I did not find an real issue in the code handling /dev/kmsg, /proc/kmsg, or syslog sycall API. There might be just few small changes: 1. There is an increased number of spurious wakeups because log_wait is shared between upstream readers and printk kthreads. And we newly wake up waiters from both vprintk_emit() and __console_unlock() code paths. It might affect especially the pooling APIs: kmsg_pool(), devkmsg_pool()). They might return 0 more often than before. 2. 4th patch replaced wake_up_interruptible(&log_wait) with wake_up_interruptible_all(&log_wait). As a result, all readers are woken at the same time. It is perfectly fine because the log buffer is lockless. And all readers should be either independent or synchronized against each other. Any of the above changes should not introduce new problems. But they might make some old problem (race) more visible. I spent quite some time reviewing the code and testing. But I neither see any problem nor I am able to reproduce it. Some more clues from Marek would be needed. Best Regards, Petr