Received: by 2002:a05:6358:11c7:b0:104:8066:f915 with SMTP id i7csp5855985rwl; Wed, 22 Mar 2023 03:21:54 -0700 (PDT) X-Google-Smtp-Source: AK7set+d3B3S6HAmq35MNNWqV0SW8r+SE63oFqlt36ba4Rj4dVMxDHDq3N4wQUWPXJNVM2qQieaO X-Received: by 2002:a17:906:a098:b0:933:12d1:d168 with SMTP id q24-20020a170906a09800b0093312d1d168mr6590966ejy.10.1679480513867; Wed, 22 Mar 2023 03:21:53 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1679480513; cv=none; d=google.com; s=arc-20160816; b=zIIglsTGjKhk/eFrpjxrmorO5+HarGngFN5F7tludrT9N8bG1sNN7u9T8Vw87gNPLQ RnwCkPQMZRDX4l5Fx2pSVUMOhQ/NSjEG6mukAyDBiEe5b7h+r+N/aYMdO4IoAvgKt3Mj 87Bd5Yfn0pDc6BG96mxGZaoPMT6xod3kvVKVzlWAeN396s13giQzao08jUjmd6a3bYwp fYK/vE277nEgNIB/TC49/Am5nOYV0BI0KGhGp9AZJFPwGq++gT/dgWfl8eNe6AD5SL8W BRc7Oj8hR2wEomrT/qL2Eu6W48k25WyNORBw9+FljC0pyOE/FqqLu1BkfJpZ74RDvLch NLFQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:in-reply-to:content-disposition:mime-version :references:message-id:subject:cc:to:from:date:dkim-signature; bh=aqEzHI7CsoF2HtxRsuwP9+5W2QuYHXY8VrSFADFSMLM=; b=do4F59GMJ0FioC1AUEzZkvCfqnHXrinUs73t8t8upnvmHJ0GvI7bWplAiwUxkxA/6f 58ox9ryvuS+i3TTbBZg2jux5j8vFoEJ+/gdxaEEezDbNYY4I70rc2MBOIMAgCxWnWeuL fJaeOoQXp5nRTcvYcYOlMmmP1Wfp/TkBnar8RCzvvnW/FChbRbnHS72mt4Q8MvUTrjP5 xlmpsZXpBvp89sY0Ae0xy9UVxVcVoDUG8r3sbXH/c+i5l0sAnGq1FXmjyQrGpgOR4e3z goHQiR4x7PzHXA4lkgn//uh77vHGniX5x682dSrnS6tXU4v8XauD89WFWg0z5urYrurK mYAg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@suse.com header.s=susede1 header.b=nTml06Ve; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=suse.com Return-Path: Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id vv8-20020a170907a68800b009327ff6d591si12372211ejc.224.2023.03.22.03.21.29; Wed, 22 Mar 2023 03:21:53 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@suse.com header.s=susede1 header.b=nTml06Ve; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=suse.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229937AbjCVKNJ (ORCPT + 99 others); Wed, 22 Mar 2023 06:13:09 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41084 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229750AbjCVKNH (ORCPT ); Wed, 22 Mar 2023 06:13:07 -0400 Received: from smtp-out1.suse.de (smtp-out1.suse.de [IPv6:2001:67c:2178:6::1c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3840072AB for ; Wed, 22 Mar 2023 03:13:04 -0700 (PDT) Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by smtp-out1.suse.de (Postfix) with ESMTPS id E1CE121B45; Wed, 22 Mar 2023 10:13:02 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.com; s=susede1; t=1679479982; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=aqEzHI7CsoF2HtxRsuwP9+5W2QuYHXY8VrSFADFSMLM=; b=nTml06VeJDqK9PEr/pRjJsAJ13bkHpV7QyzOi3MENEr+VHVAZfGLpu7kGmaBJDt56cpijR XFCxNyk/Y+5V0PanR6I9OwVkyv1QVrxosPmUVQVW9B8jXleMxpxNNu3STvpHnpzJoXVLIA +QRg5R4/1vUhgJI0ZuXXbspr/NhFqOE= Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by imap2.suse-dmz.suse.de (Postfix) with ESMTPS id BC5C7138E9; Wed, 22 Mar 2023 10:13:02 +0000 (UTC) Received: from dovecot-director2.suse.de ([192.168.254.65]) by imap2.suse-dmz.suse.de with ESMTPSA id yuIuKq7UGmT5CQAAMHmgww (envelope-from ); Wed, 22 Mar 2023 10:13:02 +0000 Date: Wed, 22 Mar 2023 11:13:02 +0100 From: Michal Hocko To: Marcelo Tosatti Cc: Christoph Lameter , Aaron Tomlin , Frederic Weisbecker , Andrew Morton , linux-kernel@vger.kernel.org, linux-mm@kvack.org, Russell King , Huacai Chen , Heiko Carstens , x86@kernel.org, Vlastimil Babka Subject: Re: [PATCH v7 00/13] fold per-CPU vmstats remotely Message-ID: References: <20230320180332.102837832@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Spam-Status: No, score=-2.5 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Mon 20-03-23 16:07:29, Marcelo Tosatti wrote: > On Mon, Mar 20, 2023 at 07:25:55PM +0100, Michal Hocko wrote: > > On Mon 20-03-23 15:03:32, Marcelo Tosatti wrote: > > > This patch series addresses the following two problems: > > > > > > 1. A customer provided evidence indicating that a process > > > was stalled in direct reclaim: > > > > > This is addressed by the trivial patch 1. > > > > [...] > > > 2. With a task that busy loops on a given CPU, > > > the kworker interruption to execute vmstat_update > > > is undesired and may exceed latency thresholds > > > for certain applications. > > > > Yes it can but why does that matter? > > It matters for the application that is executing and expects > not to be interrupted. Those workloads shouldn't enter the kernel in the first place, no? Otherwise the in kernel execution with all the direct or indirect dependencies (e.g. via locks) can throw any latency expectations off the window. > > > By having vmstat_shepherd flush the per-CPU counters to the > > > global counters from remote CPUs. > > > > > > This is done using cmpxchg to manipulate the counters, > > > both CPU locally (via the account functions), > > > and remotely (via cpu_vm_stats_fold). > > > > > > Thanks to Aaron Tomlin for diagnosing issue 1 and writing > > > the initial patch series. > > > > > > > > > Performance details for the kworker interruption: > > > > > > oslat 1094.456862: sys_mlock(start: 7f7ed0000b60, len: 1000) > > > oslat 1094.456971: workqueue_queue_work: ... function=vmstat_update ... > > > oslat 1094.456974: sched_switch: prev_comm=oslat ... ==> next_comm=kworker/5:1 ... > > > kworker 1094.456978: sched_switch: prev_comm=kworker/5:1 ==> next_comm=oslat ... > > > > > > The example above shows an additional 7us for the > > > > > > oslat -> kworker -> oslat > > > > > > switches. In the case of a virtualized CPU, and the vmstat_update > > > interruption in the host (of a qemu-kvm vcpu), the latency penalty > > > observed in the guest is higher than 50us, violating the acceptable > > > latency threshold for certain applications. > > > > I do not think we have ever promissed any specific latency guarantees > > for vmstat. These are statistics have been mostly used for debugging > > purposes AFAIK. I am not aware of any specific user space use case that > > would be latency sensitive. Your changelog doesn't go into details there > > either. > > There is a class of workloads for which response time can be > of interest. MAC scheduler is an example: > > https://par.nsf.gov/servlets/purl/10090368 Yes, I am not disputing low latency workloads in general. I am just saying that you haven't really established a very sound justification here. Of course there are workloads which do not want to conflict with any in kernel house keeping. Those have to be configured and implemented very carefully though. Vmstat as such should not collide with those workloads as long as they do not interact with the kernel in a way counters are updated. Is this hard or impossible to avoid? I can imagine that those workloads have an start up sequence where the kernel is involved and counters updated so that deferred flushing could interfere with the later and latency sensitive phase. Is that a real problem in practice? Please tell us much more why we need to make the vmstat code more complex. Thanks! -- Michal Hocko SUSE Labs