Received: by 2002:a05:6358:11c7:b0:104:8066:f915 with SMTP id i7csp7499501rwl; Thu, 23 Mar 2023 05:19:50 -0700 (PDT) X-Google-Smtp-Source: AK7set8ukdsflV/80yYcnoS++ufCCNzlmlQWfaVtr28UQLrQM35R7rHKbIAi3YEAf50M0ZhGXHbw X-Received: by 2002:aa7:d149:0:b0:4fb:9b54:ccb8 with SMTP id r9-20020aa7d149000000b004fb9b54ccb8mr11241952edo.21.1679573990685; Thu, 23 Mar 2023 05:19:50 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1679573990; cv=none; d=google.com; s=arc-20160816; b=vtUXYn6JCRIk3hbMrcU13KSEKfwewbhR71hK8+vQHsE34TzkDY2vjJxCXkHUrbkh/g Kobz2PBHRk3R/nZ/wfhiQMpRLWzIpE+T6VvtUdOiSB3wb8I5xRiczbaX8geVeNxcoyqe 6skZRrAc8YxmVqhZc0KwRT7/AWiejoPRhpkgmaKOKBeN06Izp1mSiAABzwqR8G43sRIZ 69OGq3rERb7Z0iNlbOb+/kpnXeS+FlSi4jTo0qhAUsBEL+jHDOyahg/zQcV1pF+3c0PM B0L+ypikTMnp9jrooq7GaiIJUnw0+Q+wWdySHhZHJ/t00d0iz9FiUikZMU21460CTpVK NLSw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:in-reply-to:content-disposition:mime-version :references:message-id:subject:cc:to:from:date:dkim-signature; bh=Uu+N5jOhken1tfZs5kVQQIsrfUWmzwdgGp/YYNK2ddI=; b=BU6cwGEJBtrXevV8gfTPRQLpgKOVUcGAb0D6TBkbB7LEymOiuzY98VYsxGllQso3Lu OXE7A1q8GFrm5/AAbf+ihnVojp+EWy6rQnAZBfJCG51CLEc63qA6ek7GbaGnbW72qX7w baHeNpuyqxT3YaXnCcHHuu7QvKrubaO0nN+n/Fo9IeIiRAYd4YjP5mvsb5gzeD+8x+ny 4SKVymRjr427/v52CrE1QyBF/rM4VX+U5zpK/KvjsrIKeLz6Q/c55JjLg2NTOJ9X5mKV uFhLdipxTV7+VhJu2f1k9pBskx4xn3KnhR+8/UMGfgFM6ef2K0f6cnIEyTiGg+0+dbOX JyfQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@suse.com header.s=susede1 header.b=gR445mg0; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=suse.com Return-Path: Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id u23-20020aa7d997000000b004fd23c33363si18249689eds.186.2023.03.23.05.19.26; Thu, 23 Mar 2023 05:19:50 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@suse.com header.s=susede1 header.b=gR445mg0; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=suse.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231646AbjCWMSY (ORCPT + 99 others); Thu, 23 Mar 2023 08:18:24 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58244 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231428AbjCWMSI (ORCPT ); Thu, 23 Mar 2023 08:18:08 -0400 Received: from smtp-out1.suse.de (smtp-out1.suse.de [IPv6:2001:67c:2178:6::1c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2856C2BF19 for ; Thu, 23 Mar 2023 05:17:36 -0700 (PDT) Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by smtp-out1.suse.de (Postfix) with ESMTPS id 92BE3339C9; Thu, 23 Mar 2023 12:17:33 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.com; s=susede1; t=1679573853; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=Uu+N5jOhken1tfZs5kVQQIsrfUWmzwdgGp/YYNK2ddI=; b=gR445mg0TOBYGbWjZ1RG3k1s2MieH6HxpCRvqcr4BUjL+mUSWQ1cqVgaxefHLMtEZKy6lB BJlTrqxjFaJUJWNygldd5h4KoE6jq54y+25iWhMOLvPZw22iGS4/9UmdpzdI8+IR6bI8CV HvsNaeYcN1mwyIzCWKUG6wR1OLlci3g= Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by imap2.suse-dmz.suse.de (Postfix) with ESMTPS id 7368D132C2; Thu, 23 Mar 2023 12:17:33 +0000 (UTC) Received: from dovecot-director2.suse.de ([192.168.254.65]) by imap2.suse-dmz.suse.de with ESMTPSA id skRCGV1DHGQfLwAAMHmgww (envelope-from ); Thu, 23 Mar 2023 12:17:33 +0000 Date: Thu, 23 Mar 2023 13:17:32 +0100 From: Michal Hocko To: Marcelo Tosatti Cc: Christoph Lameter , Aaron Tomlin , Frederic Weisbecker , Andrew Morton , linux-kernel@vger.kernel.org, linux-mm@kvack.org, Russell King , Huacai Chen , Heiko Carstens , x86@kernel.org, Vlastimil Babka Subject: Re: [PATCH v7 00/13] fold per-CPU vmstats remotely Message-ID: References: <20230320180332.102837832@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Spam-Status: No, score=-2.5 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Thu 23-03-23 07:52:22, Marcelo Tosatti wrote: > On Thu, Mar 23, 2023 at 08:51:14AM +0100, Michal Hocko wrote: > > On Wed 22-03-23 11:20:55, Marcelo Tosatti wrote: > > > On Wed, Mar 22, 2023 at 02:35:20PM +0100, Michal Hocko wrote: > > [...] > > > > > "Performance details for the kworker interruption: > > > > > > > > > > oslat 1094.456862: sys_mlock(start: 7f7ed0000b60, len: 1000) > > > > > oslat 1094.456971: workqueue_queue_work: ... function=vmstat_update ... > > > > > oslat 1094.456974: sched_switch: prev_comm=oslat ... ==> next_comm=kworker/5:1 ... > > > > > kworker 1094.456978: sched_switch: prev_comm=kworker/5:1 ==> next_comm=oslat ... > > > > > > > > > > The example above shows an additional 7us for the > > > > > > > > > > oslat -> kworker -> oslat > > > > > > > > > > switches. In the case of a virtualized CPU, and the vmstat_update > > > > > interruption in the host (of a qemu-kvm vcpu), the latency penalty > > > > > observed in the guest is higher than 50us, violating the acceptable > > > > > latency threshold for certain applications." > > > > > > > > Yes, I have seen that but it doesn't really give a wider context to > > > > understand why those numbers matter. > > > > > > OK. > > > > > > "In the case of RAN, a MAC scheduler with TTI=1ms, this causes >100us > > > interruption observed in a guest (which is above the safety > > > threshold for this application)." > > > > > > Is that OK? > > > > This might be a sufficient information for somebody familiar with the > > matter (not me). So no, not enough. We need to hear a more complete > > story. > > Michal, > > Please refer to > https://www.diva-portal.org/smash/get/diva2:541460/FULLTEXT01.pdf > > 2.3 Channel Dependent Scheduling > The purpose of scheduling is to decide which terminal will transmit data on which set > of resource blocks with what transport format to use. The objective is to assign > resources to the terminal such that the quality of service (QoS) requirement is fulfilled. > Scheduling decision is taken every 1 ms by base station (termed as eNodeB) as the > same length of Transmission Time Interval (TTI) in LTE system. > > In general: > > https://en.wikipedia.org/wiki/Real-time_computing Thank you, but not something I was really asking for (repeatedly). I am pretty aware of what RT computing is about. I am not really interested in a generic fluff. I am asking about specific usecases you have in mind when pushing these changes. > For example, for the MAC scheduler processing must occur every 1ms, > and a certain amount of computation takes place (and must finish before > the next 1ms timeframe). A > 50us latency spike as observed by cyclictest > is considered a "failure". OK, you are claiming that much but you are not really filling up other holes in your story. Let me just outline few questions I have. Your measurements talk about 7us overhead the vmstat processing might add. This is really far from > 50us above. You suggest that this is an effect of the workload running in a guest without more details. I am quite surprised to hear about RT expectations inside a guest system TBH. All that being said, it would be really helpful if you were more specific about the workload and why there is no other way but making vmstat infrastructure more complex (it is quite complex on its own). Thanks! -- Michal Hocko SUSE Labs