Received: by 2002:a05:6358:bb9e:b0:b9:5105:a5b4 with SMTP id df30csp233733rwb; Fri, 2 Sep 2022 13:11:36 -0700 (PDT) X-Google-Smtp-Source: AA6agR5Qon2ebBnmdS/iE+EcJQcgsJgIwOjoKMRn2EkkYXwDAJAZ3ag/qHNaXOIkd73ntVra017v X-Received: by 2002:a17:90a:ac2:b0:1fd:fad1:e506 with SMTP id r2-20020a17090a0ac200b001fdfad1e506mr6662580pje.66.1662149495838; Fri, 02 Sep 2022 13:11:35 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1662149495; cv=none; d=google.com; s=arc-20160816; b=g/Tvt8Q134kOoGxmp5s0wUsUTxp1IMgCe2FN9TSr0oTHaUfojB5kxW8Gf2eswnNJ6M lxZjnAaMUgPZliFS28HALtQbzodVGhgPvrfjmmPuUDO8dRKbIFyKvNMk7y8QIkQFnPO3 PDfwpmg28zEPqdpCbxVNklRo+OvxubYOh+0RVQd2d0yawIXWBCw9GPUpV2Cu8JjDaVlf 8UQZCCjkRwXT3VEyTjxT+c15cmHo61NfhKObCCVgbZBDnU3ALJ3VR/w7qHvpSpB9PhqY SRcyDtYKqgsfzWK1nMCJMVTx3VzGhdGpmX90THLeB8DNJDFfnKjRTHV9WY8ND26f6qoe YLYw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:in-reply-to:from :references:cc:to:content-language:subject:user-agent:mime-version :date:message-id:dkim-signature; bh=BgSQRPeP2vU+fzfPkC5136xUIsrQV+zIPc0zX8WqYvQ=; b=wlyZ34R6Awrwq8lrOZ5oVb+Z94OKLsppefgmcTmai+atBmSU0p+Ji7VFGFHKxRMzfw EUMtEYGLSwoQfIDMH5cruN9aEUnqPNZc9H73TzmnsE9IUwtvCN/xoDKbY0t3JcvtRZyJ /nf/9oCISW3wTX9gqqvGtCKYo5aYhb48CldB1O4IhkeIrxcRxXLFPVS03VSXk0ziq2uP GYlpDWzkUQCmnS98H49/d/yDKBoNkeoCKptt921ilVVCqKEuGbo2xMdbcdhnymyYMMZm /0imPko9LKw0TWWFxuPQOmkfukX7BwWI23yDyBhDGyzHG41AO7dZu2yMvT4OLh4PCqMS I2lA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel-dk.20210112.gappssmtp.com header.s=20210112 header.b=4q1ObG8W; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id l12-20020a170903244c00b001752db2c753si3401536pls.24.2022.09.02.13.11.23; Fri, 02 Sep 2022 13:11:35 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel-dk.20210112.gappssmtp.com header.s=20210112 header.b=4q1ObG8W; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230199AbiIBTyG (ORCPT + 99 others); Fri, 2 Sep 2022 15:54:06 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55466 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229950AbiIBTyB (ORCPT ); Fri, 2 Sep 2022 15:54:01 -0400 Received: from mail-pj1-x102d.google.com (mail-pj1-x102d.google.com [IPv6:2607:f8b0:4864:20::102d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id CCC7CC2763 for ; Fri, 2 Sep 2022 12:53:58 -0700 (PDT) Received: by mail-pj1-x102d.google.com with SMTP id x1-20020a17090ab00100b001fda21bbc90so6463792pjq.3 for ; Fri, 02 Sep 2022 12:53:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20210112.gappssmtp.com; s=20210112; h=content-transfer-encoding:in-reply-to:from:references:cc:to :content-language:subject:user-agent:mime-version:date:message-id :from:to:cc:subject:date; bh=BgSQRPeP2vU+fzfPkC5136xUIsrQV+zIPc0zX8WqYvQ=; b=4q1ObG8WZYb+BFWPgZh7N7aITtmlNIGPzQtvKE2aYuRrU3oMZQ76tQz2zqxy31cTLt d8rIjpGYxzTxBhay1NPxlOw7vaD/UOVOLVL9kTH97EOmLP9jqiD95DNmZFsZ1Ptc8XmN SpoCT9nh9ECoUVn4HyZ7Qsn6ksQUfbOmmaLZNC8YNvNZJ0mYksYatinKwvYndzY1XvJm nky6nJQ50OCGfppKmlv+KompjFnK/xiaX17J7WZx+2ulxNlRcgnWzxkBWjasiiHjJ4Jl WgqrWN7XLOnHrvW9WLg3mCJ+/TAuDw9lCpZSCR3GLv/KhadF9BUUSKQnQ8/gjIX8jzxs EhUw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:in-reply-to:from:references:cc:to :content-language:subject:user-agent:mime-version:date:message-id :x-gm-message-state:from:to:cc:subject:date; bh=BgSQRPeP2vU+fzfPkC5136xUIsrQV+zIPc0zX8WqYvQ=; b=wgrEF+c3SMBKvktbupPLnLOpZU3oGE+BX9e3qQrVb8biUSPwZaTO6wP9lQuZgZp2lX 1Ptb6VZ5uBvV4r1tSdpoQURo+RDm0hWWQWE38DlU5+OKphnMtdg7Q+XEw77Q3oq4vH1W Ujy5ehPpcBFTqRbK21j5zndX7uoLa5u4OpqjF8T3x0OYhe3oTW4iOF+7hnptqCbwCfwI aPkuV0qsFiqeVlDqvXiQKtVZxOftIQ9i/fZPU6Ng4JkhVtM+O1DFmn6DkhYTT9v2ZrTt AElX+ldqN7EeEBx8jDRvqIXxlqyW4gL2iA4JBC653/B30pRY4bFhkFiDoJFvXyRiJSzl n4AA== X-Gm-Message-State: ACgBeo1CbMsPcoIvblKE94y8xyazZauIq26p47eP2Pa23biLZo2kf45A Yb4/h/iWOOUBGypSCXDCbnpL4g== X-Received: by 2002:a17:90b:b16:b0:1fd:b47c:6ab with SMTP id bf22-20020a17090b0b1600b001fdb47c06abmr6698292pjb.203.1662148438176; Fri, 02 Sep 2022 12:53:58 -0700 (PDT) Received: from [192.168.1.136] ([198.8.77.157]) by smtp.gmail.com with ESMTPSA id c6-20020a170902c1c600b00172ccb3f4ebsm2008369plc.160.2022.09.02.12.53.54 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Fri, 02 Sep 2022 12:53:57 -0700 (PDT) Message-ID: Date: Fri, 2 Sep 2022 13:53:53 -0600 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux aarch64; rv:102.0) Gecko/20100101 Thunderbird/102.1.2 Subject: Re: [RFC PATCH 00/30] Code tagging framework and applications Content-Language: en-US To: Kent Overstreet Cc: Roman Gushchin , Yosry Ahmed , Michal Hocko , Mel Gorman , Peter Zijlstra , Suren Baghdasaryan , Andrew Morton , Vlastimil Babka , Johannes Weiner , dave@stgolabs.net, Matthew Wilcox , liam.howlett@oracle.com, void@manifault.com, juri.lelli@redhat.com, ldufour@linux.ibm.com, Peter Xu , David Hildenbrand , mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, changbin.du@intel.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, Steven Rostedt , bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, Christoph Lameter , Pekka Enberg , Joonsoo Kim , 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, Shakeel Butt , Muchun Song , arnd@arndb.de, jbaron@akamai.com, David Rientjes , minchan@google.com, kaleshsingh@google.com, kernel-team@android.com, Linux-MM , iommu@lists.linux.dev, kasan-dev@googlegroups.com, io-uring@vger.kernel.org, linux-arch@vger.kernel.org, xen-devel@lists.xenproject.org, linux-bcache@vger.kernel.org, linux-modules@vger.kernel.org, Linux Kernel Mailing List References: <20220831101948.f3etturccmp5ovkl@suse.de> <20220831190154.qdlsxfamans3ya5j@moria.home.lan> <20220901223720.e4gudprscjtwltif@moria.home.lan> <20220902001747.qqsv2lzkuycffuqe@moria.home.lan> <3a41b9fc-05f1-3f56-ecd0-70b9a2912a31@kernel.dk> <20220902194839.xqzgsoowous72jkz@moria.home.lan> From: Jens Axboe In-Reply-To: <20220902194839.xqzgsoowous72jkz@moria.home.lan> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,NICE_REPLY_A,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS, T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 9/2/22 1:48 PM, Kent Overstreet wrote: > On Fri, Sep 02, 2022 at 06:02:12AM -0600, Jens Axboe wrote: >> On 9/1/22 7:04 PM, Roman Gushchin wrote: >>> On Thu, Sep 01, 2022 at 08:17:47PM -0400, Kent Overstreet wrote: >>>> On Thu, Sep 01, 2022 at 03:53:57PM -0700, Roman Gushchin wrote: >>>>> I'd suggest to run something like iperf on a fast hardware. And maybe some >>>>> io_uring stuff too. These are two places which were historically most sensitive >>>>> to the (kernel) memory accounting speed. >>>> >>>> I'm getting wildly inconsistent results with iperf. >>>> >>>> io_uring-echo-server and rust_echo_bench gets me: >>>> Benchmarking: 127.0.0.1:12345 >>>> 50 clients, running 512 bytes, 60 sec. >>>> >>>> Without alloc tagging: 120547 request/sec >>>> With: 116748 request/sec >>>> >>>> https://github.com/frevib/io_uring-echo-server >>>> https://github.com/haraldh/rust_echo_bench >>>> >>>> How's that look to you? Close enough? :) >>> >>> Yes, this looks good (a bit too good). >>> >>> I'm not that familiar with io_uring, Jens and Pavel should have a better idea >>> what and how to run (I know they've workarounded the kernel memory accounting >>> because of the performance in the past, this is why I suspect it might be an >>> issue here as well). >> >> io_uring isn't alloc+free intensive on a per request basis anymore, it >> would not be a good benchmark if the goal is to check for regressions in >> that area. > > Good to know. The benchmark is still a TCP benchmark though, so still useful. > > Matthew suggested > while true; do echo 1 >/tmp/foo; rm /tmp/foo; done > > I ran that on tmpfs, and the numbers with and without alloc tagging were > statistically equal - there was a fair amount of variation, it wasn't a super > controlled test, anywhere from 38-41 seconds with 100000 iterations (and alloc > tagging was some of the faster runs). > > But with memcg off, it ran in 32-33 seconds. We're piggybacking on the same > mechanism memcg uses for stashing per-object pointers, so it looks like that's > the bigger cost. I've complained about memcg accounting before, the slowness of it is why io_uring works around it by caching. Anything we account we try NOT do in the fast path because of it, the slowdown is considerable. You care about efficiency now? I thought that was relegated to irrelevant 10M IOPS cases. -- Jens Axboe