Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
Date:   Tue, 7 Sep 2021 10:11:21 -0700
From:   Roman Gushchin <guro@fb.com>
To:     Jens Axboe <axboe@kernel.dk>
CC:     Linus Torvalds <torvalds@linux-foundation.org>,
        kernel test robot <oliver.sang@intel.com>,
        Vasily Averin <vvs@virtuozzo.com>,
        Shakeel Butt <shakeelb@google.com>,
        Alexander Viro <viro@zeniv.linux.org.uk>,
        Alexey Dobriyan <adobriyan@gmail.com>,
        Andrei Vagin <avagin@gmail.com>,
        Borislav Petkov <bp@alien8.de>, Borislav Petkov <bp@suse.de>,
        Christian Brauner <christian.brauner@ubuntu.com>,
        Dmitry Safonov <0x7f454c46@gmail.com>,
        "Eric W. Biederman" <ebiederm@xmission.com>,
        Greg Kroah-Hartman <gregkh@linuxfoundation.org>,
        "H. Peter Anvin" <hpa@zytor.com>, Ingo Molnar <mingo@redhat.com>,
        "J. Bruce Fields" <bfields@fieldses.org>,
        Jeff Layton <jlayton@kernel.org>,
        Jiri Slaby <jirislaby@kernel.org>,
        Johannes Weiner <hannes@cmpxchg.org>,
        Kirill Tkhai <ktkhai@virtuozzo.com>,
        Michal Hocko <mhocko@kernel.org>,
        Oleg Nesterov <oleg@redhat.com>,
        Serge Hallyn <serge@hallyn.com>, Tejun Heo <tj@kernel.org>,
        Thomas Gleixner <tglx@linutronix.de>,
        Vladimir Davydov <vdavydov.dev@gmail.com>,
        Yutian Yang <nglaive@gmail.com>,
        Zefan Li <lizefan.x@bytedance.com>,
        Andrew Morton <akpm@linux-foundation.org>,
        LKML <linux-kernel@vger.kernel.org>, <lkp@lists.01.org>,
        kernel test robot <lkp@intel.com>,
        "Huang, Ying" <ying.huang@intel.com>,
        Feng Tang <feng.tang@intel.com>,
        Zhengjun Xing <zhengjun.xing@linux.intel.com>
Subject: Re: [memcg] 0f12156dff: will-it-scale.per_process_ops -33.6%
 regression
Message-ID: <YTedOVCV3s6Z210f@carbon.dhcp.thefacebook.com>
References: <20210907150757.GE17617@xsang-OptiPlex-9020>
 <dbc9955d-6c28-1dd5-b842-ef39a762aa3b@kernel.dk>
 <CAHk-=wgrF65BpP6P6_a6+k+C8V+JOBiEx0X-c9idiM87uk32FQ@mail.gmail.com>
 <774dee70-69bd-9f95-d197-4cff83e4e633@kernel.dk>
Content-Type: text/plain; charset="us-ascii"
Content-Disposition: inline
In-Reply-To: <774dee70-69bd-9f95-d197-4cff83e4e633@kernel.dk>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?fk453BclGFspxLN+TuUZlsbMvn/vfur66XoA1gsb9kRM374Z1HcounpzyDOl?=
 =?us-ascii?Q?uSVRsCtyo6QBdoNC79EI0rVdDgzlsYqRBk8kdQstkbSOzxyJSQUTjufiAGvP?=
 =?us-ascii?Q?rE7WscAzux5RdLuRoTt9hLn/h3GisppD4epobLRmucvrM5qlQOOhDUnkLOxv?=
 =?us-ascii?Q?ovq2onxmKf6tubqgg07/0ut64oqwseeWCUdI0jaM4Szj2i4LQvwOvuOhLpPT?=
 =?us-ascii?Q?qK0pMXkbalSe62gebOO79hbRb6UWRnqOAxSxRBCAAd7xEYEBpknRJo/pSqht?=
 =?us-ascii?Q?/pjyjt6cM2/6wFHhJezPv2eobcPUXIvQggvsK25xgW1FnlG/J7XC6FmAhm2j?=
 =?us-ascii?Q?R9mf/VRKvddU9ZCmwu/VMhHKFPU9WpqIveBechHZe2yCZsDnTfKznHt64HJI?=
 =?us-ascii?Q?sNitfzh6ptrrlOvrzzTL4bTwAdD/cIcN4K6xCbsjn9240n9KkF0BgnLP15+m?=
 =?us-ascii?Q?fde06SB2awnWJIaUdduMDZndxtNQD5GY7bFxrma5KQP4YxXQ6SDouyPpOBJC?=
 =?us-ascii?Q?wfzcY4RgIBf35ul+duu0XUFMZ8BiXA8u6UbdbB90Y6vK7oG5frQWvRZCvuB/?=
 =?us-ascii?Q?ZD8qVHhnfEF3UxX2Gbweoy0EQrgOP1AskKSGEKzTlK/rhRiFregKmuLCGYlw?=
 =?us-ascii?Q?byaEeE63lmssqODQgx+FauQsbc5OdQiqiHzj//ct6mmPITU1y5yVEuvoU9MT?=
 =?us-ascii?Q?lROSP6QG7KoOzZRyNLbbRk2knNU+cQq2GNiHNEQ9x44nATIr5XpVkVjnL4GE?=
 =?us-ascii?Q?G2yCzS/Gu1fn5pYamoVM/DwYkEOFxlhEK0e2MKjtkYX5qzqH1tm8vucyodD2?=
 =?us-ascii?Q?Jbi7e/cWBgr9XtNHlYXDkuMS1AGsE81CxB67kETjocil2RDfvhjg3mIzNIfU?=
 =?us-ascii?Q?P2o6ogDRxc0y0sXF/x4TGNsn06En/5AJBCO0G4bQpqU2xeMSzkehGVj6NZhw?=
 =?us-ascii?Q?XtW4C86Tc3Ct53MMHRJKAjMgWN/X7T8HtWADmfEk1kx0fGcDOdqsHclqwZAC?=
 =?us-ascii?Q?kVu4g/TPTj0+dhvNGA0kC4NOGZoyQtCrWSi3RMy5hP13fX8QH1VLuQDgPS2b?=
 =?us-ascii?Q?fneNi3jR8B+I3tMh/TRF5oPk70Y8tN3H5cXvQawc8mOLMtd7UMWHMvpR93Rg?=
 =?us-ascii?Q?6V6hqdWmjdiTpkz4TEcf5sYyIqXPk5xKDM02pZZpCNHOqMRh1tJ4CMW8o1Ip?=
 =?us-ascii?Q?8zweLs0a8ZMSRwzHKiTwlSrdeh3TqVsbZpOoUqu7bIRy79BTwazBPTirtwKS?=
 =?us-ascii?Q?H0rSad0cU7PwSirw6gf+q9zEwMgxB/rFXyKJV8tyDhiAB0MXLgrjwfNMw4Ct?=
 =?us-ascii?Q?RrvGFwlVcd2iF85TNiQ/P2ZAu/2upJVPbMLA1nK9Df4C2A=3D=3D?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 158c725d-8824-466a-cb0e-08d97222857c
X-MS-Exchange-CrossTenant-AuthSource: BYAPR15MB4136.namprd15.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 07 Sep 2021 17:11:24.5977
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 8ae927fe-1255-47a7-a2af-5f3a069daaa2
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: ap0shElmae6gR3Lo2xa5TYSSe2X/wnSK4G+lgO+bJjQQ1397ix30x9iCxIZcJAZf
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BYAPR15MB3448
X-OriginatorOrg: fb.com
X-Proofpoint-GUID: jlyPk5568zMFQXJlTjOT9VAiiZ9b0k5N
X-Proofpoint-ORIG-GUID: jlyPk5568zMFQXJlTjOT9VAiiZ9b0k5N
X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.391,18.0.790
 definitions=2021-09-07_06:2021-09-07,2021-09-07 signatures=0
X-Proofpoint-Spam-Details: rule=fb_default_notspam policy=fb_default score=0 impostorscore=0
 bulkscore=0 spamscore=0 mlxlogscore=999 lowpriorityscore=0 mlxscore=0
 malwarescore=0 clxscore=1011 phishscore=0 adultscore=0 priorityscore=1501
 suspectscore=0 classifier=spam adjust=0 reason=mlx scancount=1
 engine=8.12.0-2108310000 definitions=main-2109070113
X-FB-Internal: deliver
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On Tue, Sep 07, 2021 at 10:43:46AM -0600, Jens Axboe wrote:
> On 9/7/21 10:39 AM, Linus Torvalds wrote:
> > On Tue, Sep 7, 2021 at 8:46 AM Jens Axboe <axboe@kernel.dk> wrote:
> >>
> >> Are we at all worried about these? There's been a number of them
> >> reported, basically for all the accounting enablements that have been
> >> done in this merge window.
> > 
> > We are worried about them. I'm considering reverting several of them
> > because I think the problems are
> > 
> >  (a) big
> > 
> >  (b) nontrivial
> > 
> > and the patches clearly weren't ready and people weren't aware of this issue.
> 
> I think that is prudent. When I first enabled it for io_uring it was a
> bit of a shit show in terms of performance degradations, and some work
> had to be done before it could get enabled in a sane fashion.
> 
> The accounting needs to be more efficient if we're seeing 30-50%
> slowdowns simply by enabling it on a kmem cache.

There are two polar cases:
1) a big number of relatively short-living allocations, which lifetime is well
   bounded (e.g. by a lifetime of a task),
2) a relatively small number of long-living allocations, which lifetime
   is potentially indefinite (e.g. struct mount).

We can't use the same approach for both cases, otherwise we'll run into either
performance or garbage collection problems (which also lead to performance
problems, but delayed).

I think of maybe building a generic cache layer for accounted allocations,
which can be used in cases like io_uring. Shakeel, what's your plan here?

As now, I agree that reverting patches causing a significant regression is best
way forward.