Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67;
From:   Roman Gushchin <guro@fb.com>
To:     Shakeel Butt <shakeelb@google.com>
CC:     Johannes Weiner <hannes@cmpxchg.org>,
        Andrew Morton <akpm@linux-foundation.org>,
        Linux MM <linux-mm@kvack.org>,
        Cgroups <cgroups@vger.kernel.org>,
        LKML <linux-kernel@vger.kernel.org>,
        Kernel Team <Kernel-team@fb.com>
Subject: Re: [PATCH 3/4] mm: memcontrol: fix recursive statistics correctness
 & scalabilty
Thread-Topic: [PATCH 3/4] mm: memcontrol: fix recursive statistics correctness
 & scalabilty
Thread-Index: AQHU8UKUV0F4p+EmokWSlSobKT/LA6Y48L8AgAAFtYA=
Date:   Fri, 12 Apr 2019 20:15:38 +0000
Message-ID: <20190412201534.GB24377@tower.DHCP.thefacebook.com>
References: <20190412151507.2769-1-hannes@cmpxchg.org>
 <20190412151507.2769-4-hannes@cmpxchg.org>
 <CALvZod4xu10+E41YyaamigysZAnDcdA09f5m-hGd72LeJ9VmEg@mail.gmail.com>
In-Reply-To: <CALvZod4xu10+E41YyaamigysZAnDcdA09f5m-hGd72LeJ9VmEg@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
x-ms-exchange-messagesentrepresentingtype: 1
x-originating-ip: [2620:10d:c090:200::1:2586]
x-ms-publictraffictype: Email
x-ms-office365-filtering-correlation-id: 125b0329-fb27-4491-1eea-08d6bf83a151
x-microsoft-antispam: BCL:0;PCL:0;RULEID:(2390118)(7020095)(4652040)(8989299)(4534185)(4627221)(201703031133081)(201702281549075)(8990200)(5600139)(711020)(4605104)(2017052603328)(7193020);SRVR:BYAPR15MB3159;
x-ms-traffictypediagnostic: BYAPR15MB3159:
x-microsoft-antispam-prvs: <BYAPR15MB31591CD0F47CAABABE6FB7B5BE280@BYAPR15MB3159.namprd15.prod.outlook.com>
x-forefront-prvs: 0005B05917
x-forefront-antispam-report: SFV:NSPM;SFS:(10019020)(396003)(366004)(136003)(376002)(346002)(39860400002)(189003)(199004)(52116002)(81166006)(71200400001)(105586002)(71190400001)(97736004)(68736007)(478600001)(8936002)(14454004)(7736002)(81156014)(6116002)(33656002)(106356001)(8676002)(25786009)(305945005)(486006)(54906003)(5660300002)(1076003)(476003)(46003)(316002)(4326008)(99286004)(53936002)(446003)(76176011)(86362001)(6246003)(256004)(6436002)(2906002)(9686003)(11346002)(6512007)(6486002)(102836004)(53546011)(186003)(6916009)(386003)(6506007)(229853002);DIR:OUT;SFP:1102;SCL:1;SRVR:BYAPR15MB3159;H:BYAPR15MB2631.namprd15.prod.outlook.com;FPR:;SPF:None;LANG:en;PTR:InfoNoRecords;MX:1;A:1;
received-spf: None (protection.outlook.com: fb.com does not designate
 permitted sender hosts)
x-ms-exchange-senderadcheck: 1
x-microsoft-antispam-message-info: fbf353SeamkSFWKGkIHYPCnsqViX/Opvi0LBCLohv31qIeMltFxLdC5xKM38uK7Wn4v1HoHTriqBB03H4tAFlfK1638xscrkbDAHfW/7lHldgQIX6jQqpRUOUO2RICOg4tkzmU4CjWHDVjon3UTzcTkXSDXv0NevoZRYm90reu5H6Ne6fXW0fHT3iEg2XVyK+ep0le2ckrTkeXwxoZDnVr+J96QToBog0G37fbrHimm/ntbBxQ6/eFo3dxvDd9MALZ981gjR9j/uHxmwIJOnFnEt+qtZ+SnL9dJmwcR6Mti+6AO3UEio7sI8tPccq4ygBHQLKPkl0vreO4FYsdTfVWHnFPNF/NlDURvNoVI+63OsEGlVmBj4zGqB2hOr6c6Z00lM39D2RkXdY7DEqxBHnEGmVap3WQju9iJCZxIkuMU=
Content-Type: text/plain; charset="us-ascii"
Content-ID: <83C4D58C79AA094F9D9733F4C0F7C635@namprd15.prod.outlook.com>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-MS-Exchange-CrossTenant-Network-Message-Id: 125b0329-fb27-4491-1eea-08d6bf83a151
X-MS-Exchange-CrossTenant-originalarrivaltime: 12 Apr 2019 20:15:38.9934
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 8ae927fe-1255-47a7-a2af-5f3a069daaa2
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BYAPR15MB3159
X-OriginatorOrg: fb.com
X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:,, definitions=2019-04-12_11:,,
 signatures=0
X-Proofpoint-Spam-Reason: safe
X-FB-Internal: Safe
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On Fri, Apr 12, 2019 at 12:55:10PM -0700, Shakeel Butt wrote:
> On Fri, Apr 12, 2019 at 8:15 AM Johannes Weiner <hannes@cmpxchg.org> wrot=
e:
> >
> > Right now, when somebody needs to know the recursive memory statistics
> > and events of a cgroup subtree, they need to walk the entire subtree
> > and sum up the counters manually.
> >
> > There are two issues with this:
> >
> > 1. When a cgroup gets deleted, its stats are lost. The state counters
> > should all be 0 at that point, of course, but the events are not. When
> > this happens, the event counters, which are supposed to be monotonic,
> > can go backwards in the parent cgroups.
> >
>=20
> We also faced this exact same issue as well and had the similar solution.
>=20
> > 2. During regular operation, we always have a certain number of lazily
> > freed cgroups sitting around that have been deleted, have no tasks,
> > but have a few cache pages remaining. These groups' statistics do not
> > change until we eventually hit memory pressure, but somebody watching,
> > say, memory.stat on an ancestor has to iterate those every time.
> >
> > This patch addresses both issues by introducing recursive counters at
> > each level that are propagated from the write side when stats change.
> >
> > Upward propagation happens when the per-cpu caches spill over into the
> > local atomic counter. This is the same thing we do during charge and
> > uncharge, except that the latter uses atomic RMWs, which are more
> > expensive; stat changes happen at around the same rate. In a sparse
> > file test (page faults and reclaim at maximum CPU speed) with 5 cgroup
> > nesting levels, perf shows __mod_memcg_page state at ~1%.
> >
>=20
> (Unrelated to this patchset) I think there should also a way to get
> the exact memcg stats. As the machines are getting bigger (more cpus
> and larger basic page size) the accuracy of stats are getting worse.
> Internally we have an additional interface memory.stat_exact for that.
> However I am not sure in the upstream kernel will an additional
> interface is better or something like /proc/sys/vm/stat_refresh which
> sync all per-cpu stats.

I was thinking about eventually consistent counters: sync them periodically
from a worker thread. It should keep the cost of reading small, but
should increase the accuracy. Will it work for you?