Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Wed, 2 Feb 2022 09:51:32 -0800
From:   Roman Gushchin <guro@fb.com>
To:     Michal Hocko <mhocko@suse.com>
CC:     Waiman Long <longman@redhat.com>,
        Johannes Weiner <hannes@cmpxchg.org>,
        Vladimir Davydov <vdavydov.dev@gmail.com>,
        Andrew Morton <akpm@linux-foundation.org>,
        Petr Mladek <pmladek@suse.com>,
        Steven Rostedt <rostedt@goodmis.org>,
        Sergey Senozhatsky <senozhatsky@chromium.org>,
        Andy Shevchenko <andriy.shevchenko@linux.intel.com>,
        Rasmus Villemoes <linux@rasmusvillemoes.dk>,
        <linux-kernel@vger.kernel.org>, <cgroups@vger.kernel.org>,
        <linux-mm@kvack.org>, Ira Weiny <ira.weiny@intel.com>,
        Rafael Aquini <aquini@redhat.com>
Subject: Re: [PATCH v2 3/3] mm/page_owner: Dump memcg information
Message-ID: <YfrEpOGObnc0mYAW@carbon.dhcp.thefacebook.com>
References: <YfeuK5j7cbgM+Oo+@dhcp22.suse.cz>
 <YfgT/9tEREQNiiAN@cmpxchg.org>
 <YfgnUZQBRkqhrEIb@carbon.dhcp.thefacebook.com>
 <Yfgpknwr1tMnPkqh@dhcp22.suse.cz>
 <12686956-612d-d89b-5641-470d5e913090@redhat.com>
 <YfkQJ4QhfY0dICB9@dhcp22.suse.cz>
 <268a8bdf-4c70-b967-f34c-2293b54186f0@redhat.com>
 <YfpHbtffFi2x1L4p@dhcp22.suse.cz>
 <YfqpSPLC+LAdqbJX@carbon.dhcp.thefacebook.com>
 <YfqzbwAPKpshXSLK@dhcp22.suse.cz>
Content-Type: text/plain; charset="us-ascii"
Content-Disposition: inline
In-Reply-To: <YfqzbwAPKpshXSLK@dhcp22.suse.cz>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?Y7qsojFVNHVwCD+69LDeNdR1D2VBVNXHaJV6uBZZPPX59a34XSGO44RERTGX?=
 =?us-ascii?Q?41Qd0SLhX868H5sEwLT6uEn+FmePw6tBVFdbNGP6hyEUfqeu5ZEIC5lnbN1F?=
 =?us-ascii?Q?ZkWsmpmjWFxsUsfcQYUWiXmmpRWU1f4BX5gIYq57BhxlWPpo6PFR0aGFcW1B?=
 =?us-ascii?Q?1RbjF/+IV6P1DGsKFQHIWSV5NsGXsldiGDM+oQG0X9v4WVdNmeafooQbSADT?=
 =?us-ascii?Q?R4Y7nDRDUY1+4Rzeskupebhmq0YAx2/KTBq4XOwA+vsHB0LR4rEE5jTPSgyi?=
 =?us-ascii?Q?Wp+x1uIDdfK7bZgwQgBm8WLDOq1lQy5lWPGyr+WZwy/MCMFV66PKeLqz5yce?=
 =?us-ascii?Q?DgpLstn0Nqg9aJhnYdLLUwDeR1Xz+WFgeonV/DfV/qhkrfTHj9md0L67jsJe?=
 =?us-ascii?Q?Hqw0InS4MNi+QN5N7FTiHUy4rMoCyu9FoshLx8ZQu5/GE4v/DGjR+DsljQ/A?=
 =?us-ascii?Q?6+pI8yPHj+NNu3Uw/hb8ExU/NlNH6AXKNQiTXBr6O8BmAYMOc+RhreEFBjX7?=
 =?us-ascii?Q?mKHp5tgliapOqegmrlr3ZlyKzgRpvl4COGysXlU4ru2UKSDrhWAB59f6Ka+x?=
 =?us-ascii?Q?5G1hSM5fhUsk03WJ+xU5PE3osQEpjWFT5COpyJJ1Y72xBSZ27fFtvlF0+YwY?=
 =?us-ascii?Q?v1PjUm5GmJbt2iVDyBwT5A0PnELlGg0nanpA/V7SZr6HKmN/4t6tNRZZuLHA?=
 =?us-ascii?Q?RsDR1A9OrVhJlVTmiRTK5sgp/ri0XUrhiDGd7ZK8turMWojShiFW1MNX2F2A?=
 =?us-ascii?Q?cYvpgNntcMk/ckJrdlWNMlT2LocFU++V5gV8YWc81sEygU2UGl1VsGa/SerO?=
 =?us-ascii?Q?YqDlkl4qJNxOnFS7IsQtBUYpCg9DrF/nQtCcPDek83051JkBl6dxsyhwGvSs?=
 =?us-ascii?Q?sHbycZP551H/hG6jg3hFUHeTaW+AqOXTAZgZ9tJdhtRYS5f1aKejKj5ICPpu?=
 =?us-ascii?Q?VBMEQu5ZKAUi4pfJTFcwWxt0GYVwUgwAfs0a/ljSD7TLF1G8OeL3qV/wuXr7?=
 =?us-ascii?Q?krP+dZXq8JQ+CJ6ifNqvLoI+vWRlbSY0TKMyfLmVdviOAlJPhL7UrEoTb74s?=
 =?us-ascii?Q?yDcfNuTluxZSFLs9OJneKz75TgBnXly9mPKoO452PIlh+2siHMc3t20emjk4?=
 =?us-ascii?Q?fx2z4Wpuzo3wEDB4rE3HMKTWRHj3+IuzUmkJrkgrTU4+ohrPb59eJgoESff8?=
 =?us-ascii?Q?TSEnrIqnxNbt2OZ9eY1A6yYPyhsaraz1+tpYO0YIFIj4L9OsAZlTcOzq9RUr?=
 =?us-ascii?Q?pByHYbyZtHX555DDhROmDzzqFarfy4MSMbuTSIez0jh2rLnm3xeVmyrlC+eN?=
 =?us-ascii?Q?u6ZcnJgZcg1hxnp575Z2CaWA1gnf3VfEUb0yw5wf4R4smEr+U9fGtaPq0Dn2?=
 =?us-ascii?Q?tEPfa0bvPJOlinlBInRDM2Bq16OPAvYRe3rGco2J2syiBwPaqvJ9y9V3RJ00?=
 =?us-ascii?Q?/fP5py6EhWI4zCcGhhOdzMC4r+Cs1MUWc2Bs4ZXFC7K+FaKb96GCJP7yO/AD?=
 =?us-ascii?Q?4QN26VMaW+aUHqYzelcp1Lz/8FRQDN5p1/mVzuGghmXyqoL91Sr06yqr8JR6?=
 =?us-ascii?Q?DqcaSS7eidBIaGtYfIxAja0USGLwS7zpYefIb3Ct?=
X-MS-Exchange-CrossTenant-Network-Message-Id: ca44078f-a967-4a1d-4575-08d9e674a83b
X-MS-Exchange-CrossTenant-AuthSource: BYAPR15MB4136.namprd15.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 02 Feb 2022 17:51:36.5655
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 8ae927fe-1255-47a7-a2af-5f3a069daaa2
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: 3npXZtvbS1C18sqefPk/VadDMnzZdxHlZzwQO44dGZhMGWabzVpUQ0EJfrpD1f4z
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ0PR15MB4600
X-OriginatorOrg: fb.com
X-Proofpoint-GUID: OqWeYJJXvvBujFEcsUpfg34JSu5dvZ9Q
X-Proofpoint-ORIG-GUID: OqWeYJJXvvBujFEcsUpfg34JSu5dvZ9Q
X-Proofpoint-Virus-Version: vendor=baseguard
 engine=ICAP:2.0.205,Aquarius:18.0.816,Hydra:6.0.425,FMLib:17.11.62.513
 definitions=2022-02-02_08,2022-02-01_01,2021-12-02_01
X-Proofpoint-Spam-Details: rule=fb_outbound_notspam policy=fb_outbound score=0 clxscore=1015
 priorityscore=1501 bulkscore=0 phishscore=0 spamscore=0 malwarescore=0
 mlxlogscore=487 suspectscore=0 adultscore=0 lowpriorityscore=0 mlxscore=0
 impostorscore=0 classifier=spam adjust=0 reason=mlx scancount=1
 engine=8.12.0-2201110000 definitions=main-2202020100
X-FB-Internal: deliver
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On Wed, Feb 02, 2022 at 05:38:07PM +0100, Michal Hocko wrote:
> On Wed 02-02-22 07:54:48, Roman Gushchin wrote:
> > On Wed, Feb 02, 2022 at 09:57:18AM +0100, Michal Hocko wrote:
> > > On Tue 01-02-22 11:41:19, Waiman Long wrote:
> > > > 
> > > > On 2/1/22 05:49, Michal Hocko wrote:
> > > [...]
> > > > > Could you be more specific? Offlined memcgs are still part of the
> > > > > hierarchy IIRC. So it shouldn't be much more than iterating the whole
> > > > > cgroup tree and collect interesting data about dead cgroups.
> > > > 
> > > > What I mean is that without piggybacking on top of page_owner, we will to
> > > > add a lot more code to collect and display those information which may have
> > > > some overhead of its own.
> > > 
> > > Yes, there is nothing like a free lunch. Page owner is certainly a tool
> > > that can be used. My main concern is that this tool doesn't really
> > > scale on large machines with a lots of memory. It will provide a very
> > > detailed information but I am not sure this is particularly helpful to
> > > most admins (why should people process tons of allocation backtraces in
> > > the first place). Wouldn't it be sufficient to have per dead memcg stats
> > > to see where the memory sits?
> > > 
> > > Accumulated offline memcgs is something that bothers more people and I
> > > am really wondering whether we can do more for those people to evaluate
> > > the current state.
> > 
> > Cgroup v2 has corresponding counters for years. Or do you mean something different?
> 
> Do we have anything more specific than nr_dying_descendants?

No, just nr_dying_descendants.

> I was thinking about an interface which would provide paths and stats for dead
> memcgs. But I have to confess I haven't really spent much time thinking
> about how much work that would be. I am by no means against adding memcg
> information to the page owner. I just think there must be a better way
> to present resource consumption by dead memcgs.

I'd go with a drgn script. I wrote a bunch of them some times ago and
can probably revive them and post here (will take few days).

I agree that the problem still exists and providing some tool around would be
useful.

Thanks!