Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67;
From:   Roman Gushchin <guro@fb.com>
To:     Michal Hocko <mhocko@kernel.org>
CC:     "linux-mm@kvack.org" <linux-mm@kvack.org>,
        Andrew Morton <akpm@linux-foundation.org>,
        Johannes Weiner <hannes@cmpxchg.org>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        Kernel Team <Kernel-team@fb.com>,
        "stable@vger.kernel.org" <stable@vger.kernel.org>,
        Tejun Heo <tj@kernel.org>
Subject: Re: [PATCH 1/2] mm: memcg: switch to css_tryget() in
 get_mem_cgroup_from_mm()
Thread-Topic: [PATCH 1/2] mm: memcg: switch to css_tryget() in
 get_mem_cgroup_from_mm()
Thread-Index: AQHVlPTSqR6tDyqEq0WgIBYYtGCX/Kd/odWAgABI+QCAAAVsAIAAXr+A
Date:   Thu, 7 Nov 2019 22:41:13 +0000
Message-ID: <20191107224107.GA8219@castle.DHCP.thefacebook.com>
References: <20191106225131.3543616-1-guro@fb.com>
 <20191107122125.GS8314@dhcp22.suse.cz>
 <20191107164236.GB2919@castle.dhcp.thefacebook.com>
 <20191107170200.GX8314@dhcp22.suse.cz>
In-Reply-To: <20191107170200.GX8314@dhcp22.suse.cz>
Accept-Language: en-US
Content-Language: en-US
x-ms-exchange-messagesentrepresentingtype: 1
x-originating-ip: [2620:10d:c090:180::5002]
x-ms-publictraffictype: Email
x-ms-office365-filtering-correlation-id: 35b97961-fba7-4796-44bb-08d763d397ce
x-ms-traffictypediagnostic: BN8PR15MB2756:
x-ms-exchange-transport-forked: True
x-microsoft-antispam-prvs: <BN8PR15MB275652D8D3886335414E7A48BE780@BN8PR15MB2756.namprd15.prod.outlook.com>
x-ms-oob-tlc-oobclassifiers: OLM:8273;
x-forefront-prvs: 0214EB3F68
x-forefront-antispam-report: SFV:NSPM;SFS:(10019020)(346002)(136003)(376002)(39860400002)(396003)(366004)(189003)(199004)(305945005)(186003)(81166006)(99286004)(7736002)(476003)(486006)(6486002)(46003)(66476007)(64756008)(33656002)(81156014)(1076003)(71190400001)(66446008)(66556008)(66946007)(446003)(8676002)(478600001)(11346002)(86362001)(71200400001)(256004)(4326008)(76176011)(6116002)(54906003)(386003)(6506007)(8936002)(102836004)(316002)(9686003)(2906002)(5660300002)(52116002)(6436002)(14454004)(6246003)(25786009)(6916009)(229853002)(6512007);DIR:OUT;SFP:1102;SCL:1;SRVR:BN8PR15MB2756;H:BN8PR15MB2626.namprd15.prod.outlook.com;FPR:;SPF:None;LANG:en;PTR:InfoNoRecords;MX:1;A:1;
received-spf: None (protection.outlook.com: fb.com does not designate
 permitted sender hosts)
x-ms-exchange-senderadcheck: 1
x-microsoft-antispam: BCL:0;
x-microsoft-antispam-message-info: d+TLgiSS5K47WcO8oPPSQv7g2py5j51tTANM+k6eJ8DdTIKeNdSOjpMzGi+yrA9GxazHnvsEPnQ1OypAz4+/ue4f2SjYi57IyHVj2FmGTv4W6mkyMbYwTSSt+InixW1u/2T9AZO4bHWzAAP3GZfGgTdQEzSFmixhbsObRrVDSaMjixNST2dvXg35y0sSXiVMZWbRVvFlO2+WIYCjYzyZSSI0cBld3i7RoGeOD3iMFhA7r2Yl/lHLw863q+KwnFErZWjwMNUkAc6PmoZ05OVrNwGycN3c8L2A0LOqT7JKDzKs5wURjMnyfGwodwec85Jr5vnQZHeKtf9mg0KzoGIRjsPtJrWS0tsWWZxdvfz9l9setDZ+Jji1S8Gbhs4wl0VQA/WzGlTEsEape6JDTbzgUObewn3f5kiDv+IKkA0DZ956qTR+pk/Eyqc3CfP1aEp/
Content-Type: text/plain; charset="us-ascii"
Content-ID: <37FC4BAB3B6F8C458F00D810EB53DFDE@namprd15.prod.outlook.com>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-MS-Exchange-CrossTenant-Network-Message-Id: 35b97961-fba7-4796-44bb-08d763d397ce
X-MS-Exchange-CrossTenant-originalarrivaltime: 07 Nov 2019 22:41:13.6569
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 8ae927fe-1255-47a7-a2af-5f3a069daaa2
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: 1DUzY16LztemHiJcKkepJi3Z6uWiVFexKPbxPR6ZCOJSS/ERrOjeN+xS3SywVeA+
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN8PR15MB2756
X-OriginatorOrg: fb.com
X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.95,18.0.572
 definitions=2019-11-07_07:2019-11-07,2019-11-07 signatures=0
X-Proofpoint-Spam-Details: rule=fb_default_notspam policy=fb_default score=0 spamscore=0 mlxscore=0
 clxscore=1015 bulkscore=0 priorityscore=1501 malwarescore=0 suspectscore=0
 phishscore=0 lowpriorityscore=0 mlxlogscore=904 impostorscore=0
 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1
 engine=8.12.0-1910280000 definitions=main-1911070207
X-FB-Internal: deliver
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On Thu, Nov 07, 2019 at 06:02:00PM +0100, Michal Hocko wrote:
> On Thu 07-11-19 16:42:41, Roman Gushchin wrote:
> > On Thu, Nov 07, 2019 at 01:21:25PM +0100, Michal Hocko wrote:
> > > On Wed 06-11-19 14:51:30, Roman Gushchin wrote:
> > > > We've encountered a rcu stall in get_mem_cgroup_from_mm():
> > > >=20
> > > >  rcu: INFO: rcu_sched self-detected stall on CPU
> > > >  rcu: 33-....: (21000 ticks this GP) idle=3D6c6/1/0x400000000000000=
2 softirq=3D35441/35441 fqs=3D5017
> > > >  (t=3D21031 jiffies g=3D324821 q=3D95837) NMI backtrace for cpu 33
> > > >  <...>
> > > >  RIP: 0010:get_mem_cgroup_from_mm+0x2f/0x90
> > > >  <...>
> > > >  __memcg_kmem_charge+0x55/0x140
> > > >  __alloc_pages_nodemask+0x267/0x320
> > > >  pipe_write+0x1ad/0x400
> > > >  new_sync_write+0x127/0x1c0
> > > >  __kernel_write+0x4f/0xf0
> > > >  dump_emit+0x91/0xc0
> > > >  writenote+0xa0/0xc0
> > > >  elf_core_dump+0x11af/0x1430
> > > >  do_coredump+0xc65/0xee0
> > > >  ? unix_stream_sendmsg+0x37d/0x3b0
> > > >  get_signal+0x132/0x7c0
> > > >  do_signal+0x36/0x640
> > > >  ? recalc_sigpending+0x17/0x50
> > > >  exit_to_usermode_loop+0x61/0xd0
> > > >  do_syscall_64+0xd4/0x100
> > > >  entry_SYSCALL_64_after_hwframe+0x44/0xa9
> > > >=20
> > > > The problem is caused by an exiting task which is associated with
> > > > an offline memcg.
> > >=20
> > > Hmm, how can we have a task in an offline memcg? I thought that any
> > > existing task will prevent cgroup removal from proceeding. Is this so=
me
> > > sort of race where the task managed to disassociate from the cgroup
> > > while there is still a binding to a memcg existing? What am I missing=
?
> >=20
> > It's an exiting task with the PF_EXITING flag set and it's in their lat=
e stages
> > of life.
>=20
> This is a signal delivery path AFAIU (get_signal) and the coredumping
> happens before do_exit. My understanding is that that unlinking
> happens from cgroup_exit. So either I am misreading the backtrace or
> there is some other way to leave cgroups or there is something more
> going on.

Yeah, you're right. I have no better explanation for this and the similar,
mentioned in the commit bsd accounting issue, than some very rare race cond=
ition
that allows cgroups to be offlined with a task inside.

I'll think more about it.

Thanks, it's a really good question.

>=20
> JFTR I am not really disputing the patch but I simply do not understand
> how the problem really happened.
>=20
> --=20
> Michal Hocko
> SUSE Labs