Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
Date:   Mon, 6 Jul 2020 14:38:33 -0700
From:   Roman Gushchin <guro@fb.com>
To:     Shakeel Butt <shakeelb@google.com>
CC:     Michal Hocko <mhocko@kernel.org>,
        Johannes Weiner <hannes@cmpxchg.org>,
        Yang Shi <yang.shi@linux.alibaba.com>,
        David Rientjes <rientjes@google.com>,
        Greg Thelen <gthelen@google.com>,
        Andrew Morton <akpm@linux-foundation.org>,
        Linux MM <linux-mm@kvack.org>,
        LKML <linux-kernel@vger.kernel.org>,
        Cgroups <cgroups@vger.kernel.org>
Subject: Re: [RFC PROPOSAL] memcg: per-memcg user space reclaim interface
Message-ID: <20200706213404.GA152560@carbon.lan>
References: <20200702152222.2630760-1-shakeelb@google.com>
 <20200703063548.GM18446@dhcp22.suse.cz>
 <CALvZod5gthVX5m6o50OiYsXa=0_NpXK-tVvjTF42Oj4udr4Nuw@mail.gmail.com>
 <20200703155021.GB114903@carbon.dhcp.thefacebook.com>
 <CALvZod5Z4=1CijJp0QRnx+pdH=Me6sYPXASCxVATnshU0RW-Qw@mail.gmail.com>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <CALvZod5Z4=1CijJp0QRnx+pdH=Me6sYPXASCxVATnshU0RW-Qw@mail.gmail.com>
MIME-Version: 1.0
X-MS-Exchange-MessageSentRepresentingType: 1
Received: from carbon.lan (2620:10d:c090:400::5:f033) by BY5PR20CA0005.namprd20.prod.outlook.com (2603:10b6:a03:1f4::18) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3153.21 via Frontend Transport; Mon, 6 Jul 2020 21:38:35 +0000
X-Originating-IP: [2620:10d:c090:400::5:f033]
X-MS-PublicTrafficType: Email
X-MS-Office365-Filtering-Correlation-Id: 369de30e-9f98-4c31-ef9b-08d821f4f039
X-MS-TrafficTypeDiagnostic: BYAPR15MB4136:
X-Microsoft-Antispam-PRVS: <BYAPR15MB4136AA82D396A1BA8C08FAD3BE690@BYAPR15MB4136.namprd15.prod.outlook.com>
X-FB-Source: Internal
X-MS-Oob-TLC-OOBClassifiers: OLM:7691;
X-Forefront-PRVS: 04569283F9
X-MS-Exchange-SenderADCheck: 1
X-Microsoft-Antispam: BCL:0;
X-Microsoft-Antispam-Message-Info: 5ESyWGh4Wol/vg2Yf178zbrkzCxnOFPNIgl94Z7mWRXeudRqYJxtLA4kjCIF+ZwX8zX8qx+SOOMR+LpoSwNGKLQe12OByOX//3qAVnko278IQtKUIOdgvGfQSft/JoP+z9nH54idVh4YRMp0jZChnWMdLI/Nq2/jc/OA8PUA4DffsjHp0zrCy5ZZHrdLL5A6Am+qXcdGTcgqlkVucdJSmCgtAoUmPpRAGjn5S5dFXvbLUZ2rzaaa9iElG4JqlH/zCQ+4Ig6HV6d/d9gkJlI+kjYWS3F8Ol5B2mDW4++eC5YXLIqeH82SIdprvjqYOBq6GhzB0mocDX3nudAWciLDnu1c7sz5SM0mSl71rfvhTe7Anp2nhul0uXzGIuQQ6A/ScHh+WsY0SEdpk9PC0IcOC9PO8xVOvIgY2ICtGaULOjy31t0CAs3M3WxBaWJE3dhmxOMcuhKwu8eSvLsGQXdV8UeQNkV2J6I68/OetT+mE/3TpjUOVdlqe6jKnWk3zmsj/59a1exXEigJsDTQ98bz8Ddm6aWL8xejPCJvb2FP1tW8tLEfxDMGwk6JlYD4RDqE/Mth4qlrWnCe7SSo34ue693eDwV31YQAuJabgnCDcJbPAKg/1bNjaFuZnglvTud3aTbm1TALplcMyiAX1zGo1FB2n72d+ivfHoedlrXZV6jUNYVOk0oOP0ebLJF7U7rKfQvqHhrnc7xq7Gge669CmQ==
X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:5;SRV:;IPV:NLI;SFV:SPM;H:BYAPR15MB4136.namprd15.prod.outlook.com;PTR:;CAT:OSPM;SFTY:;SFS:(39860400002)(396003)(136003)(346002)(366004)(376002)(186003)(5660300002)(8886007)(33656002)(36756003)(1076003)(316002)(83380400001)(478600001)(4326008)(16526019)(7416002)(8936002)(66476007)(66556008)(66946007)(86362001)(54906003)(8676002)(55016002)(7696005)(6916009)(9686003)(6506007)(53546011)(52116002)(2906002)(27376004)(200143005);DIR:OUT;SFP:1501;
X-MS-Exchange-AntiSpam-MessageData: wbCyazRi1Ge958U5+BnprjG44a8+2OfMZoMyi/TxvcqStQ/hAVjAkQLK8n7AQRiCTc7B/hw/XNlkUt2dAJHmPHvkWHpmzY4rypWy7nVO5wNA2hQhYQz75GOwSDIz8M5QL3VdSvTmvnKofykKjq+VOIiGY95Ng76oXPEuvi4CFu60NSRQt18PSSYybj5F18sihBU1WYOx+AZxkfjzOKPCuxJ6ZBJhQXnRQJOSjffrG5LWjiIV7ko+8JamYZUUOePwfT7nYRNRquscsP86olLwBs5AmV+DyCut6gagoyidqDCmi/ETFgM6xNTNqMOPn0Rt/lKcUIpqUTQTBidLzcwl5eiysB+/ti1EtaMrsT+ikTp+5xtlHdPi6hJIZgsW4evMsuRHnI28XO0g0BqnGlqQuRvQjOKK4VM9A4BSccsAyUAd7zBpr28pyJet9jAEOn9/g8+AYcsZJ8rxtzFdNewx6l38Dcog+HRlsjK0gh7Q4YAdNgZMImVQanvVs3c9rMDW3L80I4BXWG/5kl1dimOlmw==
X-MS-Exchange-CrossTenant-Network-Message-Id: 369de30e-9f98-4c31-ef9b-08d821f4f039
X-MS-Exchange-CrossTenant-AuthSource: BYAPR15MB4136.namprd15.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 06 Jul 2020 21:38:36.2438
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 8ae927fe-1255-47a7-a2af-5f3a069daaa2
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: K5CVvcBJbIbxDzPzuZlf8x3iYTPTUzrNAYktkNDE9BzK613Wef2l40ZmCdlQCpfX
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BYAPR15MB4136
X-OriginatorOrg: fb.com
X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.235,18.0.687
 definitions=2020-07-06_20:2020-07-06,2020-07-06 signatures=0
X-Proofpoint-Spam-Details: rule=fb_default_notspam policy=fb_default score=0 lowpriorityscore=0
 cotscore=-2147483648 priorityscore=1501 suspectscore=1 bulkscore=0
 phishscore=0 adultscore=0 impostorscore=0 spamscore=0 clxscore=1015
 mlxlogscore=999 mlxscore=0 malwarescore=0 classifier=spam adjust=0
 reason=mlx scancount=1 engine=8.12.0-2004280000
 definitions=main-2007060147
X-FB-Internal: deliver
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On Fri, Jul 03, 2020 at 09:27:19AM -0700, Shakeel Butt wrote:
> On Fri, Jul 3, 2020 at 8:50 AM Roman Gushchin <guro@fb.com> wrote:
> >
> > On Fri, Jul 03, 2020 at 07:23:14AM -0700, Shakeel Butt wrote:
> > > On Thu, Jul 2, 2020 at 11:35 PM Michal Hocko <mhocko@kernel.org> wrote:
> > > >
> > > > On Thu 02-07-20 08:22:22, Shakeel Butt wrote:
> > > > [...]
> > > > > Interface options:
> > > > > ------------------
> > > > >
> > > > > 1) memcg interface e.g. 'echo 10M > memory.reclaim'
> > > > >
> > > > > + simple
> > > > > + can be extended to target specific type of memory (anon, file, kmem).
> > > > > - most probably restricted to cgroup v2.
> > > > >
> > > > > 2) fadvise(PAGEOUT) on cgroup_dir_fd
> > > > >
> > > > > + more general and applicable to other FSes (actually we are using
> > > > > something similar for tmpfs).
> > > > > + can be extended in future to just age the LRUs instead of reclaim or
> > > > > some new use cases.
> > > >
> > > > Could you explain why memory.high as an interface to trigger pro-active
> > > > memory reclaim is not sufficient. Also memory.low limit to protect
> > > > latency sensitve workloads?
> >
> > I initially liked the proposal, but after some thoughts I've realized
> > that I don't know a good use case where memory.high is less useful.
> > Shakeel, what's the typical use case you thinking of?
> > Who and how will use the new interface?
> >
> > >
> > > Yes, we can use memory.high to trigger [proactive] reclaim in a memcg
> > > but note that it can also introduce stalls in the application running
> > > in that memcg. Let's suppose the memory.current of a memcg is 100MiB
> > > and we want to reclaim 20MiB from it, we can set the memory.high to
> > > 80MiB but any allocation attempt from the application running in that
> > > memcg can get stalled/throttled. I want the functionality of the
> > > reclaim without potential stalls.
> >
> > But reclaiming some pagecache/swapping out anon pages can always
> > generate some stalls caused by pagefaults, no?
> >
> 
> Thanks for looking into the proposal. Let me answer both of your
> questions together. I have added the two use-cases but let me explain
> the proactive reclaim a bit more as we actually use that in our
> production.
> 
> We have defined tolerable refault rates for the applications based on
> their type (latency sensitive or not). Proactive reclaim is triggered
> in the application based on their current refault rates and usage. If
> the current refault rate exceeds the tolerable refault rate then
> stop/slowdown the proactive reclaim.
> 
> For the second question, yes, each individual refault can induce the
> stall as well but we have more control on that stall as compared to
> stalls due to reclaim. For us almost all the reclaimable memory is
> anon and we use compression based swap, so, the cost of each refault
> is fixed and a couple of microseconds.
> 
> I think the next question is what about the refaults from disk or
> source with highly variable cost. Usually the latency sensitive
> applications remove such uncertainty by mlocking the pages backed by
> such backends (e.g. mlocking the executable) or at least that is the
> case for us.

Got it.

It feels like you're suggesting something similar to memory.high with
something similar to a different gfp flags. In other words, the
difference is only which pages can be reclaimed and which not. I don't
have a definitive answer here, but I wonder if we can somehow
generalize the existing interface? E.g. if the problem is with artificially
induced delays, we can have a config option/sysctl/sysfs knob/something else
which would disable it. Otherwise we risk ending up with many different kinds
of soft memory limits.

Thanks!