Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Thu, 10 Feb 2022 12:15:34 -0800
From:   Roman Gushchin <guro@fb.com>
To:     Shakeel Butt <shakeelb@google.com>
CC:     Johannes Weiner <hannes@cmpxchg.org>,
        Michal Hocko <mhocko@suse.com>,
        Chris Down <chris@chrisdown.name>,
        Andrew Morton <akpm@linux-foundation.org>,
        <cgroups@vger.kernel.org>, <linux-mm@kvack.org>,
        <linux-kernel@vger.kernel.org>
Subject: Re: [PATCH 4/4] memcg: synchronously enforce memory.high
Message-ID: <YgVyZrDPxVgP6OLG@carbon.dhcp.thefacebook.com>
References: <20220210081437.1884008-1-shakeelb@google.com>
 <20220210081437.1884008-5-shakeelb@google.com>
Content-Type: text/plain; charset="us-ascii"
Content-Disposition: inline
In-Reply-To: <20220210081437.1884008-5-shakeelb@google.com>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?D2txfGh1HMsRWhs0vOje1yy85IvLuT9w32Yi4TAmq2qyjMTX2YSjc3uDTXgI?=
 =?us-ascii?Q?SmV7yWdu1lElZL3SQIferwHUyovHTXcFF77Q9fBJTwj6oSjvfpuqdz33zXey?=
 =?us-ascii?Q?0RNE//YCdY3GPWRPsjkMs0YIvb+Ejj26Z66h+41XJZSkVoDSxKA+wXSPK5+l?=
 =?us-ascii?Q?3iKoyPwEvVz3lF7OXU3IsZ/DwHBpcv3wPkdAZDOhrp96kxc2arDolOU73LuX?=
 =?us-ascii?Q?Qc2b1fX2BkX4L5NW0oytqPSs5fuquazC/7DLIGG4K1Njg/t6URZ7Mt7h+YVw?=
 =?us-ascii?Q?BuVgeRu6z3SQB81zq6R+LJ54nwrsABKa//VPFKVE+ioE1Mlpooqtv23OHiPp?=
 =?us-ascii?Q?K7RhBjMVfHKZDUI2yZCS77lR+TUh+5sF+WhUx8L+LRTl4rv6UbPvoh2mTD3C?=
 =?us-ascii?Q?rDtuJfa0k3gtGJNiTUY5plbMlMvik370/OSN6e1ZNGl4gplLWyfbt0uvglbe?=
 =?us-ascii?Q?ZgMEKuaAknvKxOGFYt8Ov1qVSkjxXlhUVevUN8coYKh6Z0aItQWuGCrqbwIx?=
 =?us-ascii?Q?l+1fXwN8ZU6Ilgqx2YMhj/bLuop98l4KL/pjRJMH1oBdDqeoNU1ShzxhDFXT?=
 =?us-ascii?Q?NNSVXE2bkPZiK1q+ZiH7HHbi9u1wYqgkmMhCzRaxmWWPa3WajMGHlFVQk4Lv?=
 =?us-ascii?Q?EhXi7977X5ddlKl+G+gVWbIe1XCNtYiCQicXVGewOjXoN+3JJRGdoxwkMBSd?=
 =?us-ascii?Q?//82rjgwK6Q1x/w1yPhsuQ1N4CYprcw6+brDcClgLx/3hDQa56OP7e08XhHv?=
 =?us-ascii?Q?ukIGlogK5Mj3TGtejDAjXlGKpaZUX8xOqeDtA7Y+SSPGVEzz9sNbxylEDgz4?=
 =?us-ascii?Q?smCCI/pew5+Y5Cv+vxR5UF31uNfVe0J1NdI4K6usnwQZPyui9Q2IB1RWw1Rf?=
 =?us-ascii?Q?sC9BMO16AgIiHnOM+8ucBXHW9gK4wIznGsZ6Kk4WD2wr7i8uClpNKjVoDwVL?=
 =?us-ascii?Q?iJSThE1halyP17TmNNunA6vQXLln2JNWl+7ceLNp55nxCIp/54nb2iYTOVd+?=
 =?us-ascii?Q?d757ao3CzMP7IOM0xwgqLIHg94wuJYnY/hdPh5tj5iWoynlXLemDG+7gh8xa?=
 =?us-ascii?Q?wMpyV3qk7xaOx6opRwv99K1S5dV4Ca56c95St76GIAM+p38pp9ob2QvkaG7H?=
 =?us-ascii?Q?xZrmKQv/Ec90FiR+85uH2GGcfFxsKH8FucQI9mSANShoaciiD+aDeRYTWA9a?=
 =?us-ascii?Q?6g7auTMYZc9g8aZsOdJhVRdMirkNoHlaD9QNRmMIPCjHfbUWbsdyGtPJxUhZ?=
 =?us-ascii?Q?O8tmB6u8YKD+fD5zf29ZY6I58pTnzqCDawORt81nwQVeiB89Bu7Kw5A6qHco?=
 =?us-ascii?Q?u6ZkVJvTzvDAvnleZ3zb0i2UISLjNv49IfWqj5YLt5RmG+RIA+Ygk+FdOQ2t?=
 =?us-ascii?Q?/qLc9bHbuWd47JJXn291APFyd93CZeP9RMrf4Ujr6HDB8g7CFIgUdhBpBVBi?=
 =?us-ascii?Q?cBdTKp9uwTP4sLU+/PbubXUceP6kjRyNqN62OxmVA0g3L+AN4QocQcav0NLm?=
 =?us-ascii?Q?f6op5DuneWag2uT9c0ZzD/0V9UTAPkCeYICuLWzkn9vf/7Mu2GBhW/EtqZM5?=
 =?us-ascii?Q?Z6ng6VscvhicpwDqZU+vYoNSHeK4vQS/XzzF25T3?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 9eb1a59e-2d6c-4943-f7fd-08d9ecd21af1
X-MS-Exchange-CrossTenant-AuthSource: BYAPR15MB4136.namprd15.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 10 Feb 2022 20:15:39.0082
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 8ae927fe-1255-47a7-a2af-5f3a069daaa2
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: 4bZm5l7hFlWwMQdR6AVQIq9/xc5S9y/1WXQP/JA9Aq/r3UsP5HZTZE2HzYAnovPM
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM6PR15MB3259
X-OriginatorOrg: fb.com
X-Proofpoint-GUID: wSUBwFKUorxOSGXAsMF7uToRDuyVO75e
X-Proofpoint-ORIG-GUID: wSUBwFKUorxOSGXAsMF7uToRDuyVO75e
X-Proofpoint-Virus-Version: vendor=baseguard
 engine=ICAP:2.0.205,Aquarius:18.0.816,Hydra:6.0.425,FMLib:17.11.62.513
 definitions=2022-02-10_09,2022-02-09_01,2021-12-02_01
X-Proofpoint-Spam-Details: rule=fb_outbound_notspam policy=fb_outbound score=0 phishscore=0
 malwarescore=0 clxscore=1015 bulkscore=0 mlxlogscore=409 impostorscore=0
 mlxscore=0 lowpriorityscore=0 adultscore=0 suspectscore=0
 priorityscore=1501 spamscore=0 classifier=spam adjust=0 reason=mlx
 scancount=1 engine=8.12.0-2201110000 definitions=main-2202100106
X-FB-Internal: deliver
X-Spam-Status: No, score=-2.8 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_LOW,
        RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL,SPF_HELO_NONE,SPF_NONE,
        T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On Thu, Feb 10, 2022 at 12:14:37AM -0800, Shakeel Butt wrote:
> The high limit is used to throttle the workload without invoking the
> oom-killer. Recently we tried to use the high limit to right size our
> internal workloads. More specifically dynamically adjusting the limits
> of the workload without letting the workload get oom-killed. However due
> to the limitation of the implementation of high limit enforcement, we
> observed the mechanism fails for some real workloads.
> 
> The high limit is enforced on return-to-userspace i.e. the kernel let
> the usage goes over the limit and when the execution returns to
> userspace, the high reclaim is triggered and the process can get
> throttled as well. However this mechanism fails for workloads which do
> large allocations in a single kernel entry e.g. applications that
> mlock() a large chunk of memory in a single syscall. Such applications
> bypass the high limit and can trigger the oom-killer.
> 
> To make high limit enforcement more robust, this patch make the limit
> enforcement synchronous. However there are couple of open questions to
> enforce high limit synchronously. What should be the behavior of
> __GFP_NORETRY allocaion on hitting high limit? Similar question arise
> for allocations which do not allow blocking. This patch took the
> approach to keep the previous behavior i.e. let such allocations not
> throttle synchronously but rely on the return-to-userspace mechanism to
> throttle processes triggering such allocations.
> 
> This patch does not remove the return-to-userspace high limit
> enforcement due to the reason mentioned in the previous para. Also the
> allocations where the memory usage is below high limit but the swap
> usage is above swap's high limit, such allocators are throttled in the
> return-to-userspace.

Has this approach been extensively tested in the production?

Injecting sleeps at return-to-userspace moment is safe in terms of priority
inversions: a slowed down task will unlikely affect the rest of the system.

It way less predictable for a random allocation in the kernel mode, what if
the task is already holding a system-wide resource?

Someone might argue that it's not better than a system-wide memory shortage
and the same allocation might go into a direct reclaim anyway, but with
the way how memory.high is used it will happen way more often.

Thanks!