Received: by 2002:a25:c205:0:0:0:0:0 with SMTP id s5csp4561282ybf; Wed, 4 Mar 2020 06:23:54 -0800 (PST) X-Google-Smtp-Source: ADFU+vvLCjy4SdEsmI1xgzZi4SWXn0AgP4Ogi44nCdXayHsYmCbzDCokqw15yaHcFhVOHT77KLNk X-Received: by 2002:aca:aa8b:: with SMTP id t133mr1802320oie.95.1583331833919; Wed, 04 Mar 2020 06:23:53 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1583331833; cv=none; d=google.com; s=arc-20160816; b=bzm/VMSjtuws3ZmujiCjPJ+CjlqYbO9UBbk6pcgshOc3EqlkMX4hlktKwFNYOLegze vxY4uutFGhVuDnVuQgIb+8AjUcNYkXOd77D9m8sFuMPJnn5OLknMqTxK4I/bBSRfHxBM gF5sSdxpXtwwWyEhmNAjxkFRvn2FEx4vR51N3A4W/vBX+VMA5b82WglrFAtrzX+f78Gy /Bq18cf8yy1JTKmPnRJeWkam4G3rVo5fOOJdj3Dgu3mPwHopGNioYAunJVwAceTVMC7V qsPIrx+3eFlcx4nN3QOkHOKPGujbv0OOQvFEirEi7pZnpwJPQ32xf85XZzUKQA5HOTOa 5inA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:message-id:date:subject:cc:to:from :dkim-signature; bh=DuM4SHf1sVSRFd0iAfPg/3FUWAci4sdx8YlPTtxLHUA=; b=1CpwPa1RRusmfnJB60d7+gAdvFyPLx+3WKex/VvEsza/cptysd9bh4I/uaALBW0Dvn w3KAVV38mWAyUIlU5flxwhCsYwgjeiyogcQqVZgnH8b9GfJMg1SiHnMWP4sgY1m7IM6T G666x0/ScTBXz1SvQjrN+pAySjue+QSlJB2Fnwj2/rhPmLETHGnusAOIlc/xPGwfjskc Qo+/Cu4asIAPBKOvcDv23mJA0HqQXCsP6vdDtHm8/vjWN2GvWV1P7SzCBuDHGKkrzPgE /62y5HXEUa0vjZcKVf23cwp3E9bH5an8BsDYRmXjWx2Vko0Pb7cKt/YNBaPumu75y0YI SXHw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=AjBLDLRa; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id u13si685169oic.208.2020.03.04.06.23.27; Wed, 04 Mar 2020 06:23:53 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=AjBLDLRa; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729564AbgCDOXQ (ORCPT + 99 others); Wed, 4 Mar 2020 09:23:16 -0500 Received: from mail-lj1-f193.google.com ([209.85.208.193]:39685 "EHLO mail-lj1-f193.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726748AbgCDOXQ (ORCPT ); Wed, 4 Mar 2020 09:23:16 -0500 Received: by mail-lj1-f193.google.com with SMTP id f10so2219791ljn.6 for ; Wed, 04 Mar 2020 06:23:13 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id; bh=DuM4SHf1sVSRFd0iAfPg/3FUWAci4sdx8YlPTtxLHUA=; b=AjBLDLRaVZo2u3eVMtEp8eF/n6W+A+BSXG1x0tQJO17cbACepFUj9lAAQYvLH+nwIT kFtbfwM6ADxCwKKha/V5bMdPvRNkj1PBa2ds4tOgWDCS6UxryiCc2PLfDFGA1mg+/qtj tu/segWSBZ4AjX0KJ2fF8H9CqDT3NMzYEHWfBrZnJOumiDvXYN+MFgseacU3MoUd5rmR 80fqvqmWZzMTG7ig66oQpPjGKJ/7xyzSmlbRFGGcWWmn/lg+anX8O1BnCn3ZI5Yy4HL7 LwIbnYpjhvzSEUpQbMAxEbvCmBmG/Y2P2vba5qmJ9i7wmARKsj4ZCCkHv+I1LYEuEpDI 6ffw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id; bh=DuM4SHf1sVSRFd0iAfPg/3FUWAci4sdx8YlPTtxLHUA=; b=Cu6FllzU6Nyf82IwRntHrFzqCjcCPPqrVhFDJbeGPntJTvFnJwKjMgC4eiPIRr1gzM m0WXPkB8QABZClhthSwTLGqOkCs4nqpXxmda4G4KM/hIWy/NSYYg455cIWkeuQIhMYM1 ktULhLDBZ76AvIQkFk7tqfrkA7DckLtsOjWv7uGKOVrxwH8sTmOCpnS4PaGGy7rJ4l1W EA1Shc7DOzsw0goXZQEbMIsD5k4sb1QiJrphdK42hN3HL0nKTDnZ+greF7/4Zjh5WoG4 MXHNxgKj3skp2xu3ffUJrh77kNc6VG+LhUZGSDEDVza/OXDo7qMOgIzyetqkTor4+NUj rQ5w== X-Gm-Message-State: ANhLgQ2MLCCptzSc/vlL9IGHOJVw+lj2AC4IMe4WTTsAZ6txF7CJ+QKe yoiPX2QkA8G92nnlnjsD8K8= X-Received: by 2002:a2e:3807:: with SMTP id f7mr2123355lja.103.1583331791595; Wed, 04 Mar 2020 06:23:11 -0800 (PST) Received: from localhost.localdomain (188.146.98.66.nat.umts.dynamic.t-mobile.pl. [188.146.98.66]) by smtp.gmail.com with ESMTPSA id z23sm9575712ljg.99.2020.03.04.06.23.10 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 04 Mar 2020 06:23:11 -0800 (PST) From: mateusznosek0@gmail.com To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Mateusz Nosek , akpm@linux-foundation.org Subject: [RFC PATCH] mm: Micro-optimisation: Save two branches on hot - page allocation path Date: Wed, 4 Mar 2020 15:22:30 +0100 Message-Id: <20200304142230.8753-1-mateusznosek0@gmail.com> X-Mailer: git-send-email 2.17.1 Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org From: Mateusz Nosek This patch makes ALLOC_KSWAPD equal to __GFP_KSWAPD_RACLAIM (cast to 'int'). Thanks to that code like: ```if (gfp_mask & __GFP_KSWAPD_RECLAIM) alloc_flags |= ALLOC_KSWAPD;``` can be changed to: ```alloc_flags |= (__force int) (gfp_mask &__GFP_KSWAPD_RECLAIM);``` Thanks to this one branch less is generated in the assembly. In case of ALLOC_KSWAPD flag two branches are saved, first one in code that always executes in the beggining of page allocation and the second one in loop in page allocator slowpath. Signed-off-by: Mateusz Nosek --- mm/internal.h | 2 +- mm/page_alloc.c | 23 +++++++++++++++-------- 2 files changed, 16 insertions(+), 9 deletions(-) diff --git a/mm/internal.h b/mm/internal.h index 86372d164476..7fb724977743 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -535,7 +535,7 @@ unsigned long reclaim_clean_pages_from_list(struct zone *zone, #else #define ALLOC_NOFRAGMENT 0x0 #endif -#define ALLOC_KSWAPD 0x200 /* allow waking of kswapd */ +#define ALLOC_KSWAPD 0x800 /* allow waking of kswapd */ enum ttu_flags; struct tlbflush_unmap_batch; diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 79e950d76ffc..73afd883eab5 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -3609,10 +3609,14 @@ static bool zone_allows_reclaim(struct zone *local_zone, struct zone *zone) static inline unsigned int alloc_flags_nofragment(struct zone *zone, gfp_t gfp_mask) { - unsigned int alloc_flags = 0; + unsigned int alloc_flags; - if (gfp_mask & __GFP_KSWAPD_RECLAIM) - alloc_flags |= ALLOC_KSWAPD; + /* + * __GFP_KSWAPD_RECLAIM is assumed to be the same as ALLOC_KSWAPD + * to save a branch. + */ + BUILD_BUG_ON(__GFP_KSWAPD_RECLAIM != (__force gfp_t) ALLOC_KSWAPD); + alloc_flags = (__force int) (gfp_mask & __GFP_KSWAPD_RECLAIM); #ifdef CONFIG_ZONE_DMA32 if (!zone) @@ -4248,8 +4252,13 @@ gfp_to_alloc_flags(gfp_t gfp_mask) { unsigned int alloc_flags = ALLOC_WMARK_MIN | ALLOC_CPUSET; - /* __GFP_HIGH is assumed to be the same as ALLOC_HIGH to save a branch. */ + /* + * __GFP_HIGH is assumed to be the same as ALLOC_HIGH + * and __GFP_KSWAPD_RECLAIM is assumed to be the same as ALLOC_KSWAPD + * to save two branches. + */ BUILD_BUG_ON(__GFP_HIGH != (__force gfp_t) ALLOC_HIGH); + BUILD_BUG_ON(__GFP_KSWAPD_RECLAIM != (__force gfp_t) ALLOC_KSWAPD); /* * The caller may dip into page reserves a bit more if the caller @@ -4257,7 +4266,8 @@ gfp_to_alloc_flags(gfp_t gfp_mask) * policy or is asking for __GFP_HIGH memory. GFP_ATOMIC requests will * set both ALLOC_HARDER (__GFP_ATOMIC) and ALLOC_HIGH (__GFP_HIGH). */ - alloc_flags |= (__force int) (gfp_mask & __GFP_HIGH); + alloc_flags |= (__force int) + (gfp_mask & (__GFP_HIGH | __GFP_KSWAPD_RECLAIM)); if (gfp_mask & __GFP_ATOMIC) { /* @@ -4274,9 +4284,6 @@ gfp_to_alloc_flags(gfp_t gfp_mask) } else if (unlikely(rt_task(current)) && !in_interrupt()) alloc_flags |= ALLOC_HARDER; - if (gfp_mask & __GFP_KSWAPD_RECLAIM) - alloc_flags |= ALLOC_KSWAPD; - #ifdef CONFIG_CMA if (gfpflags_to_migratetype(gfp_mask) == MIGRATE_MOVABLE) alloc_flags |= ALLOC_CMA; -- 2.17.1