Received: by 2002:a05:6a10:c604:0:0:0:0 with SMTP id y4csp651913pxt; Thu, 5 Aug 2021 08:27:06 -0700 (PDT) X-Google-Smtp-Source: ABdhPJwm+jqrtgl9VaTingqpwUOCHtJKpf//MTMmjMthgo9hXOTinBqTT6MkVAP8r/WzbkLwOpO3 X-Received: by 2002:a05:6402:26c6:: with SMTP id x6mr7620708edd.175.1628177226574; Thu, 05 Aug 2021 08:27:06 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1628177226; cv=none; d=google.com; s=arc-20160816; b=pY0r78QsXzOUlwbcGpZaxdXqu2pzbMTQ4fYE2uHN7poDYFSX585d9pPz2deWjs02XT AlEVgjo1svDHzAlwE/+CD1nrMvuVwHO9MyopfRdDQJAB3BSGiaYeyt4XCgMgHpjiVuwz TgqlaHvoMKsNjkywgfN8QDQYrvQuodMsn3ZroJPVH+9C7o33io+etoHDuYzsoOsX0aSw 9a5HnCK9Cx5D3DNOuv4UgcmbJE9rve6U6XVW+KGoYyx2pDXVGm8l3TNQaZJiQtzVTQMO GMSdiI8SfAqFSshqN5gh4oZ4Xz9KcBslBuHhpP8XVUVYrxsEXsAE22hyU9/uc4A0lA5k tjeQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature:dkim-signature; bh=0tHKVexUODlCjCDTjxePVAtF0VzjMMYB15eSPQdP/ms=; b=QfYD7bGQljxoHsbr+0czwm3yWNGbuIW5x+jPTghGGWiO2FSihYOW47O8KLqapWn7Yb 6GbEJ/admXcJXUtEJqBz8pec2r0aoIH1gLx4NjLnZkcpfipWN0EM9hJeTVp2ulCNUqLa N3O9nkVhVOrDsTqxrA5gPfgB5L39JbY5ymn+TECKf/quwmciDxlYaV8juxi03KGl1Tpp CEKH7K59XghJXFekG1tY+XPEMTQw574AGyDPPhUTlOCrzjsV+QR0lelDvtFAHXycZwCV 6GTjSFQDIKshyuZHdfMsiadBuM5L8OaDUlf/RS5o59AkveRqMymQrCMHGcRnGv1yxIBn u89g== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@suse.cz header.s=susede2_rsa header.b=F0BXkna1; dkim=neutral (no key) header.i=@suse.cz; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [23.128.96.18]) by mx.google.com with ESMTP id q9si5812318ejn.647.2021.08.05.08.26.43; Thu, 05 Aug 2021 08:27:06 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18; Authentication-Results: mx.google.com; dkim=pass header.i=@suse.cz header.s=susede2_rsa header.b=F0BXkna1; dkim=neutral (no key) header.i=@suse.cz; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S242217AbhHEPZK (ORCPT + 99 others); Thu, 5 Aug 2021 11:25:10 -0400 Received: from smtp-out2.suse.de ([195.135.220.29]:59212 "EHLO smtp-out2.suse.de" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S241983AbhHEPUY (ORCPT ); Thu, 5 Aug 2021 11:20:24 -0400 Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by smtp-out2.suse.de (Postfix) with ESMTPS id DB10420254; Thu, 5 Aug 2021 15:20:08 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1628176808; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=0tHKVexUODlCjCDTjxePVAtF0VzjMMYB15eSPQdP/ms=; b=F0BXkna1+rv/gCKbvbzswSMMWr3LY8vqv/XDO23+e8nRS7wkUB0jmez+sa/GfAQc43ew8m itVofpbBA5TsztNBjc1DT2jV4UxSut8I4lSD3Oaoi4UFpe/hWUrVd2mjTmUYbq6QHZFBFm 7VG1Kzt9QhSdzXDSoQpJQczOcASrRfk= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1628176808; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=0tHKVexUODlCjCDTjxePVAtF0VzjMMYB15eSPQdP/ms=; b=IiHdMSXDN/TnwmCg1vUS1OtVOw7Ao4E3/246RyjmMbBNTpyQpoBAeEWSkmnOex2Q8c3ZKp TV+FO2zfQYMtFHCg== Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by imap2.suse-dmz.suse.de (Postfix) with ESMTPS id A593013DAC; Thu, 5 Aug 2021 15:20:08 +0000 (UTC) Received: from dovecot-director2.suse.de ([192.168.254.65]) by imap2.suse-dmz.suse.de with ESMTPSA id GATfJ6gBDGFDdQAAMHmgww (envelope-from ); Thu, 05 Aug 2021 15:20:08 +0000 From: Vlastimil Babka To: Andrew Morton , Christoph Lameter , David Rientjes , Pekka Enberg , Joonsoo Kim Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Mike Galbraith , Sebastian Andrzej Siewior , Thomas Gleixner , Mel Gorman , Jesper Dangaard Brouer , Jann Horn , Vlastimil Babka Subject: [PATCH v4 25/35] mm, slub: separate detaching of partial list in unfreeze_partials() from unfreezing Date: Thu, 5 Aug 2021 17:19:50 +0200 Message-Id: <20210805152000.12817-26-vbabka@suse.cz> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20210805152000.12817-1-vbabka@suse.cz> References: <20210805152000.12817-1-vbabka@suse.cz> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Unfreezing partial list can be split to two phases - detaching the list from struct kmem_cache_cpu, and processing the list. The whole operation does not need to be protected by disabled irqs. Restructure the code to separate the detaching (with disabled irqs) and unfreezing (with irq disabling to be reduced in the next patch). Also, unfreeze_partials() can be called from another cpu on behalf of a cpu that is being offlined, where disabling irqs on the local cpu has no sense, so restructure the code as follows: - __unfreeze_partials() is the bulk of unfreeze_partials() that processes the detached percpu partial list - unfreeze_partials() detaches list from current cpu with irqs disabled and calls __unfreeze_partials() - unfreeze_partials_cpu() is to be called for the offlined cpu so it needs no irq disabling, and is called from __flush_cpu_slab() - flush_cpu_slab() is for the local cpu thus it needs to call unfreeze_partials(). So it can't simply call __flush_cpu_slab(smp_processor_id()) anymore and we have to open-code the proper calls. Signed-off-by: Vlastimil Babka --- mm/slub.c | 73 ++++++++++++++++++++++++++++++++++++++----------------- 1 file changed, 51 insertions(+), 22 deletions(-) diff --git a/mm/slub.c b/mm/slub.c index 984173ce8465..c8637150bf28 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -2330,25 +2330,15 @@ static void deactivate_slab(struct kmem_cache *s, struct page *page, } } -/* - * Unfreeze all the cpu partial slabs. - * - * This function must be called with preemption or migration - * disabled with c local to the cpu. - */ -static void unfreeze_partials(struct kmem_cache *s, - struct kmem_cache_cpu *c) -{ #ifdef CONFIG_SLUB_CPU_PARTIAL +static void __unfreeze_partials(struct kmem_cache *s, struct page *partial_page) +{ struct kmem_cache_node *n = NULL, *n2 = NULL; - struct page *page, *partial_page, *discard_page = NULL; + struct page *page, *discard_page = NULL; unsigned long flags; local_irq_save(flags); - partial_page = slub_percpu_partial(c); - c->partial = NULL; - while (partial_page) { struct page new; struct page old; @@ -2403,10 +2393,45 @@ static void unfreeze_partials(struct kmem_cache *s, discard_slab(s, page); stat(s, FREE_SLAB); } +} -#endif /* CONFIG_SLUB_CPU_PARTIAL */ +/* + * Unfreeze all the cpu partial slabs. + */ +static void unfreeze_partials(struct kmem_cache *s) +{ + struct page *partial_page; + unsigned long flags; + + local_irq_save(flags); + partial_page = this_cpu_read(s->cpu_slab->partial); + this_cpu_write(s->cpu_slab->partial, NULL); + local_irq_restore(flags); + + if (partial_page) + __unfreeze_partials(s, partial_page); } +static void unfreeze_partials_cpu(struct kmem_cache *s, + struct kmem_cache_cpu *c) +{ + struct page *partial_page; + + partial_page = slub_percpu_partial(c); + c->partial = NULL; + + if (partial_page) + __unfreeze_partials(s, partial_page); +} + +#else /* CONFIG_SLUB_CPU_PARTIAL */ + +static inline void unfreeze_partials(struct kmem_cache *s) { } +static inline void unfreeze_partials_cpu(struct kmem_cache *s, + struct kmem_cache_cpu *c) { } + +#endif /* CONFIG_SLUB_CPU_PARTIAL */ + /* * Put a page that was just frozen (in __slab_free|get_partial_node) into a * partial page slot if available. @@ -2435,7 +2460,7 @@ static void put_cpu_partial(struct kmem_cache *s, struct page *page, int drain) * partial array is full. Move the existing * set to the per node partial list. */ - unfreeze_partials(s, this_cpu_ptr(s->cpu_slab)); + unfreeze_partials(s); oldpage = NULL; pobjects = 0; pages = 0; @@ -2470,11 +2495,6 @@ static inline void flush_slab(struct kmem_cache *s, struct kmem_cache_cpu *c) stat(s, CPUSLAB_FLUSH); } -/* - * Flush cpu slab. - * - * Called from IPI handler with interrupts disabled. - */ static inline void __flush_cpu_slab(struct kmem_cache *s, int cpu) { struct kmem_cache_cpu *c = per_cpu_ptr(s->cpu_slab, cpu); @@ -2482,14 +2502,23 @@ static inline void __flush_cpu_slab(struct kmem_cache *s, int cpu) if (c->page) flush_slab(s, c); - unfreeze_partials(s, c); + unfreeze_partials_cpu(s, c); } +/* + * Flush cpu slab. + * + * Called from IPI handler with interrupts disabled. + */ static void flush_cpu_slab(void *d) { struct kmem_cache *s = d; + struct kmem_cache_cpu *c = this_cpu_ptr(s->cpu_slab); - __flush_cpu_slab(s, smp_processor_id()); + if (c->page) + flush_slab(s, c); + + unfreeze_partials(s); } static bool has_cpu_slab(int cpu, void *info) -- 2.32.0