Received: by 10.192.165.148 with SMTP id m20csp4962053imm; Tue, 8 May 2018 18:38:46 -0700 (PDT) X-Google-Smtp-Source: AB8JxZrBbQqwYEkxPfWJvSEK3kcwB/NjD7YvuI7RhJzsI+gF749FzP7dnhlMZe58AjVqZHqoc2YT X-Received: by 2002:a65:4443:: with SMTP id e3-v6mr35452816pgq.450.1525829926065; Tue, 08 May 2018 18:38:46 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1525829926; cv=none; d=google.com; s=arc-20160816; b=DAiHG9/D5ktjK6xAMfvAe+LnPtjrIbJqgmiHXZTo0qlUrpXtcc+biFHHG9TeuRuIDV KtxJBSUBMm8TVNacBsL/jMfbMO/DP7QsYoOTkLiR3v6LreE9RI2PPZOoXMhyo2a6Bbuw SPCfo322MAS9OW1QeJHtVsLFyhuDG5WvuRgl/zv4KwqwPlJN7v9kHA1ZnEYGEh+3mMAq XocbVvU/s5jlvJi9gAnzJhcpARJvPbDuBY6+U8hHp231YqUiT7wEPFhyeihdeecKRYFj Htb514RNyTPLjzFiGqJzL14CNw94sqArt6dz89VSMQh7w3T6UXJMyL4GJI8qaKQy1+uj PehA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:dkim-signature:arc-authentication-results; bh=DukDmIDOCuZvOvYV7AxI2za4rxww2vVAjhaHcLiyQT0=; b=RXL4hm1GhbxV5iXOfzzm1Qvc++2zSHxvxwkzv7aPOQxA6U3yDsP/hUIwVQrrLV7Wii UMuaUFh8ZCHuiYkIZ1DP8uzqFKoD4qo35ov4zl48nzRHHs87U9rONfyn6djp8/O0yC2S zSzdm5Xz1YcaeADCYdwA2oSwplfyH+cpxFzYJ+c6Bn/00jAKqhlzbbqAcdJZyvLFqgMO cdy1ZAA/rPrZBALqmOthbAhcyWoC0FNcEvjPC3kjYiw7xyZIz/WDc2w9jXUR3GQprNk5 xQrQS6osT3ndqUbUEf1YEf1Ek3uCb0dU0+WJAHbf9XQ+iudA67Ij5lXZFYfX0XKTilXZ w94w== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=OgQQUPVX; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id 5si22833470pfi.285.2018.05.08.18.38.31; Tue, 08 May 2018 18:38:46 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=OgQQUPVX; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S933457AbeEIBeU (ORCPT + 99 others); Tue, 8 May 2018 21:34:20 -0400 Received: from mail-qt0-f196.google.com ([209.85.216.196]:39177 "EHLO mail-qt0-f196.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S933384AbeEIBeR (ORCPT ); Tue, 8 May 2018 21:34:17 -0400 Received: by mail-qt0-f196.google.com with SMTP id f1-v6so43693655qtj.6; Tue, 08 May 2018 18:34:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=DukDmIDOCuZvOvYV7AxI2za4rxww2vVAjhaHcLiyQT0=; b=OgQQUPVXu1Wt/pYZ4W5jyv14YX7JAGgy5HLjbGjNtTts6rquJEQMUxvekITplCxZGh B5/ZpkwxSm4oubpOgjYDR6wth9+IXwNAQnvsqszh2O0Vw8oj0Hat3uzDiM9HqbovoXC6 pbLNb59dhaa2hlI6d1WrIQusXHrNcJjwV9ybqn/lg7GmLev9kTkMstBjVXfN2NdeorJR jNTz4emOlZvcFnVawsbAQ4H3Fa5FsYMcFfV74jRVpSaR3ksVxE9SOLKQvcHn5IidhN0T GW+b3W2ektVNCcNrRWFUt7qKrPGowLF3aB+rdgPt3RPP/0AwE6zI2T3XNpk8ex6srvdp SrPQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=DukDmIDOCuZvOvYV7AxI2za4rxww2vVAjhaHcLiyQT0=; b=AIMDaIMB0l8+l3sY/xpBxH0aG7wyWDQT6pz4uJHUm1GPWKanvo+qWAK71g6SjPTgSH 3LQEJmULHAYcrYyJ1/alarXlsuJfCtQAS4K9uZa6OM7kUbZmyT/KwzqVQzy8KIx//C2R bWYS++ktFYt5WOOHhWFZz+6J4VJlM8OwBxnn60as6lYI5Mh92YGGtIQxFD+6Pi0KzXwA X3E5zOC03s3k4ssAmQ0n/2mpRh4GKdJhUr6ssWSt89tTgXvlnoAVRjjSYni68QVJVl+/ reILfItRIaFVq5+iWSpwBYTDvMJL/EXqT20A2wbFItS3RW6E2vKWAV5iWHk1KJ595a7F cZjA== X-Gm-Message-State: ALQs6tBYnBqkaU3p8a3EogiyitrEAPC7Y+hXlZG8kmQHBTYJihGNjkj+ C+j+2nuKuPeF4hHNTXx/H/bwBeERVA== X-Received: by 2002:aed:2ba3:: with SMTP id e32-v6mr39821933qtd.272.1525829655527; Tue, 08 May 2018 18:34:15 -0700 (PDT) Received: from localhost.localdomain (c-71-234-172-214.hsd1.vt.comcast.net. [71.234.172.214]) by smtp.gmail.com with ESMTPSA id x28-v6sm23719003qtx.95.2018.05.08.18.34.14 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 08 May 2018 18:34:14 -0700 (PDT) From: Kent Overstreet To: linux-kernel@vger.kernel.org, linux-block@vger.kernel.org, linux-mm@kvack.org, Jens Axboe , Ingo Molnar Cc: Kent Overstreet Subject: [PATCH 01/10] mempool: Add mempool_init()/mempool_exit() Date: Tue, 8 May 2018 21:33:49 -0400 Message-Id: <20180509013358.16399-2-kent.overstreet@gmail.com> X-Mailer: git-send-email 2.17.0 In-Reply-To: <20180509013358.16399-1-kent.overstreet@gmail.com> References: <20180509013358.16399-1-kent.overstreet@gmail.com> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Allows mempools to be embedded in other structs, getting rid of a pointer indirection from allocation fastpaths. mempool_exit() is safe to call on an uninitialized but zeroed mempool. Signed-off-by: Kent Overstreet --- include/linux/mempool.h | 34 +++++++++++++ mm/mempool.c | 108 ++++++++++++++++++++++++++++++---------- 2 files changed, 115 insertions(+), 27 deletions(-) diff --git a/include/linux/mempool.h b/include/linux/mempool.h index b51f5c430c..0c964ac107 100644 --- a/include/linux/mempool.h +++ b/include/linux/mempool.h @@ -25,6 +25,18 @@ typedef struct mempool_s { wait_queue_head_t wait; } mempool_t; +static inline bool mempool_initialized(mempool_t *pool) +{ + return pool->elements != NULL; +} + +void mempool_exit(mempool_t *pool); +int mempool_init_node(mempool_t *pool, int min_nr, mempool_alloc_t *alloc_fn, + mempool_free_t *free_fn, void *pool_data, + gfp_t gfp_mask, int node_id); +int mempool_init(mempool_t *pool, int min_nr, mempool_alloc_t *alloc_fn, + mempool_free_t *free_fn, void *pool_data); + extern mempool_t *mempool_create(int min_nr, mempool_alloc_t *alloc_fn, mempool_free_t *free_fn, void *pool_data); extern mempool_t *mempool_create_node(int min_nr, mempool_alloc_t *alloc_fn, @@ -43,6 +55,14 @@ extern void mempool_free(void *element, mempool_t *pool); */ void *mempool_alloc_slab(gfp_t gfp_mask, void *pool_data); void mempool_free_slab(void *element, void *pool_data); + +static inline int +mempool_init_slab_pool(mempool_t *pool, int min_nr, struct kmem_cache *kc) +{ + return mempool_init(pool, min_nr, mempool_alloc_slab, + mempool_free_slab, (void *) kc); +} + static inline mempool_t * mempool_create_slab_pool(int min_nr, struct kmem_cache *kc) { @@ -56,6 +76,13 @@ mempool_create_slab_pool(int min_nr, struct kmem_cache *kc) */ void *mempool_kmalloc(gfp_t gfp_mask, void *pool_data); void mempool_kfree(void *element, void *pool_data); + +static inline int mempool_init_kmalloc_pool(mempool_t *pool, int min_nr, size_t size) +{ + return mempool_init(pool, min_nr, mempool_kmalloc, + mempool_kfree, (void *) size); +} + static inline mempool_t *mempool_create_kmalloc_pool(int min_nr, size_t size) { return mempool_create(min_nr, mempool_kmalloc, mempool_kfree, @@ -68,6 +95,13 @@ static inline mempool_t *mempool_create_kmalloc_pool(int min_nr, size_t size) */ void *mempool_alloc_pages(gfp_t gfp_mask, void *pool_data); void mempool_free_pages(void *element, void *pool_data); + +static inline int mempool_init_page_pool(mempool_t *pool, int min_nr, int order) +{ + return mempool_init(pool, min_nr, mempool_alloc_pages, + mempool_free_pages, (void *)(long)order); +} + static inline mempool_t *mempool_create_page_pool(int min_nr, int order) { return mempool_create(min_nr, mempool_alloc_pages, mempool_free_pages, diff --git a/mm/mempool.c b/mm/mempool.c index 5c9dce3471..df90ace400 100644 --- a/mm/mempool.c +++ b/mm/mempool.c @@ -137,6 +137,28 @@ static void *remove_element(mempool_t *pool, gfp_t flags) return element; } +/** + * mempool_destroy - exit a mempool initialized with mempool_init() + * @pool: pointer to the memory pool which was initialized with + * mempool_init(). + * + * Free all reserved elements in @pool and @pool itself. This function + * only sleeps if the free_fn() function sleeps. + * + * May be called on a zeroed but uninitialized mempool (i.e. allocated with + * kzalloc()). + */ +void mempool_exit(mempool_t *pool) +{ + while (pool->curr_nr) { + void *element = remove_element(pool, GFP_KERNEL); + pool->free(element, pool->pool_data); + } + kfree(pool->elements); + pool->elements = NULL; +} +EXPORT_SYMBOL(mempool_exit); + /** * mempool_destroy - deallocate a memory pool * @pool: pointer to the memory pool which was allocated via @@ -150,15 +172,65 @@ void mempool_destroy(mempool_t *pool) if (unlikely(!pool)) return; - while (pool->curr_nr) { - void *element = remove_element(pool, GFP_KERNEL); - pool->free(element, pool->pool_data); - } - kfree(pool->elements); + mempool_exit(pool); kfree(pool); } EXPORT_SYMBOL(mempool_destroy); +int mempool_init_node(mempool_t *pool, int min_nr, mempool_alloc_t *alloc_fn, + mempool_free_t *free_fn, void *pool_data, + gfp_t gfp_mask, int node_id) +{ + spin_lock_init(&pool->lock); + pool->min_nr = min_nr; + pool->pool_data = pool_data; + pool->alloc = alloc_fn; + pool->free = free_fn; + init_waitqueue_head(&pool->wait); + + pool->elements = kmalloc_array_node(min_nr, sizeof(void *), + gfp_mask, node_id); + if (!pool->elements) + return -ENOMEM; + + /* + * First pre-allocate the guaranteed number of buffers. + */ + while (pool->curr_nr < pool->min_nr) { + void *element; + + element = pool->alloc(gfp_mask, pool->pool_data); + if (unlikely(!element)) { + mempool_exit(pool); + return -ENOMEM; + } + add_element(pool, element); + } + + return 0; +} +EXPORT_SYMBOL(mempool_init_node); + +/** + * mempool_init - initialize a memory pool + * @min_nr: the minimum number of elements guaranteed to be + * allocated for this pool. + * @alloc_fn: user-defined element-allocation function. + * @free_fn: user-defined element-freeing function. + * @pool_data: optional private data available to the user-defined functions. + * + * Like mempool_create(), but initializes the pool in (i.e. embedded in another + * structure). + */ +int mempool_init(mempool_t *pool, int min_nr, mempool_alloc_t *alloc_fn, + mempool_free_t *free_fn, void *pool_data) +{ + return mempool_init_node(pool, min_nr, alloc_fn, free_fn, + pool_data, GFP_KERNEL, NUMA_NO_NODE); + +} +EXPORT_SYMBOL(mempool_init); + /** * mempool_create - create a memory pool * @min_nr: the minimum number of elements guaranteed to be @@ -186,35 +258,17 @@ mempool_t *mempool_create_node(int min_nr, mempool_alloc_t *alloc_fn, gfp_t gfp_mask, int node_id) { mempool_t *pool; + pool = kzalloc_node(sizeof(*pool), gfp_mask, node_id); if (!pool) return NULL; - pool->elements = kmalloc_array_node(min_nr, sizeof(void *), - gfp_mask, node_id); - if (!pool->elements) { + + if (mempool_init_node(pool, min_nr, alloc_fn, free_fn, pool_data, + gfp_mask, node_id)) { kfree(pool); return NULL; } - spin_lock_init(&pool->lock); - pool->min_nr = min_nr; - pool->pool_data = pool_data; - init_waitqueue_head(&pool->wait); - pool->alloc = alloc_fn; - pool->free = free_fn; - /* - * First pre-allocate the guaranteed number of buffers. - */ - while (pool->curr_nr < pool->min_nr) { - void *element; - - element = pool->alloc(gfp_mask, pool->pool_data); - if (unlikely(!element)) { - mempool_destroy(pool); - return NULL; - } - add_element(pool, element); - } return pool; } EXPORT_SYMBOL(mempool_create_node); -- 2.17.0