Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67;
From:   Roman Gushchin <guro@fb.com>
To:     "Tobin C. Harding" <tobin@kernel.org>
CC:     Andrew Morton <akpm@linux-foundation.org>,
        Christopher Lameter <cl@linux.com>,
        Pekka Enberg <penberg@cs.helsinki.fi>,
        Matthew Wilcox <willy@infradead.org>,
        Tycho Andersen <tycho@tycho.ws>,
        "linux-mm@kvack.org" <linux-mm@kvack.org>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>
Subject: Re: [RFC 02/15] slub: Add isolate() and migrate() methods
Thread-Topic: [RFC 02/15] slub: Add isolate() and migrate() methods
Thread-Index: AQHU1WWOzoo+8Mo/g0+TXm2HFlFybKYG/kMA
Date:   Mon, 11 Mar 2019 21:51:09 +0000
Message-ID: <20190311215106.GA7915@tower.DHCP.thefacebook.com>
References: <20190308041426.16654-1-tobin@kernel.org>
 <20190308041426.16654-3-tobin@kernel.org>
In-Reply-To: <20190308041426.16654-3-tobin@kernel.org>
Accept-Language: en-US
Content-Language: en-US
x-ms-exchange-messagesentrepresentingtype: 1
x-originating-ip: [2620:10d:c090:200::1:b487]
x-ms-publictraffictype: Email
x-ms-office365-filtering-correlation-id: 2e2de05f-6db7-4f1a-f655-08d6a66babeb
x-microsoft-antispam: BCL:0;PCL:0;RULEID:(2390118)(7020095)(4652040)(8989299)(4534185)(4627221)(201703031133081)(201702281549075)(8990200)(5600127)(711020)(4605104)(2017052603328)(7153060)(7193020);SRVR:BYAPR15MB2262;
x-ms-traffictypediagnostic: BYAPR15MB2262:
x-microsoft-exchange-diagnostics: 1;BYAPR15MB2262;20:wnEeT3LekANxezNI+AW2CCjJaLmhkbMgV18YtI3B3g/Zx9CMBMqhhw+bwJN+pOnBfwgmwTXT/0SWoVwD0M4V+VhQpLTkGsKZ1vve9gD+khoLWVgt/gbVBssE/iZ2v+l+paBCvvLCOu4sZrE3snR8lJqKOJb2v37nAGorJAEm8v0=
x-microsoft-antispam-prvs: <BYAPR15MB2262E707A7F8CA7739F49F19BE480@BYAPR15MB2262.namprd15.prod.outlook.com>
x-forefront-prvs: 09730BD177
x-forefront-antispam-report: SFV:NSPM;SFS:(10019020)(39860400002)(396003)(366004)(136003)(346002)(376002)(54094003)(199004)(189003)(5660300002)(6116002)(53936002)(316002)(6486002)(54906003)(81166006)(478600001)(68736007)(305945005)(386003)(6512007)(99286004)(81156014)(76176011)(1076003)(52116002)(6436002)(6506007)(8936002)(9686003)(7736002)(25786009)(105586002)(229853002)(6246003)(2906002)(8676002)(97736004)(186003)(4326008)(106356001)(6916009)(14454004)(71190400001)(446003)(71200400001)(11346002)(486006)(476003)(33656002)(46003)(14444005)(256004)(102836004)(86362001);DIR:OUT;SFP:1102;SCL:1;SRVR:BYAPR15MB2262;H:BYAPR15MB2631.namprd15.prod.outlook.com;FPR:;SPF:None;LANG:en;PTR:InfoNoRecords;A:1;MX:1;
received-spf: None (protection.outlook.com: fb.com does not designate
 permitted sender hosts)
x-ms-exchange-senderadcheck: 1
x-microsoft-antispam-message-info: 4dCundw96p/+F4EB4H9V1wlBa7XyIpoEtYMIzLLfyOpGW2mAS8FVkxgo2r2Pmv1YGIb7cq4hZ++H25ZcpiItBEh8KdT+67S/FMzQRhD79z6VONuzXlET2XFGdmAO0g4GvabgMnyJigP8jprnjvzZ+1v8OvD41ybCL3PNm9FCfqtAQCfG/o2BMRA/MaL7G6WRouOA3O7xoKK5KWU7KIzorU14m4KO7dUyToxjRHhBLycQEfBGABOmcN2DINvymD1exsD9VZWNx+FfgYz5WRbba6C6wAx8cokSjsIxwCTJ0Zb9D6PvjpXKJgovCyltmwMBMM75ibKZT5sj8Hdw3qN8PPN2vhtfZB2IX6J8xYDHoJV2HI9WjyQe8MiVrKk/W9oX9W14vDHlD7RyfOoETmAzu6cJjvgP8ba7r/8wfsUCUvk=
Content-Type: text/plain; charset="us-ascii"
Content-ID: <C887680CD53FAB4D95B7FFA57CF5A72D@namprd15.prod.outlook.com>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-MS-Exchange-CrossTenant-Network-Message-Id: 2e2de05f-6db7-4f1a-f655-08d6a66babeb
X-MS-Exchange-CrossTenant-originalarrivaltime: 11 Mar 2019 21:51:09.8285
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 8ae927fe-1255-47a7-a2af-5f3a069daaa2
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BYAPR15MB2262
X-OriginatorOrg: fb.com
X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:,, definitions=2019-03-11_16:,,
 signatures=0
X-Proofpoint-Spam-Reason: safe
X-FB-Internal: Safe
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On Fri, Mar 08, 2019 at 03:14:13PM +1100, Tobin C. Harding wrote:
> Add the two methods needed for moving objects and enable the display of
> the callbacks via the /sys/kernel/slab interface.
>=20
> Add documentation explaining the use of these methods and the prototypes
> for slab.h. Add functions to setup the callbacks method for a slab
> cache.
>=20
> Add empty functions for SLAB/SLOB. The API is generic so it could be
> theoretically implemented for these allocators as well.
>=20
> Co-developed-by: Christoph Lameter <cl@linux.com>
> Signed-off-by: Tobin C. Harding <tobin@kernel.org>
> ---
>  include/linux/slab.h     | 69 ++++++++++++++++++++++++++++++++++++++++
>  include/linux/slub_def.h |  3 ++
>  mm/slab_common.c         |  4 +++
>  mm/slub.c                | 42 ++++++++++++++++++++++++
>  4 files changed, 118 insertions(+)
>=20
> diff --git a/include/linux/slab.h b/include/linux/slab.h
> index 11b45f7ae405..22e87c41b8a4 100644
> --- a/include/linux/slab.h
> +++ b/include/linux/slab.h
> @@ -152,6 +152,75 @@ void memcg_create_kmem_cache(struct mem_cgroup *, st=
ruct kmem_cache *);
>  void memcg_deactivate_kmem_caches(struct mem_cgroup *);
>  void memcg_destroy_kmem_caches(struct mem_cgroup *);
> =20
> +/*
> + * Function prototypes passed to kmem_cache_setup_mobility() to enable
> + * mobile objects and targeted reclaim in slab caches.
> + */
> +
> +/**
> + * typedef kmem_cache_isolate_func - Object migration callback function.
> + * @s: The cache we are working on.
> + * @ptr: Pointer to an array of pointers to the objects to migrate.
> + * @nr: Number of objects in array.
> + *
> + * The purpose of kmem_cache_isolate_func() is to pin each object so tha=
t
> + * they cannot be freed until kmem_cache_migrate_func() has processed
> + * them. This may be accomplished by increasing the refcount or setting
> + * a flag.
> + *
> + * The object pointer array passed is also passed to
> + * kmem_cache_migrate_func().  The function may remove objects from the
> + * array by setting pointers to NULL. This is useful if we can determine
> + * that an object is being freed because kmem_cache_isolate_func() was
> + * called when the subsystem was calling kmem_cache_free().  In that
> + * case it is not necessary to increase the refcount or specially mark
> + * the object because the release of the slab lock will lead to the
> + * immediate freeing of the object.
> + *
> + * Context: Called with locks held so that the slab objects cannot be
> + *          freed.  We are in an atomic context and no slab operations
> + *          may be performed.
> + * Return: A pointer that is passed to the migrate function. If any
> + *         objects cannot be touched at this point then the pointer may
> + *         indicate a failure and then the migration function can simply
> + *         remove the references that were already obtained. The private
> + *         data could be used to track the objects that were already pin=
ned.
> + */
> +typedef void *kmem_cache_isolate_func(struct kmem_cache *s, void **ptr, =
int nr);
> +
> +/**
> + * typedef kmem_cache_migrate_func - Object migration callback function.
> + * @s: The cache we are working on.
> + * @ptr: Pointer to an array of pointers to the objects to migrate.
> + * @nr: Number of objects in array.
> + * @node: The NUMA node where the object should be allocated.
> + * @private: The pointer returned by kmem_cache_isolate_func().
> + *
> + * This function is responsible for migrating objects.  Typically, for
> + * each object in the input array you will want to allocate an new
> + * object, copy the original object, update any pointers, and free the
> + * old object.
> + *
> + * After this function returns all pointers to the old object should now
> + * point to the new object.
> + *
> + * Context: Called with no locks held and interrupts enabled.  Sleeping
> + *          is possible.  Any operation may be performed.
> + */
> +typedef void kmem_cache_migrate_func(struct kmem_cache *s, void **ptr,
> +				     int nr, int node, void *private);
> +
> +/*
> + * kmem_cache_setup_mobility() is used to setup callbacks for a slab cac=
he.
> + */
> +#ifdef CONFIG_SLUB
> +void kmem_cache_setup_mobility(struct kmem_cache *, kmem_cache_isolate_f=
unc,
> +			       kmem_cache_migrate_func);
> +#else
> +static inline void kmem_cache_setup_mobility(struct kmem_cache *s,
> +	kmem_cache_isolate_func isolate, kmem_cache_migrate_func migrate) {}
> +#endif
> +
>  /*
>   * Please use this macro to create slab caches. Simply specify the
>   * name of the structure and maybe some flags that are listed above.
> diff --git a/include/linux/slub_def.h b/include/linux/slub_def.h
> index 3a1a1dbc6f49..a7340a1ed5dc 100644
> --- a/include/linux/slub_def.h
> +++ b/include/linux/slub_def.h
> @@ -99,6 +99,9 @@ struct kmem_cache {
>  	gfp_t allocflags;	/* gfp flags to use on each alloc */
>  	int refcount;		/* Refcount for slab cache destroy */
>  	void (*ctor)(void *);
> +	kmem_cache_isolate_func *isolate;
> +	kmem_cache_migrate_func *migrate;
> +
>  	unsigned int inuse;		/* Offset to metadata */
>  	unsigned int align;		/* Alignment */
>  	unsigned int red_left_pad;	/* Left redzone padding size */
> diff --git a/mm/slab_common.c b/mm/slab_common.c
> index f9d89c1b5977..754acdb292e4 100644
> --- a/mm/slab_common.c
> +++ b/mm/slab_common.c
> @@ -298,6 +298,10 @@ int slab_unmergeable(struct kmem_cache *s)
>  	if (!is_root_cache(s))
>  		return 1;
> =20
> +	/*
> +	 * s->isolate and s->migrate imply s->ctor so no need to
> +	 * check them explicitly.
> +	 */
>  	if (s->ctor)
>  		return 1;
> =20
> diff --git a/mm/slub.c b/mm/slub.c
> index 69164aa7cbbf..0133168d1089 100644
> --- a/mm/slub.c
> +++ b/mm/slub.c
> @@ -4325,6 +4325,34 @@ int __kmem_cache_create(struct kmem_cache *s, slab=
_flags_t flags)
>  	return err;
>  }
> =20
> +void kmem_cache_setup_mobility(struct kmem_cache *s,
> +			       kmem_cache_isolate_func isolate,
> +			       kmem_cache_migrate_func migrate)
> +{

I wonder if it's better to adapt kmem_cache_create() to take two additional
argument? I suspect mobility is not a dynamic option, so it can be
set on kmem_cache creation.

> +	/*
> +	 * Mobile objects must have a ctor otherwise the object may be
> +	 * in an undefined state on allocation.  Since the object may
> +	 * need to be inspected by the migration function at any time
> +	 * after allocation we must ensure that the object always has a
> +	 * defined state.
> +	 */
> +	if (!s->ctor) {
> +		pr_err("%s: cannot setup mobility without a constructor\n",
> +		       s->name);
> +		return;
> +	}
> +
> +	s->isolate =3D isolate;
> +	s->migrate =3D migrate;
> +
> +	/*
> +	 * Sadly serialization requirements currently mean that we have
> +	 * to disable fast cmpxchg based processing.
> +	 */

Can you, please, elaborate a bit more here?

> +	s->flags &=3D ~__CMPXCHG_DOUBLE;
> +}
> +EXPORT_SYMBOL(kmem_cache_setup_mobility);
> +
>  void *__kmalloc_track_caller(size_t size, gfp_t gfpflags, unsigned long =
caller)
>  {
>  	struct kmem_cache *s;
> @@ -5018,6 +5046,20 @@ static ssize_t ops_show(struct kmem_cache *s, char=
 *buf)
> =20
>  	if (s->ctor)
>  		x +=3D sprintf(buf + x, "ctor : %pS\n", s->ctor);
> +
> +	if (s->isolate) {
> +		x +=3D sprintf(buf + x, "isolate : ");
> +		x +=3D sprint_symbol(buf + x,
> +				(unsigned long)s->isolate);
> +		x +=3D sprintf(buf + x, "\n");
> +	}

Is there a reason why s->ctor and s->isolate/migrate are printed
using different methods?

> +
> +	if (s->migrate) {
> +		x +=3D sprintf(buf + x, "migrate : ");
> +		x +=3D sprint_symbol(buf + x,
> +				(unsigned long)s->migrate);
> +		x +=3D sprintf(buf + x, "\n");
> +	}
>  	return x;
>  }
>  SLAB_ATTR_RO(ops);
> --=20
> 2.21.0
>=20

Thanks!