Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67;
IronPort-SDR: xnXXwp6ziCShDt8RRg0xQCE/VHwlXNOzhFk1suiqE0RgGKCzwuD3VJybdjJPTqSgAVyJoh2WLf
 Jo1zwAH4P6vhudNNofIbUnt4D5Hxg+XzSvnNI5yNrJ92U5XTKGde4qW/hUk5gVxv2hWB+J4XRz
 7P2aYjd+vNxFnlqb6U0tlwaFTLTnYiWFDRf5GBVEPwhycWhHmsmxgtwQPW+6OXWMViQNclKXvR
 RV4B6ENuDiWXCsU+tB05CRAqu/mH0GdoQKEuLIWXe4+RihL6VvVJIVzBARYrnLtY4y5n27mrWb
 ees=
From:   Damien Le Moal <Damien.LeMoal@wdc.com>
To:     "Kirill A. Shutemov" <kirill@shutemov.name>
CC:     Mike Christie <mchristi@redhat.com>,
        "axboe@kernel.dk" <axboe@kernel.dk>,
        "James.Bottomley@HansenPartnership.com" 
        <James.Bottomley@HansenPartnership.com>,
        "martin.petersen@oracle.com" <martin.petersen@oracle.com>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        "linux-scsi@vger.kernel.org" <linux-scsi@vger.kernel.org>,
        "linux-block@vger.kernel.org" <linux-block@vger.kernel.org>,
        Miklos Szeredi <miklos@szeredi.hu>
Subject: Re: [RFC PATCH] Add proc interface to set PF_MEMALLOC flags
Thread-Topic: [RFC PATCH] Add proc interface to set PF_MEMALLOC flags
Thread-Index: AQHVZyuWq/Op5bBFkEiasB8XvOZqww==
Date:   Tue, 10 Sep 2019 13:37:13 +0000
Message-ID: <DM6PR04MB581765CF739A729164C6916EE7B60@DM6PR04MB5817.namprd04.prod.outlook.com>
References: <20190909162804.5694-1-mchristi@redhat.com>
 <20190910100000.mcik63ot6o3dyzjv@box.shutemov.name>
 <DM6PR04MB5817096434DE9381DDB55184E7B60@DM6PR04MB5817.namprd04.prod.outlook.com>
 <20190910124116.74pxl73rybmkl5j3@box>
Accept-Language: en-US
Content-Language: en-US
wdcipoutbound: EOP-TRUE
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-MS-Exchange-CrossTenant-Network-Message-Id: 6f85023a-9f3c-4290-e1c4-08d735f3fd77
X-MS-Exchange-CrossTenant-originalarrivaltime: 10 Sep 2019 13:37:14.0062
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: b61c8803-16f3-4c35-9b17-6f65f441df86
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: ARe3DHAvFYP3B8Lj/Ky2Cdp5xyeblowbah6KKg1aJNOoMFRgEWU9jTOQgET5yyJKxZTlKvN3wyxIxYF5eACTBQ==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM6PR04MB3964
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

+ Miklos=0A=
=0A=
On 2019/09/10 13:41, Kirill A. Shutemov wrote:=0A=
> On Tue, Sep 10, 2019 at 12:05:33PM +0000, Damien Le Moal wrote:=0A=
>> On 2019/09/10 11:00, Kirill A. Shutemov wrote:=0A=
>>> On Mon, Sep 09, 2019 at 11:28:04AM -0500, Mike Christie wrote:=0A=
>>>> There are several storage drivers like dm-multipath, iscsi, and nbd th=
at=0A=
>>>> have userspace components that can run in the IO path. For example,=0A=
>>>> iscsi and nbd's userspace deamons may need to recreate a socket and/or=
=0A=
>>>> send IO on it, and dm-multipath's daemon multipathd may need to send I=
O=0A=
>>>> to figure out the state of paths and re-set them up.=0A=
>>>>=0A=
>>>> In the kernel these drivers have access to GFP_NOIO/GFP_NOFS and the=
=0A=
>>>> memalloc_*_save/restore functions to control the allocation behavior,=
=0A=
>>>> but for userspace we would end up hitting a allocation that ended up=
=0A=
>>>> writing data back to the same device we are trying to allocate for.=0A=
>>>>=0A=
>>>> This patch allows the userspace deamon to set the PF_MEMALLOC* flags=
=0A=
>>>> through procfs. It currently only supports PF_MEMALLOC_NOIO, but=0A=
>>>> depending on what other drivers and userspace file systems need, for=
=0A=
>>>> the final version I can add the other flags for that file or do a file=
=0A=
>>>> per flag or just do a memalloc_noio file.=0A=
>>>>=0A=
>>>> Signed-off-by: Mike Christie <mchristi@redhat.com>=0A=
>>>> ---=0A=
>>>>  Documentation/filesystems/proc.txt |  6 ++++=0A=
>>>>  fs/proc/base.c                     | 53 +++++++++++++++++++++++++++++=
+=0A=
>>>>  2 files changed, 59 insertions(+)=0A=
>>>>=0A=
>>>> diff --git a/Documentation/filesystems/proc.txt b/Documentation/filesy=
stems/proc.txt=0A=
>>>> index 99ca040e3f90..b5456a61a013 100644=0A=
>>>> --- a/Documentation/filesystems/proc.txt=0A=
>>>> +++ b/Documentation/filesystems/proc.txt=0A=
>>>> @@ -46,6 +46,7 @@ Table of Contents=0A=
>>>>    3.10  /proc/<pid>/timerslack_ns - Task timerslack value=0A=
>>>>    3.11	/proc/<pid>/patch_state - Livepatch patch operation state=0A=
>>>>    3.12	/proc/<pid>/arch_status - Task architecture specific informati=
on=0A=
>>>> +  3.13  /proc/<pid>/memalloc - Control task's memory reclaim behavior=
=0A=
>>>>  =0A=
>>>>    4	Configuring procfs=0A=
>>>>    4.1	Mount options=0A=
>>>> @@ -1980,6 +1981,11 @@ Example=0A=
>>>>   $ cat /proc/6753/arch_status=0A=
>>>>   AVX512_elapsed_ms:      8=0A=
>>>>  =0A=
>>>> +3.13 /proc/<pid>/memalloc - Control task's memory reclaim behavior=0A=
>>>> +---------------------------------------------------------------------=
--=0A=
>>>> +A value of "noio" indicates that when a task allocates memory it will=
 not=0A=
>>>> +reclaim memory that requires starting phisical IO.=0A=
>>>> +=0A=
>>>>  Description=0A=
>>>>  -----------=0A=
>>>>  =0A=
>>>> diff --git a/fs/proc/base.c b/fs/proc/base.c=0A=
>>>> index ebea9501afb8..c4faa3464602 100644=0A=
>>>> --- a/fs/proc/base.c=0A=
>>>> +++ b/fs/proc/base.c=0A=
>>>> @@ -1223,6 +1223,57 @@ static const struct file_operations proc_oom_sc=
ore_adj_operations =3D {=0A=
>>>>  	.llseek		=3D default_llseek,=0A=
>>>>  };=0A=
>>>>  =0A=
>>>> +static ssize_t memalloc_read(struct file *file, char __user *buf, siz=
e_t count,=0A=
>>>> +			     loff_t *ppos)=0A=
>>>> +{=0A=
>>>> +	struct task_struct *task;=0A=
>>>> +	ssize_t rc =3D 0;=0A=
>>>> +=0A=
>>>> +	task =3D get_proc_task(file_inode(file));=0A=
>>>> +	if (!task)=0A=
>>>> +		return -ESRCH;=0A=
>>>> +=0A=
>>>> +	if (task->flags & PF_MEMALLOC_NOIO)=0A=
>>>> +		rc =3D simple_read_from_buffer(buf, count, ppos, "noio", 4);=0A=
>>>> +	put_task_struct(task);=0A=
>>>> +	return rc;=0A=
>>>> +}=0A=
>>>> +=0A=
>>>> +static ssize_t memalloc_write(struct file *file, const char __user *b=
uf,=0A=
>>>> +			      size_t count, loff_t *ppos)=0A=
>>>> +{=0A=
>>>> +	struct task_struct *task;=0A=
>>>> +	char buffer[5];=0A=
>>>> +	int rc =3D count;=0A=
>>>> +=0A=
>>>> +	memset(buffer, 0, sizeof(buffer));=0A=
>>>> +	if (count !=3D sizeof(buffer) - 1)=0A=
>>>> +		return -EINVAL;=0A=
>>>> +=0A=
>>>> +	if (copy_from_user(buffer, buf, count))=0A=
>>>> +		return -EFAULT;=0A=
>>>> +	buffer[count] =3D '\0';=0A=
>>>> +=0A=
>>>> +	task =3D get_proc_task(file_inode(file));=0A=
>>>> +	if (!task)=0A=
>>>> +		return -ESRCH;=0A=
>>>> +=0A=
>>>> +	if (!strcmp(buffer, "noio")) {=0A=
>>>> +		task->flags |=3D PF_MEMALLOC_NOIO;=0A=
>>>> +	} else {=0A=
>>>> +		rc =3D -EINVAL;=0A=
>>>> +	}=0A=
>>>=0A=
>>> Really? Without any privilege check? So any random user can tap into=0A=
>>> __GFP_NOIO allocations?=0A=
>>=0A=
>> OK. It probably should have a test on capable(CAP_SYS_ADMIN) or similar.=
 Since=0A=
>> these storage daemons are generally run as root anyway, that would still=
 work=0A=
>> for most setup I think.=0A=
>>=0A=
>>>=0A=
>>> NAK.=0A=
>>>=0A=
>>> I don't think that it's great idea in general to expose this low-level=
=0A=
>>> machinery to userspace. But it's better to get comment from people move=
=0A=
>>> familiar with reclaim path.=0A=
>>=0A=
>> Any setup with stacked file systems and one of the IO path component bei=
ng a=0A=
>> user level process can benefit from this. See the problem described in t=
his=0A=
>> patch I pushed for (unsuccessfully as it was a heavy handed solution):=
=0A=
>> https://www.spinics.net/lists/linux-fsdevel/msg148912.html=0A=
>>=0A=
>> As the discussion in this thread shows, there is no existing simple solu=
tion to=0A=
>> deal with this reclaim recursion problem. And automatic detection is too=
 hard,=0A=
>> if at all possible. With the proper access rights added, this user acces=
sible=0A=
>> interface does look very sensible to me.=0A=
> =0A=
> Looking into the thread, have you find out if there's anything on FUSE=0A=
> side that helps it to avoid deadlocks? Or FUSE just relies on luck with=
=0A=
> this?=0A=
=0A=
I did not see anything relevant. The nofs allocations seem to all be in the=
=0A=
writpage/writepages methods for the client side, to prepare requests to sen=
d to=0A=
the fuse daemon serving them. I think that that is equivalent to a regular =
FS=0A=
(e.g. XFS) using NOFS allocations during writeback on top of the emulated d=
evice=0A=
served by a user level daemon (e.g. tcmu-runner in the problem case I repor=
ted).=0A=
So it does look like a fuse daemon actually serving the request may still=
=0A=
trigger a reclaim into the fuse FS. I wonder if such problem ever was repor=
ted=0A=
or if there are some clever tricks I am missing.=0A=
=0A=
Miklos,=0A=
=0A=
Could you comment on this ? Is there a mechanism in fuse preventing the=0A=
userspace fuse daemon memory triggering a reclaim into the fuse FS being pr=
ocessed ?=0A=
=0A=
Best regards.=0A=
=0A=
-- =0A=
Damien Le Moal=0A=
Western Digital Research=0A=