Subject: Re: [fuse-devel] fuse: max_background and congestion_threshold
 settings
References: <87oa1g90nx.fsf@thinkpad.rath.org>
 <64a57faa-d3a6-a209-8728-723ed7f37c2f@virtuozzo.com>
 <87fumrmdvn.fsf@thinkpad.rath.org>
 <716677ab-f962-1628-205b-2326219f4487@virtuozzo.com>
 <877f83mb2v.fsf@thinkpad.rath.org>
 <7828c809-f699-c16f-a1aa-24ce839547ff@virtuozzo.com>
 <877f7vcewf.fsf@thinkpad.rath.org>
From: Maxim Patlasov <mpatlasov@virtuozzo.com>
CC: <fuse-devel@lists.sourceforge.net>,
        linux-fsdevel <linux-fsdevel@vger.kernel.org>,
        LKML <linux-kernel@vger.kernel.org>, <mszeredi@redhat.com>
To: "Nikolaus@rath.org >> Nikolaus Rath" <Nikolaus@rath.org>
Message-ID: <8484b43a-881e-5da6-f044-9da6597e76db@virtuozzo.com>
Date: Tue, 22 Nov 2016 15:24:53 -0800
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101
 Thunderbird/45.4.0
MIME-Version: 1.0
In-Reply-To: <877f7vcewf.fsf@thinkpad.rath.org>
Content-Type: text/plain; charset="utf-8"; format=flowed
Content-Transfer-Encoding: 7bit
SpamDiagnosticOutput: 1:99
SpamDiagnosticMetadata: NSPM
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 22 Nov 2016 23:25:06.6345 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM5PR0801MB1842
X-OriginatorOrg: virtuozzo.com
Sender: linux-kernel-owner@vger.kernel.org
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org
Content-Length: 3225
Lines: 64

On 11/22/2016 02:45 PM, Nikolaus Rath wrote:

> On Nov 16 2016, Maxim Patlasov <mpatlasov@virtuozzo.com> wrote:
>> On 11/16/2016 12:19 PM, Nikolaus Rath wrote:
>>
>>> On Nov 16 2016, Maxim Patlasov <mpatlasov@virtuozzo.com> wrote:
>>>> On 11/16/2016 11:19 AM, Nikolaus Rath wrote:
>>>>
>>>>> Hi Maxim,
>>>>>
>>>>> On Nov 15 2016, Maxim Patlasov <mpatlasov@virtuozzo.com> wrote:
>>>>>> On 11/15/2016 08:18 AM, Nikolaus Rath wrote:
>>>>>>> Could someone explain to me the meaning of the max_background and
>>>>>>> congestion_threshold settings of the fuse module?
>>>>>>>
>>>>>>> At first I assumed that max_background specifies the maximum number of
>>>>>>> pending requests (i.e., requests that have been send to userspace but
>>>>>>> for which no reply was received yet). But looking at fs/fuse/dev.c, it
>>>>>>> looks as if not every request is included in this number.
>>>>>> fuse uses max_background for cases where the total number of
>>>>>> simultaneous requests of given type is not limited by some other
>>>>>> natural means. AFAIU, these cases are: 1) async processing of direct
>>>>>> IO; 2) read-ahead. As an example of "natural" limitation: when
>>>>>> userspace process blocks on a sync direct IO read/write, the number of
>>>>>> requests fuse consumed is limited by the number of such processes
>>>>>> (actually their threads). In contrast, if userspace requests 1GB
>>>>>> direct IO read/write, it would be unreasonable to issue 1GB/128K==8192
>>>>>> fuse requests simultaneously. That's where max_background steps in.
>>>>> Ah, that makes sense. Are these two cases meant as examples, or is that
>>>>> an exhaustive list? Because I would have thought that other cases should
>>>>> be writing of cached data (when writeback caching is enabled), and
>>>>> asynchronous I/O from userspace...?
>>>> I think that's exhaustive list, but I can miss something.
>>>>
>>>> As for writing of cached data, that definitely doesn't go through
>>>> background requests. Here we rely on flusher: fuse will allocate as
>>>> many requests as the flusher wants to writeback.
>>>>
>>>> Buffered AIO READs actually block in submit_io until fully
>>>> processed. So it's just another example of "natural" limitation I told
>>>> above.
>>> Not sure I understand. What is it that's blocking? It can't be the
>>> userspace process, because then it wouldn't be asynchronous I/O...
>> Surprise! Alas, Linux kernel does NOT process buffered AIO reads in
>> async manner. You can verify it yourself by strace-ing a simple
>> program looping over io_submit + io_getevents: for direct IO (as
>> expected) io_submit returns immediately while io_getevents waits for
>> actual IO; in contrast, for buffered IO (surprisingly) io_submit waits
>> for actual IO while io_getevents returns immediately. Presumably,
>> people are supposed to use mmap-ed read/writes rather than buffered
>> AIO.
> What about buffered, asynchronous writes when writeback cache is
> disabled? It sounds as if io_submit does not block (so userspace could
> create an unlimited number), nor can the kernel coalesce them (since
> writeback caching is disabled).

I've never looked closely at it. Do you have a particular use case or 
concern?


>
> Thanks!
> -Nikolaus
>