Subject: Re: [fuse-devel] fuse: max_background and congestion_threshold
 settings
References: <87oa1g90nx.fsf@thinkpad.rath.org>
 <64a57faa-d3a6-a209-8728-723ed7f37c2f@virtuozzo.com>
 <87fumrmdvn.fsf@thinkpad.rath.org>
From: Maxim Patlasov <mpatlasov@virtuozzo.com>
CC: linux-kernel <linux-kernel@vger.kernel.org>,
        Miklos Szeredi <mszeredi@redhat.com>,
        <fuse-devel@lists.sourceforge.net>,
        linux-fsdevel <linux-fsdevel@vger.kernel.org>
To: Nikolaus Rath <Nikolaus@rath.org>
Message-ID: <716677ab-f962-1628-205b-2326219f4487@virtuozzo.com>
Date: Wed, 16 Nov 2016 11:56:10 -0800
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101
 Thunderbird/45.4.0
MIME-Version: 1.0
In-Reply-To: <87fumrmdvn.fsf@thinkpad.rath.org>
Content-Type: text/plain; charset="utf-8"; format=flowed
Content-Transfer-Encoding: 7bit
SpamDiagnosticOutput: 1:99
SpamDiagnosticMetadata: NSPM
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 16 Nov 2016 19:56:24.8113 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-Transport-CrossTenantHeadersStamped: VI1PR0801MB1853
X-OriginatorOrg: virtuozzo.com
Sender: linux-kernel-owner@vger.kernel.org
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org
Content-Length: 2912
Lines: 56

On 11/16/2016 11:19 AM, Nikolaus Rath wrote:

> Hi Maxim,
>
> On Nov 15 2016, Maxim Patlasov <mpatlasov@virtuozzo.com> wrote:
>> On 11/15/2016 08:18 AM, Nikolaus Rath wrote:
>>> Could someone explain to me the meaning of the max_background and
>>> congestion_threshold settings of the fuse module?
>>>
>>> At first I assumed that max_background specifies the maximum number of
>>> pending requests (i.e., requests that have been send to userspace but
>>> for which no reply was received yet). But looking at fs/fuse/dev.c, it
>>> looks as if not every request is included in this number.
>> fuse uses max_background for cases where the total number of
>> simultaneous requests of given type is not limited by some other
>> natural means. AFAIU, these cases are: 1) async processing of direct
>> IO; 2) read-ahead. As an example of "natural" limitation: when
>> userspace process blocks on a sync direct IO read/write, the number of
>> requests fuse consumed is limited by the number of such processes
>> (actually their threads). In contrast, if userspace requests 1GB
>> direct IO read/write, it would be unreasonable to issue 1GB/128K==8192
>> fuse requests simultaneously. That's where max_background steps in.
> Ah, that makes sense. Are these two cases meant as examples, or is that
> an exhaustive list? Because I would have thought that other cases should
> be writing of cached data (when writeback caching is enabled), and
> asynchronous I/O from userspace...?

I think that's exhaustive list, but I can miss something.

As for writing of cached data, that definitely doesn't go through 
background requests. Here we rely on flusher: fuse will allocate as many 
requests as the flusher wants to writeback.

Buffered AIO READs actually block in submit_io until fully processed. So 
it's just another example of "natural" limitation I told above. Buffered 
AIO WRITEs go through writeback mechanics anyway, so here again we rely 
on flusher behaving reasonable. And finally, direct AIO does go through 
fuse background requests as I wrote: "1) async processing of direct IO;"

>
> Also, I am not sure what you mean with async processing of direct
> I/O. Shouldn't direct I/O always go directly to the file-system? If so,
> how can it be processed asynchronously?

That's a nice optimization we implemented a few years ago: having 
incoming sync direct IO request of 1MB size, kernel fuse splits it into 
eight 128K requests and starts processing them in async manner, waiting 
for the completion of all of them before completing that incoming 1MB 
requests. This boosts performance tremendously if userspace fuse daemon 
is able to efficiently process many requests "in parallel". This 
optimization is implemented using background fuse requests. Otherwise, 
having 1GB incoming request, we would obediently allocate 8K fuse 
requests in one shot -- too dangerous and not good for latency.

Thanks,
Maxim