Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1758245AbcDADja (ORCPT ); Thu, 31 Mar 2016 23:39:30 -0400 Received: from mx0a-00082601.pphosted.com ([67.231.145.42]:49054 "EHLO mx0a-00082601.pphosted.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1754864AbcDADj3 (ORCPT ); Thu, 31 Mar 2016 23:39:29 -0400 Subject: Re: [PATCHSET v3][RFC] Make background writeback not suck To: Dave Chinner References: <1459350477-16404-1-git-send-email-axboe@fb.com> <20160331082433.GO11812@dastard> <56FD344F.70908@fb.com> <56FD4E70.3090203@fb.com> <20160401005608.GU11812@dastard> <56FDEB1A.2030404@fb.com> CC: , , From: Jens Axboe Message-ID: <56FDED6D.4070200@fb.com> Date: Thu, 31 Mar 2016 21:39:25 -0600 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101 Thunderbird/38.6.0 MIME-Version: 1.0 In-Reply-To: <56FDEB1A.2030404@fb.com> Content-Type: text/plain; charset="windows-1252"; format=flowed Content-Transfer-Encoding: 7bit X-Originating-IP: [192.168.54.13] X-Proofpoint-Spam-Reason: safe X-FB-Internal: Safe X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:,, definitions=2016-04-01_01:,, signatures=0 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 1221 Lines: 25 On 03/31/2016 09:29 PM, Jens Axboe wrote: >>> I can't seem to reproduce this at all. On an nvme device, I get a >>> fairly steady 60K/sec file creation rate, and we're nowhere near >>> being IO bound. So the throttling has no effect at all. >> >> That's too slow to show the stalls - your likely concurrency bound >> in allocation by the default AG count (4) from mkfs. Use mkfs.xfs -d >> agcount=32 so that every thread works in it's own AG. > > That's the key, with that I get 300-400K ops/sec instead. I'll run some > testing with this tomorrow and see what I can find, it did one full run > now and I didn't see any issues, but I need to run it at various > settings and see if I can find the issue. No stalls seen, I get the same performance with it disabled and with it enabled, at both default settings, and lower ones (wb_percent=20). Looking at iostat, we don't drive a lot of depth, so it makes sense, even with the throttling we're doing essentially the same amount of IO. What does 'nr_requests' say for your virtio_blk device? Looks like virtio_blk has a queue_depth setting, but it's not set by default, and then it uses the free entries in the ring. But I don't know what that is... -- Jens Axboe