Received: by 2002:a89:d88:0:b0:1fa:5c73:8e2d with SMTP id eb8csp691984lqb; Fri, 24 May 2024 10:09:26 -0700 (PDT) X-Forwarded-Encrypted: i=3; AJvYcCV8y6BpKT/7E475zbvP8De9v84UsYc0iAX8BbWPchhpXpLS7Wl20YY/6fW6BKti6gIHrWFiClCtUeYagapBHCglwBl4wMJLr9Wf5zNnhw== X-Google-Smtp-Source: AGHT+IGDomLdedNfLIsWgtj+A0/hlsrMWoA9/EIVtZtB5474d9Bnjhk1/zJ/9NLpoL95fnbjtK4c X-Received: by 2002:a05:6602:22cc:b0:7e1:b4b2:d708 with SMTP id ca18e2360f4ac-7e8c471228dmr363315339f.4.1716570566409; Fri, 24 May 2024 10:09:26 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1716570566; cv=pass; d=google.com; s=arc-20160816; b=rDrWcUjoRQuWl9Az4CDfx9/VVK/8NGZZag0lBEAbvcx0+Qomvvz/io/JaziojqdNKf kQnj5TKrzUtCBHieOlyKm2EcZOrUQ+TSLHlei9bXKrel77LdLYMwLxOV5rBJFwAPPFIq WmjDGrmG6u/yiHFtyk1U/GQ4mdovqVPW5zOo6EiCZJXGMZrvBWXLKQNs0YMseub4nunn XAK7Sss0pBWe/1gSv/gbjrdM897jQ/p3eJEYhjSvQ6NS294NAE318ki4ADS5rxMn4VWe GE0KAg/tIgs0QiiF/m3H7yEduE9uOrTqh8rWo+G3mRyFrNbtMapVko0qz2eLM5qAUlL4 PpzA== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:list-unsubscribe:list-subscribe:list-id:precedence :dkim-signature; bh=1hNE1DUO9EWuL5FjLuf7/UTJBlZf8rRfdN2q01WBfOw=; fh=RKUCPljsk+mRGAkdNgxXhgFAnOuKzGVEN7HPnkTs3yw=; b=ssKg9TmtWBDrzURpLiBTXdKZWzNs/FSKhDVKAExnIeN99r1CsDZM/2EaB+eMqoN/R0 qXcECAWUZvk22kzHt8cyXQOEBxeGmRpeQEP3xQf/dCd73cdbOYy2L+mrzyb4VHoYpYqq YusDb2ybtPJYV88zpJFqmdW0pHXvP6bm4G8KQLG/x72qQ4F65nETZivdr3Pa+LH/alvR 46CgNKImrRkyjUzjNWtlt9Yvmde4SGPy9pgfArrSBVXUbu4WyoGj4VbJX1dTl5nhzTQG OdrT5AN9sTWnVMKmmWHsvUnXvLxdfJUqw0lHq/eZd18mPcDazd7jXNvjByQksl4bUVu/ VoWA==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@gmail.com header.s=20230601 header.b=PnRJH1jO; arc=pass (i=1 spf=pass spfdomain=gmail.com dkim=pass dkdomain=gmail.com dmarc=pass fromdomain=gmail.com); spf=pass (google.com: domain of linux-nfs+bounces-3376-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) smtp.mailfrom="linux-nfs+bounces-3376-linux.lists.archive=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from sv.mirrors.kernel.org (sv.mirrors.kernel.org. [2604:1380:45e3:2400::1]) by mx.google.com with ESMTPS id 41be03b00d2f7-68229a2b7bdsi1560114a12.646.2024.05.24.10.09.26 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 24 May 2024 10:09:26 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-nfs+bounces-3376-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) client-ip=2604:1380:45e3:2400::1; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20230601 header.b=PnRJH1jO; arc=pass (i=1 spf=pass spfdomain=gmail.com dkim=pass dkdomain=gmail.com dmarc=pass fromdomain=gmail.com); spf=pass (google.com: domain of linux-nfs+bounces-3376-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) smtp.mailfrom="linux-nfs+bounces-3376-linux.lists.archive=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sv.mirrors.kernel.org (Postfix) with ESMTPS id C553C282111 for ; Fri, 24 May 2024 17:09:25 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id A82F3376EB; Fri, 24 May 2024 17:09:21 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="PnRJH1jO" X-Original-To: linux-nfs@vger.kernel.org Received: from mail-lj1-f170.google.com (mail-lj1-f170.google.com [209.85.208.170]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0934E12EBD5 for ; Fri, 24 May 2024 17:09:19 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.208.170 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1716570561; cv=none; b=oAdNsOLrEhg8PHKAkGw+D3FwQ4BxZw71rRbOs86i5SEZpWpjuFIhDP3hPWxKUSEzIIzrC6kieO780kZgv/dMPGp3eLA2/5R8a59h8F9RdW5knG03I54ioq6XjGOg96HYVCEXeVbWJL5SHlBlowPbMWLvUovcAuma0d4Vhs1GTAI= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1716570561; c=relaxed/simple; bh=qNIgV+zYuCw6AQg4t53gzr/BasCfHNnPNFYeg/DxxwE=; h=MIME-Version:References:In-Reply-To:From:Date:Message-ID:Subject: To:Cc:Content-Type; b=R2gmIo0+DR1t2kKws14k7BMcR9CZ5IMaPghpPRrfigYg9Tg38gA6q0SmQlJoicSIpFZZmMI5hIXxng6zDP0xIG6I6sYrrm3ds/tT29c7O7umxKHKR6j4bxFb/BnIk7IThsC7QDT8a7OqPvKVamnRBQnl/JYCBqUQ41stC5q69vY= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=PnRJH1jO; arc=none smtp.client-ip=209.85.208.170 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Received: by mail-lj1-f170.google.com with SMTP id 38308e7fff4ca-2e95afec7e6so8743361fa.0 for ; Fri, 24 May 2024 10:09:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1716570558; x=1717175358; darn=vger.kernel.org; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=1hNE1DUO9EWuL5FjLuf7/UTJBlZf8rRfdN2q01WBfOw=; b=PnRJH1jOYPj05BatbL5SKnjC5Q7CJwrdOeY1UkttNJ1G3JUgoQy5p6vAGun4/ih37I lLeAcyhDZitYvIGUXTiqVmCW7TH2da8VquQLD6nPvIaCRpd8J/+5HJaYf6OigmK/GlrI yozthzSvOSTegK0YbxQ/o8Vb6vaSi8F7Zz42ZYLZseBn0QPWOptQpCWU344t7ws6RqV2 nUqXaiJjNvAH/kprNflZuL+b7ix88VMJLf4hJOgxi06pFSDxs6bJLumb+f3RvsTL50jR TT0r83/KamxcOm+vralpbSoHPLc8q0Vsv3444T+JiExVQpgFPKD98rMgbDzejszMl/V4 +NYg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1716570558; x=1717175358; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=1hNE1DUO9EWuL5FjLuf7/UTJBlZf8rRfdN2q01WBfOw=; b=ZAytRJ6M1D31IXVO4HNtG+tMFq2eSo6GHuFFmUwJSDitlY3UScS4zlLlL2qkUS0xYi L8z9bvXLmZ9yQNQWVxMVL/YNuaDMvtrJKGU/VR2apnKVRd5ss+4cLwxfOWjgOpOQqSbG tnLgh8GgRhbUp/0YRXBUyCQ3b25KsPw9yOM2BtbqPTg41byuvnKkLS7x0/rkiW2kYrf1 RUqkNVYcUMFPI7beYyxZFpyVhDHVFuCGk94LTyQtgYgbU3W01Vm+zSxg6iX9mlzoHD6D AgWNPU9Ien6opcLF/h7a3yPVH/8WBvCQ3fev5nLFxJlX+kWbJAacUCSdnotzCYoanJnD f4tA== X-Gm-Message-State: AOJu0YyVuxfF7tpocsOkmbeUBQypB6M4W/2VAyr1wOqDMVg6+huLFUBq 3OBYpMCvRsBPXtQ3bwBLhqJp7AoTxpVVce/6ujSYudU+GvEDvBBUlsynPXOZOIXMFVriiAqpaRP r3yFAOZ9HlQG8O6hwgnQSy5vsF7uXQA== X-Received: by 2002:a2e:2205:0:b0:2e9:58d9:6d8 with SMTP id 38308e7fff4ca-2e958d90769mr11029411fa.7.1716570557865; Fri, 24 May 2024 10:09:17 -0700 (PDT) Precedence: bulk X-Mailing-List: linux-nfs@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 References: <20240523165436.g5xgo7aht7dtmvfb@quack3> In-Reply-To: <20240523165436.g5xgo7aht7dtmvfb@quack3> From: Dan Shelton Date: Fri, 24 May 2024 19:08:51 +0200 Message-ID: Subject: Re: Bad NFS performance for fsync(2) To: Jan Kara Cc: linux-nfs@vger.kernel.org, Chuck Lever , Jeff Layton , Trond Myklebust , Anna Schumaker Content-Type: text/plain; charset="UTF-8" On Thu, 23 May 2024 at 18:55, Jan Kara wrote: > > Hello! > > I've been debugging NFS performance regression with recent kernels. It > seems to be at least partially related to the following behavior of NFS > (which is there for a long time AFAICT). Suppose the following workload: > > fio --direct=0 --ioengine=sync --thread --directory=/test --invalidate=1 \ > --group_reporting=1 --runtime=100 --fallocate=posix --ramp_time=10 \ > --name=RandomWrites-async --new_group --rw=randwrite --size=32000m \ > --numjobs=4 --bs=4k --fsync_on_close=1 --end_fsync=1 \ > --filename_format='FioWorkloads.$jobnum' Where do you get the fio command from? Dan -- Dan Shelton - Cluster Specialist Win/Lin/Bsd