Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:18 as permitted sender) client-ip=2620:137:e000::1:18;
Message-ID: <34dd96b3-e253-de4e-d5d3-a49bc1990e6f@ddn.com>
Date:   Tue, 7 Jun 2022 23:42:16 +0200
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101
 Thunderbird/91.9.1
Subject: Re: [PATCH v4 1/1] Allow non-extending parallel direct writes on the
 same file.
Content-Language: en-US
To:     Vivek Goyal <vgoyal@redhat.com>,
        Dharmendra Singh <dharamhans87@gmail.com>
Cc:     miklos@szeredi.hu, linux-fsdevel@vger.kernel.org,
        fuse-devel@lists.sourceforge.net, linux-kernel@vger.kernel.org,
        Dharmendra Singh <dsingh@ddn.com>
References: <20220605072201.9237-1-dharamhans87@gmail.com>
 <20220605072201.9237-2-dharamhans87@gmail.com> <Yp/CYjONZHoekSVA@redhat.com>
From:   Bernd Schubert <bschubert@ddn.com>
In-Reply-To: <Yp/CYjONZHoekSVA@redhat.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 2
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?NVF6TEkrb2hqeEFlaFdhWVhlOHZzWUt0SCtBd011VWhyenNLVE1xUWNpelBl?=
 =?utf-8?B?MlhQY0sxMGQ1VUVaZURNNG5QUHpab0JXRFdoY3JKWkFkWURPNGpnSEp5anA4?=
 =?utf-8?B?dUJzOG15MWdOQnV1T00rR0k5UC9BOTRVWTdkQldRWHhKWkVoNGNSMVFEcjJk?=
 =?utf-8?B?NGRqMkF4VXhzSUFLQTBFYUV0Q05Gd0g0Y3BNUEtDSWd6aGhsb2lLdVZ2SHVE?=
 =?utf-8?B?b2U0Tmo5QktkWHRNd1dLM0NCYWI0dFA3dFIyUkNxd2M0SnZlb2JtUzI1TmdQ?=
 =?utf-8?B?L3ZYSmZFdVNUaGxXTUlpWDB0a1ZPSE9ZM0xFYjBPNU1TTWo4bEoxbjd6WEhW?=
 =?utf-8?B?WDl6V3hVYXRMWnVCYTBpVG9pZkJVRmRKUnJQME1pQmRhWHkvMjRSYkF4Yk9p?=
 =?utf-8?B?RlB2T0oxTnYrbm1tNWVTK0I0NlVqUC90bzdGcFNHdWhlek5tbFByVTY2Q2hk?=
 =?utf-8?B?NWpGa0dmT2hoWUg1dmF2V1ltTWZwZmIxNldXWjQrYUo1NzFMZTZFOUMyQTRN?=
 =?utf-8?B?a3c1UXRFMFp2aHhkWmdNOXlvcTJkUlRSN0p3L3h2K3NuWjFKOC9hQlhqbEx1?=
 =?utf-8?B?cnFNUzVWNFBZWlExYWRmTmxFdkVKbFBxWm9iVGllM0pHdGNwZW9vNjU1emZq?=
 =?utf-8?B?MFlPRWhRbTRBaXhLWDdBMllsK2Q0MDRJQW54WmZ1dkUrcmk5dGlrTGpQMXc2?=
 =?utf-8?B?dXlWTmRIRlBUMnBWMWxmM3A1UWY2TUZsdllwNjY2NEVEL0h6RUl4R2k4TTRE?=
 =?utf-8?B?S0IvazVyK0VyL0lmZVptZ3RwajFVSFM5SGhBWnJYZmJ0VlVFY3BUUmtwT1JE?=
 =?utf-8?B?REZtbFdHeUpBVU1MRFc4dHVWZiswMTloVEF5bW04ZEU4SVVmcU1TUjFzQWJX?=
 =?utf-8?B?aEUwZUx5b1VaZUgzWStLVlR0VU1jSXhpbVYyWU5peFZwbFBtbHZqMnNWYjR3?=
 =?utf-8?B?TGE1QXk5ZEN2bDEwRGhGR1FPTkVleFNUeVdxeEJ2WDBrT0VGVmVNMXNnd2JE?=
 =?utf-8?B?QldBSEQ1ZnR6cVNxYlc5S0FvanhMd0krTlZuWmFxaGZDQ3d0U0dwYlhaUTZV?=
 =?utf-8?B?d0ZqV2NEb1JsdDlFVDRRa0kzbHZiclVhRFpORXFSNDNwVlpnM29qenRycWti?=
 =?utf-8?B?VlVBdmE1UGxuKzVJaUxnUitxR211VnIwRHBXc3R6dE5LZk8xVlFrS0tnZFJK?=
 =?utf-8?B?U3l3WkRsRXp6cTZsd3hZUENkOVdGeE5raWNTZG1TTnVEcHJ2UGxzbld2WGNn?=
 =?utf-8?B?UkhIbjluSXhNaC9CU1JiRENFQ2pEbVFJNUJKZTBrZngwQ3gwY3R0L0JRRDJj?=
 =?utf-8?B?alZ1dDhGQ3FhdmdrSzRMbXNERGx2aGp1ZmtNaFVaV2l5SzBwOGdEZmZ2QXdq?=
 =?utf-8?B?clRNNEo1R0RNUHMrdTZ5dTZ2M1duejIwUGM1RW94T3VzTHAzYXRRaVVrVmw2?=
 =?utf-8?B?MmpPUzVWV0cxUGFaSUwwSEZIdmF0NGZHV0NKVElTQ3pHSHJjVnpUbHZlNURo?=
 =?utf-8?B?bVpwc3pjKzBwMk04VENTdTFDOEZSYmFSRWw5VlBDS1Mzem82UUtzYzZLRlpr?=
 =?utf-8?B?SjVINEpoM1NtdWE3eEQ2c2V5Y2tSL0ZqMWw3OHkrWDlKWnZicFh5OU5SOVZy?=
 =?utf-8?B?cGdRVVJ5azNUVFBiZWwrMHczOW1yZjBJakZaMXVxSjhnWG1JOHZ0bC8ybldO?=
 =?utf-8?B?a0lvM3gvTWlNUjdFamFaM3JGdFRLaDZxbzlIZGxybkRuSzREWGlaeGtrTno4?=
 =?utf-8?B?TngwQVp0Y3lsTnRoRG9NbkYxODlJSm1UNHl2TjZNdUQyeTZReEcxczlUREpJ?=
 =?utf-8?B?VXd0OEFHRnBVS3FjWndEeTkyUVdLS0d5WkI0bmtrZlRyT2JmOEUyVnNnckU2?=
 =?utf-8?B?VGZSN1FTbEg2WGM1Q2tJOXpkWnpkZ0Y4U2F0cTB0dUF0TDdLK055bUtOelFR?=
 =?utf-8?B?M2UxNUl2a2JiZllCTFVwS2o5b0NIbHNTNE5KeGNjYXUwa3hnV2swRzZMbFI2?=
 =?utf-8?B?Yjl6a3NmMUZSUmhpU0lvRC9sa2o3VGE4UEZabjJqNktZZG1odUhkV0gxcC8r?=
 =?utf-8?B?dlhBNERWbWNqQkpuNGZZcGlzbUlLNU90REloaFc4TTZMQUR1azNrUk04S3FG?=
 =?utf-8?B?ZWhjTzBCSjUxbnBFVDlibXFqaTBhU0cvMXIrMmhFVW5IUEw2QmVCdkdnTnlm?=
 =?utf-8?B?WThuZlA5ekwwc3p6WU9DY0lEdHV6cjdpbDBJQTVPMy95L3JNT0o1NXc3SmlU?=
 =?utf-8?B?UUZvWkxuMGZRTGR3ZnpXd3BYY3JqK2oxL2VVU0xGWmczY2h4ZDdacXlYcHN0?=
 =?utf-8?B?MjNrV0xXVTV1R21aZmdTOUVCUTRnY1hEeGZadzE3dWNQZ1VuTUJhZzRpZjNY?=
 =?utf-8?Q?DhzhLwWO65dlPVHvxYGaDsXHp20lGnxLSVgYeVkaapBj4?=
X-MS-Exchange-AntiSpam-MessageData-1: +kz+iFVuQLdZNA==
X-OriginatorOrg: ddn.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 772aaf9c-dd70-4e97-d5c4-08da48ce9a2f
X-MS-Exchange-CrossTenant-AuthSource: DM5PR1901MB2037.namprd19.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 07 Jun 2022 21:42:21.5354
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 753b6e26-6fd3-43e6-8248-3f1735d59bb4
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: /OqHWoT8FNMDma+j9r1MAZxW7sjY+dwNZnSuZuEjcmHIcIeaxg8aLZrc50VM0qYdgGhERNQa5UdB6eFmZ+54SQ==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR19MB2973
X-Spam-Status: No, score=-3.2 required=5.0 tests=BAYES_00,DKIM_SIGNED,
	DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,
	MAILING_LIST_MULTI,NICE_REPLY_A,RDNS_NONE,SPF_HELO_NONE,
	T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no
	version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
	lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org


On 6/7/22 23:25, Vivek Goyal wrote:
> On Sun, Jun 05, 2022 at 12:52:00PM +0530, Dharmendra Singh wrote:
>> From: Dharmendra Singh <dsingh@ddn.com>
>>
>> In general, as of now, in FUSE, direct writes on the same file are
>> serialized over inode lock i.e we hold inode lock for the full duration
>> of the write request. I could not found in fuse code a comment which
>> clearly explains why this exclusive lock is taken for direct writes.
>>
>> Following might be the reasons for acquiring exclusive lock but not
>> limited to
>> 1) Our guess is some USER space fuse implementations might be relying
>>     on this lock for seralization.
> 
> Hi Dharmendra,
> 
> I will just try to be devil's advocate. So if this is server side
> limitation, then it is possible that fuse client's isize data in
> cache is stale. For example, filesystem is shared between two
> clients.
> 
> - File size is 4G as seen by client A.
> - Client B truncates the file to 2G.
> - Two processes in client A, try to do parallel direct writes and will
>    be able to proceed and server will get two parallel writes both
>    extending file size.
> 
> I can see that this can happen with virtiofs with cache=auto policy.
> 
> IOW, if this is a fuse server side limitation, then how do you ensure
> that fuse kernel's i_size definition is not stale.

Hi Vivek,

I'm sorry, to be sure, can you explain where exactly a client is located 
for you? For us these are multiple daemons linked to libufse - which you 
seem to call 'server' Typically these clients are on different machines. 
And servers are for us on the other side of the network - like an NFS 
server.

So now while I'm not sure what you mean with 'client', I'm wondering 
about two generic questions

a) I need to double check, but we were under the assumption the code in 
question is a direct-io code path. I assume cache=auto would use the 
page cache and should not be effected?

b) How would the current lock help for distributed clients? Or multiple 
fuse daemons (what you seem to call server) per local machine?

For a single vfs mount point served by fuse, truncate should take the 
exclusive lock and parallel writes the shared lock - I don't see a 
problem here either.


Thanks,
Bernd