2013-03-18 07:15:53

by chendt

[permalink] [raw]
Subject: pnfs: WRITE operation will not send LAYOUTCOMMI OP to MDS if use filelayout

In the recent pnfs kernel versions, WRITE operation will not send
LAYOUTCOMMI OP to MDS if use filelayout.

In nfs_write_rpcsetup function, the value of the variable 'stable' will
always be NFS_FILE_SYNC(ncommit is 0). so NFS_INO_LAYOUTCOMMIT
flag will not be set.

The value of the variable ncommit' will be ++ in the later process
(nfs_writeback_release_common function).


Thanks!



2013-03-18 12:57:42

by Myklebust, Trond

[permalink] [raw]
Subject: RE: pnfs: WRITE operation will not send LAYOUTCOMMI OP to MDS if use filelayout

PiAtLS0tLU9yaWdpbmFsIE1lc3NhZ2UtLS0tLQ0KPiBGcm9tOiBsaW51eC1uZnMtb3duZXJAdmdl
ci5rZXJuZWwub3JnIFttYWlsdG86bGludXgtbmZzLQ0KPiBvd25lckB2Z2VyLmtlcm5lbC5vcmdd
IE9uIEJlaGFsZiBPZiBjaGVuZGl0YW5nDQo+IFNlbnQ6IE1vbmRheSwgTWFyY2ggMTgsIDIwMTMg
MzoxMiBBTQ0KPiBUbzogYmhhbGV2eUB0b25pYW4uY29tDQo+IENjOiBsaW51eC1uZnNAdmdlci5r
ZXJuZWwub3JnOyBMSU5FDQo+IFN1YmplY3Q6IHBuZnM6IFdSSVRFIG9wZXJhdGlvbiB3aWxsIG5v
dCBzZW5kIExBWU9VVENPTU1JIE9QIHRvIE1EUyBpZg0KPiB1c2UgZmlsZWxheW91dA0KPiANCj4g
SW4gdGhlIHJlY2VudCBwbmZzIGtlcm5lbCB2ZXJzaW9ucywgV1JJVEUgb3BlcmF0aW9uIHdpbGwg
bm90IHNlbmQNCj4gTEFZT1VUQ09NTUkgT1AgdG8gTURTIGlmIHVzZSBmaWxlbGF5b3V0Lg0KPiAN
Cj4gSW4gbmZzX3dyaXRlX3JwY3NldHVwIGZ1bmN0aW9uLCB0aGUgdmFsdWUgb2YgdGhlIHZhcmlh
YmxlICdzdGFibGUnIHdpbGwgYWx3YXlzDQo+IGJlIE5GU19GSUxFX1NZTkMobmNvbW1pdCBpcyAw
KS4gc28gTkZTX0lOT19MQVlPVVRDT01NSVQgZmxhZyB3aWxsIG5vdA0KPiBiZSBzZXQuDQoNCklm
IHRoZXNlIGFyZSBzbWFsbCBmaWxlcywgdGhlbiB0aGF0IGlzIGV4cGVjdGVkIGJlaGF2aW91ci4g
VGhlIHBORlMgc3BlYyBkb2VzIG5vdCBleGVtcHQgdGhlIERTIGZyb20gaGF2aW5nIHRvIGhvbm91
ciB0aGUgJ3N0YWJsZScgZmxhZy4NCg0KQ2hlZXJzDQogIFRyb25kDQo=

2013-03-18 10:51:05

by chendt

[permalink] [raw]
Subject: Re: pnfs: WRITE operation will not send LAYOUTCOMMI OP to MDS if use filelayout

于 2013年03月18日 17:01, Benny Halevy 写道:
> On 2013-03-18 09:11, chenditang wrote:
>> In the recent pnfs kernel versions, WRITE operation will not send
>> LAYOUTCOMMI OP to MDS if use filelayout.
> I presume that the file server sets NFL4_UFLG_COMMIT_THRU_MDS
> and that the DS is returning how=FILE_SYNC and that the client
> is implementing the behavior described in Errata ID: 2751
> http://www.rfc-editor.org/errata_search.php?rfc=5661
>
> Do you have a tcp dump of client, MDS, and DS traffic?
NFL4_UFLG_COMMIT_THRU_MDS is not set, "nfl_util" is 8192 and "commit_through_mds" is 0 to filelayout_decode_layout.

>> In nfs_write_rpcsetup function, the value of the variable 'stable' will
>> always be NFS_FILE_SYNC(ncommit is 0). so NFS_INO_LAYOUTCOMMIT
>> flag will not be set.
> What's the value of the "how" arg to nfs_write_rpcsetup that you see?
> At any rate, if you look at filelayout_set_layoutcommit, it will not
> call pnfs_set_layoutcommit in the following cases:
>
> if (FILELAYOUT_LSEG(hdr->lseg)->commit_through_mds ||
> wdata->res.verf->committed == NFS_FILE_SYNC)
> return;
"how" is 32, and "cinfo->mds->ncommit" is 0. pnfs_set_layoutcommit has
not beeb called.
I think "cinfo->mds->ncommit" can not be used here.

I dump the stack, and the initial value of "ncommit" is 0, then it to be
used in nfs_write_rpcsetup function,
but "ncommit++" in nfs_writeback_release_common function(if
NFS_FILE_SYNC is not set).

static const struct rpc_call_ops nfs_write_common_ops = {
.rpc_call_prepare = nfs_write_prepare,
.rpc_call_done = nfs_writeback_done_common,
.rpc_release = nfs_writeback_release_common,
};

The code processes:
step1: init_once --> nfsi->commit_info.ncommit = 0;
setp2: nfs_flush_one --> nfs_write_rpcsetup
setp3: nfs_writeback_done --> nfs_writeback_done -->
filelayout_write_done_cb --> filelayout_set_layoutcommit
setp4: nfs_writeback_release_common--> ...-->nfs_write_completion -->
.... --> cinfo->mds->ncommit++;
>
>
> Benny
>
>> The value of the variable ncommit' will be ++ in the later process
>> (nfs_writeback_release_common function).
>>
>>
>> Thanks!
>>




2013-03-18 09:01:47

by Benny Halevy

[permalink] [raw]
Subject: Re: pnfs: WRITE operation will not send LAYOUTCOMMI OP to MDS if use filelayout

On 2013-03-18 09:11, chenditang wrote:
> In the recent pnfs kernel versions, WRITE operation will not send
> LAYOUTCOMMI OP to MDS if use filelayout.

I presume that the file server sets NFL4_UFLG_COMMIT_THRU_MDS
and that the DS is returning how=FILE_SYNC and that the client
is implementing the behavior described in Errata ID: 2751
http://www.rfc-editor.org/errata_search.php?rfc=5661

Do you have a tcp dump of client, MDS, and DS traffic?

>
> In nfs_write_rpcsetup function, the value of the variable 'stable' will
> always be NFS_FILE_SYNC(ncommit is 0). so NFS_INO_LAYOUTCOMMIT
> flag will not be set.

What's the value of the "how" arg to nfs_write_rpcsetup that you see?
At any rate, if you look at filelayout_set_layoutcommit, it will not
call pnfs_set_layoutcommit in the following cases:

if (FILELAYOUT_LSEG(hdr->lseg)->commit_through_mds ||
wdata->res.verf->committed == NFS_FILE_SYNC)
return;

Benny

>
> The value of the variable ncommit' will be ++ in the later process
> (nfs_writeback_release_common function).
>
>
> Thanks!
>