2023-05-04 10:21:14

by Petr Vorel

[permalink] [raw]
Subject: LTP: tirpc_rpcb_rmtcall is failing

Hi Steve,

tirpc_rpcb_rmtcall is failing. I was able to reproduce it on
* openSUSE Tumbleweed with libtirpc 1.3.3
* Debian stable 11 (bullseye) with libtirpc 1.3.1-1

OTOH SLE 15-SP4 with libtirpc 1.2.6 is working.

PATH="/opt/ltp/testcases/bin:$PATH" rpc_test.sh -s tirpc_svc_4 -c tirpc_rpcb_rmtcall
rpc_test 1 TINFO: initialize 'lhost' 'ltp_ns_veth2' interface
rpc_test 1 TINFO: add local addr 10.0.0.2/24
rpc_test 1 TINFO: add local addr fd00:1:1:1::2/64
rpc_test 1 TINFO: initialize 'rhost' 'ltp_ns_veth1' interface
rpc_test 1 TINFO: add remote addr 10.0.0.1/24
rpc_test 1 TINFO: add remote addr fd00:1:1:1::1/64
rpc_test 1 TINFO: Network config (local -- remote):
rpc_test 1 TINFO: ltp_ns_veth2 -- ltp_ns_veth1
rpc_test 1 TINFO: 10.0.0.2/24 -- 10.0.0.1/24
rpc_test 1 TINFO: fd00:1:1:1::2/64 -- fd00:1:1:1::1/64
rpc_test 1 TINFO: timeout per run is 0h 5m 0s
rpc_test 1 TINFO: check registered RPC with rpcinfo
rpc_test 1 TINFO: registered RPC:
program vers proto port service
100000 4 tcp 111 portmapper
100000 3 tcp 111 portmapper
100000 2 tcp 111 portmapper
100000 4 udp 111 portmapper
100000 3 udp 111 portmapper
100000 2 udp 111 portmapper
100005 1 udp 20048 mountd
100005 1 tcp 20048 mountd
100005 2 udp 20048 mountd
100005 2 tcp 20048 mountd
100005 3 udp 20048 mountd
100005 3 tcp 20048 mountd
100024 1 udp 37966 status
100024 1 tcp 43643 status
100003 3 tcp 2049 nfs
100003 4 tcp 2049 nfs
100227 3 tcp 2049 nfs_acl
100021 1 udp 59603 nlockmgr
100021 3 udp 59603 nlockmgr
100021 4 udp 59603 nlockmgr
100021 1 tcp 39145 nlockmgr
100021 3 tcp 39145 nlockmgr
100021 4 tcp 39145 nlockmgr
rpc_test 1 TINFO: using libtirpc: yes
rpc_test 1 TFAIL: tirpc_rpcb_rmtcall 10.0.0.2 536875000 failed unexpectedly
1

The problem is in tirpc_rpcb_rmtcall.c [1], which calls rpcb_rmtcall(), which
returns 1 (I suppose RPC_CANTENCODEARGS - can't encode arguments - enum
clnt_stat from tirpc/rpc/clnt_stat.h):

cs = rpcb_rmtcall(nconf, argc[1], progNum, VERSNUM, PROCNUM,
(xdrproc_t) xdr_int, (char *)&var_snd,
(xdrproc_t) xdr_int, (char *)&var_rec, tv, &svcaddr);

test_status = (cs == RPC_SUCCESS) ? 0 : 1;

//This last printf gives the result status to the tests suite
//normally should be 0: test has passed or 1: test has failed
printf("%d\n", test_status);

return test_status;

Any idea what could be wrong with these very old tests?

Kind regards,
Petr

[1] https://github.com/linux-test-project/ltp/blob/12765c115f11026c090ab0ee5dd79b38d95ef31f/testcases/network/rpc/rpc-tirpc/tests_pack/rpc_suite/tirpc/tirpc_expertlevel_rpcb_rmtcall/tirpc_rpcb_rmtcall.c#L91-L93


2023-05-11 13:24:48

by Steve Dickson

[permalink] [raw]
Subject: Re: LTP: tirpc_rpcb_rmtcall is failing

Hello Petr,

On 5/4/23 6:16 AM, Petr Vorel wrote:
> Hi Steve,
>
> tirpc_rpcb_rmtcall is failing. I was able to reproduce it on
> * openSUSE Tumbleweed with libtirpc 1.3.3
> * Debian stable 11 (bullseye) with libtirpc 1.3.1-1
>
> OTOH SLE 15-SP4 with libtirpc 1.2.6 is working.
>
> PATH="/opt/ltp/testcases/bin:$PATH" rpc_test.sh -s tirpc_svc_4 -c tirpc_rpcb_rmtcall
> rpc_test 1 TINFO: initialize 'lhost' 'ltp_ns_veth2' interface
> rpc_test 1 TINFO: add local addr 10.0.0.2/24
> rpc_test 1 TINFO: add local addr fd00:1:1:1::2/64
> rpc_test 1 TINFO: initialize 'rhost' 'ltp_ns_veth1' interface
> rpc_test 1 TINFO: add remote addr 10.0.0.1/24
> rpc_test 1 TINFO: add remote addr fd00:1:1:1::1/64
> rpc_test 1 TINFO: Network config (local -- remote):
> rpc_test 1 TINFO: ltp_ns_veth2 -- ltp_ns_veth1
> rpc_test 1 TINFO: 10.0.0.2/24 -- 10.0.0.1/24
> rpc_test 1 TINFO: fd00:1:1:1::2/64 -- fd00:1:1:1::1/64
> rpc_test 1 TINFO: timeout per run is 0h 5m 0s
> rpc_test 1 TINFO: check registered RPC with rpcinfo
> rpc_test 1 TINFO: registered RPC:
> program vers proto port service
> 100000 4 tcp 111 portmapper
> 100000 3 tcp 111 portmapper
> 100000 2 tcp 111 portmapper
> 100000 4 udp 111 portmapper
> 100000 3 udp 111 portmapper
> 100000 2 udp 111 portmapper
> 100005 1 udp 20048 mountd
> 100005 1 tcp 20048 mountd
> 100005 2 udp 20048 mountd
> 100005 2 tcp 20048 mountd
> 100005 3 udp 20048 mountd
> 100005 3 tcp 20048 mountd
> 100024 1 udp 37966 status
> 100024 1 tcp 43643 status
> 100003 3 tcp 2049 nfs
> 100003 4 tcp 2049 nfs
> 100227 3 tcp 2049 nfs_acl
> 100021 1 udp 59603 nlockmgr
> 100021 3 udp 59603 nlockmgr
> 100021 4 udp 59603 nlockmgr
> 100021 1 tcp 39145 nlockmgr
> 100021 3 tcp 39145 nlockmgr
> 100021 4 tcp 39145 nlockmgr
> rpc_test 1 TINFO: using libtirpc: yes
> rpc_test 1 TFAIL: tirpc_rpcb_rmtcall 10.0.0.2 536875000 failed unexpectedly
> 1
>
> The problem is in tirpc_rpcb_rmtcall.c [1], which calls rpcb_rmtcall(), which
> returns 1 (I suppose RPC_CANTENCODEARGS - can't encode arguments - enum
> clnt_stat from tirpc/rpc/clnt_stat.h):
>
> cs = rpcb_rmtcall(nconf, argc[1], progNum, VERSNUM, PROCNUM,
> (xdrproc_t) xdr_int, (char *)&var_snd,
> (xdrproc_t) xdr_int, (char *)&var_rec, tv, &svcaddr);
>
> test_status = (cs == RPC_SUCCESS) ? 0 : 1;
>
> //This last printf gives the result status to the tests suite
> //normally should be 0: test has passed or 1: test has failed
> printf("%d\n", test_status);
>
> return test_status;
>
> Any idea what could be wrong with these very old tests?
No... No idea... but I'm unable to reproduce it. It appears
you are using different repo that the one I found on
github [1]. But...

Looking code, RPC_CANTENCODEARGS is returned when
there is an xdr problem which might means a
memory problem??

With that said... commits 21718bbb^..fa153d63 did
make a lot of changes in the locking and cache
management.

steved.

[1] https://github.com/linux-test-project/ltp
>
> Kind regards,
> Petr
>
> [1] https://github.com/linux-test-project/ltp/blob/12765c115f11026c090ab0ee5dd79b38d95ef31f/testcases/network/rpc/rpc-tirpc/tests_pack/rpc_suite/tirpc/tirpc_expertlevel_rpcb_rmtcall/tirpc_rpcb_rmtcall.c#L91-L93
>


2023-05-12 07:52:51

by Petr Vorel

[permalink] [raw]
Subject: Re: LTP: tirpc_rpcb_rmtcall is failing

Hi Steve,

> Hello Petr,

> On 5/4/23 6:16 AM, Petr Vorel wrote:
> > Hi Steve,

> > tirpc_rpcb_rmtcall is failing. I was able to reproduce it on
> > * openSUSE Tumbleweed with libtirpc 1.3.3
> > * Debian stable 11 (bullseye) with libtirpc 1.3.1-1

> > OTOH SLE 15-SP4 with libtirpc 1.2.6 is working.

> > PATH="/opt/ltp/testcases/bin:$PATH" rpc_test.sh -s tirpc_svc_4 -c tirpc_rpcb_rmtcall
> > rpc_test 1 TINFO: initialize 'lhost' 'ltp_ns_veth2' interface
> > rpc_test 1 TINFO: add local addr 10.0.0.2/24
> > rpc_test 1 TINFO: add local addr fd00:1:1:1::2/64
> > rpc_test 1 TINFO: initialize 'rhost' 'ltp_ns_veth1' interface
> > rpc_test 1 TINFO: add remote addr 10.0.0.1/24
> > rpc_test 1 TINFO: add remote addr fd00:1:1:1::1/64
> > rpc_test 1 TINFO: Network config (local -- remote):
> > rpc_test 1 TINFO: ltp_ns_veth2 -- ltp_ns_veth1
> > rpc_test 1 TINFO: 10.0.0.2/24 -- 10.0.0.1/24
> > rpc_test 1 TINFO: fd00:1:1:1::2/64 -- fd00:1:1:1::1/64
> > rpc_test 1 TINFO: timeout per run is 0h 5m 0s
> > rpc_test 1 TINFO: check registered RPC with rpcinfo
> > rpc_test 1 TINFO: registered RPC:
> > program vers proto port service
> > 100000 4 tcp 111 portmapper
> > 100000 3 tcp 111 portmapper
> > 100000 2 tcp 111 portmapper
> > 100000 4 udp 111 portmapper
> > 100000 3 udp 111 portmapper
> > 100000 2 udp 111 portmapper
> > 100005 1 udp 20048 mountd
> > 100005 1 tcp 20048 mountd
> > 100005 2 udp 20048 mountd
> > 100005 2 tcp 20048 mountd
> > 100005 3 udp 20048 mountd
> > 100005 3 tcp 20048 mountd
> > 100024 1 udp 37966 status
> > 100024 1 tcp 43643 status
> > 100003 3 tcp 2049 nfs
> > 100003 4 tcp 2049 nfs
> > 100227 3 tcp 2049 nfs_acl
> > 100021 1 udp 59603 nlockmgr
> > 100021 3 udp 59603 nlockmgr
> > 100021 4 udp 59603 nlockmgr
> > 100021 1 tcp 39145 nlockmgr
> > 100021 3 tcp 39145 nlockmgr
> > 100021 4 tcp 39145 nlockmgr
> > rpc_test 1 TINFO: using libtirpc: yes
> > rpc_test 1 TFAIL: tirpc_rpcb_rmtcall 10.0.0.2 536875000 failed unexpectedly
> > 1

> > The problem is in tirpc_rpcb_rmtcall.c [1], which calls rpcb_rmtcall(), which
> > returns 1 (I suppose RPC_CANTENCODEARGS - can't encode arguments - enum
> > clnt_stat from tirpc/rpc/clnt_stat.h):

> > cs = rpcb_rmtcall(nconf, argc[1], progNum, VERSNUM, PROCNUM,
> > (xdrproc_t) xdr_int, (char *)&var_snd,
> > (xdrproc_t) xdr_int, (char *)&var_rec, tv, &svcaddr);

> > test_status = (cs == RPC_SUCCESS) ? 0 : 1;

> > //This last printf gives the result status to the tests suite
> > //normally should be 0: test has passed or 1: test has failed
> > printf("%d\n", test_status);

> > return test_status;

> > Any idea what could be wrong with these very old tests?
> No... No idea... but I'm unable to reproduce it. It appears
> you are using different repo that the one I found on
> github [1]. But...

Thanks a lot for looking into the issue.
BTW on which Fedora/RHEL/CentOS version did you test?

No, I'm also using the official LTP repository on github [1].
And I compile on recent glibc (> 2.32, which removed SUN-RPC) and with libtirpc:

./configure
...
libtirpc: yes
glibc SUN-RPC: no

> Looking code, RPC_CANTENCODEARGS is returned when
> there is an xdr problem which might means a
> memory problem??

> With that said... commits 21718bbb^..fa153d63 did
That was released on 1.3.3, but I'm able to reproduce it on
Debian stable 11 (bullseye) with libtirpc 1.3.1-1.

Kind regards,
Petr

> make a lot of changes in the locking and cache
> management.

> steved.

> [1] https://github.com/linux-test-project/ltp

> > Kind regards,
> > Petr

> > [1] https://github.com/linux-test-project/ltp/blob/12765c115f11026c090ab0ee5dd79b38d95ef31f/testcases/network/rpc/rpc-tirpc/tests_pack/rpc_suite/tirpc/tirpc_expertlevel_rpcb_rmtcall/tirpc_rpcb_rmtcall.c#L91-L93



2023-05-12 10:25:35

by Steve Dickson

[permalink] [raw]
Subject: Re: LTP: tirpc_rpcb_rmtcall is failing

Hello,

On 5/12/23 3:43 AM, Petr Vorel wrote:
> Hi Steve,
>
>> Hello Petr,
>
>> On 5/4/23 6:16 AM, Petr Vorel wrote:
>>> Hi Steve,
>
>>> tirpc_rpcb_rmtcall is failing. I was able to reproduce it on
>>> * openSUSE Tumbleweed with libtirpc 1.3.3
>>> * Debian stable 11 (bullseye) with libtirpc 1.3.1-1
>
>>> OTOH SLE 15-SP4 with libtirpc 1.2.6 is working.
>
>>> PATH="/opt/ltp/testcases/bin:$PATH" rpc_test.sh -s tirpc_svc_4 -c tirpc_rpcb_rmtcall
>>> rpc_test 1 TINFO: initialize 'lhost' 'ltp_ns_veth2' interface
>>> rpc_test 1 TINFO: add local addr 10.0.0.2/24
>>> rpc_test 1 TINFO: add local addr fd00:1:1:1::2/64
>>> rpc_test 1 TINFO: initialize 'rhost' 'ltp_ns_veth1' interface
>>> rpc_test 1 TINFO: add remote addr 10.0.0.1/24
>>> rpc_test 1 TINFO: add remote addr fd00:1:1:1::1/64
>>> rpc_test 1 TINFO: Network config (local -- remote):
>>> rpc_test 1 TINFO: ltp_ns_veth2 -- ltp_ns_veth1
>>> rpc_test 1 TINFO: 10.0.0.2/24 -- 10.0.0.1/24
>>> rpc_test 1 TINFO: fd00:1:1:1::2/64 -- fd00:1:1:1::1/64
>>> rpc_test 1 TINFO: timeout per run is 0h 5m 0s
>>> rpc_test 1 TINFO: check registered RPC with rpcinfo
>>> rpc_test 1 TINFO: registered RPC:
>>> program vers proto port service
>>> 100000 4 tcp 111 portmapper
>>> 100000 3 tcp 111 portmapper
>>> 100000 2 tcp 111 portmapper
>>> 100000 4 udp 111 portmapper
>>> 100000 3 udp 111 portmapper
>>> 100000 2 udp 111 portmapper
>>> 100005 1 udp 20048 mountd
>>> 100005 1 tcp 20048 mountd
>>> 100005 2 udp 20048 mountd
>>> 100005 2 tcp 20048 mountd
>>> 100005 3 udp 20048 mountd
>>> 100005 3 tcp 20048 mountd
>>> 100024 1 udp 37966 status
>>> 100024 1 tcp 43643 status
>>> 100003 3 tcp 2049 nfs
>>> 100003 4 tcp 2049 nfs
>>> 100227 3 tcp 2049 nfs_acl
>>> 100021 1 udp 59603 nlockmgr
>>> 100021 3 udp 59603 nlockmgr
>>> 100021 4 udp 59603 nlockmgr
>>> 100021 1 tcp 39145 nlockmgr
>>> 100021 3 tcp 39145 nlockmgr
>>> 100021 4 tcp 39145 nlockmgr
>>> rpc_test 1 TINFO: using libtirpc: yes
>>> rpc_test 1 TFAIL: tirpc_rpcb_rmtcall 10.0.0.2 536875000 failed unexpectedly
>>> 1
>
>>> The problem is in tirpc_rpcb_rmtcall.c [1], which calls rpcb_rmtcall(), which
>>> returns 1 (I suppose RPC_CANTENCODEARGS - can't encode arguments - enum
>>> clnt_stat from tirpc/rpc/clnt_stat.h):
>
>>> cs = rpcb_rmtcall(nconf, argc[1], progNum, VERSNUM, PROCNUM,
>>> (xdrproc_t) xdr_int, (char *)&var_snd,
>>> (xdrproc_t) xdr_int, (char *)&var_rec, tv, &svcaddr);
>
>>> test_status = (cs == RPC_SUCCESS) ? 0 : 1;
>
>>> //This last printf gives the result status to the tests suite
>>> //normally should be 0: test has passed or 1: test has failed
>>> printf("%d\n", test_status);
>
>>> return test_status;
>
>>> Any idea what could be wrong with these very old tests?
>> No... No idea... but I'm unable to reproduce it. It appears
>> you are using different repo that the one I found on
>> github [1]. But...
>
> Thanks a lot for looking into the issue.
> BTW on which Fedora/RHEL/CentOS version did you test?
Fedora 38... but I will keep trying...
>
> No, I'm also using the official LTP repository on github [1].
> And I compile on recent glibc (> 2.32, which removed SUN-RPC) and with libtirpc:
>
> ./configure
> ...
> libtirpc: yes
> glibc SUN-RPC: no
>
>> Looking code, RPC_CANTENCODEARGS is returned when
>> there is an xdr problem which might means a
>> memory problem??
>
>> With that said... commits 21718bbb^..fa153d63 did
> That was released on 1.3.3, but I'm able to reproduce it on
> Debian stable 11 (bullseye) with libtirpc 1.3.1-1.
Good to know... thanks!

steved.
>
> Kind regards,
> Petr
>
>> make a lot of changes in the locking and cache
>> management.
>
>> steved.
>
>> [1] https://github.com/linux-test-project/ltp
>
>>> Kind regards,
>>> Petr
>
>>> [1] https://github.com/linux-test-project/ltp/blob/12765c115f11026c090ab0ee5dd79b38d95ef31f/testcases/network/rpc/rpc-tirpc/tests_pack/rpc_suite/tirpc/tirpc_expertlevel_rpcb_rmtcall/tirpc_rpcb_rmtcall.c#L91-L93
>
>