Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id ; Mon, 19 Nov 2001 11:38:13 -0500 Received: (majordomo@vger.kernel.org) by vger.kernel.org id ; Mon, 19 Nov 2001 11:38:04 -0500 Received: from noc1.BelWue.de ([129.143.2.1]:8183 "EHLO noc1.BelWue.DE") by vger.kernel.org with ESMTP id ; Mon, 19 Nov 2001 11:37:44 -0500 From: Birger Lammering Message-ID: <15353.13652.591045.916300@stderr.science-computing.de> Date: Mon, 19 Nov 2001 17:37:40 +0100 To: linux-kernel@vger.kernel.org CC: trond.myklebust@fys.uio.no Subject: more tcpdumpinfo for nfs3 problem: aix-server --- linux 2.4.15pre5 client In-Reply-To: In-Reply-To: <15352.56551.709659.146271@stderr.science-computing.de> X-Mailer: VM 6.89 under 21.1 (patch 14) "Cuyahoga Valley" XEmacs Lucid Mime-Version: 1.0 (generated by tm-edit 1.7) Content-Type: text/plain; charset=US-ASCII Sender: linux-kernel-owner@vger.kernel.org X-Mailing-List: linux-kernel@vger.kernel.org Hi, Alan Cox writes: > Right so someone is truncating frames. I'd start with the hub then work > outwards. tcpdump on the aix was truncating the frames. sorry, the snap length was too small --- with using -s 64000 there were no truncated frames anymore. Well, I guess (n.b.: GUESS) the problem is with nfs again... This is a consistent reproducible bug. We have tested it on several AIXen, Linuxen on different switched Networks. After having learned a bit better how to use tcpdump, here we go: nfs3 Client: Linux 2.4.15pre5 (capc25) 3com 905B / Intel EtherReal 100+ nfs3 Server: Aix 4.3 (caes04), IBM SP2 / 7044-170 100MBit/FullDuplex switched Network (Cisco Catalyst) This is the first attempt of copying a 500k file onto the nfs server: 16:24:19.652672 > capc25.muc.654 > caes04.muc.sunrpc: udp 40 (DF) 16:24:19.655073 < caes04.muc.sunrpc > capc25.muc.654: udp 24 16:24:19.655119 > capc25.muc.655 > caes04.muc.sunrpc: udp 56 (DF) 16:24:19.656813 < caes04.muc.sunrpc > capc25.muc.655: udp 28 16:24:19.656853 > capc25.muc.656 > caes04.muc.nfs: S 2952133600:2952133600(0) win 5840 (DF) 16:24:19.657695 < caes04.muc.nfs > capc25.muc.656: S 4177984247:4177984247(0) ack 2952133601 win 59860 16:24:19.657711 > capc25.muc.656 > caes04.muc.nfs: . 1:1(0) ack 1 win 5840 (DF) 16:24:19.657746 > capc25.muc.656 > caes04.muc.nfs: P 1:45(44) ack 1 win 5840 (DF) 16:24:19.658653 < caes04.muc.nfs > capc25.muc.656: P 1:29(28) ack 45 win 60032 16:24:19.658666 > capc25.muc.656 > caes04.muc.nfs: . 45:45(0) ack 29 win 5840 (DF) 16:24:19.658702 > capc25.muc.656 > caes04.muc.nfs: F 45:45(0) ack 29 win 5840 (DF) 16:24:19.658749 > capc25.muc.657 > caes04.muc.sunrpc: udp 40 (DF) 16:24:19.659739 < caes04.muc.nfs > capc25.muc.656: . 29:29(0) ack 46 win 60032 16:24:19.660017 < caes04.muc.nfs > capc25.muc.656: F 29:29(0) ack 46 win 60032 16:24:19.660026 > capc25.muc.656 > caes04.muc.nfs: . 46:46(0) ack 30 win 5840 (DF) 16:24:19.660157 < caes04.muc.sunrpc > capc25.muc.657: udp 24 16:24:19.660199 > capc25.muc.658 > caes04.muc.sunrpc: udp 56 (DF) 16:24:19.662361 < caes04.muc.sunrpc > capc25.muc.658: udp 28 16:24:19.662393 > capc25.muc.50947429 > caes04.muc.nfs: 40 null (DF) 16:24:19.664439 < caes04.muc.nfs > capc25.muc.50947429: reply ok 24 null 16:24:19.665397 > capc25.muc.799 > caes04.muc.nfs: S 2955913379:2955913379(0) win 5840 (DF) 16:24:19.667274 < caes04.muc.nfs > capc25.muc.799: S 3186497216:3186497216(0) ack 2955913380 win 59860 16:24:19.667293 > capc25.muc.799 > caes04.muc.nfs: . 1:1(0) ack 1 win 5840 (DF) 16:24:19.667328 > capc25.muc.799 > caes04.muc.nfs: P 1:109(108) ack 1 win 5840 (DF) 16:24:19.669096 < caes04.muc.nfs > capc25.muc.799: P 1:117(116) ack 109 win 60032 16:24:19.669108 > capc25.muc.799 > caes04.muc.nfs: . 109:109(0) ack 117 win 5840 (DF) 16:24:19.669155 > capc25.muc.799 > caes04.muc.nfs: P 109:217(108) ack 117 win 5840 (DF) 16:24:19.670267 < caes04.muc.nfs > capc25.muc.799: P 117:289(172) ack 217 win 60032 16:24:19.670295 > capc25.muc.799 > caes04.muc.nfs: P 217:325(108) ack 289 win 6432 (DF) 16:24:19.671674 < caes04.muc.nfs > capc25.muc.799: P 289:457(168) ack 325 win 60032 16:24:19.673364 > capc25.muc.799 > caes04.muc.nfs: P 325:461(136) ack 457 win 6432 (DF) 16:24:19.675030 < caes04.muc.nfs > capc25.muc.799: P 457:701(244) ack 461 win 60032 16:24:19.675121 > capc25.muc.799 > caes04.muc.nfs: P 461:625(164) ack 701 win 7504 (DF) 16:24:19.693914 < caes04.muc.nfs > capc25.muc.799: P 701:849(148) ack 625 win 60032 16:24:19.694170 > capc25.muc.799 > caes04.muc.nfs: . 625:2085(1460) ack 849 win 7504 (DF) 16:24:19.694186 > capc25.muc.799 > caes04.muc.nfs: . 2085:3545(1460) ack 849 win 7504 (DF) 16:24:19.774514 < caes04.muc.nfs > capc25.muc.799: . 849:849(0) ack 3545 win 57112 16:24:19.774545 > capc25.muc.799 > caes04.muc.nfs: . 3545:5005(1460) ack 849 win 7504 (DF) 16:24:19.774556 > capc25.muc.799 > caes04.muc.nfs: . 5005:6465(1460) ack 849 win 7504 (DF) 16:24:19.774565 > capc25.muc.799 > caes04.muc.nfs: . 6465:7925(1460) ack 849 win 7504 (DF) 16:24:19.983865 < caes04.muc.nfs > capc25.muc.799: . 849:849(0) ack 7925 win 52732 16:24:19.983887 > capc25.muc.799 > caes04.muc.nfs: . 7925:9385(1460) ack 849 win 7504 (DF) 16:24:19.983897 > capc25.muc.799 > caes04.muc.nfs: . 9385:10845(1460) ack 849 win 7504 (DF) 16:24:19.983906 > capc25.muc.799 > caes04.muc.nfs: . 10845:12305(1460) ack 849 win 7504 (DF) 16:24:19.983916 > capc25.muc.799 > caes04.muc.nfs: P 12305:13765(1460) ack 849 win 7504 (DF) 16:24:20.194551 < caes04.muc.nfs > capc25.muc.799: . 849:849(0) ack 13765 win 46892 16:24:20.194569 > capc25.muc.799 > caes04.muc.nfs: . 13765:15225(1460) ack 849 win 7504 (DF) 16:24:20.194578 > capc25.muc.799 > caes04.muc.nfs: P 15225:16685(1460) ack 849 win 7504 (DF) 16:24:20.194588 > capc25.muc.799 > caes04.muc.nfs: . 16685:18145(1460) ack 849 win 7504 (DF) 16:24:20.194598 > capc25.muc.799 > caes04.muc.nfs: . 18145:19605(1460) ack 849 win 7504 (DF) 16:24:20.194607 > capc25.muc.799 > caes04.muc.nfs: P 19605:21065(1460) ack 849 win 7504 (DF) 16:24:20.396361 < caes04.muc.nfs > capc25.muc.799: . 849:849(0) ack 21065 win 39592 ...and so on... 16:24:21.904205 < caes04.muc.nfs > capc25.muc.799: . 2981:2981(0) ack 462513 win 58968 16:24:21.904231 > capc25.muc.799 > caes04.muc.nfs: P 480033:481493(1460) ack 2981 win 7504 (DF) 16:24:21.904239 > capc25.muc.799 > caes04.muc.nfs: . 481493:482953(1460) ack 2981 win 7504 (DF) 16:24:21.904247 > capc25.muc.799 > caes04.muc.nfs: . 482953:484413(1460) ack 2981 win 7504 (DF) 16:24:21.904256 > capc25.muc.799 > caes04.muc.nfs: . 484413:485873(1460) ack 2981 win 7504 (DF) 16:24:21.904265 > capc25.muc.799 > caes04.muc.nfs: . 485873:487333(1460) ack 2981 win 7504 (DF) 16:24:21.904277 > capc25.muc.799 > caes04.muc.nfs: . 487333:488793(1460) ack 2981 win 7504 (DF) 16:24:21.904289 > capc25.muc.799 > caes04.muc.nfs: . 488793:490253(1460) ack 2981 win 7504 (DF) 16:24:21.904298 > capc25.muc.799 > caes04.muc.nfs: . 490253:491713(1460) ack 2981 win 7504 (DF) 16:24:21.904309 > capc25.muc.799 > caes04.muc.nfs: . 491713:493173(1460) ack 2981 win 7504 (DF) 16:24:21.904317 > capc25.muc.799 > caes04.muc.nfs: . 493173:494633(1460) ack 2981 win 7504 (DF) 16:24:21.904327 > capc25.muc.799 > caes04.muc.nfs: . 494633:496093(1460) ack 2981 win 7504 (DF) 16:24:21.904337 > capc25.muc.799 > caes04.muc.nfs: . 496093:497553(1460) ack 2981 win 7504 (DF) 16:24:21.904349 > capc25.muc.799 > caes04.muc.nfs: . 497553:499013(1460) ack 2981 win 7504 (DF) 16:24:21.904357 > capc25.muc.799 > caes04.muc.nfs: . 499013:500473(1460) ack 2981 win 7504 (DF) 16:24:21.904364 > capc25.muc.799 > caes04.muc.nfs: . 500473:501933(1460) ack 2981 win 7504 (DF) 16:24:21.904372 > capc25.muc.799 > caes04.muc.nfs: . 501933:503393(1460) ack 2981 win 7504 (DF) 16:24:21.905180 < caes04.muc.nfs > capc25.muc.799: P 2981:3145(164) ack 465433 win 56048 16:24:21.905979 > capc25.muc.799 > caes04.muc.nfs: . 503393:504853(1460) ack 2981 win 7504 (DF) 16:24:21.905986 > capc25.muc.799 > caes04.muc.nfs: . 504853:506313(1460) ack 2981 win 7504 (DF) 16:24:21.905992 > capc25.muc.799 > caes04.muc.nfs: . 506313:507773(1460) ack 2981 win 7504 (DF) 16:24:21.905998 > capc25.muc.799 > caes04.muc.nfs: . 507773:509233(1460) ack 2981 win 7504 (DF) 16:24:21.906004 > capc25.muc.799 > caes04.muc.nfs: P 509233:510693(1460) ack 2981 win 7504 (DF) 16:24:21.906009 > capc25.muc.799 > caes04.muc.nfs: P 510693:512153(1460) ack 2981 win 7504 (DF) 16:24:21.906016 > capc25.muc.799 > caes04.muc.nfs: . 512153:513613(1460) ack 2981 win 7504 (DF) 16:24:21.906021 > capc25.muc.799 > caes04.muc.nfs: . 513613:515073(1460) ack 3145 win 7504 (DF) 16:24:21.906027 > capc25.muc.799 > caes04.muc.nfs: P 515073:515133(60) ack 3145 win 7504 (DF) 16:24:21.909806 < caes04.muc.nfs > capc25.muc.799: . 3145:3145(0) ack 494633 win 59764 16:24:21.910644 < caes04.muc.nfs > capc25.muc.799: P 3145:3309(164) ack 497553 win 56844 16:24:21.912247 < caes04.muc.nfs > capc25.muc.799: . 3309:3309(0) ack 515133 win 59892 16:24:21.912855 < caes04.muc.nfs > capc25.muc.799: P 3309:3473(164) ack 515133 win 59892 16:24:21.941386 > capc25.muc.799 > caes04.muc.nfs: . 515133:515133(0) ack 3473 win 7504 (DF) 16:24:21.965016 < caes04.muc.nfs > capc25.muc.799: P 3473:3629(156) ack 515133 win 60032 16:24:21.965042 > capc25.muc.799 > caes04.muc.nfs: . 515133:515133(0) ack 3629 win 7504 (DF) 16:24:21.965128 > capc25.muc.799 > caes04.muc.nfs: P 515133:515273(140) ack 3629 win 7504 (DF) 16:24:21.998128 < caes04.muc.nfs > capc25.muc.799: P 3629:3785(156) ack 515273 win 60032 16:24:22.031385 > capc25.muc.799 > caes04.muc.nfs: . 515273:515273(0) ack 3785 win 7504 (DF) The file was copied all right. qx09820@capc25 /home/qx09820 > netstat |grep caes04 tcp 0 0 capc25.muc:729 caes04.muc:nfs TIME_WAIT tcp 0 0 capc25.muc:744 caes04.muc:nfs TIME_WAIT tcp 0 0 capc25.muc:737 caes04.muc:nfs TIME_WAIT tcp 0 0 capc25.muc:798 caes04.muc:nfs ESTABLISHED Now copying the same file for a second time: 16:27:20.716640 > capc25.muc.799 > caes04.muc.nfs: P 2956428652:2956428780(128) ack 3186501001 win 7504 (DF) 16:27:20.717226 < caes04.muc.nfs > capc25.muc.799: P 1:117(116) ack 128 win 60032 16:27:20.717243 > capc25.muc.799 > caes04.muc.nfs: . 128:128(0) ack 117 win 7504 (DF) 16:27:20.717360 > capc25.muc.799 > caes04.muc.nfs: P 128:264(136) ack 117 win 7504 (DF) 16:27:20.717966 < caes04.muc.nfs > capc25.muc.799: P 117:361(244) ack 264 win 60032 16:27:20.718041 > capc25.muc.799 > caes04.muc.nfs: P 264:428(164) ack 361 win 8576 (DF) 16:27:20.735478 < caes04.muc.nfs > capc25.muc.799: P 361:509(148) ack 428 win 60032 16:27:20.735745 > capc25.muc.799 > caes04.muc.nfs: . 428:1888(1460) ack 509 win 8576 (DF) 16:27:20.735760 > capc25.muc.799 > caes04.muc.nfs: . 1888:3348(1460) ack 509 win 8576 (DF) 16:27:20.858656 < caes04.muc.nfs > capc25.muc.799: . 509:509(0) ack 3348 win 57112 16:27:20.858672 > capc25.muc.799 > caes04.muc.nfs: . 3348:4808(1460) ack 509 win 8576 (DF) 16:27:20.858682 > capc25.muc.799 > caes04.muc.nfs: . 4808:6268(1460) ack 509 win 8576 (DF) 16:27:20.858692 > capc25.muc.799 > caes04.muc.nfs: . 6268:7728(1460) ack 509 win 8576 (DF) 16:27:21.068297 < caes04.muc.nfs > capc25.muc.799: . 509:509(0) ack 7728 win 52732 16:27:21.068318 > capc25.muc.799 > caes04.muc.nfs: . 7728:9188(1460) ack 509 win 8576 (DF) 16:27:21.068328 > capc25.muc.799 > caes04.muc.nfs: . 9188:10648(1460) ack 509 win 8576 (DF) 16:27:21.068337 > capc25.muc.799 > caes04.muc.nfs: . 10648:12108(1460) ack 509 win 8576 (DF) 16:27:21.068347 > capc25.muc.799 > caes04.muc.nfs: . 12108:13568(1460) ack 509 win 8576 (DF) 16:27:21.278296 < caes04.muc.nfs > capc25.muc.799: . 509:509(0) ack 13568 win 46892 16:27:21.278321 > capc25.muc.799 > caes04.muc.nfs: . 13568:15028(1460) ack 509 win 8576 (DF) 16:27:21.278331 > capc25.muc.799 > caes04.muc.nfs: . 15028:16488(1460) ack 509 win 8576 (DF) 16:27:21.278341 > capc25.muc.799 > caes04.muc.nfs: . 16488:17948(1460) ack 509 win 8576 (DF) 16:27:21.278350 > capc25.muc.799 > caes04.muc.nfs: . 17948:19408(1460) ack 509 win 8576 (DF) 16:27:21.278359 > capc25.muc.799 > caes04.muc.nfs: . 19408:20868(1460) ack 509 win 8576 (DF) 16:27:21.488343 < caes04.muc.nfs > capc25.muc.799: . 509:509(0) ack 20868 win 39592 16:27:21.488376 > capc25.muc.799 > caes04.muc.nfs: . 20868:22328(1460) ack 509 win 8576 (DF) 16:27:21.488387 > capc25.muc.799 > caes04.muc.nfs: . 22328:23788(1460) ack 509 win 8576 (DF) 16:27:21.488397 > capc25.muc.799 > caes04.muc.nfs: . 23788:25248(1460) ack 509 win 8576 (DF) 16:27:21.488406 > capc25.muc.799 > caes04.muc.nfs: . 25248:26708(1460) ack 509 win 8576 (DF) 16:27:21.488415 > capc25.muc.799 > caes04.muc.nfs: . 26708:28168(1460) ack 509 win 8576 (DF) 16:27:21.488424 > capc25.muc.799 > caes04.muc.nfs: . 28168:29628(1460) ack 509 win 8576 (DF) 16:27:21.690712 < caes04.muc.nfs > capc25.muc.799: . 509:509(0) ack 29628 win 30832 16:27:21.690736 > capc25.muc.799 > caes04.muc.nfs: P 29628:31088(1460) ack 509 win 8576 (DF) 16:27:21.690746 > capc25.muc.799 > caes04.muc.nfs: . 31088:32548(1460) ack 509 win 8576 (DF) 16:27:21.690756 > capc25.muc.799 > caes04.muc.nfs: . 32548:34008(1460) ack 509 win 8576 (DF) 16:27:21.690765 > capc25.muc.799 > caes04.muc.nfs: . 34008:35468(1460) ack 509 win 8576 (DF) 16:27:21.690774 > capc25.muc.799 > caes04.muc.nfs: . 35468:36928(1460) ack 509 win 8576 (DF) 16:27:21.690784 > capc25.muc.799 > caes04.muc.nfs: . 36928:38388(1460) ack 509 win 8576 (DF) 16:27:21.690793 > capc25.muc.799 > caes04.muc.nfs: . 38388:39848(1460) ack 509 win 8576 (DF) ..and so on.. 16:27:22.567415 < caes04.muc.nfs > capc25.muc.799: P 2477:2641(164) ack 434048 win 54320 16:27:22.568505 > capc25.muc.799 > caes04.muc.nfs: . 470548:472008(1460) ack 2477 win 8576 (DF) 16:27:22.568512 > capc25.muc.799 > caes04.muc.nfs: . 472008:473468(1460) ack 2477 win 8576 (DF) 16:27:22.568518 > capc25.muc.799 > caes04.muc.nfs: . 473468:474928(1460) ack 2477 win 8576 (DF) 16:27:22.568524 > capc25.muc.799 > caes04.muc.nfs: . 474928:476388(1460) ack 2477 win 8576 (DF) 16:27:22.568530 > capc25.muc.799 > caes04.muc.nfs: . 476388:477848(1460) ack 2477 win 8576 (DF) 16:27:22.568536 > capc25.muc.799 > caes04.muc.nfs: . 477848:479308(1460) ack 2641 win 8576 (DF) 16:27:22.568541 > capc25.muc.799 > caes04.muc.nfs: . 479308:480768(1460) ack 2641 win 8576 (DF) 16:27:22.568550 > capc25.muc.799 > caes04.muc.nfs: . 480768:482228(1460) ack 2641 win 8576 (DF) 16:27:22.568556 > capc25.muc.799 > caes04.muc.nfs: . 482228:483688(1460) ack 2641 win 8576 (DF) 16:27:22.570140 < caes04.muc.nfs > capc25.muc.799: . 2641:2641(0) ack 461788 win 59496 16:27:22.570163 > capc25.muc.799 > caes04.muc.nfs: . 483688:485148(1460) ack 2641 win 8576 (DF) 16:27:22.570172 > capc25.muc.799 > caes04.muc.nfs: . 485148:486608(1460) ack 2641 win 8576 (DF) 16:27:22.570180 > capc25.muc.799 > caes04.muc.nfs: . 486608:488068(1460) ack 2641 win 8576 (DF) 16:27:22.570190 > capc25.muc.799 > caes04.muc.nfs: . 488068:489528(1460) ack 2641 win 8576 (DF) 16:27:22.570200 > capc25.muc.799 > caes04.muc.nfs: . 489528:490988(1460) ack 2641 win 8576 (DF) 16:27:22.570214 > capc25.muc.799 > caes04.muc.nfs: . 490988:492448(1460) ack 2641 win 8576 (DF) 16:27:22.570222 > capc25.muc.799 > caes04.muc.nfs: . 492448:493908(1460) ack 2641 win 8576 (DF) 16:27:22.570230 > capc25.muc.799 > caes04.muc.nfs: . 493908:495368(1460) ack 2641 win 8576 (DF) 16:27:22.570238 > capc25.muc.799 > caes04.muc.nfs: . 495368:496828(1460) ack 2641 win 8576 (DF) 16:27:22.570247 > capc25.muc.799 > caes04.muc.nfs: . 496828:498288(1460) ack 2641 win 8576 (DF) 16:27:22.570257 > capc25.muc.799 > caes04.muc.nfs: P 498288:499748(1460) ack 2641 win 8576 (DF) 16:27:22.570268 > capc25.muc.799 > caes04.muc.nfs: . 499748:501208(1460) ack 2641 win 8576 (DF) 16:27:22.570279 > capc25.muc.799 > caes04.muc.nfs: . 501208:502668(1460) ack 2641 win 8576 (DF) 16:27:22.570287 > capc25.muc.799 > caes04.muc.nfs: P 502668:504128(1460) ack 2641 win 8576 (DF) 16:27:22.570294 > capc25.muc.799 > caes04.muc.nfs: . 504128:505588(1460) ack 2641 win 8576 (DF) 16:27:22.570307 > capc25.muc.799 > caes04.muc.nfs: . 505588:507048(1460) ack 2641 win 8576 (DF) 16:27:22.570433 < caes04.muc.nfs > capc25.muc.799: P 2641:2805(164) ack 464708 win 56576 16:27:22.571906 > capc25.muc.799 > caes04.muc.nfs: . 507048:508508(1460) ack 2641 win 8576 (DF) 16:27:22.571913 > capc25.muc.799 > caes04.muc.nfs: . 508508:509968(1460) ack 2641 win 8576 (DF) 16:27:22.571919 > capc25.muc.799 > caes04.muc.nfs: . 509968:511428(1460) ack 2641 win 8576 (DF) 16:27:22.571924 > capc25.muc.799 > caes04.muc.nfs: . 511428:512888(1460) ack 2641 win 8576 (DF) 16:27:22.571933 > capc25.muc.799 > caes04.muc.nfs: . 512888:514348(1460) ack 2805 win 8576 (DF) 16:27:22.571938 > capc25.muc.799 > caes04.muc.nfs: P 514348:514796(448) ack 2805 win 8576 (DF) 16:27:22.573924 < caes04.muc.nfs > capc25.muc.799: . 2805:2805(0) ack 495368 win 58832 16:27:22.574462 < caes04.muc.nfs > capc25.muc.799: P 2805:2969(164) ack 499748 win 54452 16:27:22.575192 < caes04.muc.nfs > capc25.muc.799: . 2969:2969(0) ack 514796 win 60032 16:27:22.575518 < caes04.muc.nfs > capc25.muc.799: P 2969:3133(164) ack 514796 win 60032 16:27:22.608665 > capc25.muc.799 > caes04.muc.nfs: . 514796:514796(0) ack 3133 win 8576 (DF) 16:27:26.248872 > capc25.muc.799 > caes04.muc.nfs: P 514796:514936(140) ack 3133 win 8576 (DF) 16:27:26.268210 < caes04.muc.nfs > capc25.muc.799: . 3133:3133(0) ack 514936 win 60032 16:27:26.282829 < caes04.muc.nfs > capc25.muc.799: P 3133:3289(156) ack 514936 win 60032 16:27:26.282843 > capc25.muc.799 > caes04.muc.nfs: . 514936:514936(0) ack 3289 win 8576 (DF) from now on we get lot's of these: 16:27:26.489024 > capc25.muc.576126976 > caes04.muc.nfs: 40 null (DF) 16:27:26.489647 < caes04.muc.nfs > capc25.muc.576126976: reply ok 24 null The cp command on the Linux nfs3-client side hangs and cannot be killed. We get: dmesg: nfs: server caes04 not responding, still trying then after a while: dmesg: nfs: server caes04 OK qx09820@capc25 /home/qx09820 > netstat | grep caes04 tcp 0 0 capc25.muc:798 caes04.muc:nfs ESTABLISHED The cp seems to be finished and the game can start again. :-( The interesting bit seems to be at the end, where `cp` on the Linux nfs3 client hangs. The cause of the problem might just as well be on the AIX nfs3 server as on the client. The tcpdump on the server is basically identical -- now that option -s 64000 has been used. Files of size 500kB, 1MB and 2MB have been used. Sometimes the whole file is copied before (after/while?) cp locks up, sometimes just about 800k are copied. We also got these messages after the cp died and the nfs: server caes04 OK message appeared. The line with looks weird. Selective acknowledgement? Is this normal? 16:34:56.487168 > capc25.muc.1649999872 > caes04.muc.nfs: 40 null (DF) 16:34:56.493170 < caes04.muc.nfs > capc25.muc.1649999872: reply ok 24 null 16:35:26.488376 > capc25.muc.2186870784 > caes04.muc.nfs: 40 null (DF) 16:35:26.492354 < caes04.muc.nfs > capc25.muc.2186870784: reply ok 24 null 16:35:56.489587 > capc25.muc.2723741696 > caes04.muc.nfs: 40 null (DF) 16:35:56.491055 < caes04.muc.nfs > capc25.muc.2723741696: reply ok 24 null 16:36:26.490783 > capc25.muc.3260612608 > caes04.muc.nfs: 40 null (DF) 16:36:26.505492 < caes04.muc.nfs > capc25.muc.3260612608: reply ok 24 null 16:36:56.491996 > capc25.muc.3797483520 > caes04.muc.nfs: 40 null (DF) 16:36:56.494516 < caes04.muc.nfs > capc25.muc.3797483520: reply ok 24 null 16:37:26.493194 > capc25.muc.39452672 > caes04.muc.nfs: 40 null (DF) 16:37:26.493631 < caes04.muc.nfs > capc25.muc.39452672: reply ok 24 null 16:37:56.484418 > capc25.muc.576323584 > caes04.muc.nfs: 40 null (DF) 16:37:56.485075 < caes04.muc.nfs > capc25.muc.576323584: reply ok 24 null 16:38:26.485620 > capc25.muc.1113194496 > caes04.muc.nfs: 40 null (DF) 16:38:26.486086 < caes04.muc.nfs > capc25.muc.1113194496: reply ok 24 null 16:38:56.486833 > capc25.muc.1650065408 > caes04.muc.nfs: 40 null (DF) 16:38:56.487256 < caes04.muc.nfs > capc25.muc.1650065408: reply ok 24 null 16:39:26.488042 > capc25.muc.2186936320 > caes04.muc.nfs: 40 null (DF) 16:39:26.488555 < caes04.muc.nfs > capc25.muc.2186936320: reply ok 24 null 16:39:56.489253 > capc25.muc.2723807232 > caes04.muc.nfs: 40 null (DF) 16:39:56.489655 < caes04.muc.nfs > capc25.muc.2723807232: reply ok 24 null 16:40:26.490458 > capc25.muc.3260678144 > caes04.muc.nfs: 40 null (DF) 16:40:26.490858 < caes04.muc.nfs > capc25.muc.3260678144: reply ok 24 null 16:40:56.491656 > capc25.muc.3797549056 > caes04.muc.nfs: 40 null (DF) 16:40:56.492277 < caes04.muc.nfs > capc25.muc.3797549056: reply ok 24 null 16:41:04.171081 < caes04.muc.nfs > capc25.muc.798: F 2278783378:2278783378(0) ack 3544800811 win 60032 16:41:04.171126 > capc25.muc.798 > caes04.muc.nfs: F 1:1(0) ack 1 win 9408 (DF) 16:41:04.171307 > capc25.muc.799 > caes04.muc.nfs: S 4020491914:4020491914(0) win 5840 (DF) 16:41:04.172213 < caes04.muc.nfs > capc25.muc.798: . 1:1(0) ack 2 win 60032 16:41:04.172379 < caes04.muc.nfs > capc25.muc.799: S 3316289216:3316289216(0) ack 4020491915 win 59860 16:41:04.172394 > capc25.muc.799 > caes04.muc.nfs: . 1:1(0) ack 1 win 5840 (DF) 16:41:04.172426 > capc25.muc.799 > caes04.muc.nfs: . 1:1461(1460) ack 1 win 5840 (DF) 16:41:04.172437 > capc25.muc.799 > caes04.muc.nfs: . 1461:2921(1460) ack 1 win 5840 (DF) 16:41:04.193288 < caes04.muc.nfs > capc25.muc.799: . 1:1(0) ack 2921 win 57112 16:41:04.193322 > capc25.muc.799 > caes04.muc.nfs: . 2921:4381(1460) ack 1 win 5840 (DF) 16:41:04.193330 > capc25.muc.799 > caes04.muc.nfs: . 4381:5841(1460) ack 1 win 5840 (DF) 16:41:04.193339 > capc25.muc.799 > caes04.muc.nfs: . 5841:7301(1460) ack 1 win 5840 (DF) 16:41:04.403451 < caes04.muc.nfs > capc25.muc.799: . 1:1(0) ack 7301 win 52732 16:41:04.403467 > capc25.muc.799 > caes04.muc.nfs: . 7301:8761(1460) ack 1 win 5840 (DF) 16:41:04.403477 > capc25.muc.799 > caes04.muc.nfs: . 8761:10221(1460) ack 1 win 5840 (DF) 16:41:04.403486 > capc25.muc.799 > caes04.muc.nfs: . 10221:11681(1460) ack 1 win 5840 (DF) 16:41:04.403496 > capc25.muc.799 > caes04.muc.nfs: P 11681:13141(1460) ack 1 win 5840 (DF) 16:41:04.613256 < caes04.muc.nfs > capc25.muc.799: . 1:1(0) ack 13141 win 46892 16:41:04.613294 > capc25.muc.799 > caes04.muc.nfs: . 13141:14601(1460) ack 1 win 5840 (DF) 16:41:04.613305 > capc25.muc.799 > caes04.muc.nfs: P 14601:16061(1460) ack 1 win 5840 (DF) 16:41:04.613315 > capc25.muc.799 > caes04.muc.nfs: . 16061:17521(1460) ack 1 win 5840 (DF) 16:41:04.613324 > capc25.muc.799 > caes04.muc.nfs: . 17521:18981(1460) ack 1 win 5840 (DF) 16:41:04.613334 > capc25.muc.799 > caes04.muc.nfs: P 18981:20441(1460) ack 1 win 5840 (DF) 16:41:04.815623 < caes04.muc.nfs > capc25.muc.799: . 1:1(0) ack 20441 win 39592 16:41:04.815641 > capc25.muc.799 > caes04.muc.nfs: . 20441:21901(1460) ack 1 win 5840 (DF) 16:41:04.815651 > capc25.muc.799 > caes04.muc.nfs: . 21901:23361(1460) ack 1 win 5840 (DF) 16:41:04.815660 > capc25.muc.799 > caes04.muc.nfs: . 23361:24821(1460) ack 1 win 5840 (DF) 16:41:04.815670 > capc25.muc.799 > caes04.muc.nfs: P 24821:26281(1460) ack 1 win 5840 (DF) 16:41:04.815693 > capc25.muc.799 > caes04.muc.nfs: . 26281:27741(1460) ack 1 win 5840 (DF) 16:41:04.815704 > capc25.muc.799 > caes04.muc.nfs: . 27741:29201(1460) ack 1 win 5840 (DF) 16:41:05.023080 < caes04.muc.nfs > capc25.muc.799: . 1:1(0) ack 29201 win 30832 16:41:05.023117 > capc25.muc.799 > caes04.muc.nfs: . 29201:30661(1460) ack 1 win 5840 (DF) 16:41:05.023127 > capc25.muc.799 > caes04.muc.nfs: . 30661:32121(1460) ack 1 win 5840 (DF) 16:41:05.023137 > capc25.muc.799 > caes04.muc.nfs: P 32121:32917(796) ack 1 win 5840 (DF) 16:41:05.024644 < caes04.muc.nfs > capc25.muc.799: . 1:1(0) ack 32917 win 60032 16:41:05.025192 < caes04.muc.nfs > capc25.muc.799: P 1:165(164) ack 32917 win 60032 16:41:05.025203 > capc25.muc.799 > caes04.muc.nfs: . 32917:32917(0) ack 165 win 5840 (DF) 16:41:05.025272 > capc25.muc.799 > caes04.muc.nfs: P 32917:33057(140) ack 165 win 5840 (DF) 16:41:05.048922 < caes04.muc.nfs > capc25.muc.799: P 165:321(156) ack 33057 win 60032 16:41:05.081819 > capc25.muc.799 > caes04.muc.nfs: . 33057:33057(0) ack 321 win 5840 (DF) 16:41:26.492871 > capc25.muc.39518208 > caes04.muc.nfs: 40 null (DF) 16:41:26.495098 < caes04.muc.nfs > capc25.muc.39518208: reply ok 24 null 16:41:56.494082 > capc25.muc.576389120 > caes04.muc.nfs: 40 null (DF) 16:41:56.500261 < caes04.muc.nfs > capc25.muc.576389120: reply ok 24 null 16:42:26.485359 > capc25.muc.1113260032 > caes04.muc.nfs: 40 null (DF) 16:42:26.487429 < caes04.muc.nfs > capc25.muc.1113260032: reply ok 24 null 16:42:56.486530 > capc25.muc.1650130944 > caes04.muc.nfs: 40 null (DF) 16:42:56.487097 < caes04.muc.nfs > capc25.muc.1650130944: reply ok 24 null 16:43:26.487748 > capc25.muc.2187001856 > caes04.muc.nfs: 40 null (DF) 16:43:26.489881 < caes04.muc.nfs > capc25.muc.2187001856: reply ok 24 null 16:43:34.489177 > capc25.muc.799 > caes04.muc.nfs: F 33057:33057(0) ack 321 win 5840 (DF) 16:43:34.489777 < caes04.muc.nfs > capc25.muc.799: . 321:321(0) ack 33058 win 60032 16:43:34.490370 < caes04.muc.nfs > capc25.muc.799: F 321:321(0) ack 33058 win 60032 16:43:34.490391 > capc25.muc.799 > caes04.muc.nfs: . 33058:33058(0) ack 322 win 5840 (DF) 16:43:34.494041 > capc25.muc.1022 > caes04.muc.32859: udp 80 (DF) 16:43:34.494079 > capc25.muc.1022 > caes04.muc.sunrpc: udp 84 (DF) 16:43:34.494091 > capc25.muc.1022 > caes04.muc.32859: udp 80 (DF) 16:43:34.496654 < caes04.muc.sunrpc > capc25.muc.1022: udp 28 16:43:34.507767 < caes04.muc.32859 > capc25.muc.1022: udp 72 16:43:34.507939 < caes04.muc.32859 > capc25.muc.1022: udp 24 16:43:56.488965 > capc25.muc.2723872768 > caes04.muc.nfs: 40 null (DF) 16:43:56.489405 < caes04.muc.nfs > capc25.muc.2723872768: reply ok 24 null 16:44:26.490297 > capc25.muc.3260743680 > caes04.muc.nfs: 40 null (DF) 16:44:26.490754 < caes04.muc.nfs > capc25.muc.3260743680: reply ok 24 null Slighlty clueless, Birger - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/