Received: by 2002:a05:6a10:c604:0:0:0:0 with SMTP id y4csp225755pxt; Wed, 4 Aug 2021 20:59:51 -0700 (PDT) X-Google-Smtp-Source: ABdhPJxRk1qZ6Ws5tOnhnh6+eLT2NKTu0ysWqqfAk7wdIXZSQvDA9rbKs5B2IXGpAXM5S5XB0m5S X-Received: by 2002:a17:906:c055:: with SMTP id bm21mr2680929ejb.350.1628135991441; Wed, 04 Aug 2021 20:59:51 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1628135991; cv=none; d=google.com; s=arc-20160816; b=r1kIEViEESZbSZ4InF1o2nl8OFPW5RmHXIMSP5n0mEGouE9TLNDsFIS11Jrptxo4Ed DhdGnq00uadzg+tgpp4cACAWBzUyc8eR8K4A+xOIIo/b+OAoau3eWMPMWlj3gQwE6iHm xeVvVrV4RTYptbNZGowa8MG7vgIIgNssPXAhy8QCKeOvZ7DCZ7vchpznKT9tdZczTNAm G7n2pcMabJG6l3T0/3FONm859EzZ19ylv+BO4nJ4+QDKLBhFRcxx2BylN+PQN/jxq8CH LxWs5c6GAbSQsdjzC13xj47Nk7csji6zR5UEXPL+f1e673pzKfx073I6+6sa5cgFQy2E lz8w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:thread-topic:thread-index :content-transfer-encoding:mime-version:subject:message-id:to:from :date:dkim-signature:dkim-filter; bh=tGovMBm85IvDU8LaObR+eSrzxLVJP8ixpeBBAlWEdJs=; b=PUhPrbBa96yE2Gn25V+pOobFlXAG+QQ8bltYkJffJM5bFy4cGF3XMmC/vBSy3/KWhz Pm8teYBiaS4R/rTXXZFQLB832ryISx/L0J+bRTAQm1YEznGiBZm26JsplQHr7bBfQdwE Y3OyiDxm9dI7co74k2msqqTHCcpIIvtSsC054OGfeK/7TgdAQSc3w2B+maLJYmvcwZ41 sMWnSU2A32Mtb/+VsLSqxdEr0XyryJrQf5qwlHjeADAEeOlv1lPir/WsbS8I3ztl9GCX KZ1RWbbNROJChOIuGt1J1LThFHYAtB8XekSZLkkoJU/OAPQ93cGQxraihz4IpUQTOnvv wiTQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@raptorengineering.com header.s=B8E824E6-0BE2-11E6-931D-288C65937AAD header.b=U9daUbUR; spf=pass (google.com: domain of linux-nfs-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-nfs-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=raptorengineering.com Return-Path: Received: from vger.kernel.org (vger.kernel.org. [23.128.96.18]) by mx.google.com with ESMTP id 5si4398327edv.551.2021.08.04.20.59.16; Wed, 04 Aug 2021 20:59:51 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-nfs-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18; Authentication-Results: mx.google.com; dkim=pass header.i=@raptorengineering.com header.s=B8E824E6-0BE2-11E6-931D-288C65937AAD header.b=U9daUbUR; spf=pass (google.com: domain of linux-nfs-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-nfs-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=raptorengineering.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230177AbhHEAAe (ORCPT + 99 others); Wed, 4 Aug 2021 20:00:34 -0400 Received: from mail.rptsys.com ([23.155.224.45]:43695 "EHLO mail.rptsys.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229991AbhHEAAe (ORCPT ); Wed, 4 Aug 2021 20:00:34 -0400 Received: from localhost (localhost [127.0.0.1]) by mail.rptsys.com (Postfix) with ESMTP id DD55A37B2A7AB3 for ; Wed, 4 Aug 2021 19:00:20 -0500 (CDT) Received: from mail.rptsys.com ([127.0.0.1]) by localhost (vali.starlink.edu [127.0.0.1]) (amavisd-new, port 10032) with ESMTP id sGRSBKbmb8wV for ; Wed, 4 Aug 2021 19:00:20 -0500 (CDT) Received: from localhost (localhost [127.0.0.1]) by mail.rptsys.com (Postfix) with ESMTP id 44C1B37B2A7AAF for ; Wed, 4 Aug 2021 19:00:20 -0500 (CDT) DKIM-Filter: OpenDKIM Filter v2.10.3 mail.rptsys.com 44C1B37B2A7AAF DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=raptorengineering.com; s=B8E824E6-0BE2-11E6-931D-288C65937AAD; t=1628121620; bh=tGovMBm85IvDU8LaObR+eSrzxLVJP8ixpeBBAlWEdJs=; h=Date:From:To:Message-ID:MIME-Version; b=U9daUbUR5aZ9L7vDZ3vp982nnvK4x2iLUwrf8746uGTo98OpV3yMkY7vgm+V+Fr0a NeGkLHnfTOBfc/j+/3P0abvUwcqCEoIITpnOqoJaSEF2I5D64av7s8ow7zVPfJXtWA bgTE7R57dfTu2NcaL8HAUXlOdiBmx6pJ7gldSSrg= X-Virus-Scanned: amavisd-new at rptsys.com Received: from mail.rptsys.com ([127.0.0.1]) by localhost (vali.starlink.edu [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id 9Si816jfNJz1 for ; Wed, 4 Aug 2021 19:00:20 -0500 (CDT) Received: from vali.starlink.edu (vali.starlink.edu [192.168.3.21]) by mail.rptsys.com (Postfix) with ESMTP id 26FDB37B2A7AAC for ; Wed, 4 Aug 2021 19:00:20 -0500 (CDT) Date: Wed, 4 Aug 2021 19:00:20 -0500 (CDT) From: Timothy Pearson To: linux-nfs Message-ID: <985631970.48634.1628121620017.JavaMail.zimbra@raptorengineeringinc.com> Subject: Callback slot table overflowed MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Mailer: Zimbra 8.5.0_GA_3042 (ZimbraWebClient - GC83 (Linux)/8.5.0_GA_3042) Thread-Index: pWASFVTGcuTitVsqm6vgvEReZrSGHg== Thread-Topic: Callback slot table overflowed Precedence: bulk List-ID: X-Mailing-List: linux-nfs@vger.kernel.org All, We've hit an odd issue after upgrading a main NFS server from Debian Stretch to Debian Buster. In both cases the 5.13.4 kernel was used, however after the upgrade none of our ARM thin clients can mount their root filesystems -- early in the boot process I/O errors are returned immediately following "Callback slot table overflowed" in the client dmesg. I am unable to find any useful information on this "Callback slot table overflowed" message, and have no idea why it is only impacting our ARM (armel) clients. Both 4.14 and 5.3 on the client side show the issue, other client kernel versions were not tested. Curiously, increasing the rsize/wsize values to 65536 or higher reduces (but does not eliminate) the number of callback overflow messages. The server is a ppc64el 64k page host, and none of our pcc64el or amd64 thin clients are experiencing any problems. Nothing of interest appears in the server message log. Any troubleshooting hints would be most welcome. Thank you!