Received-SPF: pass (google.com: domain of linux-nfs-owner@vger.kernel.org designates 2620:137:e000::1:18 as permitted sender) client-ip=2620:137:e000::1:18;
IronPort-PHdr: A9a23:Hw95iBKsqdMxqL9eP9mcuWsyDhhOgF28FgIW659yjbVIf+zj+pn5J
 0XQ6L1ri0OBRoTU7f9Iyo+0+6DtUGAN+9CN5XYFdpEfWxoMk85DmQsmDYaMAlH6K/i/aSs8E
 YxCWVZp8mv9P1JSHZP7bkHS5GCu4C4bAVPyORcmTtk=
IronPort-Data: A9a23:R+a7q6MvoTdxCF/vrR1Wl8FynXyQoLVcMsEvi/4bfWQNrUpz3mYFy
 mAbXWyOPfeLMWXzKo92YdnipxwC7Z7VmtA1GXM5pCpnJ55oRWopJjg4wmPYZX76whjrERoPA
 /3z7rAsFehsJpPmjk7F3oPJ8D8shclkepKmULSeYnooGFc/IMscoUsLd9AR09YAbeeRXlvlV
 ePa+6Uz73f8hlaYmkpNg06ygEsHUMba4Vv0jXRiDRx/h2IyolFOZH4pyQBdGFOjKmVcNrbSq
 +8uV9hV9EuBl/smIovNfroW7iTm61MdVOSDoiM+ZkSsvvRNjgEw3YobCPkgVQBari2suddX7
 O9hhZPlHG/FPoWU8Agcez9xNngmeIFjpvrAK3X5ttGPxUrbdXeq2+9pEEw9IYwf/KBwHH1K8
 vsbbjsKa3hvhcrvmO79FrYq25RldZGxVG8ckigIITXxFfkhTIzRa6DD+d8ewSs9lslVW/vSe
 qL1bBI0NkmfPUAUZg9/5JQWgcOIvyKhUBli+EuqvbE240Ph/jF9z+24WDbSUofTHp4K9qqCn
 UrC/mLkElQcOse31zWI6DSvi/XJkCe9X5gdfIBU7dZviVyXg2AWVhsfUALjpeHj0xbkHdVCN
 0YT5ywi67Ao81CmRcX8WBv+p2OYuhkbWJxbFOhSBByxJrT8zgSWKkg4TQJ9L9EDrOoQQhoO7
 HO5kIa8bdBwi4G9RXWY/7aSiDq9PykJMGMPDRM5oRs5D8rL/Nht1kiWJjp3OOvk1YOsQ2qYL
 yWi9XBm390uYdg3O7JXFLwtqx6hvdDsSQ8z6x6/somNs1siPNHNi2BF+THmARtoKY+YShyNu
 SYCks3HtOQWV8jVzmqKXfkHG6yv67CdKjrAjFVzHp4nsTOw53qkeoMW6zZ7TKuIDirmUWC5C
 KMwkVoOjHO2AJdMRfMnC25WI5lxpZUM7fy/CpjpgiNmO/CdjjOv8iB0flK31GvwikUqmqxXE
 c7FLZrzXStEVf43nWPeqwIhPVkDln5WKYT7FcCT8vhb+ebEDJJoYetYbArTNrxhhE96iFyEr
 YgCb6NmNCmzoMWlO3KMrub/3HgPLHMhAovxpdAffemZOg18EX0gDPm5/F/SU90Nokihrc+Rp
 ivVchYAljLX3CSbQS3XNCwLQO6xDP5X8CNjVQRxbA3A8yVyMe6HsvxFH7NpJuZPyQCW5aUpJ
 xXzU57cWasnp/Wu02h1UKQRW6Q7L0r13FrXZXL6CNX9FrY5LzH0FhbfVlOH3EEz4uCf76PSf
 5XIOtvnfKc+
IronPort-HdrOrdr: A9a23:tTfhoairPF/XvZKOxH/8nPSoOHBQX0R13DAbv31ZSRFFG/FwyP
 rCoB1L73XJYWgqM03I+eruBEDyewK4yXcT2/hoAV7CZniehILMFu1fBOTZsl7d8kHFh4lgPM
 RbAtVD4b/LbWSS5PySiGfYLz9J+qj8zEnCv5a5854Cd3APV0gt1XYaNu7NeXcGPzWuSKBJYq
 a0145inX6NaH4XZsO0Cj0sWPXCncTCkNbDbQQdDxAqxQGShXfwgYSKWySw71M7aXdi0L0i+W
 /Kn0jQ4biiieiyzlv523XI55pbtdP9wp9oBdCKiOISNjLw4zzYLbhJavmnhnQYseuv4FElnJ
 3lpAohBd167zfrcmS8sXLWqnzd+Qdrz0Wn5U6TgHPlr8C8bik9EdB9iYVQdQacw1Y8vflnuZ
 g7k16xht5yN1ftjS7979/HW1VBjUyvu0cvluYVkjh2TZYeUrlMtoYSlXklXavoJBiKprzPLd
 MeTf01vJ1tABOnhjHizyNSKeWXLzsO9kzseDlAhiSXuwIm7kyRgXFohvD3pU1wi67VfaM0lN
 gsAp4Y6I2mcfVmE56VJN1xNfdfWVa9Ni4lDgqpUCXa/ec8Sjnwgq+y3Kg49emxPLMSyp93tI
 XmOWkoxVIPRw==
Message-ID: <19e14932-ed88-60a1-844a-0e17deee269d@math.utexas.edu>
Date:   Tue, 15 Feb 2022 16:13:25 -0600
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101
 Thunderbird/91.5.0
Content-Language: en-US
To:     "linux-nfs@vger.kernel.org" <linux-nfs@vger.kernel.org>
From:   Patrick Goetz <pgoetz@math.utexas.edu>
Subject: How are client requests load balanced across multiple nfsd processes?
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?cGtiSEdhK3dXRHVidEJFakFiVGU4NUIzYUtmSmk0WFoyNTJCaml5Tmc1MmVO?=
 =?utf-8?B?dWtyUWxoMXAwUVIyMFhlNlR1bytkRWVnQ1BTMWN4YUJtbWNUR0V1dFhyRTVC?=
 =?utf-8?B?bDMxd2Rob3JpdktPd0pFOHZjeVl1N2xSb1oyNSttaENISWtKanBBQVQvTGZU?=
 =?utf-8?B?U2d2anVHemx3NFd0bW1LbWRibjN3YitCTS9YUTdDOElzOHh0cHM1dTl6OExJ?=
 =?utf-8?B?c1ZUWmFtRk5tTVRwMXh2YnlOeU8wbkZtSkFLYTkyaThtUHJ0RGdGZzR3b3Jr?=
 =?utf-8?B?OVFjVG9lQ3Y5dVhmdnNpb2pKOXhqNTNzaGpPYUVFZWs5eXByTWp6RU1aQWN1?=
 =?utf-8?B?U0hKbGFUUWNDQWlidjhpZ016SWtHWHdnTVBiOWlJZW56WjdRQll0VktHUVhh?=
 =?utf-8?B?bTlzSHJkWjBPUGZRdEYzUnZjUk9kZ3lXNXlSVGFVRDN3ZlI5OUVnb01YU3h2?=
 =?utf-8?B?aFd6aHU0aVR5R2VJWDJTQ0xwRW5tSU5ETkpEL1NUWmNyNW9XVzFiL3VWQllN?=
 =?utf-8?B?SUMrNCtyMEFodEMrVDFsbUZIaWxvSW5Ba2h6NzNFa0d6WEVjcU1IM09Td3Za?=
 =?utf-8?B?d2oxT0lHWEt6a3lPMVBNRlBvRmtCbkNiZUdaNGtLUy9DN29JdzBnT0dKUGpz?=
 =?utf-8?B?eHRwaEFoNTdWVEJINjZUMGcwU3BHVU52eTZsR0hqTmNrN2dZY3ZuU2Q5aFJO?=
 =?utf-8?B?dzNSVmNsWGVHYVFIWjBQUHlpZDliRGhhbTViL2ZGbzVmRDJSbGRvWFZoMUh0?=
 =?utf-8?B?RUROaHpaanVSZjcxTXhhREMxZEdMVFp2TlhnNC9FMnVWZDV6WXlZSGNsblc5?=
 =?utf-8?B?L0xMOXJBRjVkdDVJc3J1TmowNXhyYnZzMk9IcjQ4TlpSWTdra243VWZyQVhu?=
 =?utf-8?B?YnB4SVUrU1MyQXQrc1Q4dStPT2hROE1UU1A5bDlnRndYRkRDTnJLVkk0blpm?=
 =?utf-8?B?TlN1NndsYlRPYnJtU3R0eE5pSkhnREk2WFBpNkVXdGx6bnMwVTVqakVhOUdQ?=
 =?utf-8?B?bzFRa2wxOXV1T0dZQ0Y1THY1N3dCVkhNdC9pQmkxc2VKK25XWW11dnJIbDNw?=
 =?utf-8?B?SkVldDZSUWZRNjV2a25qSDc0d3hFdWVpRFJyRGxBeVgzc2R6Ri8vV3VIVGht?=
 =?utf-8?B?SElERURMZlErdTl2L012MTJ0Rmd6VGpDclpKeUFRdEVVNlpBS0tmWkJLVWdZ?=
 =?utf-8?B?U0RGMVpSd29OcldvMjRBa2k5d0NDd2N6eXFPTEZjalo3dkFTck1pQkJuMUd5?=
 =?utf-8?B?c1NZMmNrZWg0T3J6NDdJaWdYa3lIeEVqMTBRMTBpdFZtQ3pwN1lIOUZld0x1?=
 =?utf-8?B?cEZFa2FPZ0JHVGJYWmlHY0ZjZnczTURlL0VSZXlCb2FKVm9PQVlUZnNLZnQy?=
 =?utf-8?B?ZVUvQ29ZVTgxN29JaWsxUko3L3R0MC9VSTI4MXpkUFdyN2hxYjRLaG4vUzVt?=
 =?utf-8?B?VXg1a1dmNnJEbDc2RHJ6azFyUktSTDBtN0NhUUtaSlpoMXVSZXFmYThibmpR?=
 =?utf-8?B?MzU3eVdQMlhkQ3dlN0JGczFleFEvR2xHTkNVRDBNMU9TKzhSQXcwS1JQQmNG?=
 =?utf-8?B?SWdMTitiTkdFK0hROHJ4YjgyTzcrZWk2aDRHWGJjV3BGM3IyUXN3c0dYb1hC?=
 =?utf-8?B?Z2NWalNEZXVCeVJxUnpsblhEZUE2RDJCVXBPUU5PZEFTQ2pFcEMxUHdDSENJ?=
 =?utf-8?B?YndjVTV1TU5VRlJMTkZHK3JlRWU1YU9ZbnJqK2tLT1lTK01TL1M0T0IraGtr?=
 =?utf-8?B?U2RJcUNEWHgzMWZWRUcvYU9MS0FCTCs2Y1hyd0NVY3c2cldqY0FNV0R6eXl6?=
 =?utf-8?B?YUl5VTVvVUFFQWErd3cydmpxWGFBWUVsT2I4dEdnbWoxMEFaNWpoSHNxbHho?=
 =?utf-8?B?Slh3TzFVLzNhT1l0WDhocjVSdVVnWFJrbm5zWUh1aUhJOVBkaTVQdEpONFhO?=
 =?utf-8?B?S1ZmUjE0V0xsUlozTmNlNnEwRklRWXNtV1JvY1RPTzhVcVVXaTROb2VZcVh5?=
 =?utf-8?B?cjVqd1lzaURpTDlBeHNnblFrNnNlRm4vZ0k1QjVERDB0RWhpOHB3NW0zSEFq?=
 =?utf-8?B?Q25Ra0syUDlkLzlpZFVtUjNUV0hnTVNBZ0lUSHVDRzBNb0xTVWZkb1RWN3px?=
 =?utf-8?B?eVlPaUpaQ1grYXFxczJiRUd3cHpENDdTeUROVlNwQlNnb1pSRWd6RGhlWGtm?=
 =?utf-8?Q?/met718Rdj97h6kqPm2WYDU=3D?=
X-OriginatorOrg: math.utexas.edu
X-MS-Exchange-CrossTenant-Network-Message-Id: c12fe226-b354-4257-1b75-08d9f0d0643f
X-MS-Exchange-CrossTenant-AuthSource: BYAPR06MB3848.namprd06.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 15 Feb 2022 22:13:27.6919
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 31d7e2a5-bdd8-414e-9e97-bea998ebdfe1
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: FRgTssZPnGX9kDgwzcLHT3XpADcZ6YqgnRkXTZNKCkqcAKsDXoAOs2HZ101DlI1RvirzWTQp4Ga+fu71DJUSkg==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: MW4PR06MB8947
X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,DKIM_SIGNED,
	DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,RDNS_NONE,
	SPF_HELO_NONE,T_SCC_BODY_TEXT_LINE autolearn=no autolearn_force=no
	version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
	lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-nfs.vger.kernel.org>
X-Mailing-List: linux-nfs@vger.kernel.org

However inappropriate this question is, I'm asking anyway, as this has 
been driving me nuts for 2 weeks and I can't find an answer.

When I set

   RPCNFSDCOUNT=16

what I thought this did was spin up an nfsd thread master with 15 
threads and the thread master would pass out client requests to the 
threads, but looking at /proc/$PID/status -> TGID clearly indicates 
these are all entirely separate processes. (I wasn't sure if ps showed 
threads as separate processes; apparently it doesn't.)

So the question is how do different client requests get farmed out to 
different nfsd daemons for service? Who's actually listening on port 2049?

If there's a reference other than the source code where I can read up on 
this, I'm interested. I looked in a couple of linux programming books 
including Richard Stevens and couldn't find what I was looking for.

This was all prompted by some vendor trying to sell me an EC (Erasure 
Coding) n+m system who commented "NFS isn't multi-threaded, NFS can only 
communicate with one server, for a shared/mounted filesystem, so it will 
always be limited to the speed of that NFS Server. POSIX/Multi-threaded 
means the filesystem is parallel and can be reading/writing to multiple 
nodes at once in a storage cluster/setup. The opposite of NFS."

I think pNFS addresses this, but then how does one implement pNFS?