Received-SPF: pass (google.com: domain of linux-nfs-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <7ba38377-7992-7f0f-d905-cceb42510f39@linaro.org>
Date:   Mon, 9 Jan 2023 09:14:16 +0100
MIME-Version: 1.0
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101
 Thunderbird/102.6.1
Subject: Re: Regression: NULL pointer dereference after NFS_V4_2_READ_PLUS
 (commit 7fd461c47)
To:     Trond Myklebust <trondmy@hammerspace.com>
Cc:     Anna Schumaker <Anna.Schumaker@Netapp.com>,
        linux-nfs <linux-nfs@vger.kernel.org>,
        Linux Kernel Mailing List <linux-kernel@vger.kernel.org>,
        "regressions@lists.linux.dev" <regressions@lists.linux.dev>
References: <f591b13c-4600-e2a4-8efa-aac6ad828dd1@linaro.org>
 <82526863-d07a-0a5d-2990-1555b1387f26@linaro.org>
 <2C5E9725-F152-4D2E-882E-CF92A35481BF@hammerspace.com>
Content-Language: en-US
From:   Krzysztof Kozlowski <krzysztof.kozlowski@linaro.org>
In-Reply-To: <2C5E9725-F152-4D2E-882E-CF92A35481BF@hammerspace.com>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit
Precedence: bulk

On 08/01/2023 18:09, Trond Myklebust wrote:
> Hi Krzysztof,
> 
>> On Jan 8, 2023, at 08:25, Krzysztof Kozlowski <krzysztof.kozlowski@linaro.org> wrote:
>>
>> [You don't often get email from krzysztof.kozlowski@linaro.org. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification]
>>
>> On 07/01/2023 16:44, Krzysztof Kozlowski wrote:
>>> Hi,
>>>
>>> Bisect identified commit 7fd461c47c6c ("NFSv4.2: Change the default
>>> KConfig value for READ_PLUS") as one leading to NULL pointer exception
>>> when mounting NFS root on NFSv4 client:
>>>
>>> [   25.739003] systemd[1]: Set hostname to <odroidhc1>.
>>> [   25.771714] systemd[1]: Failed to bump fs.file-max, ignoring: Invalid
>>> argument
>>> [   26.199478] 8<--- cut here ---
>>> [   26.201366] Unable to handle kernel NULL pointer dereference at
>>> virtual address 00000004
>>> ...
>>> [   26.555522]  mmiocpy from xdr_inline_decode+0xec/0x16c
>>> [   26.560628]  xdr_inline_decode from nfs4_xdr_dec_read_plus+0x178/0x358
>>> [   26.567130]  nfs4_xdr_dec_read_plus from call_decode+0x204/0x304
>>>
>>> Full OOPS attached. Full log available here:
>>> https://krzk.eu/#/builders/21/builds/3901/steps/15/logs/serial0
>>>
>>> Disabling NFS_V4_2_READ_PLUS fixes the issue, so obviously the commit is
>>> not the cause, but rather making it default caused the regression.
>>>
>>> I did not make the bisect yet which commit introduced it, if every
>>> config includes NFS_V4_2_READ_PLUS.
>>
>> When every kernel is built with NFS_V4_2_READ_PLUS, bisect pointed to:
>> d3b00a802c84 ("NFS: Replace the READ_PLUS decoding code")
>>
>> commit d3b00a802c845a6021148ce2e669b5a0b5729959
>> Author: Anna Schumaker <Anna.Schumaker@Netapp.com>
>> Date:   Thu Jul 21 14:21:34 2022 -0400
>>
>>    NFS: Replace the READ_PLUS decoding code
>>
>>    We now take a 2-step process that allows us to place data and hole
>>    segments directly at their final position in the xdr_stream without
>>    needing to do a bunch of redundant copies to expand holes. Due to the
>>    variable lengths of each segment, the xdr metadata might cross page
>>    boundaries which I account for by setting a small scratch buffer so
>>    xdr_inline_decode() won't fail.
>>
>>    Signed-off-by: Anna Schumaker <Anna.Schumaker@Netapp.com>
>>    Signed-off-by: Trond Myklebust <trond.myklebust@hammerspace.com>
>>
>> With a trace:
>> [   25.898462] systemd[1]: Set hostname to <odroidhc1>.
>> [   25.933746] systemd[1]: Failed to bump fs.file-max, ignoring: Invalid
>> argument
>> [   25.986237] random: crng init done
>> [   26.264564] 8<--- cut here ---
>> [   26.266823] Unable to handle kernel NULL pointer dereference at
>> virtual address 00000fe8
>> ...
>> [   26.597263]  nfs4_xdr_dec_read_plus from call_decode+0x204/0x304
>> [   26.603222]  call_decode from __rpc_execute+0xd0/0x890
>> [   26.608328]  __rpc_execute from rpc_async_schedule+0x1c/0x34
>> [   26.613960]  rpc_async_schedule from process_one_work+0x294/0x790
>> [   26.620030]  process_one_work from worker_thread+0x54/0x518
>> [   26.625570]  worker_thread from kthread+0xf4/0x128
>> [   26.630336]  kthread from ret_from_fork+0x14/0x2c
>>
> 
> Is this test being run against a 6.2-rc2 server, or is it an older server platform? We know there were bugs in older server implementations, so the question is whether this might be a problem with handling a bad/corrupt RPC reply from the server, or whether it is happening against code that is supposed to have been fixed?

I would say that buggy server should not cause NULL pointer dereferences
on the client. Otherwise this is a perfect recipe for a rogue server in
the network to start crashing clients and running exploits... Imagine a
compromised machine (through some other means) in a local company
network running now a server with NFS share "HR salary data" or "HR
planned layoffs", where unsuspected people in that network access it
leading to exploit of NFS code on their side...

Server is Raspberry Pi 3 kernel: 5.10.92-2-rpi-legacy-ARCH

Which points that it is not latest stable, so anyway I need to update.

Best regards,
Krzysztof