From: Josef Bacik <jbacik@fb.com>
To: Adam Borowski <kilobyte@angband.pl>
CC: "linux-block@vger.kernel.org" <linux-block@vger.kernel.org>,
        "nbd-general@lists.sourceforge.net" 
        <nbd-general@lists.sourceforge.net>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>
Subject: Re: nbd drops connection on most writes
Thread-Topic: nbd drops connection on most writes
Thread-Index: AQHTAhr+9zeMz0rdfkCsH9zyatoTo6JeM+4H
Date: Fri, 21 Jul 2017 12:22:51 +0000
Message-ID: <F5862F0D-20D5-4CEC-9FAB-C5ED306DDFB6@fb.com>
References: <20170721121439.ofwm3lfuzqjsvjok@angband.pl>
In-Reply-To: <20170721121439.ofwm3lfuzqjsvjok@angband.pl>
Accept-Language: en-US
Content-Language: en-US
spamdiagnosticoutput: 1:99
spamdiagnosticmetadata: NSPM
Content-Type: text/plain; charset="utf-8"
MIME-Version: 1.0
X-MS-Exchange-CrossTenant-originalarrivaltime: 21 Jul 2017 12:22:51.1438
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 8ae927fe-1255-47a7-a2af-5f3a069daaa2
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM5SPR00MB335
X-OriginatorOrg: fb.com
X-Proofpoint-Spam-Reason: safe
X-FB-Internal: Safe
X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:,, definitions=2017-07-21_06:,,
 signatures=0
Sender: linux-kernel-owner@vger.kernel.org
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org
Content-Transfer-Encoding: 8bit
X-MIME-Autoconverted: from base64 to 8bit by nfs id v6LCNHmN009562
Content-Length: 5013
Lines: 83

Oh shit the default timeout is 0 if you don't set it in the client.  Use the timeout option with nbd client and it should fix it for you.  I'll send something up to make this a sane default.  Thanks,

Josef

Sent from my iPhone

> On Jul 21, 2017, at 8:15 AM, Adam Borowski <kilobyte@angband.pl> wrote:
> 
> Hi!
> I'm afraid that 4.13-rc1 nbd aborts connection on writes for me:
> 
> [  251.938384] block nbd0: Send data failed (result -11)
> [  251.943484] block nbd0: Request send failed trying another connection
> [  251.950034] block nbd0: Receive control failed (result -32)
> [  251.955676] block nbd0: Attempted send on invalid socket
> [  251.961022] print_req_error: I/O error, dev nbd0, sector 2206344
> [  251.961025] block nbd0: shutting down sockets
> 
> Not all kinds of writes trigger the problem.  For example, you can dd to the
> nbd block device, likewise badblocks -w succeeds without a hitch.  Yet at
> least btrfs and swap disconnect nearly immediately.  Reads seem to work: for
> example, btrfs can usually mount and scrub successfully, yet minor writes
> that happen on a filesystem mounted rw even without explicit user-level
> writes cause a disconnect in a short time.  "Real" writes to the filesystem
> trigger it apparently outright.  Likewise, to use swap you need to write to
> it first, thus it fails quickly.
> 
> Reproduced on arm64 (Pine64) first.  As this SoC just switched from an
> out-of-tree ethernet driver to a completely different new one (dwmac-sun8i),
> and such a switch can't be bisected, I assumed that's the culprit and did
> not complain while in -next.
> 
> However, turns out the same happens on a bog-standard amd64, both on bare
> metal and in qemu.
> 
> In all of these cases, the server is an amd64 Debian stretch, kernel
> 4.9.30-2+deb9u2, nbd-server 1:3.15.2-3.
> 
> Bisect blames dc88e34d "nbd: set sk->sk_sndtimeo for our sockets", and
> indeed, reverting that patch makes everything fine again.
> 
> 
> Bisect log:
> # bad: [63a86362130f4c17eaa57f3ef5171ec43111a54e] Merge tag 'pm-4.13-rc2' of git://git.kernel.org/pub/scm/linux/kernel/git/rafael/linux-pm
> # good: [6f7da290413ba713f0cdd9ff1a2a9bb129ef4f6c] Linux 4.12
> git bisect start 'linus/master' 'v4.12'
> # bad: [55a7b2125cf4739a8478d2d7223310ae7393408c] Merge tag 'arm64-upstream' of git://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux
> git bisect bad 55a7b2125cf4739a8478d2d7223310ae7393408c
> # bad: [1849f800fba32cd5a0b647f824f11426b85310d8] Merge tag 'armsoc-dt' of git://git.kernel.org/pub/scm/linux/kernel/git/arm/arm-soc
> git bisect bad 1849f800fba32cd5a0b647f824f11426b85310d8
> # bad: [cbcd4f08aa637b74f575268770da86a00fabde6d] Merge tag 'staging-4.13-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/gregkh/staging
> git bisect bad cbcd4f08aa637b74f575268770da86a00fabde6d
> # bad: [1b044f1cfc65a7d90b209dfabd57e16d98b58c5b] Merge branch 'timers-core-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
> git bisect bad 1b044f1cfc65a7d90b209dfabd57e16d98b58c5b
> # bad: [892ad5acca0b2ddb514fae63fa4686bf726d2471] Merge branch 'locking-core-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
> git bisect bad 892ad5acca0b2ddb514fae63fa4686bf726d2471
> # bad: [e442cbf910c71fba5926cf757dd7f8fcce22fc5f] pktcdvd: remove the call to blk_queue_bounce
> git bisect bad e442cbf910c71fba5926cf757dd7f8fcce22fc5f
> # bad: [d86c4d8ef31b3d99c681c859cb4e936dafc2d7a4] nvme: move reset workqueue handling to common code
> git bisect bad d86c4d8ef31b3d99c681c859cb4e936dafc2d7a4
> # bad: [fdd050b5b3c96813ae6756ed68157d32ba31b9f2] Merge branch 'uuid-types' of bombadil.infradead.org:public_git/uuid into nvme-base
> git bisect bad fdd050b5b3c96813ae6756ed68157d32ba31b9f2
> # bad: [a104c9f22c7d073d4ae308ca36383ce5cc4631cc] nvme-rdma: fix merge error
> git bisect bad a104c9f22c7d073d4ae308ca36383ce5cc4631cc
> # good: [b040ad9cf6a169cc000a5324fcada695dfa1f4b3] loop: fix error handling regression
> git bisect good b040ad9cf6a169cc000a5324fcada695dfa1f4b3
> # bad: [36ffc6c1c0e67acdacb53348350d0a37206dbadf] block_dev: propagate bio_iov_iter_get_pages error in __blkdev_direct_IO
> git bisect bad 36ffc6c1c0e67acdacb53348350d0a37206dbadf
> # bad: [f729b66fca43d850d564b264c2033980c00a14b0] gfs2: remove the unused sd_log_error field
> git bisect bad f729b66fca43d850d564b264c2033980c00a14b0
> # bad: [401741547f95c0883fe143ac446d92c772937556] nvme-lightnvm: use blk_execute_rq in nvme_nvm_submit_user_cmd
> git bisect bad 401741547f95c0883fe143ac446d92c772937556
> # bad: [dc88e34d69d87c370deaa9d613dac8e3a0411f59] nbd: set sk->sk_sndtimeo for our sockets
> git bisect bad dc88e34d69d87c370deaa9d613dac8e3a0411f59
> # first bad commit: [dc88e34d69d87c370deaa9d613dac8e3a0411f59] nbd: set sk->sk_sndtimeo for our sockets
> 
> 
> Meow!
> -- 
> ⢀⣴⠾⠻⢶⣦⠀ 
> ⣾⠁⢠⠒⠀⣿⡁ A dumb species has no way to open a tuna can.
> ⢿⡄⠘⠷⠚⠋⠀ A smart species invents a can opener.
> ⠈⠳⣄⠀⠀⠀⠀ A master species delegates.