Received: by 2002:a25:8b91:0:0:0:0:0 with SMTP id j17csp5841056ybl; Sun, 22 Dec 2019 16:13:13 -0800 (PST) X-Google-Smtp-Source: APXvYqy2PKbFFTDonURPjV2+ah0wqmReN0jnHuwCIgGikpFtKD0flK3z/VjC1fHxMj0zcAfLHFBd X-Received: by 2002:a9d:7c91:: with SMTP id q17mr13170730otn.293.1577059992935; Sun, 22 Dec 2019 16:13:12 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1577059992; cv=none; d=google.com; s=arc-20160816; b=Xk1uBK4iQKZcZ5Dk4LEEJe/mUbtn1fgHfvPp4G0GhwHvdMSuZCWctJkovyvFB9Juud 5FiSJ/evf6g1x22asPwIDL0kzKYTzv3mVBH1nQ7ave0FVCt10Cm3YJpqCnYJV0CKeQjQ 7RtfJAn7a208nLWcT7sZyelOdNJbEWg46QzH5e0t5bKrPk62EDffwRNRdzy3EQj5G5Ps XylMgeW32eRIZ6QSlX6bZBcCwOf5bkAgv/MxY+7JHL+Jk1duK9ZHkFpiw7Dx/2kwhASQ nDyA0rnPRYfn/3fO3YXfzBp6I6RBl02Mk4iNYzo80rWW17vnJ/DZ/G0z+OO/B14Mqy3g SUHg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:message-id:user-agent:references :in-reply-to:subject:cc:to:from:date:content-transfer-encoding :mime-version:dkim-signature; bh=4nuc1AoyG636k7YM3XrC/oDbYE0mXjxK+LkO7UE/R7g=; b=TfytEvlsbgs1zdL181lg9JTLDcG1/z9whbjbF7kBggLCU/GmHtO3EzfnLtpq8Ezvkw SqxP7QgEOHIsxe2e9ZXsIzpx/+8Oi3b6xSFkPuGXrTmoejNYskaY/IHFcZngzdo3W1b+ cuqyYfXvsA18ksI9JswG33IKAqqoF68uqfw+VbX9sOi9MPa1a/dkZG7mSOYXS8fm08M3 Lzmiq8FJ+FIAyFeWPHVNWU/AaQbkHBBvgmT3QREZo8d0ekKo6tMNk74qjB2djWYw42Oj 4ydJxbFtthQ7+3ktgLTr+o67grnO8dDwQ2V4pZUIEyL0cyfgEimI1LVlDbkGL8kegoEh RrUA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@natalenko.name header.s=dkim-20170712 header.b="rC+8/udN"; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=natalenko.name Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id f15si3952356oib.61.2019.12.22.16.12.29; Sun, 22 Dec 2019 16:13:12 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@natalenko.name header.s=dkim-20170712 header.b="rC+8/udN"; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=natalenko.name Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726539AbfLWAKX (ORCPT + 99 others); Sun, 22 Dec 2019 19:10:23 -0500 Received: from vulcan.natalenko.name ([104.207.131.136]:39474 "EHLO vulcan.natalenko.name" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726114AbfLWAKX (ORCPT ); Sun, 22 Dec 2019 19:10:23 -0500 Received: from mail.natalenko.name (vulcan.natalenko.name [IPv6:fe80::5400:ff:fe0c:dfa0]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by vulcan.natalenko.name (Postfix) with ESMTPSA id A17CB66A2C3; Mon, 23 Dec 2019 01:10:20 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=natalenko.name; s=dkim-20170712; t=1577059820; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=4nuc1AoyG636k7YM3XrC/oDbYE0mXjxK+LkO7UE/R7g=; b=rC+8/udNxj1AKXwCy4zpbuZBGwDe69x2NVua5uDpde2nu2jnucbjQ4R2PaFph7S4pQWTHe v3v7N+uwBriOz51Yf8BWdL9NnE5q47iIPZmdKas96L3v72TqOy1u04X19IbMjpjb9P8ruy fptiLJaqSFKrFhiW8dVqWqX2OTRrrQU= MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII; format=flowed Content-Transfer-Encoding: 7bit Date: Mon, 23 Dec 2019 01:10:20 +0100 From: Oleksandr Natalenko To: Paolo Valente Cc: linux-kernel , linux-block , SIMONE RICHETTI <206161@studenti.unimore.it>, tytso@mit.edu Subject: Re: Injecting delays into block layer In-Reply-To: <942604AE-5A91-4E05-869F-74A7EAC5A247@linaro.org> References: <3D695D19-B226-4093-9C27-CE561ED08CB7@linaro.org> <942604AE-5A91-4E05-869F-74A7EAC5A247@linaro.org> User-Agent: Roundcube Webmail/1.4.1 Message-ID: <21d34f9558a286c60352b5634f291f71@natalenko.name> X-Sender: oleksandr@natalenko.name Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Hi. On 06.12.2019 17:17, Paolo Valente wrote: > Simone (in CC) and I have worked a little bit on reproducing the I/O > freeze you report. Simone made a small change in SCSI_debug, which > makes the latter serve I/O with a highly varying random delay (100ms - > 1s), about twice a second. > > Then, to generate some fluctuating and heavy I/O, he ran the > comm_startup_lat.sh script of my S suite with SCSI_debug a few times. > Unfortunately, he didn't succeed in reproducing the problem. If you > want, we can send you a patch with his change for SCSI_debug. > > Any news on your side? FWIW, I guess I'm safe to exclude BFQ at the moment since I've encountered a very similar issue without having BFQ enabled. Also, I think this might be unrelated to the block layer at all. I suspect there's some race between MADV_MERGEABLE and MADV_DONTNEED since this is what's hammering the affected tasks and what I see from the call traces. I'll investigate further and probably talk to MM people instead. Sorry for the noise. -- Oleksandr Natalenko (post-factum)