Received: by 2002:a25:8b91:0:0:0:0:0 with SMTP id j17csp6601105ybl; Mon, 23 Dec 2019 08:29:47 -0800 (PST) X-Google-Smtp-Source: APXvYqyRSFuEBWttF3D9FAtqgH3W811uDw4ItFBK+BoBbvQraxq2s4v5bf+hxo3DNEfJQdKfNcPv X-Received: by 2002:a05:6830:1353:: with SMTP id r19mr33908620otq.288.1577118587270; Mon, 23 Dec 2019 08:29:47 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1577118587; cv=none; d=google.com; s=arc-20160816; b=QgAg95b6tnI/PvtgANH0HZyfxazQ7v3RS9sfSKL45tfhJSl4AR2uBrfEowzssWpr3C G296CNmJUWXvbE/iZfu05cjaJkRuiQYVnEnSK5GFqXfApJ5vVXX7nVKWXfY+BS5z1V0o Ta/CY4Ronn2J6sN9RJ9XsU7eztIDdLaWYNgmf+so7yT7FUT46FubNbJ8s3ZiUS0tE0br 8FaRwD7m5SMbAgQSKxEAMxU01btZmKJ1w+yuYi8XD0RPQV4ae0k5K/6M9QxZEbKaUBHx oqwHHMcvWFveh3HbyqOkrhGJD+TZ2AAE/bDBBaY9fYH6L9uDyFu8+YtVtYcHGa7m3Hff x1rA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:content-transfer-encoding:mime-version :user-agent:references:in-reply-to:date:cc:to:from:subject :message-id:dkim-signature; bh=vw4KPbMTBUV8sBhtBOLETSLchNFJLtEKQMemtAhU5f0=; b=Dwvd9tvL+eghBDukizongXUjDnQy1wzbm/TWAclq+ow62YXdNKBJLWYn2W3hxbOVbt +cmHkm5+Y3rJmFFopb/wr5aIsOiIM0/+h5WYFKK+zENtz8gkDXinqyZr6mWZFle4GHHY OgjRFb0PWsZQHfr9IMDX3p5UBP7OGAisZGbgfFnHPhPb+mkhnj5wxT1KLb1Js/CsqvL2 lzLQb/ptXV1ak/tdB6QEeIsnkAu2bhvWVja+B3y/9cMfQ1zKrrztS13kY8kzxfXZ9WxE tewahfq4MBjv5xEUWhI9FaTwikuZa8wFTcOh0qrX3brihqIfw7Hiru4vUd4bd5Q4n8sm UQBg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@unipv-it.20150623.gappssmtp.com header.s=20150623 header.b=HILnxJrb; spf=pass (google.com: best guess record for domain of linux-ext4-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-ext4-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id w131si9010061oie.47.2019.12.23.08.29.37; Mon, 23 Dec 2019 08:29:47 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-ext4-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@unipv-it.20150623.gappssmtp.com header.s=20150623 header.b=HILnxJrb; spf=pass (google.com: best guess record for domain of linux-ext4-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-ext4-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726829AbfLWQ3d (ORCPT + 99 others); Mon, 23 Dec 2019 11:29:33 -0500 Received: from mail-wr1-f52.google.com ([209.85.221.52]:34547 "EHLO mail-wr1-f52.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726828AbfLWQ3b (ORCPT ); Mon, 23 Dec 2019 11:29:31 -0500 Received: by mail-wr1-f52.google.com with SMTP id t2so17204397wrr.1 for ; Mon, 23 Dec 2019 08:29:30 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=unipv-it.20150623.gappssmtp.com; s=20150623; h=message-id:subject:from:to:cc:date:in-reply-to:references :user-agent:mime-version:content-transfer-encoding; bh=vw4KPbMTBUV8sBhtBOLETSLchNFJLtEKQMemtAhU5f0=; b=HILnxJrbx3T6SzDiiixqtE05Gi5pWp10VpWyCZky/8WbvfiXsgKT041EAJlLBenpwQ fyU2PejkZ54nk+wiOErY6mKIWP73+QNq3mi8A4cdmt0y8eUmj56M2YRjR/7XtLnKK9r1 6RZzJozOV2vq6X/DKdGOzhXs3GjtX5sYUQhKMeDGfXoLpyaplYL8yVZ9tYz1NdJFs3ae zqgD1F1eCJlN69OjzY9L1WtnD+WlED1DrZ/bFQHLJOnI3lTBXysarcxGNIXV2N+MfKHh 3eb+kByeZq8Velzl9fn9SpadwonD+0kqkyNosFr8l7WM2B8OoZST46JtWDgoJB45NJ4F S+vw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:subject:from:to:cc:date:in-reply-to :references:user-agent:mime-version:content-transfer-encoding; bh=vw4KPbMTBUV8sBhtBOLETSLchNFJLtEKQMemtAhU5f0=; b=A/jMnqOc9qk04579sV3mSDBfYaZ7oDiPoWXTNfVp4zoNJcZ5J+W87z61rjzom6ztge i03AUHlvgm6Wf0C8tTur7riuJUcVeQP5mr1hQvPZsQSNu3b0GnkbtBfdJa8oWxxjz9cG PDicpAdk9F2r33YSb5C4+TbFCQKxQ7jUFR165v6iG3tzKp8hsFYDFMZYL+dZLOxomBva +Y+HCuQtjbutEgHJpFPyXATYLnHfxYfNgDh9nksk0zjWedWoVn2fIOv4eLZ17b00lS9B /npEbC5BrPuyj/PJb5OPgONkhodcWvGA2czOw3dwJLcQbJsI92XekLlFKvLz/vIJX1yS Gocw== X-Gm-Message-State: APjAAAVcfF176WBuFHAHsiymVyBeGGZOizSCsdJjWEFpQLfK9q7DKLQ7 mH4Qown8rmx+MC7yNhN8J6updw== X-Received: by 2002:adf:ea88:: with SMTP id s8mr31113362wrm.293.1577118569579; Mon, 23 Dec 2019 08:29:29 -0800 (PST) Received: from angus.unipv.it (angus.unipv.it. [193.206.67.163]) by smtp.gmail.com with ESMTPSA id p26sm19436362wmc.24.2019.12.23.08.29.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 23 Dec 2019 08:29:29 -0800 (PST) Message-ID: <4c85fd3f2ec58694cc1ff7ab5c88d6e11ab6efec.camel@unipv.it> Subject: Re: AW: Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6 From: Andrea Vai To: "Theodore Y. Ts'o" , Ming Lei Cc: "Schmid, Carsten" , Finn Thain , Damien Le Moal , Alan Stern , Jens Axboe , Johannes Thumshirn , USB list , SCSI development list , Himanshu Madhani , Hannes Reinecke , Omar Sandoval , "Martin K. Petersen" , Greg KH , Hans Holmberg , Kernel development list , linux-ext4@vger.kernel.org, linux-fsdevel@vger.kernel.org Date: Mon, 23 Dec 2019 17:29:27 +0100 In-Reply-To: <20191223162619.GA3282@mit.edu> References: <8196b014b1a4d91169bf3b0d68905109aeaf2191.camel@unipv.it> <20191210080550.GA5699@ming.t460p> <20191211024137.GB61323@mit.edu> <20191211040058.GC6864@ming.t460p> <20191211160745.GA129186@mit.edu> <20191211213316.GA14983@ming.t460p> <20191218094830.GB30602@ming.t460p> <20191223130828.GA25948@ming.t460p> <20191223162619.GA3282@mit.edu> Content-Type: text/plain; charset="UTF-8" User-Agent: Evolution 3.32.5 (3.32.5-1.fc30) MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Sender: linux-ext4-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-ext4@vger.kernel.org Il giorno lun, 23/12/2019 alle 11.26 -0500, Theodore Y. Ts'o ha scritto: > On Mon, Dec 23, 2019 at 09:08:28PM +0800, Ming Lei wrote: > > > > From the above trace: > > > > b'blk_mq_sched_request_inserted' > > b'blk_mq_sched_request_inserted' > > b'dd_insert_requests' > > b'blk_mq_sched_insert_requests' > > b'blk_mq_flush_plug_list' > > b'blk_flush_plug_list' > > b'io_schedule_prepare' > > b'io_schedule' > > b'rq_qos_wait' > > b'wbt_wait' > > b'__rq_qos_throttle' > > b'blk_mq_make_request' > > b'generic_make_request' > > b'submit_bio' > > b'ext4_io_submit' > > b'ext4_writepages' > > b'do_writepages' > > b'__filemap_fdatawrite_range' > > b'ext4_release_file' > > b'__fput' > > b'task_work_run' > > b'exit_to_usermode_loop' > > b'do_syscall_64' > > b'entry_SYSCALL_64_after_hwframe' > > b'cp' [19863] > > 4400 > > > > So this write is clearly from 'cp' process, and it should be one > > ext4 fs issue. > > We need a system call trace of the cp process, to understand what > system call is resulting in fput, (eg., I assume it's close(2) but > let's be sure), and often it's calling that system call. > > What cp process is it? Is it from shellutils? Is it from busybox? > > - Ted I run the cp command from a bash script, or from a bash shell. I don't know if this answer your question, otherwise feel free to tell me a way to find the answer to give you. Thanks, Andrea