Received: by 2002:ac0:a594:0:0:0:0:0 with SMTP id m20-v6csp4217694imm; Mon, 14 May 2018 04:31:00 -0700 (PDT) X-Google-Smtp-Source: AB8JxZrG+LGXX/kAgZXoOUv1kQErwzByJvr4z669tqWCHJUMEIOvCJlVYDjfdX5UWRlW+uA+1cCf X-Received: by 2002:a62:e107:: with SMTP id q7-v6mr10103150pfh.226.1526297460398; Mon, 14 May 2018 04:31:00 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1526297460; cv=none; d=google.com; s=arc-20160816; b=RaxdHSUhep7gdpGGVHixFePPHcOpmjRFQZ/AOx+0OszncBGWra2QvccInxJuA0B6Sx Y9whwQ0Zpo4j0ToU1PpZ6N4iP7MrHU4NHilxYwLEPXDz6g2snuA0TrCwAWQ51/4Jp1PJ D94S7AS28/sh4W9XNozCmxHnsvjBGciHbZmP1F+U2KKJc+xFL5MXmO+77TROe6D6t/QS 4qkKLGvOpubtVRjntiLSbXscDNJMVpE0+IckOHgVty5uqd1WIyCx4CzR5+xgCcsW1w2V Q6ZqhInsuuX8OX5WSY4+mQolOTskCQ49UQ7soL6Hw6qzBpJ+SDiHxsGgD8i5pepI92Ps KZkA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:content-transfer-encoding :content-language:mime-version:user-agent:date:message-id:cc:to :subject:from:dkim-signature:arc-authentication-results; bh=hSlGKwLog2P5LDbsZjJ4C+eqj/Y26eRRVptfXXVfFj8=; b=0Ow+QowxcV+8laOL1DtR9PfTV5wIXs9psWAoG/Ct2qMSzWm2MFHpv4luHsAOQvj0WG jFzYXnhYw7aLHH2s4v21h3zDZWh8P31pj/HS3eJouotmBi9lXBczV+8lVfUFdpaX/3uo bPw0wv240qGNyqBpUF4W0NbmNazkZln0PedfnFY/Ic+6ypCdtnMiZopNBpl9TsOT0hSP 52eMp3iz8UMP6fc3sLyyxNwOBR3FDirrarNqtPCGMTTD1JgTIKqJFqwBz3fZ7Q/BRtJn yujLVt0S5Xgw4U/J2+blfjka2ef0Ot9CO/10+bjyqcs60hwLZyr+jnwG/wyQCpS7EoVO 5ORw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@ti.com header.s=ti-com-17Q1 header.b=USa7v8pY; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=NONE dis=NONE) header.from=ti.com Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id s1-v6si9686447pfb.39.2018.05.14.04.30.46; Mon, 14 May 2018 04:31:00 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@ti.com header.s=ti-com-17Q1 header.b=USa7v8pY; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=NONE dis=NONE) header.from=ti.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752517AbeENL32 (ORCPT + 99 others); Mon, 14 May 2018 07:29:28 -0400 Received: from lelnx194.ext.ti.com ([198.47.27.80]:42843 "EHLO lelnx194.ext.ti.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752095AbeENL3Y (ORCPT ); Mon, 14 May 2018 07:29:24 -0400 Received: from dlelxv90.itg.ti.com ([172.17.2.17]) by lelnx194.ext.ti.com (8.15.1/8.15.1) with ESMTP id w4EBTNPT020162; Mon, 14 May 2018 06:29:23 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ti.com; s=ti-com-17Q1; t=1526297363; bh=hSlGKwLog2P5LDbsZjJ4C+eqj/Y26eRRVptfXXVfFj8=; h=From:Subject:To:CC:Date; b=USa7v8pYO/N2bN9iNJf86PZSoBRnHRQNVPe7K66YvbC6ulR/xl99/IMWwYvfv9alP rOPebdtJwZ6gRroD5Kb+oltMHeR9akHns+DsjOfVKN/RxKqEIumum1ukEv8mBdIrP9 rTgOMzwD0h/XIWaLBF3cKdmeP4EKkM9zK0wm5rqk= Received: from DLEE107.ent.ti.com (dlee107.ent.ti.com [157.170.170.37]) by dlelxv90.itg.ti.com (8.14.3/8.13.8) with ESMTP id w4EBTNVY015462; Mon, 14 May 2018 06:29:23 -0500 Received: from DLEE107.ent.ti.com (157.170.170.37) by DLEE107.ent.ti.com (157.170.170.37) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P256) id 15.1.1466.3; Mon, 14 May 2018 06:29:23 -0500 Received: from dflp33.itg.ti.com (10.64.6.16) by DLEE107.ent.ti.com (157.170.170.37) with Microsoft SMTP Server (version=TLS1_0, cipher=TLS_RSA_WITH_AES_256_CBC_SHA) id 15.1.1466.3 via Frontend Transport; Mon, 14 May 2018 06:29:23 -0500 Received: from [172.24.190.215] (ileax41-snat.itg.ti.com [10.172.224.153]) by dflp33.itg.ti.com (8.14.3/8.13.8) with ESMTP id w4EBTLTo013019; Mon, 14 May 2018 06:29:21 -0500 From: Faiz Abbas Subject: MMC performance degradation due to too many requests To: "linux-kernel@vger.kernel.org" , linux-omap , CC: Ulf Hansson , Linus Walleij Message-ID: <0ea1d2bf-fa0f-4b3d-556a-5dca5afbf2a9@ti.com> Date: Mon, 14 May 2018 17:00:56 +0530 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.7.0 MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Language: en-US Content-Transfer-Encoding: 7bit X-EXCLAIMER-MD-CONFIG: e1e8a2fd-e40a-4ac6-ac9b-f7e9cc9ee180 Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Hi Everyone, I am debugging a performance degradation issue in some SD cards which happened around 4.13 kernel on TI's AM335x-evm. I compare the time taken for a simple dd command (after dropping caches) to run between kernel versions 4.13 and 4.14. I see that there is a consistent 15-20% reduction in speed. The amount of decrease depends on the card used. command: # echo 3 > /proc/sys/vm/drop_caches; dd if=/dev/zero of=/dev/mmcblk1p2 bs=1M count=100; v4.13 ~22s for 100M transfer v4.14 ~28s for 100M transfer Also, I see many times the number of interrupts for mmc1 in 4.14. v4.14 34: 2122 INTC 64 Level mmc1 v4.13 162: 457 INTC 64 Level mmc1 This increase of interrupts also exists in the latest kernel. I also printed out the size of the DMA transfers in the omap_hsmmc driver during the dd command. I see that the maximum size of the dma transfer has decreased from 1 MB to 256kB between 4.13 and 4.14. Full log: v4.13 https://pastebin.ubuntu.com/p/gK2k9GZcNd/ v4.14 https://pastebin.ubuntu.com/p/3YzCbQ79rk/ It looks like something has started splitting the request into 4 times as many smaller buffers and this is what is causing the performance decrease. I also ran the ftrace function profile tool on mmc* and blk* functions and see that the function mmc_blk_issue_rq() is where the program spends most of its time and the calls to that and other related functions have increased around 4 times with the average time per call being reduced. Function profile Log: v4.13 https://pastebin.ubuntu.com/p/R5JG6FpTvy/ v4.14 https://pastebin.ubuntu.com/p/Kc6QYYwYcN/ I am not sure why calls to mmc_blk_issue_rq() have increased by so much. Any ideas why this could be happening? Thanks, Faiz