Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67;
From:   Pankaj Suryawanshi <pankaj.suryawanshi@einfochips.com>
To:     Michal Hocko <mhocko@kernel.org>
CC:     "linux-mm@kvack.org" <linux-mm@kvack.org>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        "minchan@kernel.org" <minchan@kernel.org>,
        Kirill Tkhai <ktkhai@virtuozzo.com>
Subject: Re: [External] Re: mm/cma.c: High latency for cma allocation
Thread-Topic: [External] Re: mm/cma.c: High latency for cma allocation
Thread-Index: AQHU3YpG7XMhet9mv0+1ZFJR3Y1R16YRXCGAgAACGm2AAAebAIAABP0wgAAIBgCAAU6K1oAAEw1b
Date:   Tue, 19 Mar 2019 11:45:03 +0000
Message-ID: <SG2PR02MB30981381635A6BC3783D42CDE8400@SG2PR02MB3098.apcprd02.prod.outlook.com>
References: <SG2PR02MB3098E44824F5AA69BC04F935E8470@SG2PR02MB3098.apcprd02.prod.outlook.com>
 <20190318130757.GG8924@dhcp22.suse.cz>
 <SG2PR02MB309886996889791555D5B53EE8470@SG2PR02MB3098.apcprd02.prod.outlook.com>
 <20190318134242.GI8924@dhcp22.suse.cz>
 <SG2PR02MB30986F43403B92F31499E42AE8470@SG2PR02MB3098.apcprd02.prod.outlook.com>,<20190318142916.GK8924@dhcp22.suse.cz>,<SG2PR02MB3098DCB820E3367B09DDA45AE8400@SG2PR02MB3098.apcprd02.prod.outlook.com>
In-Reply-To: <SG2PR02MB3098DCB820E3367B09DDA45AE8400@SG2PR02MB3098.apcprd02.prod.outlook.com>
Accept-Language: en-GB, en-US
Content-Language: en-GB
received-spf: None (protection.outlook.com: einfochips.com does not designate
 permitted sender hosts)
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-MS-Exchange-CrossTenant-Network-Message-Id: f8d42a61-6cb7-449f-521a-08d6ac605393
X-MS-Exchange-CrossTenant-originalarrivaltime: 19 Mar 2019 11:45:03.6814
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 0adb040b-ca22-4ca6-9447-ab7b049a22ff
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SG2PR02MB2700
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org


________________________________________
From: Pankaj Suryawanshi
Sent: 19 March 2019 15:59
To: Michal Hocko
Cc: linux-mm@kvack.org; linux-kernel@vger.kernel.org; minchan@kernel.org; K=
irill Tkhai
Subject: Re: [External] Re: mm/cma.c: High latency for cma allocation


________________________________________
From: Michal Hocko <mhocko@kernel.org>
Sent: 18 March 2019 19:59
To: Pankaj Suryawanshi
Cc: linux-mm@kvack.org; linux-kernel@vger.kernel.org; minchan@kernel.org; K=
irill Tkhai
Subject: Re: [External] Re: mm/cma.c: High latency for cma allocation

On Mon 18-03-19 14:02:09, Pankaj Suryawanshi wrote:
>> > I have the system(vanilla kernel) with 2GB of RAM, reserved 1GB for CM=
A. No swap or zram.
>> > Sorry, I don't have information where the time is spent.
>> > time is calculated in between cma_alloc call.
>> > I have just cma_alloc trace information/function graph.
>
>> Then please collect that data because it is really hard to judge
>> anything from the numbers you have provided.
>
> Any pointers from which i can get this details ?

I would start by enabling built in tracepoints for the migration or use
a system wide perf monitoring with call graph data.

Calling Sequence is as below.

cma_alloc() -->
alloc_contig_range() -->
start_isolate_page_range() -->
__alloc_contig_migrate_range() -->
isolate_migratepages_range() -->
reclaim_clean_pages_from_list() -->
shrink_page_list()

There is no built in tracepoints except cma_alloc.
How to know where it taking time ?

I have tried for latency count for 385MB:

reclaim- reclaim_clean_pages_from_list()
migrate- migrate_pages()
migrateranges- isolate_migratepages_range()
overall - __alloc_contig_migrate_range()

Note: output is in us

[ 1151.420923] LATENCY reclaim=3D 43 migrate=3D128 migrateranges=3D23
[ 1151.421209] LATENCY reclaim=3D 11 migrate=3D253 migrateranges=3D14
[ 1151.427856] LATENCY reclaim=3D 45 migrate=3D12 migrateranges=3D12
[ 1151.434485] LATENCY reclaim=3D 44 migrate=3D33 migrateranges=3D12
[ 1151.440975] LATENCY reclaim=3D 45 migrate=3D0 migrateranges=3D11
[ 1151.447513] LATENCY reclaim=3D 39 migrate=3D35 migrateranges=3D11
[ 1151.453919] LATENCY reclaim=3D 46 migrate=3D0 migrateranges=3D12
[ 1151.460474] LATENCY reclaim=3D 39 migrate=3D41 migrateranges=3D11
[ 1151.466947] LATENCY reclaim=3D 54 migrate=3D32 migrateranges=3D17
[ 1151.473464] LATENCY reclaim=3D 45 migrate=3D21 migrateranges=3D12
[ 1151.480016] LATENCY reclaim=3D 41 migrate=3D39 migrateranges=3D12
[ 1151.486551] LATENCY reclaim=3D 41 migrate=3D36 migrateranges=3D12
[ 1151.493199] LATENCY reclaim=3D 13 migrate=3D188 migrateranges=3D12
[ 1151.500034] LATENCY reclaim=3D 60 migrate=3D94 migrateranges=3D13
[ 1151.506686] LATENCY reclaim=3D 78 migrate=3D9 migrateranges=3D12
[ 1151.513313] LATENCY reclaim=3D 33 migrate=3D147 migrateranges=3D12
[ 1151.519839] LATENCY reclaim=3D 52 migrate=3D98 migrateranges=3D12
[ 1151.526556] LATENCY reclaim=3D 46 migrate=3D126 migrateranges=3D12
[ 1151.533254] LATENCY reclaim=3D 22 migrate=3D230 migrateranges=3D12
[ 1151.540145] LATENCY reclaim=3D 0 migrate=3D305 migrateranges=3D13
[ 1151.546997] LATENCY reclaim=3D 1 migrate=3D301 migrateranges=3D13
[ 1151.553686] LATENCY reclaim=3D 40 migrate=3D201 migrateranges=3D12
[ 1151.560395] LATENCY reclaim=3D 35 migrate=3D149 migrateranges=3D12
[ 1151.567076] LATENCY reclaim=3D 77 migrate=3D43 migrateranges=3D16
[ 1151.573836] LATENCY reclaim=3D 34 migrate=3D190 migrateranges=3D12
[ 1151.580510] LATENCY reclaim=3D 51 migrate=3D120 migrateranges=3D12
[ 1151.587240] LATENCY reclaim=3D 33 migrate=3D147 migrateranges=3D13
[ 1151.594036] LATENCY reclaim=3D 20 migrate=3D241 migrateranges=3D13
[ 1151.600749] LATENCY reclaim=3D 75 migrate=3D41 migrateranges=3D13
[ 1151.607402] LATENCY reclaim=3D 77 migrate=3D32 migrateranges=3D12
[ 1151.613956] LATENCY reclaim=3D 72 migrate=3D35 migrateranges=3D12
[ 1151.620642] LATENCY reclaim=3D 59 migrate=3D162 migrateranges=3D12
[ 1151.627181] LATENCY reclaim=3D 76 migrate=3D9 migrateranges=3D11
[ 1151.633795] LATENCY reclaim=3D 80 migrate=3D0 migrateranges=3D12
[ 1151.640278] LATENCY reclaim=3D 87 migrate=3D18 migrateranges=3D12
[ 1151.646758] LATENCY reclaim=3D 82 migrate=3D10 migrateranges=3D11
[ 1151.653307] LATENCY reclaim=3D 71 migrate=3D31 migrateranges=3D12
[ 1151.659911] LATENCY reclaim=3D 61 migrate=3D77 migrateranges=3D12
[ 1151.666514] LATENCY reclaim=3D 94 migrate=3D42 migrateranges=3D15
[ 1151.673089] LATENCY reclaim=3D 67 migrate=3D59 migrateranges=3D12
[ 1151.679655] LATENCY reclaim=3D 81 migrate=3D14 migrateranges=3D12
[ 1151.686253] LATENCY reclaim=3D 49 migrate=3D93 migrateranges=3D12
[ 1151.692815] LATENCY reclaim=3D 61 migrate=3D54 migrateranges=3D12
[ 1151.699438] LATENCY reclaim=3D 42 migrate=3D99 migrateranges=3D10
[ 1151.705881] OVERALL overall=3D285157

cma_alloc latency is =3D 297385 us

Please let me know is there any workaround/solution to reduce large size bu=
ffer cma_alloc  latency ?

--
Michal Hocko
SUSE Labs
***************************************************************************=
***************************************************************************=
******* eInfochips Business Disclaimer: This e-mail message and all attachm=
ents transmitted with it are intended solely for the use of the addressee a=
nd may contain legally privileged and confidential information. If the read=
er of this message is not the intended recipient, or an employee or agent r=
esponsible for delivering this message to the intended recipient, you are h=
ereby notified that any dissemination, distribution, copying, or other use =
of this message or its attachments is strictly prohibited. If you have rece=
ived this message in error, please notify the sender immediately by replyin=
g to this message and please delete it from your computer. Any views expres=
sed in this message are those of the individual sender unless otherwise sta=
ted. Company has taken enough precautions to prevent the spread of viruses.=
 However the company accepts no liability for any damage caused by any viru=
s transmitted by this email. **********************************************=
***************************************************************************=
************************************