Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1757633AbbLBJFw (ORCPT ); Wed, 2 Dec 2015 04:05:52 -0500 Received: from mailout1.w1.samsung.com ([210.118.77.11]:19654 "EHLO mailout1.w1.samsung.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1756855AbbLBJFp convert rfc822-to-8bit (ORCPT ); Wed, 2 Dec 2015 04:05:45 -0500 X-AuditID: cbfec7f4-f79026d00000418a-98-565eb466172d From: Pavel Fedin To: "'Sunil Kovvuri'" , "'Eric Dumazet'" Cc: "'Linux Netdev List'" , "'LKML'" , "'LAKML'" , "'Sunil Goutham'" , "'Sunil Goutham'" References: <1448961223-41888-4-git-send-email-sunil.kovvuri@gmail.com> <01d101d12c46$2de43b40$89acb1c0$@samsung.com> <1448983985.25582.35.camel@edumazet-glaptop2.roam.corp.google.com> In-reply-to: Subject: RE: [PATCH 3/6] net: thunderx: Increase transmit queue length Date: Wed, 02 Dec 2015 12:05:41 +0300 Message-id: <00cb01d12ce0$9f090540$dd1b0fc0$@samsung.com> MIME-version: 1.0 Content-type: text/plain; charset=UTF-8 Content-transfer-encoding: 8BIT X-Mailer: Microsoft Outlook 14.0 Thread-index: AQGScnHROc58U50eD5J6ppH35rF/UQG2SVlqAT5pB6wB6Z4d1QJ5t4arnvoVzLA= Content-language: ru X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFvrLLMWRmVeSWpSXmKPExsVy+t/xy7ppW+LCDFpfSljse3+WzWLT42us Fpd3zWGzOLZAzOLy5R/MFgc+7mWxWNT0jsWB3WPG70UsHhtO9LN67Jx1l91j85J6j8+b5AJY o7hsUlJzMstSi/TtErgyPjXcZy6YJVhxYPs+9gbGLr4uRk4OCQETidfH/7JD2GISF+6tZ+ti 5OIQEljKKLFm/Uoo5zujxLvFrWBVbALqEqe/fmABsUUEIiSeH/gKFmcWeMUoca1DEsQWEpjK JHGgrbqLkYODUyBY4tkMW5CwsICbxJytq9lAbBYBVYmG/7/AWnkFLCX+fnjMBGELSvyYfI8F YqS6xKR5i5ghbG2JJ+8usEIcqiCx4+xrRogT/CTeHP8AVS8iMe3fPeYJjEKzkIyahWTULCSj ZiFpWcDIsopRNLU0uaA4KT3XUK84Mbe4NC9dLzk/dxMjJFK+7GBcfMzqEKMAB6MSD+8Knrgw IdbEsuLK3EOMEhzMSiK8XjJAId6UxMqq1KL8+KLSnNTiQ4zSHCxK4rxzd70PERJITyxJzU5N LUgtgskycXBKNTCufiQ9tSKzJOtcineB8zThU9ujprsXP2duuse+JHhZd7P/dW3L1mfC2saf 6z6s97OO/q9pseRryGTVQ+12H+cK65gyGUcHOMzpajzEfcWpQm7njVVPdB1OX4s8oirQeiNG /tysyVfjH7dnTdx2pKJA6ozJ3meRVc+mqGllcUfPuThh9sdLC/YpsRRnJBpqMRcVJwIA1aMo O5ACAAA= Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 2368 Lines: 40 Hello! > Probably you might have to set "coherent_pool" size in bootargs to a > higher value. > Can you please check. I have tried to do this. I was able to enlarge the pool up to 4MB, and still got allocation failures. At 8MB pool preallocation stops working: --- cut --- Call trace: [] __alloc_pages_nodemask+0x4f4/0x7d4 [] atomic_pool_init+0x60/0x1a4 [] arm64_dma_init+0x20/0x28 [] do_one_initcall+0x8c/0x1a4 [] kernel_init_freeable+0x154/0x1f4 [] kernel_init+0x10/0xd8 DMA: failed to allocate 8192 KiB pool for atomic coherent allocation --- cut --- and i get even worse faults in the driver. I know that it is possible to allocate larger pools by setting CONFIG_FORCE_MAX_ZONEORDER, but: a) This is done on per-platform basis. For ThunderX we used to have a patch (http://www.spinics.net/lists/arm-kernel/msg415457.html), which never made it upstream, because vGIC fixes stopped requiring it at some point. And also we may want to use the nicvf driver not only on actual hardware, but also inside virtual machine in KVM. So do we need to set CONFIG_FORCE_MAX_ZONEORDER for virt too? And what if at some point qemu emulates not only "virt", but some other machine (let's say AMD X-Gene), and we run it on ThunderX and want to use nicvf with this model? b) IMHO it's not good to have a driver which simply does not work without some obscure option in boot arguments. So, i see several possible ways to solve this: 1. Introduce some mechanism which would allow the driver to tell the kernel that it needs coherent pool of large size. Can be problematic because the driver can be a module, and pool allocation happens early. 2. Can we use some other method for allocating queues, which would not require such a huge coherent pool? 3. The driver could check value of atomic_pool_size and adjust own memory requirements accordingly. This indeed looks like a quick hack, but would at least make things running quickly. Kind regards, Pavel Fedin Expert Engineer Samsung Electronics Research center Russia -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/