Received: by 2002:a25:c205:0:0:0:0:0 with SMTP id s5csp2831046ybf; Mon, 2 Mar 2020 16:41:05 -0800 (PST) X-Google-Smtp-Source: ADFU+vtY79nPZ+2kL1SQnbIjWS+WiIIj4oyt6Y8Q6z+8ldxc+d2PGtleB2sI7Th75yokHZkV0pfF X-Received: by 2002:a9d:4541:: with SMTP id p1mr1466774oti.199.1583196065695; Mon, 02 Mar 2020 16:41:05 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1583196065; cv=none; d=google.com; s=arc-20160816; b=TnNrhnhGArMN06upv3Prwptk6RTk0I4Zk5mP/gMSd/6FWsfTwfJA5K+q4AYlGS4fYz /KWP233JVePbRD5wt8uAyQ04Nb2L9ipKAjbKYJE6j3WHsWg/zxeWxMjAujsmuH3z4oUm UjYOs4NarXZkJZSVzFHLlBgN+gXwAQkLlY90vwhLri+/nT+9VsNW0LIrQBVSmJi2GEjC +qYhivJttnQ/VmfcnXxuWK87OOE5RMKWt8uRZg99gcJrqRGH18CLl+3WpZD9bpCHyVmL q73OHj25N62ZM+Xq5p5EgTBsEVUZNbJeUBQfgbkD7Oc1Y/NzzHcBDkZbehPTosBVwwDe 2P+Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:in-reply-to:content-transfer-encoding :content-disposition:mime-version:references:message-id:subject:cc :to:from:date; bh=WBFwY9Q0wEf4ZUVil1F1c9/5Z/mErLi60ZnIFhI3p6I=; b=i9iEnwObCQP7HwRvDUVnWoHQkjjogKhPdnbnQJv2GDuHepZtio+RHNEmC+3dH+nXNZ oTzbkXCdjesH1OSXbVA++2mHbsGAC1pCsISDKxrRFJPfY8hHzuma/BrgYXifzfbwFjab K5w5SiGKrO7ij+TXf7iaq8H0rTvuyAS60Z/gosf3xqouChJRwZX52GSVFkPy1UWPgFaJ H2xUnByCW+cOlwO5eFqUhQH7X8NSD2Qtw55tNzbMBViVZikzqBqBlpk4ZD1PFL9Y7Y83 /n1wTFXxfLsbryQHbQ5yHJj6/AJWVQR5aFcghWlinRQE8QnEteXtOXDJvvl0jG/I5SQA E7Kw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id a3si7577934otk.234.2020.03.02.16.40.51; Mon, 02 Mar 2020 16:41:05 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726974AbgCCAkA (ORCPT + 99 others); Mon, 2 Mar 2020 19:40:00 -0500 Received: from relay8-d.mail.gandi.net ([217.70.183.201]:39001 "EHLO relay8-d.mail.gandi.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726773AbgCCAj7 (ORCPT ); Mon, 2 Mar 2020 19:39:59 -0500 X-Originating-IP: 50.39.173.182 Received: from localhost (50-39-173-182.bvtn.or.frontiernet.net [50.39.173.182]) (Authenticated sender: josh@joshtriplett.org) by relay8-d.mail.gandi.net (Postfix) with ESMTPSA id 773441BF206; Tue, 3 Mar 2020 00:39:55 +0000 (UTC) Date: Mon, 2 Mar 2020 16:39:52 -0800 From: Josh Triplett To: "Machulsky, Zorik" Cc: "Belgazal, Netanel" , "Kiyanovski, Arthur" , "Tzalik, Guy" , "Bshara, Saeed" , "netdev@vger.kernel.org" , "linux-kernel@vger.kernel.org" Subject: Re: [PATCH] ena: Speed up initialization 90x by reducing poll delays Message-ID: <20200303003952.GA264245@localhost> References: <20200229002813.GA177044@localhost> <8B4A52CD-FC5A-4256-B7DE-A659B50654CE@amazon.com> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <8B4A52CD-FC5A-4256-B7DE-A659B50654CE@amazon.com> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Mon, Mar 02, 2020 at 11:16:32PM +0000, Machulsky, Zorik wrote: > > On 2/28/20, 4:29 PM, "Josh Triplett" wrote: > > Before initializing completion queue interrupts, the ena driver uses > polling to wait for responses on the admin command queue. The ena driver > waits 5ms between polls, but the hardware has generally finished long > before that. Reduce the poll time to 10us. > > On a c5.12xlarge, this improves ena initialization time from 173.6ms to > 1.920ms, an improvement of more than 90x. This improves server boot time > and time to network bringup. > > Thanks Josh, > We agree that polling rate should be increased, but prefer not to do it aggressively and blindly. > For example linear backoff approach might be a better choice. Please let us re-work a little this > patch and bring it to review. Thanks! That's fine, as long as it has the same net improvement on boot time. I'd appreciate the opportunity to test any alternate approach you might have. (Also, as long as you're working on this, you might wish to make a similar change to the EFA driver, and to the FreeBSD drivers.) > Before: > [ 0.531722] calling ena_init+0x0/0x63 @ 1 > [ 0.531722] ena: Elastic Network Adapter (ENA) v2.1.0K > [ 0.531751] ena 0000:00:05.0: Elastic Network Adapter (ENA) v2.1.0K > [ 0.531946] PCI Interrupt Link [LNKD] enabled at IRQ 11 > [ 0.547425] ena: ena device version: 0.10 > [ 0.547427] ena: ena controller version: 0.0.1 implementation version 1 > [ 0.709497] ena 0000:00:05.0: Elastic Network Adapter (ENA) found at mem febf4000, mac addr 06:c4:22:0e:dc:da, Placement policy: Low Latency > [ 0.709508] initcall ena_init+0x0/0x63 returned 0 after 173616 usecs > > After: > [ 0.526965] calling ena_init+0x0/0x63 @ 1 > [ 0.526966] ena: Elastic Network Adapter (ENA) v2.1.0K > [ 0.527056] ena 0000:00:05.0: Elastic Network Adapter (ENA) v2.1.0K > [ 0.527196] PCI Interrupt Link [LNKD] enabled at IRQ 11 > [ 0.527211] ena: ena device version: 0.10 > [ 0.527212] ena: ena controller version: 0.0.1 implementation version 1 > [ 0.528925] ena 0000:00:05.0: Elastic Network Adapter (ENA) found at mem febf4000, mac addr 06:c4:22:0e:dc:da, Placement policy: Low Latency > [ 0.528934] initcall ena_init+0x0/0x63 returned 0 after 1920 usecs