Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67;
Date:   Mon, 2 Mar 2020 16:39:52 -0800
From:   Josh Triplett <josh@joshtriplett.org>
To:     "Machulsky, Zorik" <zorik@amazon.com>
Cc:     "Belgazal, Netanel" <netanel@amazon.com>,
        "Kiyanovski, Arthur" <akiyano@amazon.com>,
        "Tzalik, Guy" <gtzalik@amazon.com>,
        "Bshara, Saeed" <saeedb@amazon.com>,
        "netdev@vger.kernel.org" <netdev@vger.kernel.org>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>
Subject: Re: [PATCH] ena: Speed up initialization 90x by reducing poll delays
Message-ID: <20200303003952.GA264245@localhost>
References: <20200229002813.GA177044@localhost>
 <8B4A52CD-FC5A-4256-B7DE-A659B50654CE@amazon.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
In-Reply-To: <8B4A52CD-FC5A-4256-B7DE-A659B50654CE@amazon.com>
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk

On Mon, Mar 02, 2020 at 11:16:32PM +0000, Machulsky, Zorik wrote:
> 
> ﻿On 2/28/20, 4:29 PM, "Josh Triplett" <josh@joshtriplett.org> wrote:
> 
>     Before initializing completion queue interrupts, the ena driver uses
>     polling to wait for responses on the admin command queue. The ena driver
>     waits 5ms between polls, but the hardware has generally finished long
>     before that. Reduce the poll time to 10us.
>     
>     On a c5.12xlarge, this improves ena initialization time from 173.6ms to
>     1.920ms, an improvement of more than 90x. This improves server boot time
>     and time to network bringup.
>  
> Thanks Josh,
> We agree that polling rate should be increased, but prefer not to do it aggressively and blindly.
> For example linear backoff approach might be a better choice. Please let us re-work a little this 
> patch and bring it to review. Thanks!

That's fine, as long as it has the same net improvement on boot time.

I'd appreciate the opportunity to test any alternate approach you might
have.

(Also, as long as you're working on this, you might wish to make a
similar change to the EFA driver, and to the FreeBSD drivers.)

>     Before:
>     [    0.531722] calling  ena_init+0x0/0x63 @ 1
>     [    0.531722] ena: Elastic Network Adapter (ENA) v2.1.0K
>     [    0.531751] ena 0000:00:05.0: Elastic Network Adapter (ENA) v2.1.0K
>     [    0.531946] PCI Interrupt Link [LNKD] enabled at IRQ 11
>     [    0.547425] ena: ena device version: 0.10
>     [    0.547427] ena: ena controller version: 0.0.1 implementation version 1
>     [    0.709497] ena 0000:00:05.0: Elastic Network Adapter (ENA) found at mem febf4000, mac addr 06:c4:22:0e:dc:da, Placement policy: Low Latency
>     [    0.709508] initcall ena_init+0x0/0x63 returned 0 after 173616 usecs
>     
>     After:
>     [    0.526965] calling  ena_init+0x0/0x63 @ 1
>     [    0.526966] ena: Elastic Network Adapter (ENA) v2.1.0K
>     [    0.527056] ena 0000:00:05.0: Elastic Network Adapter (ENA) v2.1.0K
>     [    0.527196] PCI Interrupt Link [LNKD] enabled at IRQ 11
>     [    0.527211] ena: ena device version: 0.10
>     [    0.527212] ena: ena controller version: 0.0.1 implementation version 1
>     [    0.528925] ena 0000:00:05.0: Elastic Network Adapter (ENA) found at mem febf4000, mac addr 06:c4:22:0e:dc:da, Placement policy: Low Latency
>     [    0.528934] initcall ena_init+0x0/0x63 returned 0 after 1920 usecs