Received: by 2002:a05:6a10:8c0a:0:0:0:0 with SMTP id go10csp7490343pxb; Thu, 18 Feb 2021 11:24:50 -0800 (PST) X-Google-Smtp-Source: ABdhPJwLHkmAaDC0sasdlj2WFXWEt0DekndnGRnNs8BZC0JmCR2nQ3BQX/crBA5TjuoZkyET/Ckz X-Received: by 2002:a17:906:2bce:: with SMTP id n14mr5291768ejg.171.1613676290581; Thu, 18 Feb 2021 11:24:50 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1613676290; cv=none; d=google.com; s=arc-20160816; b=ikEe9n5XKx2Y4U/w5K1P2s0xZlVzkWEVPJYNwZpxtxrbU+PRbPu8zH1bx/RQRcW+b5 APFJc9na2MPN6r0idnlaKKoIlMZSghF0U1sjm0BY8JcojPBcZ47qWSBri7lnUOtZpFjF 5UMTQhF4xsXoiF0agAeVLF4Ixj6ikVbnfrA7urUcGUMdRCXjFKXr0hvhA6g9pQUYjoyD +BbjpwNaTMtVrAEKfa2Kn5EfVV4Mk1ADrXAqwbN6mMerqnem3Y4SLheTxEcv0qOr09Yz Kh2GGy8bsBPVvKWHszE6CSgpLXPogclQzbA+S3H1fBFZp8ghSsl3QVQSnSlV5hBpkQPK kVsg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:user-agent:in-reply-to:content-disposition :mime-version:references:message-id:subject:cc:to:from:date; bh=LBYltN8mfCk3heyqa20842UxQnuATWD/IrORxEkZZos=; b=Bp4DEpvWF527xy1pd4+2kjFgBC73uccyfVUXAeeKWRnpRlh08sUjHRTRogOx+yalx+ 30HG32nETUqJhXpXXca1bnc1xEeGL9+Ig6E2zn8LQDsA9UTotHoujqEHIXzUa9YR5dd0 aixK+ZX7wZzhH7APG4z/59JORE/ROHHjAg7vxjhstbbq9k1d3vwtXyhGDIhm9i9Zk7Ic P5vwh+FAPuQqSuPzNnvr1gh2fRFq50CmzlOovAFrG37mGgQpIpP7RWB4y2B1nCGNX4w8 gWHblureaTXQkapFeUfzuwsQ9k29ehgnr0Rkh8NU0C8PPXecV2YHO9xYb5uGuXkZX9Pk weGA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [23.128.96.18]) by mx.google.com with ESMTP id f8si4196415edw.351.2021.02.18.11.24.26; Thu, 18 Feb 2021 11:24:50 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18; Authentication-Results: mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232112AbhBRTW4 (ORCPT + 99 others); Thu, 18 Feb 2021 14:22:56 -0500 Received: from wtarreau.pck.nerim.net ([62.212.114.60]:49865 "EHLO 1wt.eu" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231152AbhBRSOV (ORCPT ); Thu, 18 Feb 2021 13:14:21 -0500 Received: (from willy@localhost) by pcw.home.local (8.15.2/8.15.2/Submit) id 11IIDU99015369; Thu, 18 Feb 2021 19:13:30 +0100 Date: Thu, 18 Feb 2021 19:13:30 +0100 From: Willy Tarreau To: Florian Fainelli Cc: Scott Branden , Greg Kroah-Hartman , BCM Kernel Feedback , LKML , Linux ARM Subject: Re: 5.10 LTS Kernel: 2 or 6 years? Message-ID: <20210218181330.GA15217@1wt.eu> References: <8cf503db-ac4c-a546-13c0-aac6da5c073b@broadcom.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.10.1 (2018-07-13) Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Hi Florian! On Thu, Feb 18, 2021 at 09:42:00AM -0800, Florian Fainelli wrote: > > Other difficulty with the LTS version is the frequency it is updated. We would not > > pickup the changes that frequently to test. A quarterly, bi-annually, or when a critical fix > > is identified would be when we update and perform any meaningful testing when in maintainence. > > Well you really have the best of both worlds here, you are not forced to > update your downstream fork of the stable kernel every week, though > bonus points if you do. > > For instance I try to test all the stable release candidates for 4.9, > 5.4 and 5.10, and merge the stable tags every week when they show up. We > do release to our customers every 6 weeks though, so usually they will > jump several stable releases, and they are fine with that. > > Yes it takes time, but I would rather do that than have to continuously > respond to questions about is this CVE fixed in kernel X.Y.Z like it > used to be before we did that. It also helps catch problem faster before > customers do, the usual benefits of continuous integration/delivery. Totally agreed! In our use case at haproxy, it's slightly different but not that much. We're much less sensitive to kernel bugs except for the network parts and any long-term stability issue. However we're extremely sensitive to openssl bugs and haproxy bugs. Thus we use them as triggers to emit a new release and we systematically update the kernel to the latest one. Our local patches usually apply pretty well on top of that. We face maybe 1-3 rejects a year which take half an hour of extra manual work, and roughly one regression every 3 years, essentially caused by one of our local patches applying to the wrong place due to changes and not caught by QA tests before being put online. I think that in ~15 years (we started with 2.4), only a single customer was ever affected by a regression caused by the kernel, it is so low we could almost laugh about it. Quite frankly this is unrivaled, and it illustrates the huge benefit in almost blindly following LTS this way! More quality, less work, and faster response to critical bugs! For sure there's no "kernel hero" in our dev team, but who really wants that anyway ? Cheers, Willy