Received: by 2002:a25:8b91:0:0:0:0:0 with SMTP id j17csp842257ybl; Fri, 10 Jan 2020 07:37:14 -0800 (PST) X-Google-Smtp-Source: APXvYqwFRAIJmbhYUrISf6ycnrZDRQopdWS3J4dr2NDq8I1lx52O+zBBpNMbLgkXTem/0wOqYNf7 X-Received: by 2002:aca:51c9:: with SMTP id f192mr2765750oib.10.1578670634534; Fri, 10 Jan 2020 07:37:14 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1578670634; cv=none; d=google.com; s=arc-20160816; b=wTt+lW5WlqJEZk4oNOHRJOEwHVjCzjcNXJ8xZ5xo4lYQXoENn7qn5LfK2FYSqsBbKp /V9AW8Eb/9uoeL7RuNnRgv+qLRp04wGMA6lOkCkF9OqLoGIUOXZ3LXwxzicMV5g03KaX dxsCORhjPoK/AiC8y+5HyurDaMVQtlpF4woPLC1eT95TKkqiID3pXPfOmNbpUMsGtZBz OMi43MsCfF61wLSDpkW83v7Mvm01aDiu2ep2GelZs8zF8sOSSsRNOVNrYKh28n/SEyNL Gk3Xovs4lBQJ+4i/BaMOALq2m1FGY/B4+uQ0FQfY7ukbTgAR84yAREhYwd9t0E6iQ2Eo 84mA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:user-agent:in-reply-to :content-disposition:mime-version:references:message-id:subject:cc :to:from:date:ironport-sdr:dkim-signature; bh=SgZqy3XTPlr7N1P3ccJM4s/BqmaQ1P0z1Cl70TTTVVg=; b=LtaKamGVJdHs49qeMWZdGTmpmFlZ+jPo9OwWwVQ8zapigwqGw5Do1MmsLoX7pjea89 1iqXOitEykOZgvWMPn5uC0GSC8bnFSrd0Vmv3UzmVRaIgBNHwVFTZi3lKCMDzLxypOsJ xBE0fZYC9RFTb+ehMtIpBc1PtYgzeEGxnP8G4CXr1EFqr7HrMp0GmUsJLfmPqlJmZeJ0 2w9K8EvqWsiD3WcHJdjXCZusneZ/HnAznYgcz0wQO65PYTNkb04e30Shs1Cs5Iy+vzca j5xogWrzVBJgAJ4cXzy5/Cca/aNy8RuxhBkunLQcsb7Q1bpxofIHdTTZLJvdFDvOr7jy CYiQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@amazon.com header.s=amazon201209 header.b=NONlX9UG; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=amazon.com Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id g1si1512583otj.115.2020.01.10.07.37.03; Fri, 10 Jan 2020 07:37:14 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@amazon.com header.s=amazon201209 header.b=NONlX9UG; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=amazon.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728327AbgAJPgH (ORCPT + 99 others); Fri, 10 Jan 2020 10:36:07 -0500 Received: from smtp-fw-9102.amazon.com ([207.171.184.29]:7641 "EHLO smtp-fw-9102.amazon.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728151AbgAJPgG (ORCPT ); Fri, 10 Jan 2020 10:36:06 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amazon.com; i=@amazon.com; q=dns/txt; s=amazon201209; t=1578670566; x=1610206566; h=date:from:to:cc:subject:message-id:references: mime-version:in-reply-to; bh=SgZqy3XTPlr7N1P3ccJM4s/BqmaQ1P0z1Cl70TTTVVg=; b=NONlX9UG1D5DZuADMfoS9coLx+Va8a87a4NT1so3/7QqC5vI/PyRzPNQ 7ajz471P2nPny8LQhrByUSqUjrp1HL822VVg3V7BtwWzvwDp5c1nwo9bc k4IInC5ZhdY1gTnQaZgwRznNqNzrb9aj9KtYO+XjxeanVG1pNvNHTld6s 8=; IronPort-SDR: vO0XLbMQgitppzP2QpWLx/9TWANK+p+D6rx2ItFgJb7JazVRZhu9IZvwDPQiY2YXRlIwTKxqQ2 K0Gg7agBJwZQ== X-IronPort-AV: E=Sophos;i="5.69,417,1571702400"; d="scan'208";a="17985115" Received: from sea32-co-svc-lb4-vlan3.sea.corp.amazon.com (HELO email-inbound-relay-1e-303d0b0e.us-east-1.amazon.com) ([10.47.23.38]) by smtp-border-fw-out-9102.sea19.amazon.com with ESMTP; 10 Jan 2020 15:35:54 +0000 Received: from EX13MTAUWA001.ant.amazon.com (iad55-ws-svc-p15-lb9-vlan3.iad.amazon.com [10.40.159.166]) by email-inbound-relay-1e-303d0b0e.us-east-1.amazon.com (Postfix) with ESMTPS id 1C7D0A0716; Fri, 10 Jan 2020 15:35:45 +0000 (UTC) Received: from EX13D10UWA003.ant.amazon.com (10.43.160.248) by EX13MTAUWA001.ant.amazon.com (10.43.160.118) with Microsoft SMTP Server (TLS) id 15.0.1367.3; Fri, 10 Jan 2020 15:35:21 +0000 Received: from EX13MTAUWC001.ant.amazon.com (10.43.162.135) by EX13D10UWA003.ant.amazon.com (10.43.160.248) with Microsoft SMTP Server (TLS) id 15.0.1367.3; Fri, 10 Jan 2020 15:35:21 +0000 Received: from localhost (10.85.220.176) by mail-relay.amazon.com (10.43.162.232) with Microsoft SMTP Server id 15.0.1367.3 via Frontend Transport; Fri, 10 Jan 2020 15:35:21 +0000 Date: Fri, 10 Jan 2020 07:35:20 -0800 From: Eduardo Valentin To: Peter Zijlstra CC: Anchal Agarwal , , , , , , , , , , , , , , , , , , , , , , , , , , , Subject: Re: [RFC PATCH V2 11/11] x86: tsc: avoid system instability in hibernation Message-ID: <20200110153520.GC8214@u40b0340c692b58f6553c.ant.amazon.com> References: <20200107234526.GA19034@dev-dsk-anchalag-2a-9c2d1d96.us-west-2.amazon.com> <20200108105011.GY2827@hirez.programming.kicks-ass.net> MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Disposition: inline In-Reply-To: <20200108105011.GY2827@hirez.programming.kicks-ass.net> User-Agent: Mutt/1.5.24 (2015-08-30) Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Hey Peter, On Wed, Jan 08, 2020 at 11:50:11AM +0100, Peter Zijlstra wrote: > On Tue, Jan 07, 2020 at 11:45:26PM +0000, Anchal Agarwal wrote: > > From: Eduardo Valentin > > > > System instability are seen during resume from hibernation when system > > is under heavy CPU load. This is due to the lack of update of sched > > clock data, and the scheduler would then think that heavy CPU hog > > tasks need more time in CPU, causing the system to freeze > > during the unfreezing of tasks. For example, threaded irqs, > > and kernel processes servicing network interface may be delayed > > for several tens of seconds, causing the system to be unreachable. > > > The fix for this situation is to mark the sched clock as unstable > > as early as possible in the resume path, leaving it unstable > > for the duration of the resume process. This will force the > > scheduler to attempt to align the sched clock across CPUs using > > the delta with time of day, updating sched clock data. In a post > > hibernation event, we can then mark the sched clock as stable > > again, avoiding unnecessary syncs with time of day on systems > > in which TSC is reliable. > > This makes no frigging sense what so bloody ever. If the clock is > stable, we don't care about sched_clock_data. When it is stable you get > a linear function of the TSC without complicated bits on. > > When it is unstable, only then do we care about the sched_clock_data. > Yeah, maybe what is not clear here is that we covering for situation where clock stability changes over time, e.g. at regular boot clock is stable, hibernation happens, then restore happens in a non-stable clock. > > Reviewed-by: Erik Quanstrom > > Reviewed-by: Frank van der Linden > > Reviewed-by: Balbir Singh > > Reviewed-by: Munehisa Kamata > > Tested-by: Anchal Agarwal > > Signed-off-by: Eduardo Valentin > > --- > > NAK, the code very much relies on never getting marked stable again > after it gets set to unstable. > Well actually, at the PM_POST_HIBERNATION, we do the check and set stable if known to be stable. The issue only really happens during the restoration path under scheduling pressure, which takes forever to finish, as described in the commit. Do you see a better solution for this issue? > > diff --git a/kernel/sched/clock.c b/kernel/sched/clock.c > > index 1152259a4ca0..374d40e5b1a2 100644 > > --- a/kernel/sched/clock.c > > +++ b/kernel/sched/clock.c > > @@ -116,7 +116,7 @@ static void __scd_stamp(struct sched_clock_data *scd) > > scd->tick_raw = sched_clock(); > > } > > > > -static void __set_sched_clock_stable(void) > > +void set_sched_clock_stable(void) > > { > > struct sched_clock_data *scd; > > > > @@ -236,7 +236,7 @@ static int __init sched_clock_init_late(void) > > smp_mb(); /* matches {set,clear}_sched_clock_stable() */ > > > > if (__sched_clock_stable_early) > > - __set_sched_clock_stable(); > > + set_sched_clock_stable(); > > > > return 0; > > } > > -- > > 2.15.3.AMZN > > -- All the best, Eduardo Valentin