Received: by 10.223.185.116 with SMTP id b49csp3951376wrg; Tue, 13 Feb 2018 10:14:58 -0800 (PST) X-Google-Smtp-Source: AH8x224RCpFBpqrxLdeRsYYyIEQ9JRW/Oac/Ce5f1c4EOnNoP0sGU59NmxjqtYknRqWBmxRAzZ+Y X-Received: by 10.98.231.25 with SMTP id s25mr2064816pfh.177.1518545698401; Tue, 13 Feb 2018 10:14:58 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1518545698; cv=none; d=google.com; s=arc-20160816; b=jTYtRs669zFTF0sze0R7RSv/COzKyhG/rqzw63BOmEVWp7anB0SQ7ReVALrENWrQxX qe6phnYLlHeqLd8M7n5E0AHzG+i0lX6556nF5fsRqoGtC3l6ST3C4YY/A+t516ffsTDU pVXBnaeagfCqDNhsqRXnARc7ktFxnWHXcZk8iLwvKiRevOJQSUi+JBfywFwAy6tJGCTA bNoStC+QZIrabTInwSwMlYxxg3dNumsWwvwCjoLwKXmhd2LcwXP0fZTRx4sR3p8+Tc2E cQpQWeRaTvv5g81STDd8s5iM0TAe255mkiYG/05zPt36sNt1i/n7NidZb60DaFA5ROGz +nGg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:user-agent:in-reply-to :content-disposition:mime-version:references:message-id:subject:cc :to:from:date:dkim-signature:arc-authentication-results; bh=zuu1RAeHsu1nRYUfIW9FxGcB+f7vdtpnnShmBSdI5Uo=; b=XG+FqXVMnVFWq0aYl3gG8MN0PrbA7DR2mmgQjaxYRQYJ9tfot7PgWzpElzQ2PZC6yL VMsCBM7wBQqt3RTIJXI/xjgrWVyVFOWizDzFUod3Phi/14uiP6EB6BMhRBXo0rHJgM5a mN6OIevhdjwJdz2QwuHm913DOE6hPGU/geomOZT0Og/FtFfUuRwZbBqBlN17gtPZsOPq qDC8qvg74rJtFLWbzVHuqRwBxoT4OAqcQYkJUaJimYfjsy5U0ksjdOR2xyojE9G/ORLF wZXYjahTyvqjsgtWz1qe8y8Y/EFTqmffR6y6Lu1lUo4yhNxW7N7kEQy6IrHsYVdpASL5 pQKg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=PTRMXFNL; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id h9si1472939pgq.683.2018.02.13.10.14.44; Tue, 13 Feb 2018 10:14:58 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=PTRMXFNL; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S965444AbeBMSN7 (ORCPT + 99 others); Tue, 13 Feb 2018 13:13:59 -0500 Received: from mail-it0-f66.google.com ([209.85.214.66]:36571 "EHLO mail-it0-f66.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S965217AbeBMSN5 (ORCPT ); Tue, 13 Feb 2018 13:13:57 -0500 Received: by mail-it0-f66.google.com with SMTP id n206so11892121itg.1; Tue, 13 Feb 2018 10:13:57 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to:user-agent; bh=zuu1RAeHsu1nRYUfIW9FxGcB+f7vdtpnnShmBSdI5Uo=; b=PTRMXFNLC29GCovNyLI6DYcnrxnguXLfpsGjU8CCRl4d9+VJ8bLeDNUoHM20rWgG22 436EFkInWhURnBmGHQjUlmHqXxQBf45Eb3eIMPV3mkx7Ot0Qi2PqetRIvciqUM0wN/hC Az6BtX0GjyPTZZRghP72xq/LzUdzTsRSjUVWLQTA9s+rilNnE6loPJZPumsq1C9wVfzo ANjTV2vc6JQOaoZjiXlRQcFtlx4+KlLmPZTFsqEuHfRYTwTcJJEZ+4KDJwdd5Vv/67YW wDIvi/pGFVh0xz8ecCtVBpxsU6r8P0IiAuw7hT6YbbolaiMWz35ue60+yUb/LoFuKxPn 7m4g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to:user-agent; bh=zuu1RAeHsu1nRYUfIW9FxGcB+f7vdtpnnShmBSdI5Uo=; b=SGOi+L+g2VhfZLOMwxsQo/8phjP5oXZ/xDJRsreN1uXvT/9PnEwPcxXRUV9rC00N4V lwK/YHN17tFayc9ne+3nwTi50lXxiyYdXKzYoQl/owZpMOiFYKjHD3Klhds7aoLL1c/1 JYFS4jjl4q2Y2UPmozVPoXYqIM7sKfAm1M+kfQQgH6VRv4KAWw6+AVUm1FzNJ+YqEAOO MpglR29remPELWNysx/fWS+VJljDx7Mxd6HoPw/SPrceLNAhOtmUeai5gLzoTM8l/UKA xTz2qXSjm95FisGFyxqd37Wrd4rzzN1vYByQmv5y8nTPvUOIZ0wZ2nSLnXrbTzm1pU6G LrSw== X-Gm-Message-State: APf1xPDU9PrA324qYUSBYk4W6US265Oc7llFPBePz5VditJuOXeMEw6i vfzS5gPPrEckZH+LPumzI6M= X-Received: by 10.36.4.82 with SMTP id 79mr2623568itb.89.1518545636909; Tue, 13 Feb 2018 10:13:56 -0800 (PST) Received: from localhost.uwnet.wisc.edu (dyn-72-33-2-110.uwnet.wisc.edu. [72.33.2.110]) by smtp.gmail.com with ESMTPSA id a25sm71103itj.6.2018.02.13.10.13.55 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 13 Feb 2018 10:13:56 -0800 (PST) Date: Tue, 13 Feb 2018 12:13:52 -0600 From: Dennis Zhou To: Eric Dumazet Cc: Tejun Heo , Daniel Borkmann , Dmitry Vyukov , syzbot , Alexei Starovoitov , netdev , LKML , syzkaller-bugs@googlegroups.com Subject: Re: lost connection to test machine (4) Message-ID: <20180213181352.GA60936@localhost.uwnet.wisc.edu> References: <001a113f8734783e94056505f8fd@google.com> <00c45ca8-305d-1818-e974-a9903c8494b8@iogearbox.net> <20180212170325.GW695913@devbig577.frc2.facebook.com> <20180212200548.GG695913@devbig577.frc2.facebook.com> <1518528926.3715.173.camel@gmail.com> <20180213173438.GA60641@localhost.uwnet.wisc.edu> <1518544167.3715.180.camel@gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <1518544167.3715.180.camel@gmail.com> User-Agent: Mutt/1.5.24 (2015-08-30) Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Tue, Feb 13, 2018 at 09:49:27AM -0800, Eric Dumazet wrote: > On Tue, 2018-02-13 at 11:34 -0600, Dennis Zhou wrote: > > Hi Eric, > > > > On Tue, Feb 13, 2018 at 05:35:26AM -0800, Eric Dumazet wrote: > > > > > > Also I would consider using this fix as I had warnings of cpus being > > > stuck there for more than 50 ms : > > > > > > > > > diff --git a/mm/percpu-vm.c b/mm/percpu-vm.c > > > index 9158e5a81391ced4e268e3d5dd9879c2bc7280ce..6309b01ceb357be01e857e5f899429403836f41f 100644 > > > --- a/mm/percpu-vm.c > > > +++ b/mm/percpu-vm.c > > > @@ -92,6 +92,7 @@ static int pcpu_alloc_pages(struct pcpu_chunk *chunk, > > > *pagep = alloc_pages_node(cpu_to_node(cpu), gfp, 0); > > > if (!*pagep) > > > goto err; > > > + cond_resched(); > > > } > > > } > > > return 0; > > > > > > > > > > This function gets called from pcpu_populate_chunk while holding the > > pcpu_alloc_mutex and is called from two scenarios. First, when an > > allocation occurs to a place without backing pages, and second when the > > workqueue item is scheduled to replenish the number of empty pages. So, > > I don't think this is a good idea. > > > > That _is_ a good idea, we do this already in vmalloc(), and vmalloc() > can absolutely be called while some mutex(es) are held. > > > > My understanding is if we're seeing warnings here, that means we're > > struggling to find backing pages. I believe adding __GFP_NORETRY on the > > workqueue path as Tejun mentioned above would help with warnings as > > well, but not if they are caused by the allocation path. > > > > That is a separate concern. > > My patch simply avoids latency spikes when huge percpu allocations are > happening, on systems with say 1024 cpus. > > I see. I misunderstood thinking this was for the same concern. Thanks, Dennis