Received: by 2002:a05:6a10:5bc5:0:0:0:0 with SMTP id os5csp5666020pxb; Sun, 7 Nov 2021 17:32:01 -0800 (PST) X-Google-Smtp-Source: ABdhPJyBjVY4G91eTjjcEUa6Hg1dESAlw+HMJRNJQ6n/jN4WFBZo/66jNDBqRNVXu5wOdMZ+xc7o X-Received: by 2002:a17:907:1693:: with SMTP id hc19mr63501754ejc.396.1636335121189; Sun, 07 Nov 2021 17:32:01 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1636335121; cv=none; d=google.com; s=arc-20160816; b=fDejp9jJe1ZFUwrbGGErYqIfv8ZU19ltcKDGgORAcQ55tktI3gzo2QUoe8rDO6OYAJ EMlFVqPZsL+bIANsAUgFlTsRf4KQggMSbLberPfpezZiJMuOrz2AjWVdl+STfXiFqkIQ TDwPpuw3v/r1m8Ky70+ZbIrphPRg58SsRbpLuZ2O4as/96gc9E1IoMxi5Kk9esOC8W0p fbpldFjnXW9XEzPR9ohpNdZqqIgXvt6vMwF57u927fsEJ23//rtTWLUz3C3ZsS64869x q7aZhjTpWt2WoqKUcO72hl/5gvRINL52vZzIsV3NAPbwV9PZfQy3MpfUyQlzGjcNlx1t 6FuA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:subject:message-id:date:from:in-reply-to :references:mime-version:dkim-signature; bh=4kIZsRDm8rxhr02Tgc4l3Zw6ev1GuI6KgvJet+nCQSI=; b=QoDTXktNb7t8t9d8NE6XZaMtGGDhIBoa1JWrMM0NObJ7WlDfqsLXzTGWGYGdcyYUGF iKhmF4nV59id/saZhEUHH7CXT7/ZgeXLvu5qBrDwigFBGiUC5PUG3YpHtsY2GcySXi9K CrkPHOAeaQJgMzVqcnBtODH03k/fM9/MXeW9J5msCULs4ZiRczryyNtglVnHUJ+2ceBD sbL95j2MQtZxg704lseBun51AkXDSY1PJPU/quMM80KoOUEjtGB8TBkYJafclFDmZdhl bWCyGOSvLq9i1X6OyploIvksJAmm2nV5IR3l08+ZXIMfEu+o1s3juJL+Nzn+nENzfumK 9Xrw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20210112 header.b=PN0kglOF; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from vger.kernel.org (vger.kernel.org. [23.128.96.18]) by mx.google.com with ESMTP id g14si29496998ejt.419.2021.11.07.17.31.26; Sun, 07 Nov 2021 17:32:01 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20210112 header.b=PN0kglOF; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235013AbhKGSLI (ORCPT + 99 others); Sun, 7 Nov 2021 13:11:08 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52676 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230234AbhKGSLH (ORCPT ); Sun, 7 Nov 2021 13:11:07 -0500 Received: from mail-ed1-x52e.google.com (mail-ed1-x52e.google.com [IPv6:2a00:1450:4864:20::52e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2F0D0C061570; Sun, 7 Nov 2021 10:08:24 -0800 (PST) Received: by mail-ed1-x52e.google.com with SMTP id f8so53639234edy.4; Sun, 07 Nov 2021 10:08:24 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=4kIZsRDm8rxhr02Tgc4l3Zw6ev1GuI6KgvJet+nCQSI=; b=PN0kglOFSxYGrMihnffAFbvbsZmvquumWw4BXLnxtjDMQ/QPChvVn8E4R6JdhtZ2pb L3nRyC99Gb0yEzIaQ1PU6UC7iy/gSsb9PYIh5fBfK/exd17nX7+mHdp/WPVqZj25fMPB SQOMFjU+xzcRCaxAfUFrPSgUhWhiZkT5dzFDARG4AHojKWEU4BGYZ1sLuLv81CcNBdtj 8u694lQy/O0uDqmRYmwPDA21TbFuMaO3JafSCM4IyKWvXu+vJvW1H2DWCZEEzNbiwPH8 1W7VrVu25SNm1rXDqhRD2tYJ84xOvBfT0N+42tbHOW+ac2FrKfrrPiCm10a1n2F8hvUe LaMw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=4kIZsRDm8rxhr02Tgc4l3Zw6ev1GuI6KgvJet+nCQSI=; b=ZBeX0LV2PNdh6XYZP1P4fKAMQF3UQBrxRdXjza4WWTL/awGnE9Vti0K9cWMeI5R1z+ 9/diIKPucIsCO+F2u4P/iy0A3C4Q9CHitj1LHCfUEWI886YzKATli/Eo3i/1RnFYzgiK cgjanXBO9UYk1N70iRqkZ1sX0qaXbWvSvcZEkh5YLcnPqOxedT+vYkt4EjenU9fE/Yh3 /CiS7YpR98npvib4uxUhlUzZGTtOw+L6zKd1PbsbP0Xbj4ELpWANNre1btElWhSR+XVz nb/iMrjS/+UjStJt/75Z4v0wo/fzdDC5adOMsTR/pOvVlFQMyeFxGFFi0XV/lyexctfc mqeg== X-Gm-Message-State: AOAM5336C8Q9DJe6I9GtbM8KJVNafYkMkdq6ZcGmce8zLh/h+pxGMPjg Gaf5Wd/KI8E1vA4ezOqqLJC2ATTp5zjjSRWw28c= X-Received: by 2002:a17:906:d0c3:: with SMTP id bq3mr88882067ejb.277.1636308502800; Sun, 07 Nov 2021 10:08:22 -0800 (PST) MIME-Version: 1.0 References: <20211105105136.12137-1-21cnbao@gmail.com> In-Reply-To: From: Barry Song <21cnbao@gmail.com> Date: Mon, 8 Nov 2021 07:08:09 +1300 Message-ID: Subject: Re: [RFC PATCH] sched&net: avoid over-pulling tasks due to network interrupts To: Peter Zijlstra Cc: David Miller , kuba@kernel.org, Eric Dumazet , pabeni@redhat.com, fw@strlen.de, Ingo Molnar , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Daniel Bristot de Oliveira , Thomas Gleixner , netdev@vger.kernel.org, LKML , Linuxarm , Guodong Xu , yangyicong , shenyang39@huawei.com, tangchengchang@huawei.com, Barry Song , Libo Chen , Tim Chen Content-Type: text/plain; charset="UTF-8" Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Sat, Nov 6, 2021 at 1:25 AM Peter Zijlstra wrote: > > On Fri, Nov 05, 2021 at 06:51:36PM +0800, Barry Song wrote: > > From: Barry Song > > > > In LPC2021, both Libo Chen and Tim Chen have reported the overpull > > of network interrupts[1]. For example, while running one database, > > ethernet is located in numa0, numa1 might be almost idle due to > > interrupts are pulling tasks to numa0 because of wake_up affine. > > I have seen the same problem. One way to solve this problem is > > moving to a normal wakeup in network rather than using a sync > > wakeup which will be more aggressively pulling tasks in scheduler > > core. > > > > On kunpeng920 with 4numa, ethernet is located at numa0, storage > > disk is located at numa2. While using sysbench to connect this > > mysql machine, I am seeing numa1 is idle though numa0,2 and 3 > > are quite busy. > > > > > I am not saying this patch is exactly the right approach, But I'd > > like to use this RFC to connect the people of net and scheduler, > > and start the discussion in this wider range. > > Well the normal way would be to use multi-queue crud and/or receive > packet steering to get the interrupt/wakeup back to the cpu that data > came from. The test case has been a multi-queue ethernet and irqs are balanced to NUMA0 by irqbalanced or pinned to NUMA0 where the card is located by the script like: #!/bin/bash irq_list=(`cat /proc/interrupts | grep network_name| awk -F: '{print $1}'`) cpunum=0 for irq in ${irq_list[@]} do echo $cpunum > /proc/irq/$irq/smp_affinity_list echo `cat /proc/irq/$irq/smp_affinity_list` (( cpunum+=1 )) done I have heard some people are working around this issue by pinning multi-queue IRQs to multiple NUMAs which can spread interrupts and avoid over-pulling tasks to one NUMA only, but lose ethernet locality? Hi, @Tim, it seems in LPC2021 you mentioned you are using this solution? And some other people are pinning ethernet IRQs to a couple of CPUs within the NUMA ethernet belongs to, and then isolate these CPUs from tasks and use those CPUs for interrupts only. This can avoid wake-up pulling at all. I think we need some generic way to resolve this problem. Hi, @Libo , what is your solution to work around this issue? Thanks Barry