Received: by 2002:a05:6a10:1d13:0:0:0:0 with SMTP id pp19csp2740358pxb; Tue, 24 Aug 2021 06:34:43 -0700 (PDT) X-Google-Smtp-Source: ABdhPJyh7RyF5KiZciZ3w4fuuw3W/rP6HkV8Cxg/CpeZHq4n+4DqXwIwzjPKcWXEa8xCElAB0kRH X-Received: by 2002:a05:6402:3128:: with SMTP id dd8mr43612431edb.40.1629812083030; Tue, 24 Aug 2021 06:34:43 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1629812083; cv=none; d=google.com; s=arc-20160816; b=lbu7u35GYAE3f83LqMAxopuooFCVdcZUE3uz1cmnrpAo44aLyIA7KOwaD2bfvvXqZK h8U7oJ/9Q2W0DYnmUlNYJ6dumYu76tum6HiiNS6SbTD53LJCmWR9D1u6Yqez4kU7qdTe ynobYTGFtVXDbqnjyYqJvtqiiqSOsC0mn+po4DN+SozSz/hgt6HSvW+jIIIGJwzLUdRq dHdHxuvQIDripJ45jpW4nisUsyNuBkY8mRs2z7LmgP7oDiZQABJa+D54kBg7Ab6SVsgV 6s/Sqseh4HfnZPeRCQw7epe0Kq4vH6MHwFxYLP/AmIngLatdTEjaKeGa7a99qr6zgAnv sb0g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:content-language :in-reply-to:mime-version:user-agent:date:message-id:references:to :subject:cc:from:dkim-signature; bh=QeV74JQJHRoEmeq7H6kmk/7VDFwFpt82W+iJfSrcr/0=; b=hJzkRwI+pEhoBxFt6wuKQ/EFCMiSB9lv91v7bqOofVtFes4b3Rd2dnYR2vHRKoOtod VcWLUz/TMUkl5HAfv7FU/jMqgdpHGKFk4p/jJtbVxj/iyxOSdAXBXoeFVcI/JHNnTeUm yJO/PXf+sr9kFp5LEkK1nfsf6eubjkdGlyWlB53awQBON7LDiyE/4YUceh+H8RUBkAsd 7vNPPWvkOtrTfSzXH4hmARDWWvzis4LxYpjdnTavMibrrqmxtp/WG4IBdRoojo3huOTO DRWUG3G+BF9Wg8qsyvqi0P68QVSUmjCaKlCaDrPdqRpk1++xI83SDgyL/gYvX8DYC25D KBaA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=dI2EdCmD; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Return-Path: Received: from vger.kernel.org (vger.kernel.org. [23.128.96.18]) by mx.google.com with ESMTP id p17si14038122edq.556.2021.08.24.06.34.19; Tue, 24 Aug 2021 06:34:43 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=dI2EdCmD; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236661AbhHXNdc (ORCPT + 99 others); Tue, 24 Aug 2021 09:33:32 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]:46401 "EHLO us-smtp-delivery-124.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229517AbhHXNdb (ORCPT ); Tue, 24 Aug 2021 09:33:31 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1629811967; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=QeV74JQJHRoEmeq7H6kmk/7VDFwFpt82W+iJfSrcr/0=; b=dI2EdCmDUHfLegT7gE2mirTT4JS1N/wZjyfD3GVXwNAN9eaFKMwqMcj1hmV/50knpBiZ2g /ESdT/mNx2+hRPhGcaMFG1p7BMqNSgmHDxf2srr5YX0az+RRIoYMd01f5xdzczzNMI72Sy 5hA39ROmWZE2QQPeHc2POvCHYHgS4ew= Received: from mail-wr1-f71.google.com (mail-wr1-f71.google.com [209.85.221.71]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-240-4nvEhA47PoC62Gve1t20fA-1; Tue, 24 Aug 2021 09:32:43 -0400 X-MC-Unique: 4nvEhA47PoC62Gve1t20fA-1 Received: by mail-wr1-f71.google.com with SMTP id n10-20020a5d660a0000b02901551ef5616eso5749890wru.20 for ; Tue, 24 Aug 2021 06:32:43 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:cc:subject:to:references:message-id:date :user-agent:mime-version:in-reply-to:content-language :content-transfer-encoding; bh=QeV74JQJHRoEmeq7H6kmk/7VDFwFpt82W+iJfSrcr/0=; b=T5VJJ9ADfinacGNyOcFobj+dqxy7VByCf9V2zF5jn3fOctiq3xdPKvE/0J94Ttnu/B Ent64r6zGgj3ObAqFh6ajkdp3E9Uo8Ho1coB7VLAxdls1AFjPOeQ5Iy82OKoGqihApYl 3ThcZ3q0tKxGCSftmKTBBehvu9o82sU3NG1tIpJ07/MwbnYdRwLD0BRkt/NY5O46QLon TjsnBl/aUZLxfg+EFrYTiYOYQY2iMEZwl/HD+lr7NjoIgb84xG2bD/RXhiJvSqfYsZpw M6uMPCDNaKhdbU+A108/PX3CR8IeRfS+G0OyPrbpyNPDYnYAEUGluefJhVZR+jlIf+Go 3gKw== X-Gm-Message-State: AOAM5305PV0iBtcRglnRJsSwDjhLaByijsObg1wa2E2x5eGF0EX4cx2n lcTHLrZncdf3pkrszXvuQ2Udzna5uSXQDLuZyeqorygZxuaO878qeqJEEMRgradCP6zoHVuL9Hb hTixgl35vnFUxLbt4aC3PHLDi X-Received: by 2002:a1c:9acc:: with SMTP id c195mr4245400wme.69.1629811962647; Tue, 24 Aug 2021 06:32:42 -0700 (PDT) X-Received: by 2002:a1c:9acc:: with SMTP id c195mr4245358wme.69.1629811962408; Tue, 24 Aug 2021 06:32:42 -0700 (PDT) Received: from [192.168.42.238] (3-14-107-185.static.kviknet.dk. [185.107.14.3]) by smtp.gmail.com with ESMTPSA id z7sm2316517wmi.4.2021.08.24.06.32.41 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Tue, 24 Aug 2021 06:32:42 -0700 (PDT) From: Jesper Dangaard Brouer X-Google-Original-From: Jesper Dangaard Brouer Cc: brouer@redhat.com, intel-wired-lan@lists.osuosl.org, netdev@vger.kernel.org, linux-kernel@vger.kernel.org, bpf@vger.kernel.org, Jason Xing , Shujin Li , =?UTF-8?B?w43DsWlnbyBIdWd1ZXQ=?= Subject: Re: [PATCH] ixgbe: let the xdpdrv work with more than 64 cpus To: kerneljasonxing@gmail.com, jesse.brandeburg@intel.com, anthony.l.nguyen@intel.com, davem@davemloft.net, kuba@kernel.org, ast@kernel.org, daniel@iogearbox.net, hawk@kernel.org, john.fastabend@gmail.com, andrii@kernel.org, kafai@fb.com, songliubraving@fb.com, yhs@fb.com, kpsingh@kernel.org References: <20210824104918.7930-1-kerneljasonxing@gmail.com> Message-ID: <59dff551-2d52-5ecc-14ac-4a6ada5b1275@redhat.com> Date: Tue, 24 Aug 2021 15:32:40 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Thunderbird/78.11.0 MIME-Version: 1.0 In-Reply-To: <20210824104918.7930-1-kerneljasonxing@gmail.com> Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 24/08/2021 12.49, kerneljasonxing@gmail.com wrote: > From: Jason Xing > > Originally, ixgbe driver doesn't allow the mounting of xdpdrv if the > server is equipped with more than 64 cpus online. So it turns out that > the loading of xdpdrv causes the "NOMEM" failure. > > Actually, we can adjust the algorithm and then make it work, which has > no harm at all, only if we set the maxmium number of xdp queues. This is not true, it can cause harm, because XDP transmission queues are used without locking. See drivers ndo_xdp_xmit function ixgbe_xdp_xmit(). As driver assumption is that each CPU have its own XDP TX-queue. This patch is not a proper fix. I do think we need a proper fix for this issue on ixgbe. > Fixes: 33fdc82f08 ("ixgbe: add support for XDP_TX action") > Co-developed-by: Shujin Li > Signed-off-by: Shujin Li > Signed-off-by: Jason Xing > --- > drivers/net/ethernet/intel/ixgbe/ixgbe_lib.c | 2 +- > drivers/net/ethernet/intel/ixgbe/ixgbe_main.c | 3 --- > 2 files changed, 1 insertion(+), 4 deletions(-) > > diff --git a/drivers/net/ethernet/intel/ixgbe/ixgbe_lib.c b/drivers/net/ethernet/intel/ixgbe/ixgbe_lib.c > index 0218f6c..5953996 100644 > --- a/drivers/net/ethernet/intel/ixgbe/ixgbe_lib.c > +++ b/drivers/net/ethernet/intel/ixgbe/ixgbe_lib.c > @@ -299,7 +299,7 @@ static void ixgbe_cache_ring_register(struct ixgbe_adapter *adapter) > > static int ixgbe_xdp_queues(struct ixgbe_adapter *adapter) > { > - return adapter->xdp_prog ? nr_cpu_ids : 0; > + return adapter->xdp_prog ? min_t(int, MAX_XDP_QUEUES, nr_cpu_ids) : 0; > } > > #define IXGBE_RSS_64Q_MASK 0x3F > diff --git a/drivers/net/ethernet/intel/ixgbe/ixgbe_main.c b/drivers/net/ethernet/intel/ixgbe/ixgbe_main.c > index 14aea40..b36d16b 100644 > --- a/drivers/net/ethernet/intel/ixgbe/ixgbe_main.c > +++ b/drivers/net/ethernet/intel/ixgbe/ixgbe_main.c > @@ -10130,9 +10130,6 @@ static int ixgbe_xdp_setup(struct net_device *dev, struct bpf_prog *prog) > return -EINVAL; > } > > - if (nr_cpu_ids > MAX_XDP_QUEUES) > - return -ENOMEM; > - > old_prog = xchg(&adapter->xdp_prog, prog); > need_reset = (!!prog != !!old_prog); > >