Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
From:   Dongli Zhang <dongli.zhang@oracle.com>
To:     netdev@vger.kernel.org, bpf@vger.kernel.org
Cc:     linux-kernel@vger.kernel.org, davem@davemloft.net, kuba@kernel.org,
        rostedt@goodmis.org, mingo@redhat.com, ast@kernel.org,
        daniel@iogearbox.net, andrii@kernel.org, imagedong@tencent.com,
        joao.m.martins@oracle.com, joe.jin@oracle.com, dsahern@gmail.com,
        edumazet@google.com
Subject: [PATCH net-next v6 3/3] net: tun: track dropped skb via kfree_skb_reason()
Date:   Fri,  4 Mar 2022 06:55:07 -0800
Message-Id: <20220304145507.1883-4-dongli.zhang@oracle.com>
In-Reply-To: <20220304145507.1883-1-dongli.zhang@oracle.com>
References: <20220304145507.1883-1-dongli.zhang@oracle.com>
Content-Type: text/plain
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?XsZJFWYJP5Ilr0zPO6alzpVjzK5knGJVuCcOBaBQpFuRCPZtc4LJhx9p3vBn?=
 =?us-ascii?Q?vP8leycAarMt2nWEOBL2+hfo9Kc+82r1yA6QWsbk4vk/IC/RO83RYhN6SJUu?=
 =?us-ascii?Q?2rk+IfIzQJlt0AVm5Ysv5yIjxVi75l1qCOPoge8mO9O22GQZMQRBHUc8rN7n?=
 =?us-ascii?Q?5ShKNQjOk4bUZL3moMCKYDgYcKfmYj1fvMWg+y6q8d/8oyaKJNVr3bzw615/?=
 =?us-ascii?Q?RpBsoNRgP9RWoWHstjNuhCVgFM/w4/QqKz2GyH8KswFnR2cEfSC/JVrHdyl4?=
 =?us-ascii?Q?AsMBcvPHd/wZz3Qrv9jI1QhWPjSCs5AmyA07cuikxGcTl35St7iFZkxEOBEs?=
 =?us-ascii?Q?Y34PPvdTgPF9wZnlraA3To0GaFgBJgzreeZ1Q3VSOlbqfE4S+JGgnjVdaTJ1?=
 =?us-ascii?Q?nCp2E9zI9E50Yl1TZEs0zOrGoM8oT1U5l6GFnYvXPx0eS9ix+BlWEdbvcW6p?=
 =?us-ascii?Q?F+Jef/dwEieDZcsniV78dT5DPDZGAI8Tg6Jw0RLPt3obEoREz81WhHv0x1TE?=
 =?us-ascii?Q?PC+rY5A8Mpu/XatE9RNvdwG2165xo3N+5lSyGghV6ocHIRdh9iLNu59GphYS?=
 =?us-ascii?Q?XKyxZVAsmOKDymny8CbKp+xHhIWxr4l5LaGHUK0B70jUbACKQvxQ1rKIeHtR?=
 =?us-ascii?Q?mF/X/iF5/nM3nF6ZJvkri9Gm2bMFuxSXhSU3qesrwawvnpvzDVWLvgeizIP3?=
 =?us-ascii?Q?IytNv//L9rB5QIPawbb103peLVwKmwJITUrVdIGfYQbvx71UF5tyJUMESJ1a?=
 =?us-ascii?Q?zCoLMle5TNOKsYo4nzrAiNA6KCLWrkeYFB2eGzfvzx2fCbKVtH5IUryb0D8E?=
 =?us-ascii?Q?O+2Hsnq1IWi0coC2I/fmYKlFh1cV60c2P7RASGCK1O546YrYEx3mtyItUiTc?=
 =?us-ascii?Q?qF6zOuEJRgM/PiOisZqBZsFYiRZ2IYuq0wrvQsXykFKCrrJuCBRSyVHp3rvZ?=
 =?us-ascii?Q?PV8pdpV8yzXOYRQAjD5I/42nrgV1TxxIJmY83aeb0kSXS2kWv1s2yNPIfLRm?=
 =?us-ascii?Q?dinVGFhQx/PRck9O353wYeTXiLUCtzp2VkBI87AzLyKO3+0fYbRPT46g2SQ5?=
 =?us-ascii?Q?WLtchFXPg4rfe+GdU5+jRq1XOMNAnXNd4iAIA8LwURZ818GH+TARB97h3jmU?=
 =?us-ascii?Q?tmGA7tO72kFI2JWKsDvrMdvK/HgPfHBOPRVw9CToDDRrHWlxwUepZ1FZbyGi?=
 =?us-ascii?Q?Zq6JBhX83grMTASKKtyz796fuyxccEnbe5BZChjOFy3D/Ud9+4Y3nyeAppjp?=
 =?us-ascii?Q?vBIeS7yDoVEWNGiidN7jiDEq65neLYkqr2MfSAkJKxXXMl63E1xwd3nh+zlN?=
 =?us-ascii?Q?Hnt8WOB/8t7jKNkH9GCUjt1sVgm4m26hq+mBFU18MTiRG/8XOCu5wjFXwmR1?=
 =?us-ascii?Q?wQVuV6Rfc0LVnZuibhrbHS7Zq5Fvtv9sfGJZKavcUViXhbz1zda2JsvzXHEO?=
 =?us-ascii?Q?SW7bVlg8FzCL+H8fC/U9zzG/lCKtAWszBkY1KQSA8MnEWx4j3Fne3ecZOmlF?=
 =?us-ascii?Q?S6x6jzyKaZSB0BhhFwBrV1OAiZQRISEUUNDUm01MT/g6TGH8AClzyrMBroku?=
 =?us-ascii?Q?XAfUL7y/heyWHNf4Db7/zSahditsyqRyj577nwJTZ144yxyDZk+EQD0ot02t?=
 =?us-ascii?Q?uer5bnltpSjwB0kJYDwFelY=3D?=
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 7078ffa6-a4c9-4087-dd53-08d9fdeefee4
X-MS-Exchange-CrossTenant-AuthSource: BYAPR10MB2663.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 04 Mar 2022 14:55:17.1653
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: xY9Zzbt8TzZKY309Onb1n9CucJq2gCLTbavV+Ewfe7brg2/+HAazETX5tLrQsd2+DnypaeeWpfv0z0NVrlyMVg==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM6PR10MB3977
X-Proofpoint-Virus-Version: vendor=nai engine=6300 definitions=10276 signatures=690470
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 malwarescore=0 mlxlogscore=999
 adultscore=0 spamscore=0 phishscore=0 suspectscore=0 bulkscore=0
 mlxscore=0 classifier=spam adjust=0 reason=mlx scancount=1
 engine=8.12.0-2202240000 definitions=main-2203040080
X-Proofpoint-ORIG-GUID: C08cUcD-gz7Btndm792AiSkD97cmXIWo
X-Proofpoint-GUID: C08cUcD-gz7Btndm792AiSkD97cmXIWo
X-Spam-Status: No, score=-2.8 required=5.0 tests=BAYES_00,DKIM_SIGNED,
        DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_LOW,
        RCVD_IN_MSPIKE_H5,RCVD_IN_MSPIKE_WL,SPF_HELO_NONE,SPF_NONE,
        T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

The TUN can be used as vhost-net backend. E.g, the tun_net_xmit() is the
interface to forward the skb from TUN to vhost-net/virtio-net.

However, there are many "goto drop" in the TUN driver. Therefore, the
kfree_skb_reason() is involved at each "goto drop" to help userspace
ftrace/ebpf to track the reason for the loss of packets.

The below reasons are introduced:

- SKB_DROP_REASON_DEV_READY
- SKB_DROP_REASON_NOMEM
- SKB_DROP_REASON_HDR_TRUNC
- SKB_DROP_REASON_TAP_FILTER
- SKB_DROP_REASON_TAP_TXFILTER

Cc: Joao Martins <joao.m.martins@oracle.com>
Cc: Joe Jin <joe.jin@oracle.com>
Signed-off-by: Dongli Zhang <dongli.zhang@oracle.com>
---
Changed since v1:
  - revise the reason name
Changed since v2:
  - declare drop_reason as type "enum skb_drop_reason"
Changed since v3:
  - rename to TAP_FILTER and TAP_TXFILTER
  - honor reverse xmas tree style declaration for 'drop_reason' in
    tun_net_xmit()
Changed since v4:
  - expand comment on DEV_READY
  - change SKB_TRIM to NOMEM
  - chnage SKB_PULL to HDR_TRUNC

 drivers/net/tun.c          | 37 ++++++++++++++++++++++++++++---------
 include/linux/skbuff.h     | 18 ++++++++++++++++++
 include/trace/events/skb.h |  5 +++++
 3 files changed, 51 insertions(+), 9 deletions(-)

diff --git a/drivers/net/tun.c b/drivers/net/tun.c
index 6e06c846fe82..bab92e489fba 100644
--- a/drivers/net/tun.c
+++ b/drivers/net/tun.c
@@ -1058,6 +1058,7 @@ static unsigned int run_ebpf_filter(struct tun_struct *tun,
 static netdev_tx_t tun_net_xmit(struct sk_buff *skb, struct net_device *dev)
 {
 	struct tun_struct *tun = netdev_priv(dev);
+	enum skb_drop_reason drop_reason;
 	int txq = skb->queue_mapping;
 	struct netdev_queue *queue;
 	struct tun_file *tfile;
@@ -1067,8 +1068,10 @@ static netdev_tx_t tun_net_xmit(struct sk_buff *skb, struct net_device *dev)
 	tfile = rcu_dereference(tun->tfiles[txq]);
 
 	/* Drop packet if interface is not attached */
-	if (!tfile)
+	if (!tfile) {
+		drop_reason = SKB_DROP_REASON_DEV_READY;
 		goto drop;
+	}
 
 	if (!rcu_dereference(tun->steering_prog))
 		tun_automq_xmit(tun, skb);
@@ -1078,22 +1081,32 @@ static netdev_tx_t tun_net_xmit(struct sk_buff *skb, struct net_device *dev)
 	/* Drop if the filter does not like it.
 	 * This is a noop if the filter is disabled.
 	 * Filter can be enabled only for the TAP devices. */
-	if (!check_filter(&tun->txflt, skb))
+	if (!check_filter(&tun->txflt, skb)) {
+		drop_reason = SKB_DROP_REASON_TAP_TXFILTER;
 		goto drop;
+	}
 
 	if (tfile->socket.sk->sk_filter &&
-	    sk_filter(tfile->socket.sk, skb))
+	    sk_filter(tfile->socket.sk, skb)) {
+		drop_reason = SKB_DROP_REASON_SOCKET_FILTER;
 		goto drop;
+	}
 
 	len = run_ebpf_filter(tun, skb, len);
-	if (len == 0)
+	if (len == 0) {
+		drop_reason = SKB_DROP_REASON_TAP_FILTER;
 		goto drop;
+	}
 
-	if (pskb_trim(skb, len))
+	if (pskb_trim(skb, len)) {
+		drop_reason = SKB_DROP_REASON_NOMEM;
 		goto drop;
+	}
 
-	if (unlikely(skb_orphan_frags_rx(skb, GFP_ATOMIC)))
+	if (unlikely(skb_orphan_frags_rx(skb, GFP_ATOMIC))) {
+		drop_reason = SKB_DROP_REASON_SKB_UCOPY_FAULT;
 		goto drop;
+	}
 
 	skb_tx_timestamp(skb);
 
@@ -1104,8 +1117,10 @@ static netdev_tx_t tun_net_xmit(struct sk_buff *skb, struct net_device *dev)
 
 	nf_reset_ct(skb);
 
-	if (ptr_ring_produce(&tfile->tx_ring, skb))
+	if (ptr_ring_produce(&tfile->tx_ring, skb)) {
+		drop_reason = SKB_DROP_REASON_FULL_RING;
 		goto drop;
+	}
 
 	/* NETIF_F_LLTX requires to do our own update of trans_start */
 	queue = netdev_get_tx_queue(dev, txq);
@@ -1122,7 +1137,7 @@ static netdev_tx_t tun_net_xmit(struct sk_buff *skb, struct net_device *dev)
 drop:
 	atomic_long_inc(&dev->tx_dropped);
 	skb_tx_error(skb);
-	kfree_skb(skb);
+	kfree_skb_reason(skb, drop_reason);
 	rcu_read_unlock();
 	return NET_XMIT_DROP;
 }
@@ -1720,6 +1735,7 @@ static ssize_t tun_get_user(struct tun_struct *tun, struct tun_file *tfile,
 	u32 rxhash = 0;
 	int skb_xdp = 1;
 	bool frags = tun_napi_frags_enabled(tfile);
+	enum skb_drop_reason drop_reason;
 
 	if (!(tun->flags & IFF_NO_PI)) {
 		if (len < sizeof(pi))
@@ -1823,9 +1839,10 @@ static ssize_t tun_get_user(struct tun_struct *tun, struct tun_file *tfile,
 
 		if (err) {
 			err = -EFAULT;
+			drop_reason = SKB_DROP_REASON_SKB_UCOPY_FAULT;
 drop:
 			atomic_long_inc(&tun->dev->rx_dropped);
-			kfree_skb(skb);
+			kfree_skb_reason(skb, drop_reason);
 			if (frags) {
 				tfile->napi.skb = NULL;
 				mutex_unlock(&tfile->napi_mutex);
@@ -1872,6 +1889,7 @@ static ssize_t tun_get_user(struct tun_struct *tun, struct tun_file *tfile,
 	case IFF_TAP:
 		if (frags && !pskb_may_pull(skb, ETH_HLEN)) {
 			err = -ENOMEM;
+			drop_reason = SKB_DROP_REASON_HDR_TRUNC;
 			goto drop;
 		}
 		skb->protocol = eth_type_trans(skb, tun->dev);
@@ -1925,6 +1943,7 @@ static ssize_t tun_get_user(struct tun_struct *tun, struct tun_file *tfile,
 	if (unlikely(!(tun->dev->flags & IFF_UP))) {
 		err = -EIO;
 		rcu_read_unlock();
+		drop_reason = SKB_DROP_REASON_DEV_READY;
 		goto drop;
 	}
 
diff --git a/include/linux/skbuff.h b/include/linux/skbuff.h
index 67cfff4065b6..34f572271c0c 100644
--- a/include/linux/skbuff.h
+++ b/include/linux/skbuff.h
@@ -424,7 +424,25 @@ enum skb_drop_reason {
 	SKB_DROP_REASON_DEV_HDR,	/* device driver specific
 					 * header/metadata is invalid
 					 */
+	/* the device is not ready to xmit/recv due to any of its data
+	 * structure that is not up/ready/initialized, e.g., the IFF_UP is
+	 * not set, or driver specific tun->tfiles[txq] is not initialized
+	 */
+	SKB_DROP_REASON_DEV_READY,
 	SKB_DROP_REASON_FULL_RING,	/* ring buffer is full */
+	SKB_DROP_REASON_NOMEM,		/* error due to OOM */
+	SKB_DROP_REASON_HDR_TRUNC,      /* failed to trunc/extract the header
+					 * from networking data, e.g., failed
+					 * to pull the protocol header from
+					 * frags via pskb_may_pull()
+					 */
+	SKB_DROP_REASON_TAP_FILTER,     /* dropped by (ebpf) filter directly
+					 * attached to tun/tap, e.g., via
+					 * TUNSETFILTEREBPF
+					 */
+	SKB_DROP_REASON_TAP_TXFILTER,	/* dropped by tx filter implemented
+					 * at tun/tap, e.g., check_filter()
+					 */
 	SKB_DROP_REASON_MAX,
 };
 
diff --git a/include/trace/events/skb.h b/include/trace/events/skb.h
index 240e7e7591fc..e1670e1e4934 100644
--- a/include/trace/events/skb.h
+++ b/include/trace/events/skb.h
@@ -55,7 +55,12 @@
 	EM(SKB_DROP_REASON_SKB_GSO_SEG, SKB_GSO_SEG)		\
 	EM(SKB_DROP_REASON_SKB_UCOPY_FAULT, SKB_UCOPY_FAULT)	\
 	EM(SKB_DROP_REASON_DEV_HDR, DEV_HDR)			\
+	EM(SKB_DROP_REASON_DEV_READY, DEV_READY)		\
 	EM(SKB_DROP_REASON_FULL_RING, FULL_RING)		\
+	EM(SKB_DROP_REASON_NOMEM, NOMEM)			\
+	EM(SKB_DROP_REASON_HDR_TRUNC, HDR_TRUNC)		\
+	EM(SKB_DROP_REASON_TAP_FILTER, TAP_FILTER)		\
+	EM(SKB_DROP_REASON_TAP_TXFILTER, TAP_TXFILTER)		\
 	EMe(SKB_DROP_REASON_MAX, MAX)
 
 #undef EM
-- 
2.17.1