Received: by 2002:a05:6a10:206:0:0:0:0 with SMTP id 6csp2209231pxj; Sun, 16 May 2021 18:40:43 -0700 (PDT) X-Google-Smtp-Source: ABdhPJz8kx7o+2Ef2PoqUyT88texoXtqAaN8r2Jesu9J8c5oCHocCwR3IHJIrj2q5LJwmW6WUdc/ X-Received: by 2002:a17:906:3042:: with SMTP id d2mr60004823ejd.234.1621215643108; Sun, 16 May 2021 18:40:43 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1621215643; cv=none; d=google.com; s=arc-20160816; b=m5gWGg5VXTM9YJk240mfGkCXITq13LhbK284xraKKmwkmgaWwJv0z5P+CDEAio2tBF jfl9oVbCBbXjTMtvikOCNzcrkkAQtFWFZH8s8bkllaYtcvBHqDFY+9ZJjv5GIt9QRZVa otF40yfSDNZjEy06VXIYmIn25RF5pGHulZ1ls/hgd3DnNMa4yT6jgB39He+Dnjj/eeIR 87DdDNAIU829CxWD0jr+zewavHwUT3GIJYc1Tt5Dtij/MwVNZ2kOkxCsaWussEdU6SRW uKs8pKj/u3Q/4jWmQqaGIxS1LKcXtb79tQcUKw3b3zud19hbhoNt/fUH4QFyrKPrZykZ 6XPw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=erftlBlDj8yOJHAczfEDcCSRCgt7DwRXUNkxjkBndbc=; b=cxge4pfuItuz4yMXKYpAbuZRNWVW+kdzn6KSFTCnDyw2u8l0UIswFQYf2v9z7ej02d 7cZR5giaBXvAdiaKwUqLHHdEbwXr8nePMEOjZnqO4vnmKvlHoZB9yL5bfuT8kyA30HV/ 7GQdWhy614etm93LdadMUKfApSYooGtw61AB6FF6GBH0Mfx9IMP5KkKSYcbE4Y7LiLgC 7Pk2TgqotuTpqQ37oBSv/xTLShleZ4Thg8EsVPUM+TbUvLfYrQ3+hvrbrjr9dvrgv/hK FMy3lzv8Qc/ki6nrsqgou/ERv19am+awBXFwlq6h+z/0J9CsRG3ZQ/ca6vx+BZk3wnlN Prvg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@amazon.co.jp header.s=amazon201209 header.b=FN+lmpKM; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=amazon.co.jp Return-Path: Received: from vger.kernel.org (vger.kernel.org. [23.128.96.18]) by mx.google.com with ESMTP id jo3si9681639ejb.468.2021.05.16.18.40.19; Sun, 16 May 2021 18:40:43 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18; Authentication-Results: mx.google.com; dkim=pass header.i=@amazon.co.jp header.s=amazon201209 header.b=FN+lmpKM; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=amazon.co.jp Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234937AbhEQAZ6 (ORCPT + 99 others); Sun, 16 May 2021 20:25:58 -0400 Received: from smtp-fw-9102.amazon.com ([207.171.184.29]:15109 "EHLO smtp-fw-9102.amazon.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234926AbhEQAZ4 (ORCPT ); Sun, 16 May 2021 20:25:56 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amazon.co.jp; i=@amazon.co.jp; q=dns/txt; s=amazon201209; t=1621211081; x=1652747081; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=erftlBlDj8yOJHAczfEDcCSRCgt7DwRXUNkxjkBndbc=; b=FN+lmpKMGESUIohC9K6+fQ+BwzOpmmX7fygHFNrmDsNjrCyPKjIgVySQ 7z6Jd1ayudy5RD/8OW2BI6vX4awQ+9qfOfAO/fuUabkdNfYpZyysk2cOg NFN3JOkQ5WzcNrvA9+TcvkzzhGU64I9vnQlOwhnjlio8dVb6buPJPOzsi s=; X-IronPort-AV: E=Sophos;i="5.82,306,1613433600"; d="scan'208";a="135171672" Received: from pdx4-co-svc-p1-lb2-vlan2.amazon.com (HELO email-inbound-relay-1e-27fb8269.us-east-1.amazon.com) ([10.25.36.210]) by smtp-border-fw-9102.sea19.amazon.com with ESMTP; 17 May 2021 00:24:40 +0000 Received: from EX13MTAUWB001.ant.amazon.com (iad12-ws-svc-p26-lb9-vlan3.iad.amazon.com [10.40.163.38]) by email-inbound-relay-1e-27fb8269.us-east-1.amazon.com (Postfix) with ESMTPS id 69855A1816; Mon, 17 May 2021 00:24:37 +0000 (UTC) Received: from EX13D04ANC001.ant.amazon.com (10.43.157.89) by EX13MTAUWB001.ant.amazon.com (10.43.161.249) with Microsoft SMTP Server (TLS) id 15.0.1497.18; Mon, 17 May 2021 00:24:36 +0000 Received: from 88665a182662.ant.amazon.com (10.43.162.28) by EX13D04ANC001.ant.amazon.com (10.43.157.89) with Microsoft SMTP Server (TLS) id 15.0.1497.18; Mon, 17 May 2021 00:24:31 +0000 From: Kuniyuki Iwashima To: "David S . Miller" , Jakub Kicinski , Eric Dumazet , Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Martin KaFai Lau CC: Benjamin Herrenschmidt , Kuniyuki Iwashima , Kuniyuki Iwashima , , , Subject: [PATCH v6 bpf-next 05/11] tcp: Migrate TCP_ESTABLISHED/TCP_SYN_RECV sockets in accept queues. Date: Mon, 17 May 2021 09:22:52 +0900 Message-ID: <20210517002258.75019-6-kuniyu@amazon.co.jp> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20210517002258.75019-1-kuniyu@amazon.co.jp> References: <20210517002258.75019-1-kuniyu@amazon.co.jp> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Content-Type: text/plain X-Originating-IP: [10.43.162.28] X-ClientProxiedBy: EX13D21UWA004.ant.amazon.com (10.43.160.252) To EX13D04ANC001.ant.amazon.com (10.43.157.89) Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org When we call close() or shutdown() for listening sockets, each child socket in the accept queue are freed at inet_csk_listen_stop(). If we can get a new listener by reuseport_migrate_sock() and clone the request by reqsk_clone(), we try to add it into the new listener's accept queue by inet_csk_reqsk_queue_add(). If it fails, we have to call __reqsk_free() to call sock_put() for its listener and free the cloned request. After putting the full socket into ehash, tcp_v[46]_syn_recv_sock() sets NULL to ireq_opt/pktopts in struct inet_request_sock, but ipv6_opt can be non-NULL. So, we have to set NULL to ipv6_opt of the old request to avoid double free. Note that we do not update req->rsk_listener and instead clone the req to migrate because another path may reference the original request. If we protected it by RCU, we would need to add rcu_read_lock() in many places. Link: https://lore.kernel.org/netdev/20201209030903.hhow5r53l6fmozjn@kafai-mbp.dhcp.thefacebook.com/ Suggested-by: Martin KaFai Lau Signed-off-by: Kuniyuki Iwashima --- net/ipv4/inet_connection_sock.c | 71 ++++++++++++++++++++++++++++++++- 1 file changed, 70 insertions(+), 1 deletion(-) diff --git a/net/ipv4/inet_connection_sock.c b/net/ipv4/inet_connection_sock.c index fa806e9167ec..07e97b2f3635 100644 --- a/net/ipv4/inet_connection_sock.c +++ b/net/ipv4/inet_connection_sock.c @@ -695,6 +695,53 @@ int inet_rtx_syn_ack(const struct sock *parent, struct request_sock *req) } EXPORT_SYMBOL(inet_rtx_syn_ack); +static struct request_sock *inet_reqsk_clone(struct request_sock *req, + struct sock *sk) +{ + struct sock *req_sk, *nreq_sk; + struct request_sock *nreq; + + nreq = kmem_cache_alloc(req->rsk_ops->slab, GFP_ATOMIC | __GFP_NOWARN); + if (!nreq) { + /* paired with refcount_inc_not_zero() in reuseport_migrate_sock() */ + sock_put(sk); + return NULL; + } + + req_sk = req_to_sk(req); + nreq_sk = req_to_sk(nreq); + + memcpy(nreq_sk, req_sk, + offsetof(struct sock, sk_dontcopy_begin)); + memcpy(&nreq_sk->sk_dontcopy_end, &req_sk->sk_dontcopy_end, + req->rsk_ops->obj_size - offsetof(struct sock, sk_dontcopy_end)); + + sk_node_init(&nreq_sk->sk_node); + nreq_sk->sk_tx_queue_mapping = req_sk->sk_tx_queue_mapping; +#ifdef CONFIG_XPS + nreq_sk->sk_rx_queue_mapping = req_sk->sk_rx_queue_mapping; +#endif + nreq_sk->sk_incoming_cpu = req_sk->sk_incoming_cpu; + refcount_set(&nreq_sk->sk_refcnt, 0); + + nreq->rsk_listener = sk; + + /* We need not acquire fastopenq->lock + * because the child socket is locked in inet_csk_listen_stop(). + */ + if (sk->sk_protocol == IPPROTO_TCP && tcp_rsk(nreq)->tfo_listener) + rcu_assign_pointer(tcp_sk(nreq->sk)->fastopen_rsk, nreq); + + return nreq; +} + +static void reqsk_migrate_reset(struct request_sock *req) +{ +#if IS_ENABLED(CONFIG_IPV6) + inet_rsk(req)->ipv6_opt = NULL; +#endif +} + /* return true if req was found in the ehash table */ static bool reqsk_queue_unlink(struct request_sock *req) { @@ -1036,14 +1083,36 @@ void inet_csk_listen_stop(struct sock *sk) * of the variants now. --ANK */ while ((req = reqsk_queue_remove(queue, sk)) != NULL) { - struct sock *child = req->sk; + struct sock *child = req->sk, *nsk; + struct request_sock *nreq; local_bh_disable(); bh_lock_sock(child); WARN_ON(sock_owned_by_user(child)); sock_hold(child); + nsk = reuseport_migrate_sock(sk, child, NULL); + if (nsk) { + nreq = inet_reqsk_clone(req, nsk); + if (nreq) { + refcount_set(&nreq->rsk_refcnt, 1); + + if (inet_csk_reqsk_queue_add(nsk, nreq, child)) { + reqsk_migrate_reset(req); + } else { + reqsk_migrate_reset(nreq); + __reqsk_free(nreq); + } + + /* inet_csk_reqsk_queue_add() has already + * called inet_child_forget() on failure case. + */ + goto skip_child_forget; + } + } + inet_child_forget(sk, req, child); +skip_child_forget: reqsk_put(req); bh_unlock_sock(child); local_bh_enable(); -- 2.30.2