Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.37 as permitted sender) client-ip=23.128.96.37;
Message-ID: <30596a99-ac3f-5394-b0b7-e23037f2ae8e@oracle.com>
Date:   Tue, 17 Oct 2023 13:14:56 +0100
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101
 Thunderbird/102.15.0
Subject: Re: [PATCH 2/4] kbuild: avoid too many execution of
 scripts/pahole-flags.sh
Content-Language: en-GB
To:     Masahiro Yamada <masahiroy@kernel.org>,
        linux-kbuild@vger.kernel.org
Cc:     linux-kernel@vger.kernel.org, Alex Gaynor <alex.gaynor@gmail.com>,
        Alexei Starovoitov <ast@kernel.org>,
        Alice Ryhl <aliceryhl@google.com>,
        Andreas Hindborg <a.hindborg@samsung.com>,
        Andrii Nakryiko <andrii@kernel.org>,
        Benno Lossin <benno.lossin@proton.me>,
        =?UTF-8?Q?Bj=c3=b6rn_Roy_Baron?= <bjorn3_gh@protonmail.com>,
        Boqun Feng <boqun.feng@gmail.com>,
        Daniel Borkmann <daniel@iogearbox.net>,
        Gary Guo <gary@garyguo.net>, Hao Luo <haoluo@google.com>,
        Jiri Olsa <jolsa@kernel.org>,
        John Fastabend <john.fastabend@gmail.com>,
        KP Singh <kpsingh@kernel.org>,
        Martin KaFai Lau <martin.lau@linux.dev>,
        Miguel Ojeda <ojeda@kernel.org>,
        Nathan Chancellor <nathan@kernel.org>,
        Nick Desaulniers <ndesaulniers@google.com>,
        Nicolas Schier <nicolas@fjasle.eu>, Song Liu <song@kernel.org>,
        Stanislav Fomichev <sdf@google.com>,
        Wedson Almeida Filho <wedsonaf@gmail.com>,
        Yonghong Song <yonghong.song@linux.dev>, bpf@vger.kernel.org,
        rust-for-linux@vger.kernel.org
References: <20231017103742.130927-1-masahiroy@kernel.org>
 <20231017103742.130927-2-masahiroy@kernel.org>
From:   Alan Maguire <alan.maguire@oracle.com>
In-Reply-To: <20231017103742.130927-2-masahiroy@kernel.org>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?OFNlK1RTZnBJVUVqM21sZ3JTMmJFYzQ1R2lhRm9CZHVKV0RXWXJ3ZFBTekhX?=
 =?utf-8?B?S21Pb25icXAwckFqZHc4WHBBQVQ4Yk9zNFVIVzhGTys5d3NHVGltTzQxTVlt?=
 =?utf-8?B?bnJyYVBDT1NOMlJ6ZXE2aTdHNUxJeGIyVmhCYk1USEFRbGltdHNLNlZtblE3?=
 =?utf-8?B?bWRFTytxTXQ1ZHZ3WjdodzI0RVMxck0xUFE2Y211WHJJRWxYemJxeVdHUjdK?=
 =?utf-8?B?R0p4d0ptY3d0Y0Vxdml2R05zODBiRmtCdy80Ymx0TjluQmpXa29VSm83b0hM?=
 =?utf-8?B?OFdoUEc0OFM3U0g0MHJqN2hNTXpmOUdnZFV1WmNzcVh4bzVFdXhwdkt4dmN3?=
 =?utf-8?B?YVU3M2Jld1Z6WW84QlF2STdWMHlvbUdScmFIU0l6R1MxVXZpdC9CWjJBNkFC?=
 =?utf-8?B?SzBOTTIrZ2hkbS9nS3Z4M1VxMXF0SDE5d09YdnI0NjBRNHJNVElMUnJ3Vk9r?=
 =?utf-8?B?eXRZNktpTXZqd0FxdStBOFU5alg4M1YyZEk1eDhuaTNhdzRLKzRLZSt2d2ZJ?=
 =?utf-8?B?YzY2YUJHR0orcEIvR0xWcnlCa3JYS2ppZnhSSG53TnVPcm80MXBtWXQxc0s0?=
 =?utf-8?B?a3NTbGd5Tkc5dDRXd2s4MkZvN1ZCQWhFNTFmbzRkTkJUQ0VSdkZWdkhkWCtv?=
 =?utf-8?B?VnVCQ1hhZ09aM0JUY1plWHFONXh1Nmt1WVMwdTJuODJHbllmU3F4UEIraG02?=
 =?utf-8?B?bTBvaE9jMXpkN0k3MERWVkY0NUdDTkNISERFL1VhaGUrY0Y3bHBocno5Z3c0?=
 =?utf-8?B?WE1IdDZRSTduRUlpWjFKbUt2WEhhYUVycVpnTXZMYnJmQjhXZHVDc3lBNzFT?=
 =?utf-8?B?Tm8rbHNrS2tBc1RYdFYrMEliRTRhSkQwcXFwRlZ3dm8zZWhuQ21XaitoRzBp?=
 =?utf-8?B?ZEZtOHk4VDFoZFlDb3VRZDNMWUx3S2l6bFJYMURQdGdhSCtERzc5RmdvN0VD?=
 =?utf-8?B?SVo4b25XK2xxTEdrQm1WcEZ0Uk8xTENyTHNxK3dVcWFKZTFYT2tDUXhHaDhz?=
 =?utf-8?B?TnVHMy85SVI5RWxlM01HNGhaMytnMGY5ZHdEMEY3TWZPTmtNSXBjWWpyT1N4?=
 =?utf-8?B?dmREVEdCQThteHQ3eVZra3ovcGsyemFyMnNVcncyVW8wT3pwaHhEN1Z6Qkh1?=
 =?utf-8?B?ZElHZ3NmSWFUenNvSUphOWE1REVzR1JwMlluQmM0MmN0NldzS255VExBY2Ew?=
 =?utf-8?B?RElRRG81T1RCV25kT21jaWpYUDdCbkVmN3dtOXpjVndMR2lCNXorMndZRFJR?=
 =?utf-8?B?VVB6SGFnUEF0QlZlTEkzcnZWeE1WQW9VUVg4ZGViU0FBQnUvZ0gwM0NnVG00?=
 =?utf-8?B?MlBkOW11bHlJYU9jMUtnemFQL1E0clkzM0hyZ1FSaGdZWHI3VHdaN1A0UWRP?=
 =?utf-8?B?V2JlN1dnbHAwMDJSQmYxUGI4UUFvK2IrSHFMWVlEWkI5SXAzWGFaYjZQY0sr?=
 =?utf-8?B?MFJsM1E0TjcrZG5zVnVmb0k1THFBUDFPMnJJVlpJZFJJOEFPc1RNQkYrOStr?=
 =?utf-8?B?Ni9QVytyb3EzVGVyVGRtV3d4VEQzTXRvaFp3MmZCRTMvc3p4ZU5WV1E2Wm9l?=
 =?utf-8?B?TG15VXBJSnpJY283ZkxWbnE4YVRNY3ZQK1VYcldhcnY5SzRaTXM0eDNFVE9R?=
 =?utf-8?B?cWovQ2Q4eFJhc0hoaWYzRWVocFUzcU04MEVVc0Q3T3JNcWl1a3lxRE1Hdmhs?=
 =?utf-8?B?UHhjdTFnUVZXSTRzQVhEelFnWFZrY2ZtMnlXdmVUcDAvWDNrdzJtNU5MMHlI?=
 =?utf-8?B?OURrY3kxeDIwSFZvdHlOeisySmFlc05pcXl3amdvYmI1R1BtNmNjb0EwdWhX?=
 =?utf-8?B?TFlLa1NMQXIrVjZqTHBuN2RpcjNqY0hZY05mK3hrVHpJMDYwemdnTzhUYTY3?=
 =?utf-8?B?a1d5S3MzNnNmbm5xeXZKUUZxdGdHdDVsMmo0YVViTjYrU0lYczlER0tOcnpu?=
 =?utf-8?B?ZENkQXZyci9FM3dkZ1UzMU56YVhJc1lPY083NWkvRnJFejlkdFgzOUR6UGkw?=
 =?utf-8?B?WGQrNVlSQUM2aXlzSzVBSGlHMmFQT2VnV3pZL1pSUFpRWU93bEQraUxLeVBK?=
 =?utf-8?B?dzdyeGZrN00xNTg2VU9vVEFyV0hLUGwzSjQ2TnRuRkFMZTJVZzV0c0VmdDR4?=
 =?utf-8?B?bGtnbEdKc25PNjI4T0NRS2F4aFdtWVRBVjFsbHJUcTRiTkRYZm12cnNDcTRp?=
 =?utf-8?Q?/3FW73oS75kNwMWqFY4Fjes=3D?=
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: =?utf-8?B?bG5LbjFFb0I4ak1pUFBKaEYzYi9wekIxbEZ0ZDdkdEN4WXFTN3YvTlhZZWF0?=
 =?utf-8?B?OHhQSFFhTGppVHd3ZmtJOEtreU1lZDUwZ2dzNWszVDcxK0M4ZXlSQVdNOWY0?=
 =?utf-8?B?VFNoZGM0citUaTBDYUEvZS9NcHhHZzhLQ2hvVVdWRklQZkNwN0NLVDJTajRO?=
 =?utf-8?B?a3BBRGtxRnkrM2lJVDY2c0RTME4rYTZXRzVlRTV6VnhseVhSREh2bWpTR0xn?=
 =?utf-8?B?QXpBc3d3aGtmSlJXLzhCWkFObEoydnRpZ2FGVEtzR09QWnlOdWZwcWlBMWZR?=
 =?utf-8?B?SGp0Qld6SXVrQ0FPRU9iOTEzQ0ROVm1KQU1KVjJvbjk3THFmRk1mTm1CYmMv?=
 =?utf-8?B?b1Q0RG9peUZSQ0pYNmp3NVdLLy8wenlNbTkzc1hGbW5Cblc2K1ptd2JML3lV?=
 =?utf-8?B?R3pOTkV4djdrV3RuWjZSc01sREhFWno2V0g5VXJ4S1M0bjRMNlRvT0I3Zkwy?=
 =?utf-8?B?bERVMStDY0w1Vy9pMzQrdGRBL0kyZGR6SXNqNWtZKzRDckU0T0NYclZ5Qmk4?=
 =?utf-8?B?TFhVeGdnZG5NaXc2Wk5iRmtKVVJZVUxzWW5nd0VaMkNSR25ZNFpmS2NzS3da?=
 =?utf-8?B?VFkxVlNCZnpUZmRiSUhDcXhTZXpleVd0cGQvT1d0UDlqWjg1cVN3S2hEU2hU?=
 =?utf-8?B?RGNJaHMzRGpaWlREUVZwT3F2QzhnZC9LTUM1NW5zb0xPNG4vYVh0K2p2ZTZL?=
 =?utf-8?B?OHRiY1plelBPSitvdUUwN1NFYWs0Yzhid1lzZU92d0ZLV2tYbUVzK2dVYTZi?=
 =?utf-8?B?UmlJT0dSeWdiaU9ERExqT1ZMUGMxT3Y1WjFHMDl0U0IxOElISU83SExsQ1dS?=
 =?utf-8?B?eFQ3TUYyQXcwNHpCc1U0d05BWm4wcVZJVTVpcDBGZnpNdWg2ay9qd3R2ZTQ5?=
 =?utf-8?B?TGdINE5LZWFjRkNLaS9QeUhTdlViMmhmSjNyOFdxUXhyR0ZIdzI0NEV3SXMz?=
 =?utf-8?B?VVJCdzl3WGtTWFgxdHlTeThWM0hCTXFuaU9DWDlQbnQzTEJxYlpqMWR3TlZa?=
 =?utf-8?B?c3JNaVZucWdTb3dXQzBCdW9xM3ZLTjZhMm5hMExKOVEwdmhlSUN0ZGkreW91?=
 =?utf-8?B?SE9XWFYybkVPZnVnRFhmd2IrcUtKWGFPN01vZ3h1U3hzcDY5bFlpcTFrNlls?=
 =?utf-8?B?dWVXWE0xdkhmcUNNc0pCSFFydDE1eSt1NnFsOWRPalF2QkcrSS9NY0NObVJL?=
 =?utf-8?B?TTNlcUtjSHF2eDBoMjlmeFpZYXNLVFovZlpOcTZqODFIdVVhb1cwM1dGZ2R4?=
 =?utf-8?B?aWFnYXNaTG81TFlnZm5KT1JOYTV5TVF4azdmMUlhUnd1UEIxb2hYM0kyWDJL?=
 =?utf-8?B?bW0vU1hneG01dkpwdTJ1bTY5T2l3RWNaeEppUVlvQXhrQlZCWFFUbElzUDhI?=
 =?utf-8?B?TVJiSStIQXF2WXdWZ0sxUEpuZzZjVEw0ak9yNE9ldmo3eDNRUktvVnY4akZU?=
 =?utf-8?Q?x0ekKoNX?=
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-Network-Message-Id: fb904966-81ef-41fe-46d9-08dbcf0ab17b
X-MS-Exchange-CrossTenant-AuthSource: BLAPR10MB5267.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 17 Oct 2023 12:15:03.9346
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: jihJX6nfN7s2h0JKMbJx15FwmA28IDjmdOlBWHTh3Qg4uM/26gokEK6/Ri7LiS0xOGZ1/8U8QivfAslOBf0O/g==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH0PR10MB6984
X-Proofpoint-Virus-Version: vendor=baseguard
 engine=ICAP:2.0.272,Aquarius:18.0.980,Hydra:6.0.619,FMLib:17.11.176.26
 definitions=2023-10-16_13,2023-10-17_01,2023-05-22_02
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 bulkscore=0 mlxscore=0 phishscore=0
 malwarescore=0 suspectscore=0 mlxlogscore=999 spamscore=0 adultscore=0
 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2309180000
 definitions=main-2310170103
X-Proofpoint-ORIG-GUID: hlx4ZlICf0UbzLah4zQqIKkS49czGibZ
X-Proofpoint-GUID: hlx4ZlICf0UbzLah4zQqIKkS49czGibZ
X-Spam-Status: No, score=-5.4 required=5.0 tests=BAYES_00,DKIM_SIGNED,
        DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,NICE_REPLY_A,
        RCVD_IN_DNSWL_BLOCKED,RCVD_IN_MSPIKE_H5,RCVD_IN_MSPIKE_WL,
        SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org
X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (snail.vger.email [0.0.0.0]); Tue, 17 Oct 2023 05:17:03 -0700 (PDT)

On 17/10/2023 11:37, Masahiro Yamada wrote:
> scripts/pahole-flags.sh is executed so many times.
> 
> You can check how many times it is invoked during the build, as follows:
> 
>   $ cat <<EOF >> scripts/pahole-flags.sh
>   > echo "scripts/pahole-flags.sh was executed" >&2
>   > EOF
> 
>   $ make -s
>   scripts/pahole-flags.sh was executed
>   scripts/pahole-flags.sh was executed
>   scripts/pahole-flags.sh was executed
>   scripts/pahole-flags.sh was executed
>   scripts/pahole-flags.sh was executed
>     [ lots of repeated lines suppressed... ]
> 
> This scripts is exectuted more than 20 times during the kernel build
> because PAHOLE_FLAGS is a recursively expanded variable and exported
> to sub-processes.
> 
> With the GNU Make >= 4.4, it is executed more than 60 times because
> exported variables are also passed to other $(shell ) invocations.
> Without careful coding, it is known to cause an exponential fork
> explosion. [1]
> 
> The use of $(shell ) in an exported recursive variable is likely wrong
> because $(shell ) is always evaluated due to the 'export' keyword, and
> the evaluation can occur multiple times by the nature of recursive
> variables.
> 
> Convert the shell script to a Makefile, which is included only when
> CONFIG_DEBUG_INFO_BTF=y.
> 
> [1]: https://savannah.gnu.org/bugs/index.php?64746
> 
> Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>

Looks like a neat change to me. I tested with CONFIG_DEBUG_INFO_BTF=y
builds for full kernel build and 'make M=modpath' module-only builds,
and both work well with correct pahole flags. Feel free to add
either/both of

Reviewed-by: Alan Maguire <alan.maguire@oracle.com>
Tested-by: Alan Maguire <alan.maguire@oracle.com>

> ---
> 
>  Makefile                |  4 +---
>  scripts/Makefile.btf    | 19 +++++++++++++++++++
>  scripts/pahole-flags.sh | 30 ------------------------------
>  3 files changed, 20 insertions(+), 33 deletions(-)
>  create mode 100644 scripts/Makefile.btf
>  delete mode 100755 scripts/pahole-flags.sh
> 
> diff --git a/Makefile b/Makefile
> index fed9a6cc3665..eaddec67e5e1 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -513,8 +513,6 @@ LZ4		= lz4c
>  XZ		= xz
>  ZSTD		= zstd
>  
> -PAHOLE_FLAGS	= $(shell PAHOLE=$(PAHOLE) $(srctree)/scripts/pahole-flags.sh)
> -
>  CHECKFLAGS     := -D__linux__ -Dlinux -D__STDC__ -Dunix -D__unix__ \
>  		  -Wbitwise -Wno-return-void -Wno-unknown-attribute $(CF)
>  NOSTDINC_FLAGS :=
> @@ -605,7 +603,6 @@ export KBUILD_RUSTFLAGS RUSTFLAGS_KERNEL RUSTFLAGS_MODULE
>  export KBUILD_AFLAGS AFLAGS_KERNEL AFLAGS_MODULE
>  export KBUILD_AFLAGS_MODULE KBUILD_CFLAGS_MODULE KBUILD_RUSTFLAGS_MODULE KBUILD_LDFLAGS_MODULE
>  export KBUILD_AFLAGS_KERNEL KBUILD_CFLAGS_KERNEL KBUILD_RUSTFLAGS_KERNEL
> -export PAHOLE_FLAGS
>  
>  # Files to ignore in find ... statements
>  
> @@ -1002,6 +999,7 @@ KBUILD_CPPFLAGS += $(call cc-option,-fmacro-prefix-map=$(srctree)/=)
>  # include additional Makefiles when needed
>  include-y			:= scripts/Makefile.extrawarn
>  include-$(CONFIG_DEBUG_INFO)	+= scripts/Makefile.debug
> +include-$(CONFIG_DEBUG_INFO_BTF)+= scripts/Makefile.btf
>  include-$(CONFIG_KASAN)		+= scripts/Makefile.kasan
>  include-$(CONFIG_KCSAN)		+= scripts/Makefile.kcsan
>  include-$(CONFIG_KMSAN)		+= scripts/Makefile.kmsan
> diff --git a/scripts/Makefile.btf b/scripts/Makefile.btf
> new file mode 100644
> index 000000000000..82377e470aed
> --- /dev/null
> +++ b/scripts/Makefile.btf
> @@ -0,0 +1,19 @@
> +# SPDX-License-Identifier: GPL-2.0
> +
> +pahole-ver := $(CONFIG_PAHOLE_VERSION)
> +pahole-flags-y :=
> +
> +# pahole 1.18 through 1.21 can't handle zero-sized per-CPU vars
> +ifeq ($(call test-le, $(pahole-ver), 121),y)
> +pahole-flags-$(call test-ge, $(pahole-ver), 118)	+= --skip_encoding_btf_vars
> +endif
> +
> +pahole-flags-$(call test-ge, $(pahole-ver), 121)	+= --btf_gen_floats
> +
> +pahole-flags-$(call test-ge, $(pahole-ver), 122)	+= -j
> +
> +pahole-flags-$(CONFIG_PAHOLE_HAS_LANG_EXCLUDE)		+= --lang_exclude=rust
> +
> +pahole-flags-$(call test-ge, $(pahole-ver), 125)	+= --skip_encoding_btf_inconsistent_proto --btf_gen_optimized
> +
> +export PAHOLE_FLAGS := $(pahole-flags-y)
> diff --git a/scripts/pahole-flags.sh b/scripts/pahole-flags.sh
> deleted file mode 100755
> index 728d55190d97..000000000000
> --- a/scripts/pahole-flags.sh
> +++ /dev/null
> @@ -1,30 +0,0 @@
> -#!/bin/sh
> -# SPDX-License-Identifier: GPL-2.0
> -
> -extra_paholeopt=
> -
> -if ! [ -x "$(command -v ${PAHOLE})" ]; then
> -	exit 0
> -fi
> -
> -pahole_ver=$($(dirname $0)/pahole-version.sh ${PAHOLE})
> -
> -if [ "${pahole_ver}" -ge "118" ] && [ "${pahole_ver}" -le "121" ]; then
> -	# pahole 1.18 through 1.21 can't handle zero-sized per-CPU vars
> -	extra_paholeopt="${extra_paholeopt} --skip_encoding_btf_vars"
> -fi
> -if [ "${pahole_ver}" -ge "121" ]; then
> -	extra_paholeopt="${extra_paholeopt} --btf_gen_floats"
> -fi
> -if [ "${pahole_ver}" -ge "122" ]; then
> -	extra_paholeopt="${extra_paholeopt} -j"
> -fi
> -if [ "${pahole_ver}" -ge "124" ]; then
> -	# see PAHOLE_HAS_LANG_EXCLUDE
> -	extra_paholeopt="${extra_paholeopt} --lang_exclude=rust"
> -fi
> -if [ "${pahole_ver}" -ge "125" ]; then
> -	extra_paholeopt="${extra_paholeopt} --skip_encoding_btf_inconsistent_proto --btf_gen_optimized"
> -fi
> -
> -echo ${extra_paholeopt}