Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67;
Subject: Re: perf test LLVM & clang 6 failing
To:     Arnaldo Carvalho de Melo <acme@kernel.org>,
        Daniel Borkmann <daniel@iogearbox.net>
CC:     Wang Nan <wangnan0@huawei.com>,
        Alexei Starovoitov <alexei.starovoitov@gmail.com>,
        Adrian Hunter <adrian.hunter@intel.com>,
        David Ahern <dsahern@gmail.com>, Jiri Olsa <jolsa@kernel.org>,
        Ingo Molnar <mingo@kernel.org>,
        Namhyung Kim <namhyung@kernel.org>,
        Linux Kernel Mailing List <linux-kernel@vger.kernel.org>
References: <20171124144717.GO8789@kernel.org>
 <7cd16670-37c9-21f5-21c8-d50385ebc951@iogearbox.net>
 <20171124190923.GR8789@kernel.org>
From:   Yonghong Song <yhs@fb.com>
Message-ID: <7f9640eb-7da7-b3ca-02bd-dece9cbf91af@fb.com>
Date:   Sat, 25 Nov 2017 22:29:51 -0800
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.12; rv:52.0)
 Gecko/20100101 Thunderbird/52.4.0
MIME-Version: 1.0
In-Reply-To: <20171124190923.GR8789@kernel.org>
Content-Type: text/plain; charset="utf-8"; format=flowed
Content-Language: en-US
Content-Transfer-Encoding: 7bit
Received-SPF: None (protection.outlook.com: fb.com does not designate
 permitted sender hosts)
SpamDiagnosticOutput: 1:99
SpamDiagnosticMetadata: NSPM
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 26 Nov 2017 06:29:56.1618 (UTC)
X-MS-Exchange-CrossTenant-Network-Message-Id: f45d67da-7f34-4b63-8504-08d534971d2e
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 8ae927fe-1255-47a7-a2af-5f3a069daaa2
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM5PR15MB1836
X-OriginatorOrg: fb.com
X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:,, definitions=2017-11-26_02:,,
 signatures=0
X-Proofpoint-Spam-Reason: safe
X-FB-Internal: Safe
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org


On 11/24/17 11:09 AM, Arnaldo Carvalho de Melo wrote:
> Em Fri, Nov 24, 2017 at 04:16:52PM +0100, Daniel Borkmann escreveu:
>> [ +Yonghong ]
>>
>> On 11/24/2017 03:47 PM, Arnaldo Carvalho de Melo wrote:
>>> FYI, just noticed, recently updated clang to version 6, from its
>>> upstream git repo.
>>
>> Do you recall what was your LLVM version prior to this where it was
>> working fine? (Wild guess from below would be the BPF inline asm
> 
> IIRC it was 4.0

I tried the following example (the almost the same one as the failed
compilation one before except a few SEC(...) marking which should not
impact the result):

$ cat bpf_prog.c
#include <uapi/linux/fs.h>
#include <uapi/asm/ptrace.h>

int bpf_func__vfs_llseek(void *ctx)
{
         return 0;
}
$ cat run.sh
KERNEL_INC_OPTIONS="-nostdinc -isystem 
/usr/lib/gcc/x86_64-redhat-linux/4.8.5/include 
-I/home/yhs/work/net-next/arch/x86/include 
-I/home/yhs/work/linux-bld/arch/x86/include 
-I/home/yhs/work/net-next/include -I/home/yhs/work/linux-bld/include 
-I/home/yhs/work/net-next/arch/x86/include/uapi 
-I/home/yhs/work/linux-bld/arch/x86/include/generated/uapi 
-I/home/yhs/work/net-next/include/uapi 
-I/home/yhs/work/linux-bld/include/generated/uapi -include 
/home/yhs/work/net-next/include/linux/kconfig.h"

clang -D__KERNEL__ -D__NR_CPUS__=4 -xc \
       $KERNEL_INC_OPTIONS \
       -Wno-unused-value -Wno-pointer-sign \
       -c bpf_prog.c -O2 -target bpf
$

You can actually get KERNEL_INC_OPTIONS by compiling kernel samples/bpf.

This program failed at both llvm 4.0 and latest trunk (6.0).

The same symptom:
In file included from bpf_prog.c:2:
In file included from 
/home/yhs/work/net-next/arch/x86/include/uapi/asm/ptrace.h:5:
In file included from /home/yhs/work/net-next/include/linux/compiler.h:237:
In file included from 
/home/yhs/work/net-next/arch/x86/include/asm/barrier.h:5:
In file included from 
/home/yhs/work/net-next/arch/x86/include/asm/alternative.h:10:
/home/yhs/work/net-next/arch/x86/include/asm/asm.h:145:50: error: 
unknown register name 'esp' in asm
register unsigned long current_stack_pointer asm(_ASM_SP);
                                                  ^
/home/yhs/work/net-next/arch/x86/include/asm/asm.h:44:18: note: expanded 
from macro '_ASM_SP'
#define _ASM_SP         __ASM_REG(sp)

I suggest that you use the same compilation process as
samples/bpf and tools/testing/selftests/bpf:

$ cat run1.sh
KERNEL_INC_OPTIONS="-nostdinc -isystem 
/usr/lib/gcc/x86_64-redhat-linux/4.8.5/include 
-I/home/yhs/work/net-next/arch/x86/include 
-I/home/yhs/work/linux-bld/arch/x86/include 
-I/home/yhs/work/net-next/include -I/home/yhs/work/linux-bld/include 
-I/home/yhs/work/net-next/arch/x86/include/uapi 
-I/home/yhs/work/linux-bld/arch/x86/include/generated/uapi 
-I/home/yhs/work/net-next/include/uapi 
-I/home/yhs/work/linux-bld/include/generated/uapi -include 
/home/yhs/work/net-next/include/linux/kconfig.h"

clang -D__KERNEL__ -D__NR_CPUS__=4 -xc \
       $KERNEL_INC_OPTIONS \
       -Wno-unused-value -Wno-pointer-sign \
       -c bpf_prog.c -O2 -emit-llvm -S
llc -filetype=obj -march=bpf bpf_prog.ll
$

This is especially true if you are doing tracing there ptrace.h may be 
included in your header files.

The "clang" compiler will be able to handle host inline asm properly and
filter out functions which uses host inline asm's. If any file-level
inline asms are still left (e.g., arm64), the llvm 6.0 should be able
to handle this but not llvm 4.0.

> 
>> support that was added recently to LLVM (2865ab6996) vs asm() used
>> in headers included in the stdin header causing trouble due to arch
>> mixup?)
>>
>>> root@jouet ~]# perf test -v LLVM
>>> 37: LLVM search and compile                               :
>>> 37.1: Basic BPF llvm compile                              :
>>> --- start ---
>>> test child forked, pid 5255
>>> Kernel build dir is set to /lib/modules/4.14.0+/build
>>> set env: KBUILD_DIR=/lib/modules/4.14.0+/build
>>> unset env: KBUILD_OPTS
>>> include option is set to  -nostdinc -isystem /usr/lib/gcc/x86_64-redhat-linux/7/include -I/home/acme/git/linux/arch/x86/include -I./arch/x86/include/generated  -I/home/acme/git/linux/include -I./include -I/home/acme/git/linux/arch/x86/include/uapi -I./arch/x86/include/generated/uapi -I/home/acme/git/linux/include/uapi -I./include/generated/uapi -include /home/acme/git/linux/include/linux/kconfig.h
>>> set env: NR_CPUS=4
>>> set env: LINUX_VERSION_CODE=0x40e00
>>> set env: CLANG_EXEC=/usr/local/bin/clang
>>> set env: CLANG_OPTIONS=-xc
>>> set env: KERNEL_INC_OPTIONS= -nostdinc -isystem /usr/lib/gcc/x86_64-redhat-linux/7/include -I/home/acme/git/linux/arch/x86/include -I./arch/x86/include/generated  -I/home/acme/git/linux/include -I./include -I/home/acme/git/linux/arch/x86/include/uapi -I./arch/x86/include/generated/uapi -I/home/acme/git/linux/include/uapi -I./include/generated/uapi -include /home/acme/git/linux/include/linux/kconfig.h
>>> set env: WORKING_DIR=/lib/modules/4.14.0+/build
>>> set env: CLANG_SOURCE=-
>>> llvm compiling command template: echo '/*
>>>   * bpf-script-example.c
>>>   * Test basic LLVM building
>>>   */
>>> #ifndef LINUX_VERSION_CODE
>>> # error Need LINUX_VERSION_CODE
>>> # error Example: for 4.2 kernel, put 'clang-opt="-DLINUX_VERSION_CODE=0x40200" into llvm section of ~/.perfconfig'
>>> #endif
>>> #define BPF_ANY 0
>>> #define BPF_MAP_TYPE_ARRAY 2
>>> #define BPF_FUNC_map_lookup_elem 1
>>> #define BPF_FUNC_map_update_elem 2
>>>
>>> static void *(*bpf_map_lookup_elem)(void *map, void *key) =
>>> 	(void *) BPF_FUNC_map_lookup_elem;
>>> static void *(*bpf_map_update_elem)(void *map, void *key, void *value, int flags) =
>>> 	(void *) BPF_FUNC_map_update_elem;
>>>
>>> struct bpf_map_def {
>>> 	unsigned int type;
>>> 	unsigned int key_size;
>>> 	unsigned int value_size;
>>> 	unsigned int max_entries;
>>> };
>>>
>>> #define SEC(NAME) __attribute__((section(NAME), used))
>>> struct bpf_map_def SEC("maps") flip_table = {
>>> 	.type = BPF_MAP_TYPE_ARRAY,
>>> 	.key_size = sizeof(int),
>>> 	.value_size = sizeof(int),
>>> 	.max_entries = 1,
>>> };
>>>
>>> SEC("func=SyS_epoll_wait")
>>> int bpf_func__SyS_epoll_wait(void *ctx)
>>> {
>>> 	int ind =0;
>>> 	int *flag = bpf_map_lookup_elem(&flip_table, &ind);
>>> 	int new_flag;
>>> 	if (!flag)
>>> 		return 0;
>>> 	/* flip flag and store back */
>>> 	new_flag = !*flag;
>>> 	bpf_map_update_elem(&flip_table, &ind, &new_flag, BPF_ANY);
>>> 	return new_flag;
>>> }
>>> char _license[] SEC("license") = "GPL";
>>> int _version SEC("version") = LINUX_VERSION_CODE;
>>> ' | $CLANG_EXEC -D__KERNEL__ -D__NR_CPUS__=$NR_CPUS -DLINUX_VERSION_CODE=$LINUX_VERSION_CODE $CLANG_OPTIONS $KERNEL_INC_OPTIONS -Wno-unused-value -Wno-pointer-sign -working-directory $WORKING_DIR -c "$CLANG_SOURCE" -target bpf -O2 -o -
>>> test child finished with 0
>>> ---- end ----
>>> LLVM search and compile subtest 0: Ok
>>> 37.2: kbuild searching                                    :
>>> --- start ---
>>> test child forked, pid 5728
>>> Kernel build dir is set to /lib/modules/4.14.0+/build
>>> set env: KBUILD_DIR=/lib/modules/4.14.0+/build
>>> unset env: KBUILD_OPTS
>>> include option is set to  -nostdinc -isystem /usr/lib/gcc/x86_64-redhat-linux/7/include -I/home/acme/git/linux/arch/x86/include -I./arch/x86/include/generated  -I/home/acme/git/linux/include -I./include -I/home/acme/git/linux/arch/x86/include/uapi -I./arch/x86/include/generated/uapi -I/home/acme/git/linux/include/uapi -I./include/generated/uapi -include /home/acme/git/linux/include/linux/kconfig.h
>>> set env: NR_CPUS=4
>>> set env: LINUX_VERSION_CODE=0x40e00
>>> set env: CLANG_EXEC=/usr/local/bin/clang
>>> set env: CLANG_OPTIONS=-xc
>>> set env: KERNEL_INC_OPTIONS= -nostdinc -isystem /usr/lib/gcc/x86_64-redhat-linux/7/include -I/home/acme/git/linux/arch/x86/include -I./arch/x86/include/generated  -I/home/acme/git/linux/include -I./include -I/home/acme/git/linux/arch/x86/include/uapi -I./arch/x86/include/generated/uapi -I/home/acme/git/linux/include/uapi -I./include/generated/uapi -include /home/acme/git/linux/include/linux/kconfig.h
>>> set env: WORKING_DIR=/lib/modules/4.14.0+/build
>>> set env: CLANG_SOURCE=-
>>> llvm compiling command template: echo '/*
>>>   * bpf-script-test-kbuild.c
>>>   * Test include from kernel header
>>>   */
>>> #ifndef LINUX_VERSION_CODE
>>> # error Need LINUX_VERSION_CODE
>>> # error Example: for 4.2 kernel, put 'clang-opt="-DLINUX_VERSION_CODE=0x40200" into llvm section of ~/.perfconfig'
>>> #endif
>>> #define SEC(NAME) __attribute__((section(NAME), used))
>>>
>>> #include <uapi/linux/fs.h>
>>> #include <uapi/asm/ptrace.h>
>>>
>>> SEC("func=vfs_llseek")
>>> int bpf_func__vfs_llseek(void *ctx)
>>> {
>>> 	return 0;
>>> }
>>>
>>> char _license[] SEC("license") = "GPL";
>>> int _version SEC("version") = LINUX_VERSION_CODE;
>>> ' | $CLANG_EXEC -D__KERNEL__ -D__NR_CPUS__=$NR_CPUS -DLINUX_VERSION_CODE=$LINUX_VERSION_CODE $CLANG_OPTIONS $KERNEL_INC_OPTIONS -Wno-unused-value -Wno-pointer-sign -working-directory $WORKING_DIR -c "$CLANG_SOURCE" -target bpf -O2 -o -
>>> In file included from <stdin>:12:
>>> In file included from /home/acme/git/linux/arch/x86/include/uapi/asm/ptrace.h:5:
>>> In file included from /home/acme/git/linux/include/linux/compiler.h:237:
>>> In file included from /home/acme/git/linux/arch/x86/include/asm/barrier.h:5:
>>> In file included from /home/acme/git/linux/arch/x86/include/asm/alternative.h:10:
>>> /home/acme/git/linux/arch/x86/include/asm/asm.h:145:50: error: unknown register name 'esp' in asm
>>> register unsigned long current_stack_pointer asm(_ASM_SP);
>>>                                                   ^
>>> /home/acme/git/linux/arch/x86/include/asm/asm.h:44:18: note: expanded from macro '_ASM_SP'
>>> #define _ASM_SP         __ASM_REG(sp)
>>>                          ^
>>> /home/acme/git/linux/arch/x86/include/asm/asm.h:27:32: note: expanded from macro '__ASM_REG'
>>> #define __ASM_REG(reg)         __ASM_SEL_RAW(e##reg, r##reg)
>>>                                 ^
>>> /home/acme/git/linux/arch/x86/include/asm/asm.h:18:29: note: expanded from macro '__ASM_SEL_RAW'
>>> # define __ASM_SEL_RAW(a,b) __ASM_FORM_RAW(a)
>>>                              ^
>>> /home/acme/git/linux/arch/x86/include/asm/asm.h:11:32: note: expanded from macro '__ASM_FORM_RAW'
>>> # define __ASM_FORM_RAW(x)     #x
>>>                                 ^
>>> <scratch space>:4:1: note: expanded from here
>>> "esp"
>>> ^
>>> 1 error generated.
>>> ERROR:	unable to compile -
>>> Hint:	Check error message shown above.
>>> Hint:	You can also pre-compile it into .o using:
>>>       		clang -target bpf -O2 -c -
>>>       	with proper -I and -D options.
>>> Failed to compile test case: 'kbuild searching'
>>> test child finished with -1
>>> ---- end ----
>>> LLVM search and compile subtest 1: FAILED!
>>> 37.3: Compile source for BPF prologue generation          :
>>> --- force skipped ---
>>> LLVM search and compile subtest 2: Skip
>>> 37.4: Compile source for BPF relocation                   :
>>> --- force skipped ---
>>> LLVM search and compile subtest 3: Skip
>>> [root@jouet ~]# clang -v
>>> clang version 6.0.0 (https://urldefense.proofpoint.com/v2/url?u=http-3A__llvm.org_git_clang.git&d=DwIBAg&c=5VD0RTtNlTh3ycd41b3MUw&r=DA8e1B5r073vIqRrFz7MRA&m=z5qVLSWJOX6g2fVdnv6PkJEj6w9ZeuUbyGZiXHbXR8o&s=e4qbhCUq0iI5gnbkZLhFvoo0HDFpRyIjElZsWLCJlZc&e= 56cc8f8880db2ebc433eeb6b6a707c101467a186) (https://urldefense.proofpoint.com/v2/url?u=http-3A__llvm.org_git_llvm.git&d=DwIBAg&c=5VD0RTtNlTh3ycd41b3MUw&r=DA8e1B5r073vIqRrFz7MRA&m=z5qVLSWJOX6g2fVdnv6PkJEj6w9ZeuUbyGZiXHbXR8o&s=1kpNkh0Slwca73HtsdX4NxJzsEM5vRjayUu-GQldnU8&e= 3656d83960a4f3fedf6d8f19043abf52379f78c3)
>>> Target: x86_64-unknown-linux-gnu
>>> Thread model: posix
>>> InstalledDir: /usr/local/bin
>>> Found candidate GCC installation: /usr/lib/gcc/x86_64-redhat-linux/7
>>> Selected GCC installation: /usr/lib/gcc/x86_64-redhat-linux/7
>>> Candidate multilib: .;@m64
>>> Candidate multilib: 32;@m32
>>> Selected multilib: .;@m64
>>> [root@jouet ~]#
>>>

From 1584975725621232985@xxx Fri Nov 24 19:10:41 +0000 2017
X-GM-THRID: 1584959210990306928
X-Gmail-Labels: Inbox,Category Forums,HistoricalUnread