Received: by 10.192.165.148 with SMTP id m20csp3456620imm; Mon, 7 May 2018 12:37:32 -0700 (PDT) X-Google-Smtp-Source: AB8JxZqBa4vU7mGIuDq63HDegf9tULCr3VKcaKSCOuzOvm3suF3UgWhTEs4kJW5+sutcnzBx9wDC X-Received: by 2002:a6b:1e8f:: with SMTP id e137-v6mr40876106ioe.192.1525721852376; Mon, 07 May 2018 12:37:32 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1525721852; cv=none; d=google.com; s=arc-20160816; b=uIIghht7fVVMUaES5JwNBhffqjeBeFOT5+TvY1k/JdwKvcNm+i0CsBd+gLpJ7w4xK2 0d3c1gEM2ul+S2vNRcvoTp4Qa8oWLHl3S9B3ASBB/Tm3ZlkyIGNnJJmw8DAeOImbwKNA QbmzgOjdXr5qTtSdnH2rDINJMkRehOhKkIb4JLoF7xBIuZU/1f5Eg56MRxgZu6hZiIIX xNrIdwEVxlT0hW34YFeaZHBgW9p6HIM3J4Qk0qEtnjGvnEkXD5tPuVqJ0RkkeRlEr81b bq0Zs8uUtKXiLDaG4DT95hXzC0T1lHFPCvV0bgNVQmFisarHq+OhMyLdoGsTgatzpmYs NY8A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:cc:to:subject:message-id:date:from :references:in-reply-to:reply-to:mime-version:dkim-signature :arc-authentication-results; bh=lXrZ2D0xKl9lAAeEn1nqwd6EAIQHBy0LK/8Uk6cDnho=; b=xJbHCEWYbCxw63KqmXYzMaU1HdQvkKa+7d5xqIjGZQUOZ+B7PIi/qKBTS7JOkI2OGP znmIFPJnhLCkm21YMpdkccvGrXl6P7kkNg90IC9xNYWatXbGcPUfKJHkwo5D+ReVvDp8 a+B36nVftt2j0h5cOR0a3x+HLatyl1DxnHEx2FFnFuHw6j3DXfoKWVXIPahqRygk6YSL BnN7CV88hNnk2KBd0owO9Kq8kCrOCMlVRVAZTzovBZClwZtGx6xpRUtKh9hxWVdxdgJB dpq9djJ5NiFgyTtQPOlWpOGcNCwAqzEJqO5TkDaNw5WTQt6JORuVmpEqGETdFhv8Cs8d fZrA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=DUKfSW+K; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id c184-v6si7888805itg.55.2018.05.07.12.37.17; Mon, 07 May 2018 12:37:32 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=DUKfSW+K; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752728AbeEGThF (ORCPT + 99 others); Mon, 7 May 2018 15:37:05 -0400 Received: from mail-lf0-f42.google.com ([209.85.215.42]:37081 "EHLO mail-lf0-f42.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752494AbeEGThE (ORCPT ); Mon, 7 May 2018 15:37:04 -0400 Received: by mail-lf0-f42.google.com with SMTP id r2-v6so6203526lff.4 for ; Mon, 07 May 2018 12:37:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:reply-to:in-reply-to:references:from:date:message-id :subject:to:cc; bh=lXrZ2D0xKl9lAAeEn1nqwd6EAIQHBy0LK/8Uk6cDnho=; b=DUKfSW+KZ/4aQmKqdV8cu89LuNHzAMzGggWCbV4mdwxssy7KhIgXgaGoBOxu6BaxNC XBHKKpvahqORN2sQyWcu2d+EP13TkEYFwtrAO20n/tKGtYJGOXlBnFPRFvdtbslrl46f 4wM7mM49pcmkjEWWDoDFdklqwx+9XrFHRcaNVZWIxqqxXxETBS46v7YrsuAkp+LlE5ya AxBE9kmwGxdeDiQdfdEvwREDNxDoSCTbddeR8a0ijfOG1k0u5Y1zPJccvPSsWVhG2btm JjLNktPU9MNAw8aqZXGrBkAgaGETAbTjmwt2MPHlBWFwCn1BZCpnvWOrAOXk5ExBXocB cO8A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:reply-to:in-reply-to:references :from:date:message-id:subject:to:cc; bh=lXrZ2D0xKl9lAAeEn1nqwd6EAIQHBy0LK/8Uk6cDnho=; b=P+EMJcrvYu8jyBaM+pQAmZV3JWvcfN73yssfIO+IvNVKisYdRU3APhpDdgCqvR/vNO mLsKRaxNGB29kypXET+LMbicVgZvflLevq6oD36bWDOH9v/k27vLwCqPq32U8lVrHdlP gRuDzek9k9RPHLVrSVvjrlU/6KJ8C5tSsdEIb5tukKDTKE1pyx1QcR074Qu3wil/lgHM cDZ3x2o8wXtfkhw7iCDfffrx+eYmDTa0CPZjOET3n0UTtjQPow1PGY+JuuF+z9/wNr8E vT6l+8pSa/uUWrEcA2SdUQWsg72SkjohfsqEwgxUiiR3w/+cAuBGIPBtRulbh9LRNWEa dn6Q== X-Gm-Message-State: ALQs6tDQVdgd4ehcn0Gie1cuwyiB6OMckZGEWEkLSR+Xi0wuZwt5GAEG Y4DAvUiF3Mp8t/effU7BEVil4UOaohI9oqHupIw= X-Received: by 2002:a2e:9949:: with SMTP id r9-v6mr13166138ljj.53.1525721822215; Mon, 07 May 2018 12:37:02 -0700 (PDT) MIME-Version: 1.0 Received: by 2002:a19:d58e:0:0:0:0:0 with HTTP; Mon, 7 May 2018 12:37:01 -0700 (PDT) Reply-To: sedat.dilek@gmail.com In-Reply-To: <20180507174909.GF19594@google.com> References: <20180423174224.GA243180@google.com> <20180424230645.GB243180@google.com> <20180507174909.GF19594@google.com> From: Sedat Dilek Date: Mon, 7 May 2018 21:37:01 +0200 Message-ID: Subject: Re: Clang patch stacks for LTS kernels (v4.4 and v4.9) and status update To: Matthias Kaehlcke Cc: lkml , Arnd Bergmann , Nick Desaulniers , Manoj Gupta , AlistairStrachan Content-Type: text/plain; charset="UTF-8" Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Mon, May 7, 2018 at 7:49 PM, Matthias Kaehlcke wrote: > On Sun, May 06, 2018 at 09:42:09AM +0200, Sedat Dilek wrote: >> On Wed, Apr 25, 2018 at 1:06 AM, Matthias Kaehlcke wrote: >> > On Tue, Apr 24, 2018 at 01:54:29PM +0200, Sedat Dilek wrote: >> >> Hi Matthias, >> >> >> >> a big thank you for giving all the informations! >> >> >> >> I used your mka/llvm/v4.14_ext Git tree... >> >> >> >> https://chromium.googlesource.com/chromiumos/third_party/kernel/+log/sandbox/mka/llvm/v4.14_ext >> >> >> >> ...and was able to compile with clang-6.0 from Debian/buster64 with... >> >> >> >> ...reverting the clang-3/clang-4 patches... >> >> >> >> user$ for p in 69e44656ae43 222b88977a00 cdfcf1e45537 26f14c9225a6 >> >> 0385a18e9995 68dab143c9b4 ; do echo [ $p ] ; LC_ALL=C git revert >> >> --no-edit $p ; done >> >> >> >> 0001-Revert-CLANG3-core-clang-work-around-x86-regparm-int.patch >> >> 0002-Revert-CLANG4-futex-don-t-optimize-futex_detect_cmpx.patch >> >> 0003-Revert-CLANG4-Disable-lkdtm-when-ftrace-is-enabled.patch >> >> 0004-Revert-CLANG4-arm64-prefetch-Use-__builtin_arm_prefe.patch >> >> 0005-Revert-CLANG4-kbuild-Add-meabi-gnu-to-the-clang-para.patch >> >> 0006-Revert-CLANG4-crypto-arm64-aes-ce-Explicitly-pass-th.patch >> >> >> >> ...and needed two additional patches from upstream: >> >> >> >> 0007-kbuild-clang-remove-crufty-HOSTCFLAGS.patch <--- Label with >> >> BACKPORT (XXX: Backported; Required when using HOSTCC in make-line, >> >> see below) >> >> 0008-x86-xen-remove-the-use-of-VLAIS.patch <--- Label with UPSTREAM >> >> (XXX: cherry-picked) >> >> >> >> These 3 patches in your Git branch are in Linux v4.14.36... >> >> >> >> 9af5ddf981ed BACKPORT: kbuild: disable clang's default use of >> >> -fmerge-all-constants >> >> f24088a3842c BACKPORT: kbuild: Set KBUILD_CFLAGS before incl. arch Makefile >> >> d4dfe384346d UPSTREAM: kbuild: fix linker feature test macros when >> >> cross compiling with Clang >> >> >> >> ...and can be dropped. >> >> I do not know your work-flow/policy: Maybe you want to keep your >> >> patch-stack against vanilla Linux v4.14 - without stables from >> >> linux-stable-4.14.y. >> > >> > Yes, these patch stacks are based on vanilla Linux v4.14, one reason >> > is that it isn't a moving target. Also I think it makes it easier for >> > folks not merging LTS (though they probably should!) to locate all >> > patches, and it's fairly trivial to skip the (currently) few patches >> > not needed with the -stable tree. >> > >> >> For easy switching "mycompiler" I use a wrapper-script: >> >> >> >> root# cat /usr/bin/mycompiler >> >> #!/bin/bash >> >> >> >> clang-6.0 "$@" >> >> - EOF - >> >> >> >> user$ cp -v /boot/config-4.14.35-1-iniza-amd64 .config >> >> user$ MAKE="make V=1" ; COMPILER="mycompiler" ; >> >> MAKE_OPTS="CC=$COMPILER HOSTCC=$COMPILER" >> >> user$ yes "" | $MAKE $MAKE_OPTS oldconfig && $MAKE $MAKE_OPTS >> >> silentoldconfig < /dev/null >> >> >> >> The diffconfig to my current kernel-config-4.14 looks like this... >> >> >> >> user$ ./scripts/diffconfig /boot/config-4.14.35-1-iniza-amd64 .config >> >> -ARCH_HAS_REFCOUNT y >> >> -BPF_JIT_ALWAYS_ON n >> >> -EXOFS_DEBUG n >> >> -EXOFS_FS m >> >> -GENERIC_CPU_VULNERABILITIES y >> >> -KASAN n >> >> -ORE m >> >> -PAGE_TABLE_ISOLATION y >> >> -RETPOLINE y >> >> -UNWINDER_FRAME_POINTER n >> >> -UNWINDER_GUESS n >> >> -UNWINDER_ORC y >> >> +FRAME_POINTER y >> >> +FRAME_POINTER_UNWINDER y >> >> +GUESS_UNWINDER n >> >> +HAVE_ARCH_KMEMCHECK y >> >> +HAVE_RELIABLE_STACKTRACE y >> >> +ORC_UNWINDER n >> >> >> >> Unfortunately, I cannot boot into the generated kernel on bare metal. >> >> >> >> Checking with QEMU (version: 2.12~rc3) and catching earlyprintk, I see this... >> >> >> >> user$ echo $KPATH >> >> $HOME/src/linux-kernel/important-files >> >> >> >> user$ ls -al >> >> insgesamt 344916 >> >> drwxr-xr-x 2 sdi sdi 4096 Apr 24 13:15 . >> >> drwxr-xr-x 20 sdi sdi 4096 Apr 24 13:13 .. >> >> -rw-r--r-- 1 sdi sdi 4528416 Apr 24 12:42 bzImage >> >> lrwxrwxrwx 1 sdi sdi 35 Apr 24 13:15 initrd.img -> >> >> initrd.img-4.14.0-1-iniza-llvmlinux >> >> -rw-r--r-- 1 sdi sdi 25572955 Apr 24 13:08 >> >> initrd.img-4.14.0-1-iniza-llvmlinux >> >> -rw-r--r-- 1 sdi sdi 2887195 Apr 24 12:42 System.map >> >> -rwxr-xr-x 1 sdi sdi 326116744 Apr 24 12:42 vmlinux >> >> >> >> user$ sudo qemu-system-x86_64 -enable-kvm -M pc -kernel $KPATH/bzImage >> >> -initrd $KPATH/initrd.img -m 512 -net none -serial stdio -append >> >> "root=/dev/ram0 console=ttyS0 hung_task_panic=1 >> >> earlyprintk=ttyS0,115200" >> >> >> >> Probing EDD (edd=off to disable)... ok >> >> [ 0.000000] Linux version 4.14.0-1-iniza-llvmlinux >> >> (sedat.dilek@gmail.com@iniza) (clang version 6.0.0-1 >> >> (tags/RELEASE_600/final)) #1 SMP Tue Apr 24 12:42:21 CEST 2018 >> >> [ 0.000000] Command line: root=/dev/ram0 console=ttyS0 >> >> hung_task_panic=1 earlyprintk=ttyS0,115200 >> >> [ 0.000000] x86/fpu: x87 FPU will use FXSAVE >> >> [ 0.000000] e820: BIOS-provided physical RAM map: >> >> [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable >> >> [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved >> >> [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved >> >> [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000001ffdffff] usable >> >> [ 0.000000] BIOS-e820: [mem 0x000000001ffe0000-0x000000001fffffff] reserved >> >> [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved >> >> [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved >> >> [ 0.000000] bootconsole [earlyser0] enabled >> >> [ 0.000000] NX (Execute Disable) protection: active >> >> [ 0.000000] random: fast init done >> >> [ 0.000000] SMBIOS 2.8 present. >> >> [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS >> >> 1.11.1-1 04/01/2014 >> >> [ 0.000000] Hypervisor detected: KVM >> >> [ 0.000000] tsc: Fast TSC calibration using PIT >> >> [ 0.000000] e820: last_pfn = 0x1ffe0 max_arch_pfn = 0x400000000 >> >> [ 0.000000] x86/PAT: PAT not supported by CPU. >> >> [ 0.000000] x86/PAT: Configuration [0-7]: WB WT UC- UC WB WT UC- UC >> >> Memory KASLR using RDTSC... >> >> [ 0.000000] found SMP MP-table at [mem 0x000f5d60-0x000f5d6f] >> >> mapped at [ffffffffff000d60] >> >> [ 0.000000] RAMDISK: [mem 0x1e77c000-0x1ffdffff] >> >> [ 0.000000] ACPI: Early table checksum verification disabled >> >> [ 0.000000] ACPI: RSDP 0x00000000000F5B90 000014 (v00 BOCHS ) >> >> [ 0.000000] ACPI: RSDT 0x000000001FFE157C 000030 (v01 BOCHS >> >> BXPCRSDT 00000001 BXPC 00000001) >> >> [ 0.000000] ACPI: FACP 0x000000001FFE1458 000074 (v01 BOCHS >> >> BXPCFACP 00000001 BXPC 00000001) >> >> [ 0.000000] ACPI: DSDT 0x000000001FFE0040 001418 (v01 BOCHS >> >> BXPCDSDT 00000001 BXPC 00000001) >> >> [ 0.000000] ACPI: FACS 0x000000001FFE0000 000040 >> >> [ 0.000000] ACPI: APIC 0x000000001FFE14CC 000078 (v01 BOCHS >> >> BXPCAPIC 00000001 BXPC 00000001) >> >> [ 0.000000] ACPI: HPET 0x000000001FFE1544 000038 (v01 BOCHS >> >> BXPCHPET 00000001 BXPC 00000001) >> >> [ 0.000000] No NUMA configuration found >> >> [ 0.000000] Faking a node at [mem 0x0000000000000000-0x000000001ffdffff] >> >> [ 0.000000] NODE_DATA(0) allocated [mem 0x1e777000-0x1e77bfff] >> >> [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 >> >> [ 0.000000] kvm-clock: cpu 0, msr 0:1e76f001, primary cpu clock >> >> [ 0.000000] kvm-clock: using sched offset of 528742140 cycles >> >> [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff >> >> max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns >> >> [ 0.000000] Zone ranges: >> >> [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] >> >> [ 0.000000] DMA32 [mem 0x0000000001000000-0x000000001ffdffff] >> >> [ 0.000000] Normal empty >> >> [ 0.000000] Device empty >> >> [ 0.000000] Movable zone start for each node >> >> [ 0.000000] Early memory node ranges >> >> [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] >> >> [ 0.000000] node 0: [mem 0x0000000000100000-0x000000001ffdffff] >> >> [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x000000001ffdffff] >> >> [ 0.000000] ACPI: PM-Timer IO Port: 0x608 >> >> [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) >> >> [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 >> >> [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) >> >> [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) >> >> [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) >> >> [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) >> >> [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) >> >> [ 0.000000] Using ACPI (MADT) for SMP configuration information >> >> [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 >> >> [ 0.000000] smpboot: Allowing 1 CPUs, 0 hotplug CPUs >> >> [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] >> >> [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] >> >> [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] >> >> [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] >> >> [ 0.000000] e820: [mem 0x20000000-0xfeffbfff] available for PCI devices >> >> [ 0.000000] Booting paravirtualized kernel on KVM >> >> [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff >> >> max_cycles: 0xffffffff, max_idle_ns: 7645519600211568 ns >> >> [ 0.000000] setup_percpu: NR_CPUS:512 nr_cpumask_bits:512 >> >> nr_cpu_ids:1 nr_node_ids:1 >> >> [ 0.000000] percpu: Embedded 37 pages/cpu @ffff96dd9e400000 s114392 >> >> r8192 d28968 u2097152 >> >> [ 0.000000] KVM setup async PF for cpu 0 >> >> [ 0.000000] kvm-stealtime: cpu 0, msr 1e40d900 >> >> [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 128872 >> >> [ 0.000000] Policy zone: DMA32 >> >> [ 0.000000] Kernel command line: root=/dev/ram0 console=ttyS0 >> >> hung_task_panic=1 earlyprintk=ttyS0,115200 >> >> [ 0.000000] PID hash table entries: 2048 (order: 2, 16384 bytes) >> >> [ 0.000000] Memory: 474480K/523768K available (7639K kernel code, >> >> 1005K rwdata, 2936K rodata, 1636K init, 688K bss, 49288K reserved, 0K >> >> cma-reserved) >> >> [ 0.000000] ftrace: allocating 27586 entries in 108 pages >> >> [ 0.004000] Hierarchical RCU implementation. >> >> [ 0.004000] RCU restricting CPUs from NR_CPUS=512 to nr_cpu_ids=1. >> >> [ 0.004000] RCU: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1 >> >> [ 0.004000] NR_IRQS: 33024, nr_irqs: 256, preallocated irqs: 16 >> >> [ 0.004000] Console: colour VGA+ 80x25 >> >> [ 0.004000] console [ttyS0] enabled >> >> [ 0.004000] console [ttyS0] enabled >> >> [ 0.004000] bootconsole [earlyser0] disabled >> >> [ 0.004000] bootconsole [earlyser0] disabled >> >> [ 0.004000] clocksource: hpet: mask: 0xffffffff max_cycles: >> >> 0xffffffff, max_idle_ns: 19112604467 ns >> >> [ 0.004000] general protection fault: 0000 [#1] SMP >> >> [ 0.004000] Modules linked in: >> >> [ 0.004000] CPU: 0 PID: 0 Comm: swapper/0 Not tainted >> >> 4.14.0-1-iniza-llvmlinux #1 >> >> [ 0.004000] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), >> >> BIOS 1.11.1-1 04/01/2014 >> >> [ 0.004000] task: ffffffffaa610480 task.stack: ffffffffaa600000 >> >> [ 0.004000] RIP: 0010:irq_work_tick+0x9d/0x110 >> >> [ 0.004000] RSP: 0000:ffff96dd9e403e58 EFLAGS: 00010046 >> >> [ 0.004000] RAX: 0000000000000082 RBX: ffff96dd9e411d80 RCX: adecc9cc04e2ca00 >> >> [ 0.004000] RDX: 000000000001ba00 RSI: fffffffffffffed4 RDI: ffff96dd9e41ba38 >> >> [ 0.004000] RBP: ffff96dd9e403e78 R08: 0000000000000000 R09: 0000000000000018 >> >> [ 0.004000] R10: 0000000000000000 R11: 0000000000000018 R12: ffffffffaa61bd40 >> >> [ 0.004000] R13: 0000000000000000 R14: ffffffffaa610480 R15: 0000000000000000 >> >> [ 0.004000] FS: 0000000000000000(0000) GS:ffff96dd9e400000(0000) >> >> knlGS:0000000000000000 >> >> [ 0.004000] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 >> >> [ 0.004000] CR2: 00000000ffffffff CR3: 0000000008a09000 CR4: 00000000000006b0 >> >> [ 0.004000] Call Trace: >> >> [ 0.004000] >> >> [ 0.004000] update_process_times+0x6e/0xa0 >> >> [ 0.004000] tick_periodic+0x78/0x90 >> >> [ 0.004000] tick_handle_periodic+0x26/0x80 >> >> [ 0.004000] timer_interrupt+0x13/0x20 >> >> [ 0.004000] __handle_irq_event_percpu+0x106/0x230 >> >> [ 0.004000] handle_irq_event+0x5a/0xc0 >> >> [ 0.004000] handle_level_irq+0x11a/0x190 >> >> [ 0.004000] handle_irq+0x1f/0x30 >> >> [ 0.004000] do_IRQ+0x4b/0xd0 >> >> [ 0.004000] common_interrupt+0x93/0x93 >> >> [ 0.004000] >> >> [ 0.004000] RIP: 0010:native_restore_fl+0x12/0x20 >> >> [ 0.004000] RSP: 0000:ffffffffaa603e10 EFLAGS: 00000286 ORIG_RAX: >> >> ffffffffffffffcf >> >> [ 0.004000] RAX: 0000000000000001 RBX: ffff96dd9d0c0608 RCX: 0000000000000000 >> >> [ 0.004000] RDX: ffff96dd9d0c0400 RSI: 0000000000000286 RDI: 0000000000000286 >> >> [ 0.004000] RBP: ffffffffaa603e18 R08: 0000000000000001 R09: 000000000000003f >> >> [ 0.004000] R10: 0000000000000286 R11: 0000000000000007 R12: ffff96dd9d0c0514 >> >> [ 0.004000] R13: ffff96dd9d0c04e0 R14: ffffffffaa61bd40 R15: ffff96dd9d0c0400 >> >> [ 0.004000] _raw_spin_unlock_irqrestore+0x1a/0x20 >> >> [ 0.004000] __setup_irq+0x5d9/0x780 >> >> [ 0.004000] setup_irq+0x5c/0x90 >> >> [ 0.004000] hpet_time_init+0x32/0x40 >> >> [ 0.004000] x86_late_time_init+0x10/0x20 >> >> [ 0.004000] start_kernel+0x45d/0x580 >> >> [ 0.004000] x86_64_start_kernel+0x30f/0x320 >> >> [ 0.004000] secondary_startup_64+0xa5/0xa5 >> >> [ 0.004000] Code: f0 f0 4c 0f b1 7b f8 4c 89 e3 4d 85 e4 75 cf 48 >> >> c7 c1 88 5b 01 00 65 48 03 0c 25 d8 a1 00 00 ff 14 25 10 08 62 aa f6 >> >> c4 02 75 4d <48> 83 39 00 74 3e 31 db 48 87 19 48 85 db 74 34 0f 1f 00 >> >> 48 8d >> >> [ 0.004000] RIP: irq_work_tick+0x9d/0x110 RSP: ffff96dd9e403e58 >> >> [ 0.004000] ---[ end trace 8c7851007fbc6b6f ]--- >> >> [ 0.004000] Kernel panic - not syncing: Fatal exception in interrupt >> >> [ 0.004000] ---[ end Kernel panic - not syncing: Fatal exception in interrupt >> >> >> >> Is this saying someting to you? >> > >> > It doesn't ring a bell, but I can repro it (with different offsets): >> > >> > [ 0.004000] general protection fault: 0000 [#1] SMP >> > [ 0.004000] Modules linked in: >> > [ 0.004000] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 4.14.0+ #10 >> > [ 0.004000] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.10.2-1 04/01/2014 >> > [ 0.004000] task: ffffffffbb410480 task.stack: ffffffffbb400000 >> > [ 0.004000] RIP: 0010:irq_work_tick+0xaf/0x120 >> > [ 0.004000] RSP: 0000:ffff96fb5fc03e58 EFLAGS: 00010046 >> > [ 0.004000] RAX: 0000000000000082 RBX: ffff96fb5fc11d80 RCX: 47e992c1bc778d00 >> > [ 0.004000] RDX: 000000000001ba00 RSI: fffffffffffffed4 RDI: ffff96fb5fc1ba38 >> > [ 0.004000] RBP: ffff96fb5fc03e78 R08: 0000000000000000 R09: 0000000000000018 >> > [ 0.004000] R10: 0000000000000000 R11: 0000000000000018 R12: ffffffffbb41bd40 >> > [ 0.004000] R13: 0000000000000000 R14: ffffffffbb410480 R15: 0000000000000000 >> > [ 0.004000] FS: 0000000000000000(0000) GS:ffff96fb5fc00000(0000) knlGS:0000000000000000 >> > [ 0.004000] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 >> > [ 0.004000] CR2: 00000000ffffffff CR3: 000000001ec09000 CR4: 00000000000006b0 >> > [ 0.004000] Call Trace: >> > [ 0.004000] >> > [ 0.004000] update_process_times+0x6e/0xa0 >> > [ 0.004000] tick_periodic+0x78/0x90 >> > [ 0.004000] tick_handle_periodic+0x26/0x80 >> > [ 0.004000] timer_interrupt+0x13/0x20 >> > [ 0.004000] __handle_irq_event_percpu+0x106/0x230 >> > [ 0.004000] handle_irq_event+0x5a/0xc0 >> > [ 0.004000] handle_level_irq+0x11a/0x190 >> > [ 0.004000] handle_irq+0x1f/0x30 >> > [ 0.004000] do_IRQ+0x4b/0xd0 >> > [ 0.004000] common_interrupt+0x93/0x93 >> > [ 0.004000] >> > [ 0.004000] RIP: 0010:native_restore_fl+0xd/0x20 >> > [ 0.004000] RSP: 0000:ffffffffbb403e08 EFLAGS: 00000282 ORIG_RAX: ffffffffffffffcf >> > [ 0.004000] RAX: 0000000000000001 RBX: 0000000000000000 RCX: 0000000000000000 >> > [ 0.004000] RDX: ffff96fb5f01b800 RSI: 0000000000000282 RDI: 0000000000000282 >> > [ 0.004000] RBP: ffffffffbb403e10 R08: 0000000000000001 R09: 000000000000003f >> > [ 0.004000] R10: ffff96fb5f01b800 R11: 0000000000000007 R12: ffff96fb5f01ba08 >> > [ 0.004000] R13: ffff96fb5f01b8e0 R14: ffffffffbb41bd40 R15: ffff96fb5f01b800 >> > [ 0.004000] _raw_spin_unlock_irqrestore+0x1a/0x20 >> > [ 0.004000] __setup_irq+0x610/0x7b0 >> > [ 0.004000] setup_irq+0x5b/0x90 >> > [ 0.004000] hpet_time_init+0x32/0x40 >> > [ 0.004000] x86_late_time_init+0x10/0x20 >> > [ 0.004000] start_kernel+0x460/0x580 >> > [ 0.004000] x86_64_start_kernel+0x30f/0x320 >> > [ 0.004000] secondary_startup_64+0xa5/0xa5 >> > [ 0.004000] Code: 4c 0f b1 7b f8 4c 89 e3 4d 85 e4 75 cf 48 c7 c1 88 5b 01 00 65 48 03 0c 25 d8 a1 00 00 ff 14 25 10 08 42 bb a9 00 02 00 00 75 4b <48> 83 39 00 74 3c 31 db 48 87 19 48 85 db 74 32 90 48 8d 7b f8 >> > [ 0.004000] RIP: irq_work_tick+0xaf/0x120 RSP: ffff96fb5fc03e58 >> > [ 0.004000] ---[ end trace 08945838e05bf5b2 ]--- >> > [ 0.004000] Kernel panic - not syncing: Fatal exception in interrupt >> > [ 0.004000] ---[ end Kernel panic - not syncing: Fatal exception in interrupt >> > >> > The exception occurs at 'irq_work_tick+0xaf' >> > >> > objdump -d -S --start-address=0x$(grep irq_work_tick System.map | sed -e >> > "s/ \+.*//") vmlinux | less >> > >> > ... >> > void irq_work_tick(void) >> > { >> > ffffffff81193200: 55 push %rbp >> > ... >> > static inline struct llist_node *llist_del_all(struct llist_head *head) >> > { >> > return xchg(&head->first, NULL); >> > ffffffff8119324a: 48 87 19 xchg %rbx,(%rcx) >> > while (llnode != NULL) { >> > ffffffff8119324d: 48 85 db test %rbx,%rbx >> > ffffffff81193250: 74 3f je ffffffff81193291 >> > ffffffff81193252: 0f 1f 40 00 nopl 0x0(%rax) >> > ffffffff81193256: 66 2e 0f 1f 84 00 00 nopw %cs:0x0(%rax,%rax,1) >> > ffffffff8119325d: 00 00 00 >> > work = llist_entry(llnode, struct irq_work, llnode); >> > ffffffff81193260: 48 8d 7b f8 lea -0x8(%rbx),%rdi >> > flags = work->flags & ~IRQ_WORK_PENDING; >> > ffffffff81193264: 4c 8b 7b f8 mov -0x8(%rbx),%r15 >> > return node->next; >> > ffffffff81193268: 4c 8b 23 mov (%rbx),%r12 >> > ffffffff8119326b: 4d 89 fe mov %r15,%r14 >> > ffffffff8119326e: 49 83 e6 fe and $0xfffffffffffffffe,%r14 >> > xchg(&work->flags, flags); >> > ffffffff81193272: 4c 89 f0 mov %r14,%rax >> > ffffffff81193275: 48 87 43 f8 xchg %rax,-0x8(%rbx) >> > work->func(work); >> > ffffffff81193279: ff 53 08 callq *0x8(%rbx) >> > (void)cmpxchg(&work->flags, flags, flags & ~IRQ_WORK_BUSY); >> > ffffffff8119327c: 49 83 e7 fc and $0xfffffffffffffffc,%r15 >> > ffffffff81193280: 4c 89 f0 mov %r14,%rax >> > ffffffff81193283: f0 4c 0f b1 7b f8 lock cmpxchg %r15,-0x8(%rbx) >> > ffffffff81193289: 4c 89 e3 mov %r12,%rbx >> > while (llnode != NULL) { >> > ffffffff8119328c: 4d 85 e4 test %r12,%r12 >> > ffffffff8119328f: 75 cf jne ffffffff81193260 >> > irq_work_run_list(raised); >> > irq_work_run_list(this_cpu_ptr(&lazy_list)); >> > ffffffff81193291: 48 c7 c1 88 5b 01 00 mov $0x15b88,%rcx >> > ffffffff81193298: 65 48 03 0c 25 d8 a1 add %gs:0xa1d8,%rcx >> > ffffffff8119329f: 00 00 >> > ffffffff811932a1: ff 14 25 10 08 c2 81 callq *0xffffffff81c20810 >> > BUG_ON(!irqs_disabled()); >> > ffffffff811932a8: a9 00 02 00 00 test $0x200,%eax >> > ffffffff811932ad: 75 4b jne ffffffff811932fa >> > return ACCESS_ONCE(head->first) == NULL; >> > ffffffff811932af: 48 83 39 00 cmpq $0x0,(%rcx) >> > >> > >> > I'm no x86/assembly expert, anyway my interpretation is: >> > >> > 'irq_work_tick+0xaf' corresponds to address 0xffffffff811932af, so the >> > exception occurs during the execution of 'cmpq $0x0,(%rcx)' or >> > 'ACCESS_ONCE(head->first) == NULL'. The instruction checks if the >> > memory location in RCX contains 0. Supposedly the register should >> > contain the address of 'head->first', however the value is >> > 0x47e992c1bc778d00, which doesn't look like a valid address. >> > >> > RCX is set shortly before: >> > >> > ffffffff81193291: 48 c7 c1 88 5b 01 00 mov $0x15b88,%rcx >> > ffffffff81193298: 65 48 03 0c 25 d8 a1 add %gs:0xa1d8,%rcx >> > >> > As stated in https://www.kernel.org/doc/Documentation/this_cpu_ops.txt >> > the GS segment register is used by the kernel for per-cpu >> > variables. And effectively System.map tells us the offset 0x15b88 >> > corresponds to the per-cpu variable lazy_list: >> > >> > 0000000000015b88 d lazy_list >> > >> > The register dump shows GS with a value of 0xffff96fb5fc00000, which >> > looks reasonable. >> > >> > But wait, right after setting RCX we do this: >> > >> > ffffffff811932a1: ff 14 25 10 08 c2 81 callq *0xffffffff81c20810 >> > BUG_ON(!irqs_disabled()); >> > >> > According to my (limited) understanding of x86 calling conventions RCX >> > is a caller-saved register, thus the caller should save it on the >> > stack to preserve its value across a function call. >> > (https://en.wikipedia.org/wiki/X86_calling_conventions) >> > >> > The call to '*0xffffffff81c20810' appears to be related with >> > KVM/paravirtualization: >> > >> > grep ffffffff81c20810 System.map >> > ffffffff81c20810 D pv_irq_ops >> > >> > Since there is no offset it must be calling the first function in the >> > structure, which is 'safe_fl' and points to 'native_save_fl' >> > (https://elixir.bootlin.com/linux/v4.14.36/source/arch/x86/kernel/paravirt.c#L316) >> > >> > objdump -d -S --start-address=0x$(grep native_save_fl System.map | sed >> > -e "s/ \+.*//") vmlinux | less >> > >> > static inline unsigned long native_save_fl(void) >> > { >> > ffffffff81060240: 55 push %rbp >> > ffffffff81060241: 48 89 e5 mov %rsp,%rbp >> > ffffffff81060244: 48 83 ec 10 sub $0x10,%rsp >> > ffffffff81060248: 65 48 8b 04 25 28 00 mov %gs:0x28,%rax >> > ffffffff8106024f: 00 00 >> > ffffffff81060251: 48 89 45 f8 mov %rax,-0x8(%rbp) >> > /* >> > * "=rm" is safe here, because "pop" adjusts the stack before >> > * it evaluates its effective address -- this is part of the >> > * documented behavior of the "pop" instruction. >> > */ >> > asm volatile("# __raw_save_flags\n\t" >> > ffffffff81060255: 9c pushfq >> > ffffffff81060256: 8f 45 f0 popq -0x10(%rbp) >> > "pushf ; pop %0" >> > : "=rm" (flags) >> > : /* no input */ >> > : "memory"); >> > >> > return flags; >> > ffffffff81060259: 48 8b 45 f0 mov -0x10(%rbp),%rax >> > ffffffff8106025d: 65 48 8b 0c 25 28 00 mov %gs:0x28,%rcx >> > ffffffff81060264: 00 00 >> > ffffffff81060266: 48 3b 4d f8 cmp -0x8(%rbp),%rcx >> > ffffffff8106026a: 75 06 jne ffffffff81060272 >> > ffffffff8106026c: 48 83 c4 10 add $0x10,%rsp >> > ffffffff81060270: 5d pop %rbp >> > ffffffff81060271: c3 retq >> > ffffffff81060272: e8 09 ec 01 00 callq ffffffff8107ee80 <__stack_chk_fail> >> > ffffffff81060277: 66 0f 1f 84 00 00 00 nopw 0x0(%rax,%rax,1) >> > ffffffff8106027e: 00 0 >> > >> > At 0xffffffff8106025d this clobbers RCX! I don't know why clang >> > doesn't save the value on the stack before calling native_save_fl(), >> > but that seems to be the problem. >> > >> > Again, I'm not an expert in this area and ventured into territory >> > unknown to me, so please excuse if I got something totally wrong ... >> > >> >> Hi Matthias, >> >> again you helped me a lot. >> >> I recalled that I had success in the early days of llvmlinux with... >> >> CONFIG_PARAVIRT=n >> >> ...as suggested by Steve Rostedt. >> >> This needed some more hackish kconfig kungfu these days - patches attached. >> >> In the end I could boot into a 4.14.37-1-iniza-llvmlinux kernel >> (kernel-config and dmesg-log attached). >> >> Concerning your analysis... >> >> Is this a clang-bug? >> Or can this be solved in the linux-kernel code? >> What are your suggestion? >> And can you help on getting this fixed with the help of your Chromium >> team and/or bring it to the right persons? > > Apparently stack-protector code is clobbering RCX, however the > paravirt code expects the register to be preserved: > > struct pv_irq_ops { > /* > * ... > * NOTE: These functions callers expect the callee to preserve > * more registers than the standard C calling convention. > */ > struct paravirt_callee_save save_fl; > > We think the solution is to mark native_save_fl with > '__nostackprotector', however this requires adding support for > such an attribute to clang. Our compiler team is working on this. > OK, cool. Can you inform me when there is progress on this? Thanks. >> IMHO an analysis like you did might help, too. >> I am not an expert in this area to say it is good/correct but I could >> follow and read it like a thriller. >> Where is Lieutenant Columbo when you need him :-)? >> >> Moreover, a test-case can be helpful, e.g. "Is clang clobbering RCX?" >> when it's a clang-bug. >> >> I tried to find some suitable test-case myself. >> >> The clang-source has a test-dir, but this is new to me. >> >> I tried to run a single test like test/Sema/asm.c in [1] with >> llvm-tools Debian-package which ships "lit.py" script. >> >> Not sure if this is a suitable for x86-64 (amd64) as the first line says... >> >> // RUN: %clang_cc1 %s -Wno-private-extern -triple i386-pc-linux-gnu >> -verify -fsyntax-only >> >> root# cd /path/to/clang-source >> >> root# python /usr/lib/llvm-7/build/utils/lit/lit.py -sv test/Sema >> lit.py: /usr/lib/llvm-7/build/utils/lit/lit/TestingConfig.py:101: >> fatal: unable to parse config file >> '/path/to/clang-source/test/lit.cfg.py', traceback: Traceback (most >> recent call last): >> File "/usr/lib/llvm-7/build/utils/lit/lit/TestingConfig.py", line >> 88, in load_from_path >> exec(compile(data, path, 'exec'), cfg_globals, None) >> File "/home/sdi/src/linux-kernel/clang/test/lit.cfg.py", line 25, in >> config.test_format = lit.formats.ShTest(not llvm_config.use_lit_shell) >> AttributeError: 'NoneType' object has no attribute 'use_lit_shell' >> >> Not sure if you have any experiences in this area, but I think asking >> on llvm-dev is a good idea. >> >> If we have a reproducible test-case we could ask on llvm-dev? >> >> In the case of linux-kernel... >> Ask Linux/x86 folks for a test-case to see if the compiler clobbers RCX? >> Show your analysis to Linux/(x86)/KVM folks? > > I'm not an expert, but I don't think there is a generic test case for > this special scenario involving paravirt and stack protector. It's > probably best to just fix the problem and move on. Yeah, I was able to run a single clang-test like Sema/asm.c. There is a thread on llvm-dev "[clang] Running a single testcase" where I had success with using the built clang binary etc. A nice excursion in that area. Thanks again. - Sedat -