Received: by 2002:a05:7412:37c9:b0:e2:908c:2ebd with SMTP id jz9csp1412520rdb; Wed, 20 Sep 2023 08:26:16 -0700 (PDT) X-Google-Smtp-Source: AGHT+IGk0ZGwwCnnv6/q/pk3W11GLhTHnktQ92DfbW+RC/wZ8LOO0d1qfd62eO21TeSSA8VmPB/k X-Received: by 2002:a05:6a20:7488:b0:15d:1646:285a with SMTP id p8-20020a056a20748800b0015d1646285amr631950pzd.21.1695223575349; Wed, 20 Sep 2023 08:26:15 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1695223575; cv=none; d=google.com; s=arc-20160816; b=fAx2nJ5alNUh9jMij8JidTtOmqr+Sk/t5/xsPBE0uSTiQMY4FJ5jALzAIclcZAUcxs uGOooIEky3NA4aSw6qO+v9p+R0vRZ1rZAG/u1O2KA/axI6+Cce62/S1ynVKN2ok9Ka66 KqWuNyPoZmEERvLxa6kXJg/0g9jFwAVNstW9HC3J1/adBBwfDKHExX+uNyKazQFlIB1Y Sj+Ez4zUWsJr6kRA6lRP6br+1Q3x3PxiCFDzkRQnX2aKBP3N+g88ET90nv34vvRqgtnx GaU1dlC68R/KkjzeBM7mVUvE7QBSV3iRx5AqH97yKSHNFlQjhCADRNMOEfEyj2heY42J KLuw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:in-reply-to:content-disposition:mime-version :references:message-id:subject:cc:to:from:date; bh=j2j4Ej6obbfB5pg09mHPgeAngrBBAV1Q/SygrZYCFDk=; fh=ZUi3rsv7W7h7VM5lJ1BvpPOxmVI4JlUDx1qQO26DohQ=; b=kJ+c23OPU1X1/cRfXIh+hi7NexNyCpT9W6hPAaBnLjHjcbrHZE+FWzAaT6QQ0hSd1a VCssXHLRFGgyEVm8eGd1ITfgjtbo0X7UVRob1NS24xqNAD/WV1U0+mWUbWQWdOCS6IO6 pfOVlFZXOXneVVPSXjuHfzYguJyI8jPTawaySg8IJHCOzL7qcKm+jv6cpLsZIBHBW+oL 1WVtXg/suxw04zsP7LqJS5JUZgnax/FXldz7gb0OMPkV24jE81mhadTQx/2IjkZMD4zc 8B2a4B4nEMyi6eIajqAHFueT6cIDJdoR+TtKuNtgJ25/+fFlhAwjNEhbn11g1laIdsvB k33Q== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:4 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=arm.com Return-Path: Received: from howler.vger.email (howler.vger.email. [2620:137:e000::3:4]) by mx.google.com with ESMTPS id cd22-20020a056a00421600b00690c0051dcdsi3671820pfb.143.2023.09.20.08.26.14 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 20 Sep 2023 08:26:15 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:4 as permitted sender) client-ip=2620:137:e000::3:4; Authentication-Results: mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:4 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=arm.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by howler.vger.email (Postfix) with ESMTP id DDC31836E5D6; Wed, 20 Sep 2023 07:32:57 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at howler.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234713AbjITOc7 (ORCPT + 99 others); Wed, 20 Sep 2023 10:32:59 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59938 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232318AbjITOc6 (ORCPT ); Wed, 20 Sep 2023 10:32:58 -0400 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id 18676AD; Wed, 20 Sep 2023 07:32:52 -0700 (PDT) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id D61631FB; Wed, 20 Sep 2023 07:33:28 -0700 (PDT) Received: from FVFF77S0Q05N.cambridge.arm.com (FVFF77S0Q05N.cambridge.arm.com [10.1.27.185]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 89A723F5A1; Wed, 20 Sep 2023 07:32:49 -0700 (PDT) Date: Wed, 20 Sep 2023 15:32:47 +0100 From: Mark Rutland To: Naresh Kamboju Cc: LTP List , open list , Linux ARM , rcu , lkft-triage@lists.linaro.org, chrubis , Greg Kroah-Hartman , Peter Zijlstra , Josh Poimboeuf , Jason Baron , Steven Rostedt , Ard Biesheuvel , Catalin Marinas , Will Deacon , Dan Carpenter , Arnd Bergmann Subject: Re: arm64: Unable to handle kernel execute from non-executable memory at virtual address ffff8000834c13a0 Message-ID: References: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00, RCVD_IN_DNSWL_BLOCKED,SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (howler.vger.email [0.0.0.0]); Wed, 20 Sep 2023 07:32:58 -0700 (PDT) Hi Naresh, On Wed, Sep 20, 2023 at 11:29:12AM +0200, Naresh Kamboju wrote: > [ my two cents ] > While running LTP pty07 test cases on arm64 juno-r2 with Linux next-20230919 > the following kernel crash was noticed. > > I have been noticing this issue intermittently on Juno-r2 for more than a month. > Anyone have noticed this crash ? How intermittent is this? 1/2, 1/10, 1/100, rarer still? Are you running *just* the pty07 test, or are you running a whole LTP suite and the issue first occurs around pty07? Given you've been hitting this for a month, have you tried testing mainline? Do you have a known-good kernel that we can start a bisect from? Do you *only* see this on Juno-r2 and are you testing on other hardware? > Reported-by: Linux Kernel Functional Testing > > [ 0.000000] Linux version 6.6.0-rc2-next-20230919 (tuxmake@tuxmake) > (aarch64-linux-gnu-gcc (Debian 13.2.0-2) 13.2.0, GNU ld (GNU Binutils > for Debian) 2.41) #1 SMP PREEMPT @1695107157 > [ 0.000000] KASLR disabled due to lack of seed > [ 0.000000] Machine model: ARM Juno development board (r2) > ... > LTP running pty > ... > > pty07.c:92: TINFO: Saving active console 1 > ../../../include/tst_fuzzy_sync.h:640: TINFO: Stopped sampling at 552 > (out of 1024) samples, sampling time reached 50% of the total time > limit > ../../../include/tst_fuzzy_sync.h:307: TINFO: loop = 552, delay_bias = 0 > ../../../include/tst_fuzzy_sync.h:295: TINFO: start_a - start_b: { avg > = 127ns, avg_dev = 84ns, dev_ratio = 0.66 } > ../../../include/tst_fuzzy_sync.h:295: TINFO: end_a - start_a : { avg > = 17296156ns, avg_dev = 5155058ns, dev_ratio = 0.30 } > ../../../include/tst_fuzzy_sync.h:295: TINFO: end_b - start_b : { avg > = 101202336ns, avg_dev = 6689286ns, dev_ratio = 0.07 } > ../../../include/tst_fuzzy_sync.h:295: TINFO: end_a - end_b : { avg > = -83906064ns, avg_dev = 10230694ns, dev_ratio = 0.12 } > ../../../include/tst_fuzzy_sync.h:295: TINFO: spins : { avg > = 2765565 , avg_dev = 339285 , dev_ratio = 0.12 } > [ 384.133538] Unable to handle kernel execute from non-executable > memory at virtual address ffff8000834c13a0 > [ 384.133559] Mem abort info: > [ 384.133568] ESR = 0x000000008600000f > [ 384.133578] EC = 0x21: IABT (current EL), IL = 32 bits > [ 384.133590] SET = 0, FnV = 0 > [ 384.133600] EA = 0, S1PTW = 0 > [ 384.133610] FSC = 0x0f: level 3 permission fault > [ 384.133621] swapper pgtable: 4k pages, 48-bit VAs, pgdp=0000000082375000 > [ 384.133634] [ffff8000834c13a0] pgd=10000009fffff003, > p4d=10000009fffff003, pud=10000009ffffe003, pmd=10000009ffff8003, > pte=00780000836c1703 > [ 384.133697] Internal error: Oops: 000000008600000f [#1] PREEMPT SMP > [ 384.133707] Modules linked in: tda998x onboard_usb_hub cec hdlcd > crct10dif_ce drm_dma_helper drm_kms_helper fuse drm backlight dm_mod > ip_tables x_tables > [ 384.133767] CPU: 3 PID: 589 Comm: (udev-worker) Not tainted > 6.6.0-rc2-next-20230919 #1 > [ 384.133779] Hardware name: ARM Juno development board (r2) (DT) > [ 384.133784] pstate: 40000005 (nZcv daif -PAN -UAO -TCO -DIT -SSBS BTYPE=--) > [ 384.133796] pc : in_lookup_hashtable+0x178/0x2000 This indicates that the faulting address ffff8000834c13a0 is in_lookup_hashtable+0x178/0x2000, which would been we've somehow marked the kernel text as non-executable, which we never do intentionally. I suspect that implies memory corruption. Have you tried running this with KASAN enabled? > [ 384.133818] lr : rcu_core (arch/arm64/include/asm/preempt.h:13 > (discriminator 1) kernel/rcu/tree.c:2146 (discriminator 1) > kernel/rcu/tree.c:2403 (discriminator 1)) > [ 384.133832] sp : ffff800083533e60 > [ 384.133836] x29: ffff800083533e60 x28: ffff0008008a6180 x27: 000000000000000a > [ 384.133854] x26: 0000000000000000 x25: 0000000000000000 x24: ffff800083533f10 > [ 384.133871] x23: ffff800082404008 x22: ffff800082ebea80 x21: ffff800082f55940 > [ 384.133889] x20: ffff00097ed75440 x19: 0000000000000001 x18: 0000000000000000 > [ 384.133905] x17: ffff8008fc95c000 x16: ffff800083530000 x15: 00003d0900000000 > [ 384.133922] x14: 0000000000030d40 x13: 0000000000000000 x12: 003d090000000000 > [ 384.133939] x11: 0000000000000000 x10: 0000000000000008 x9 : ffff80008015b05c > [ 384.133955] x8 : ffff800083533da8 x7 : 0000000000000000 x6 : 0000000000000100 > [ 384.133971] x5 : ffff800082ebf000 x4 : ffff800082ebf2e8 x3 : 0000000000000000 > [ 384.133987] x2 : ffff000825bf8618 x1 : ffff8000834c13a0 x0 : ffff00082b6d7170 > [ 384.134005] Call trace: > [ 384.134009] in_lookup_hashtable+0x178/0x2000 > [ 384.134022] rcu_core_si (kernel/rcu/tree.c:2421) > [ 384.134035] __do_softirq (arch/arm64/include/asm/jump_label.h:21 > include/linux/jump_label.h:207 include/trace/events/irq.h:142 > kernel/softirq.c:554) > [ 384.134046] ____do_softirq (arch/arm64/kernel/irq.c:81) > [ 384.134058] call_on_irq_stack (arch/arm64/kernel/entry.S:888) > [ 384.134070] do_softirq_own_stack (arch/arm64/kernel/irq.c:86) > [ 384.134082] irq_exit_rcu (arch/arm64/include/asm/percpu.h:44 > kernel/softirq.c:612 kernel/softirq.c:634 kernel/softirq.c:644) > [ 384.134094] el0_interrupt (arch/arm64/include/asm/daifflags.h:28 > arch/arm64/kernel/entry-common.c:133 > arch/arm64/kernel/entry-common.c:144 > arch/arm64/kernel/entry-common.c:763) > [ 384.134110] __el0_irq_handler_common (arch/arm64/kernel/entry-common.c:769) > [ 384.134124] el0t_64_irq_handler (arch/arm64/kernel/entry-common.c:774) > [ 384.134137] el0t_64_irq (arch/arm64/kernel/entry.S:592) > [ 384.134153] Code: 00000000 00000000 00000000 00000000 (2b6d7170) > All code > ======== > ... > 10: 70 71 jo 0x83 > 12: 6d insl (%dx),%es:(%rdi) > 13: 2b .byte 0x2b > > Code starting with the faulting instruction > =========================================== > 0: 70 71 jo 0x73 > 2: 6d insl (%dx),%es:(%rdi) > 3: 2b .byte 0x2b As a general thing, can you *please* fix this code dump to decode arm64 as arm64? Given the instructions before this are all UDF #0, I suspect the page table entry has been corrupted and this is pointing at entirely the wrong page. Thanks, Mark. > [ 384.134161] ---[ end trace 0000000000000000 ]--- > [ 384.134168] Kernel panic - not syncing: Oops: Fatal exception in interrupt > [ 384.134173] SMP: stopping secondary CPUs > [ 384.134184] Kernel Offset: disabled > [ 384.134187] CPU features: 0x8000020c,3c020000,0000421b > [ 384.134194] Memory Limit: none > > Links: > - https://qa-reports.linaro.org/lkft/linux-next-master/build/next-20230919/testrun/20054202/suite/log-parser-test/tests/ > - https://qa-reports.linaro.org/lkft/linux-next-master/build/next-20230919/testrun/20054202/suite/log-parser-test/test/check-kernel-oops/log > - https://storage.tuxsuite.com/public/linaro/lkft/builds/2VbZdpWwncUx8oSxsSXCWV3N5DH/ > - https://lkft.validation.linaro.org/scheduler/job/6666807#L2461 > > -- > Linaro LKFT > https://lkft.linaro.org