Received: by 2002:ac0:aed5:0:0:0:0:0 with SMTP id t21csp3154691imb; Tue, 5 Mar 2019 02:14:07 -0800 (PST) X-Google-Smtp-Source: APXvYqzJ4dmbd0CRza8nou0JGP+QLiTK/nekMQ7gRosEKVUujtbCB00iKLEoDXZsFVSydtnoJgLd X-Received: by 2002:a62:13da:: with SMTP id 87mr1056208pft.173.1551780847325; Tue, 05 Mar 2019 02:14:07 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1551780847; cv=none; d=google.com; s=arc-20160816; b=NJeDaoG8+MEXJd+VdxJZajicTzK8UFmq235Uvll9ZtpdlbS8yjMy+03qz32SLna5Q+ 2eHTbEg+PfLN1uWQa7MDCELj8oDmnJpySbWrGbHXTQ94yOWiL5lOFa0r8C5zOfNH1kQy PfyF6WGYcfl6GjaSlxSWC1tlD5pXDlhdjHSQ7hpM+SD266OZAeWdxl4W7iyiWH/ZixV6 gqn0nJm6MerW//kKMnmuOUee6y8KbyEStC+YfPB1M1w7Kf84Q8Xmr61DRD4i27pHzZt/ 3E4/7SIt7Z4rRrRKDXJ+/ALzEJeLTRTY1DkHSxTV0nqgtTG1EkesSdjqpmy0szG+dkHJ j/IQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version; bh=pB4RkAugqZOA7wZDlhlEUYbhlVKZt+DNUdrm+u8dH7k=; b=ldAzZ58Qr+GI159VVZlq6VCFPkXBlQEdWPEpceg6BKE82hskFEOcTxkNaq9YRmAEEK MxmLYaE6myIJzeR1hh71HNeLza/4ldIkK2Dtr1Al0jmrZou3Virm2mtc506GAFCaBxQT c4IEp6O2zoGKppEUepx/M0NzFqB40nWGrL+E5RAbOOPNYU41tnL8khCJx9fNmtFDmZM5 ea3BK2Sl6+usLpJCVFrNCj/sWi1n8u1K7+BWbgufNG8b1XpBbeEx5zLv7k8WopuoS2xc 84k5Ov6tuFzrh1iGodCOGTLMUnr+fO8iLS/zmsooc2P22sO6QBIqnY5NTN+3M+UToKaW te9A== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=canonical.com Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id c6si7767115plr.166.2019.03.05.02.13.51; Tue, 05 Mar 2019 02:14:07 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=canonical.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727489AbfCEKMb (ORCPT + 99 others); Tue, 5 Mar 2019 05:12:31 -0500 Received: from youngberry.canonical.com ([91.189.89.112]:37013 "EHLO youngberry.canonical.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727213AbfCEKMb (ORCPT ); Tue, 5 Mar 2019 05:12:31 -0500 Received: from mail-yw1-f72.google.com ([209.85.161.72]) by youngberry.canonical.com with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.76) (envelope-from ) id 1h173k-0007ZI-Q5 for linux-kernel@vger.kernel.org; Tue, 05 Mar 2019 10:12:28 +0000 Received: by mail-yw1-f72.google.com with SMTP id t9so12308162ywe.19 for ; Tue, 05 Mar 2019 02:12:28 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=pB4RkAugqZOA7wZDlhlEUYbhlVKZt+DNUdrm+u8dH7k=; b=hM9/M0FxYGgKIWGV9nedRbqLzbJhe4UhHTCvUYlHvd17xrFXYJU7uXmfWz4T4x3QPs ruGqq/2+b59Eyca0OSB4L3g3vFBVWSP0sHzszSLIIlnMXJTFDTh7zuJ6n+K6LWAl5R4H mqF4AuORcm/fg5ooO3MDMb6EZOwcN5Fu7z+Th3ebVtSVxA621vYOUL93EbQRfUopt4P2 FZTMMgnP93KEDa17RZo5pdYwgwp9WYQnlhyFF0Z3wesnmjnttXqJlnm22FXcaef0sRvD e9g4S0iu2UMU8O/zlBFqSs2Dslznakr1JVRvmz6XFjur2Ad5SFB6p4IQ6Z31MvKgIDoq MV1w== X-Gm-Message-State: APjAAAWX0OpQRe3EFDTY+ccm87oUocMVGyeGq3tqetyML7JFvJofwu7O /EMeI/QDenRVDrb0mNCjSmd0c+Qf8Ztv/io2gv5O0KpHZKoYN0K1xgvEK8Eehpikr9Agtv83ZjY l5pESzPj8F2FqkDpknEoLYY5LFscx7WlzMJIgfMPNXt9SWtvnmRv8wog9ug== X-Received: by 2002:a25:6085:: with SMTP id u127mr1397361ybb.291.1551780747024; Tue, 05 Mar 2019 02:12:27 -0800 (PST) X-Received: by 2002:a25:6085:: with SMTP id u127mr1397345ybb.291.1551780746717; Tue, 05 Mar 2019 02:12:26 -0800 (PST) MIME-Version: 1.0 References: <20190305104010.6342e9b9@gollum> In-Reply-To: <20190305104010.6342e9b9@gollum> From: Christian Ehrhardt Date: Tue, 5 Mar 2019 11:12:00 +0100 Message-ID: Subject: Re: [ovs-dev] openvswitch crash on i386 To: Juerg Haefliger , James Page Cc: pshelar@ovn.org, davem@davemloft.net, netdev@vger.kernel.org, "" , linux-kernel@vger.kernel.org Content-Type: text/plain; charset="UTF-8" Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Tue, Mar 5, 2019 at 10:58 AM Juerg Haefliger wrote: > > Hi, > > Running the following commands in a loop will crash an i386 5.0 kernel > typically within a few iterations: > > ovs-vsctl add-br test > ovs-vsctl del-br test > > [ 106.215748] BUG: unable to handle kernel paging request at e8a35f3b > [ 106.216733] #PF error: [normal kernel read fault] > [ 106.217464] *pdpt = 0000000019a76001 *pde = 0000000000000000 > [ 106.218346] Oops: 0000 [#1] SMP PTI > [ 106.218911] CPU: 0 PID: 2050 Comm: systemd-udevd Tainted: G E 5.0.0 #25 > [ 106.220103] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.11.1-1ubuntu1 04/01/2014 > [ 106.221447] EIP: kmem_cache_alloc_trace+0x7a/0x1b0 > [ 106.222178] Code: 01 00 00 8b 07 64 8b 50 04 64 03 05 28 61 e8 d2 8b 08 89 4d ec 85 c9 0f 84 03 01 00 00 8b 45 ec 8b 5f 14 8d 4a 01 8b 37 01 c3 <33> 1b 33 9f b4 00 00 00 64 0f c7 0e 75 cb 8b 75 ec 8b 47 14 0f 18 > [ 106.224752] EAX: e8a35f3b EBX: e8a35f3b ECX: 0000869f EDX: 0000869e > [ 106.225683] ESI: d2e96ef0 EDI: da401a00 EBP: d9b85dd0 ESP: d9b85db0 > [ 106.226662] DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068 EFLAGS: 00010282 > [ 106.227710] CR0: 80050033 CR2: e8a35f3b CR3: 185b8000 CR4: 000006f0 > [ 106.228703] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000 > [ 106.229604] DR6: fffe0ff0 DR7: 00000400 > [ 106.230114] Call Trace: > [ 106.230525] ? kernfs_fop_open+0xb4/0x390 > [ 106.231176] kernfs_fop_open+0xb4/0x390 > [ 106.231856] ? security_file_open+0x7c/0xc0 > [ 106.232562] do_dentry_open+0x131/0x370 > [ 106.233229] ? kernfs_fop_write+0x180/0x180 > [ 106.233905] vfs_open+0x25/0x30 > [ 106.234432] path_openat+0x2fd/0x1450 > [ 106.235084] ? cp_new_stat64+0x115/0x140 > [ 106.235754] ? cp_new_stat64+0x115/0x140 > [ 106.236427] do_filp_open+0x6a/0xd0 > [ 106.237026] ? cp_new_stat64+0x115/0x140 > [ 106.237748] ? strncpy_from_user+0x3d/0x180 > [ 106.238539] ? __alloc_fd+0x36/0x120 > [ 106.239256] do_sys_open+0x175/0x210 > [ 106.239955] sys_openat+0x1b/0x20 > [ 106.240596] do_fast_syscall_32+0x7f/0x1e0 > [ 106.241313] entry_SYSENTER_32+0x6b/0xbe > [ 106.242017] EIP: 0xb7fae871 > [ 106.242559] Code: 8b 98 58 cd ff ff 89 c8 85 d2 74 02 89 0a 5b 5d c3 8b 04 24 c3 8b 14 24 c3 8b 34 24 c3 8b 3c 24 c3 51 52 55 89 e5 0f 34 cd 80 <5d> 5a 59 c3 90 90 90 90 8d 76 00 58 b8 77 00 00 00 cd 80 90 8d 76 > [ 106.245551] EAX: ffffffda EBX: ffffff9c ECX: bffdcb60 EDX: 00088000 > [ 106.246651] ESI: 00000000 EDI: b7f9e000 EBP: 00088000 ESP: bffdc970 > [ 106.247706] DS: 007b ES: 007b FS: 0000 GS: 0033 SS: 007b EFLAGS: 00000246 > [ 106.248851] Modules linked in: openvswitch(E) > [ 106.249621] CR2: 00000000e8a35f3b > [ 106.250218] ---[ end trace 6a8d05679a59cda7 ]--- > > I've bisected this down to the following commit that seems to have introduced > the issue: > > commit 120645513f55a4ac5543120d9e79925d30a0156f (refs/bisect/bad) > Author: Jarno Rajahalme > Date: Fri Apr 21 16:48:06 2017 -0700 > > openvswitch: Add eventmask support to CT action. > > Add a new optional conntrack action attribute OVS_CT_ATTR_EVENTMASK, > which can be used in conjunction with the commit flag > (OVS_CT_ATTR_COMMIT) to set the mask of bits specifying which > conntrack events (IPCT_*) should be delivered via the Netfilter > netlink multicast groups. Default behavior depends on the system > configuration, but typically a lot of events are delivered. This can be > very chatty for the NFNLGRP_CONNTRACK_UPDATE group, even if only some > types of events are of interest. > > Netfilter core init_conntrack() adds the event cache extension, so we > only need to set the ctmask value. However, if the system is > configured without support for events, the setting will be skipped due > to extension not being found. > > Signed-off-by: Jarno Rajahalme > Reviewed-by: Greg Rose > Acked-by: Joe Stringer > Signed-off-by: David S. Miller Hi Juerg, the symptom, the identified breaking commit and actually all of it seems to be [1] which James, Joseph and I worked on already. I wanted to make you aware of the past context that already exists. Back then we already reverted the change, found it to be working then. Afterwards Joseph brought it up with Jarno [2] and got some patch it seems, but that (whatever change it was - I have never seen it) wasn't enough and still crashing. Then we lost traction on the case and now you had to re-debug it I'm afraid :-/ [1]: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1736390 [2]: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1736390/comments/55 > Reverting that commit from 5.0 makes the problem go away. I'm not able to > reproduce the crash on x86_64. > > ...Juerg > _______________________________________________ > dev mailing list > dev@openvswitch.org > https://mail.openvswitch.org/mailman/listinfo/ovs-dev -- Christian Ehrhardt Software Engineer, Ubuntu Server Canonical Ltd