Received: by 2002:ac0:aa62:0:0:0:0:0 with SMTP id w31-v6csp1526497ima; Sun, 21 Oct 2018 13:59:52 -0700 (PDT) X-Google-Smtp-Source: AJdET5cKk6a1tfflP+NG7/VPFx61qL39IjQHlQcR/jivSffYUd6Ux9kiLlYlEqYR2N+QEizLTGtr X-Received: by 2002:a65:57cd:: with SMTP id q13-v6mr2956212pgr.229.1540155592873; Sun, 21 Oct 2018 13:59:52 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1540155592; cv=none; d=google.com; s=arc-20160816; b=vmQ1VHjBBNToiJpqYVI1Q+1AUFAoLYAYGz60sfhqobzwWGD6ZK154C1Jrr9kCgtkMb RXJRbD7WJ8Ejm0wmzqqzls5ScQOkUGvxpbwayM/luqp1H3wJwJ9h1KCe319pDMUomEEB xppG8leZtRlQxTOlyPHgcvQXTPdt2Oe+6KEIaAhM9NM7VHEMs4mIYBGhapPLId25VLlz kVevalGWCZRosj3Oy27i0SOLa8W6t3dXXoEkuLBq9Ie+s9Vtn4xwmtPFaiNYohkad/Xg tw49/M1tfzAO6RdwggqI1qH/7hkLbIkuunI7gHQ24qg88YoCLx92HD7C98RONe9Jvfwl tdlA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:mime-version:references:in-reply-to :organization:message-id:date:subject:cc:to:from:dkim-signature; bh=2nqoFgpBiXfd1C2MfN3Jpy4HOyYJm6LJQYx02HZy45w=; b=k4x/9J1TE3hoMr3NToFPHwaHa+vo5wt0nla0r8pX2gVxd3jODEy0oiIHbBEbsTTQD1 Hb/CnzDI1pm1DFgdq/UHJ7QsqBHdO80uOD9XeJ/7mAYIuC1In0IOZglCQN+SSbeZ2Ruo Ip8/6LOyfZNE002LYssKTaGJUuOlQogM7TrSQHcL9q3sNt7WgK1r5KqKK+uqxFRViyrw xGA8jm6SZSnYsaTORoNUKSoIE241vghSyDn2EkMCSdK97dUTKn/bhvoeyFuLhFnIGVsE Yu1U2NnW2h9dmbUyYKiNMAUCNUvI/EN7ahQdl/+cp9HfNCIYe1kwWi1LNXQgBS4wQ5If o85w== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kdab.com header.s=dkim header.b=auolbXg9; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=kdab.com Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id p1-v6si28696093plb.10.2018.10.21.13.59.37; Sun, 21 Oct 2018 13:59:52 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@kdab.com header.s=dkim header.b=auolbXg9; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=kdab.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727422AbeJVErx (ORCPT + 99 others); Mon, 22 Oct 2018 00:47:53 -0400 Received: from mail.kdab.com ([176.9.126.58]:20007 "EHLO mail.kdab.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726188AbeJVErx (ORCPT ); Mon, 22 Oct 2018 00:47:53 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kdab.com; h= content-type:content-type:mime-version:references:in-reply-to :organization:message-id:date:date:subject:subject:from:from; s= dkim; t=1540153936; x=1541017937; bh=2nqoFgpBiXfd1C2MfN3Jpy4HOyY Jm6LJQYx02HZy45w=; b=auolbXg98sDsj2Pts0ebhbm1WdYysokA8LLOKQfsdS/ gLzkI4/xvnWoi8JDbHajwrQymvCnJvD2dQQUP140MozpNuTW74eKrtoj4AaP9zjH prxMPri3GPgdSm06mndV/lnPwztOV8YfeUG4tNzkbkDs1P+pE2+3/We+vs3f87CQ = X-Virus-Scanned: amavisd-new at kdab.com From: Milian Wolff To: Milian Wolff Cc: linux-kernel@vger.kernel.org, Jiri Olsa , namhyung@kernel.org, linux-perf-users@vger.kernel.org, Arnaldo Carvalho Subject: Re: Broken dwarf unwinding - wrong stack pointer register value? Date: Sun, 21 Oct 2018 22:32:10 +0200 Message-ID: <1759553.TYpvCCURxk@agathebauer> Organization: KDAB In-Reply-To: <2335309.gnWok9HYb4@agathebauer> References: <2335309.gnWok9HYb4@agathebauer> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="nextPart1699730.aQPqxBsjS8"; micalg="sha256"; protocol="application/pkcs7-signature" Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org --nextPart1699730.aQPqxBsjS8 Content-Transfer-Encoding: 7Bit Content-Type: text/plain; charset="us-ascii" On Sonntag, 21. Oktober 2018 00:39:51 CEST Milian Wolff wrote: > Hey all, > > I'm on the quest to figure out why perf regularly fails to unwind (some) > samples. I am seeing very strange behavior, where an apparently wrong stack > pointer value is read from the register - see below for more information and > the end of this (long) mail for my open questions. Any help would be > greatly appreciated. > > I am currently using this trivial C++ code to reproduce the issue: > > ``` > #include > #include > #include > #include > > using namespace std; > > int main() > { > uniform_real_distribution uniform(-1E5, 1E5); > default_random_engine engine; > double s = 0; > for (int i = 0; i < 10000000; ++i) { > s += norm(complex(uniform(engine), uniform(engine))); > } > cout << s << '\n'; > return 0; > } > ``` > > I compile it with `g++ -O2 -g` and then record it with `perf record --call- > graph dwarf`. Using perf script, I then see e.g.: With my patch to regularly flush the perf script output buffer, we can now easily find all broken backtraces and the corresponding debug output via: $ perf script --ns -v |& awk -v RS='' '/\[unknown\]/ {print "\n"$0}' I've pasted the output to the above command from my machine here: https://paste.kde.org/pmyxwkk1k This contains 139 samples with broken unwinding, out of 2350 samples in total, so about 6% of all samples are broken. In many cases, the first accessed memory is 0 because a too-low offset into the stack is computed from the SP value, similar to the scenario I described in my initial mail. In other cases we read garbadge addresses such as unwind: access_mem addr 0x7ffc80811cf0 val 408195dfbda90580, offset 24 In all cases except for the the two samples at the very start and end of this log, the last offset encountered in access_mem is lower than 72. Remember what I wrote in the initial mail - if I manually hack the access_mem function to use 72 for one of the broken samples, it made unwinding magically work again... With this addition of data - can anyone sched some light on what's potentially going on here? How can we improve this situation? Thanks -- Milian Wolff | milian.wolff@kdab.com | Senior Software Engineer KDAB (Deutschland) GmbH, a KDAB Group company Tel: +49-30-521325470 KDAB - The Qt, C++ and OpenGL Experts --nextPart1699730.aQPqxBsjS8 Content-Type: application/pkcs7-signature; name="smime.p7s" Content-Disposition: attachment; filename="smime.p7s" Content-Transfer-Encoding: base64 MIAGCSqGSIb3DQEHAqCAMIACAQExDzANBglghkgBZQMEAgEFADCABgkqhkiG9w0BBwEAAKCCDEIw ggXmMIIDzqADAgECAhBqm+E4O/8ra58B1dm4p1JWMA0GCSqGSIb3DQEBDAUAMIGFMQswCQYDVQQG EwJHQjEbMBkGA1UECBMSR3JlYXRlciBNYW5jaGVzdGVyMRAwDgYDVQQHEwdTYWxmb3JkMRowGAYD VQQKExFDT01PRE8gQ0EgTGltaXRlZDErMCkGA1UEAxMiQ09NT0RPIFJTQSBDZXJ0aWZpY2F0aW9u IEF1dGhvcml0eTAeFw0xMzAxMTAwMDAwMDBaFw0yODAxMDkyMzU5NTlaMIGXMQswCQYDVQQGEwJH QjEbMBkGA1UECBMSR3JlYXRlciBNYW5jaGVzdGVyMRAwDgYDVQQHEwdTYWxmb3JkMRowGAYDVQQK ExFDT01PRE8gQ0EgTGltaXRlZDE9MDsGA1UEAxM0Q09NT0RPIFJTQSBDbGllbnQgQXV0aGVudGlj YXRpb24gYW5kIFNlY3VyZSBFbWFpbCBDQTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEB AL6znlesKHZ1QBbHOAOY08YYdiFQ8yV5C0y1oNF9Olg+nKcxLqf2NHbZhGra0D00SOTq9bus3/mx gUsg/Wh/eXQ0pnp8tZ8XZWAnlyKMpjL+qUByRjXCA6RQyDMqVaVUkbIr5SU0RDX/kSsKwer3H1pT /HUrBN0X8sKtPTdGX8XAWt/VdMLBrZBlgvnkCos+KQWWCo63OTTqRvaq8aWccm+KOMjTcE6s2mj6 RkalweyDI7X+7U5lNo6jzC8RTXtVV4/Vwdax720YpMPJQaDaElmOupyTf1Qib+cpukNJnQmwygjD 8m046DQkLnpXNCAGjuJy1F5NATksUsbfJAr7FLUCAwEAAaOCATwwggE4MB8GA1UdIwQYMBaAFLuv fgI9+qbxPISOre44mOzZMjLUMB0GA1UdDgQWBBSCr2yM+MX+lmF86B89K3FIXsSLwDAOBgNVHQ8B Af8EBAMCAYYwEgYDVR0TAQH/BAgwBgEB/wIBADARBgNVHSAECjAIMAYGBFUdIAAwTAYDVR0fBEUw QzBBoD+gPYY7aHR0cDovL2NybC5jb21vZG9jYS5jb20vQ09NT0RPUlNBQ2VydGlmaWNhdGlvbkF1 dGhvcml0eS5jcmwwcQYIKwYBBQUHAQEEZTBjMDsGCCsGAQUFBzAChi9odHRwOi8vY3J0LmNvbW9k b2NhLmNvbS9DT01PRE9SU0FBZGRUcnVzdENBLmNydDAkBggrBgEFBQcwAYYYaHR0cDovL29jc3Au Y29tb2RvY2EuY29tMA0GCSqGSIb3DQEBDAUAA4ICAQB4XLKBKDRPPO5fVs6fl1bsj6JrF/bz9kkI BtTYLzXN30D+03Hj6OxCDBEaIeNmsBhrJmuubvyE7HtoSmR809AgcYboW+rcTNZ/8u/Hv+GTrNI/ AhqX2/kiQNxmgUPt/eJPs92Qclj0HnVyy9TnSvGkSDU7I5Px+TbO+88G4zipA2psZaWeEykgzClZ lPz1FjTCkk77ZXp5cQYYexE6zeeN4/0OqqoAloFrjAF4o50YJafX8mnahjp3I2Y2mkjhk0xQfhNq bzlLWPoT3m7j7U26u7zg6swjOq8hITYc3/np5tM5aVyu6t99p17bTbY7+1RTWBviN9YJzK8HxzOb XYWBf/L+VGOYNsQDTxAk0Hbvb1j6KjUhg7fO294F29QIhhmiNOr84JHoy+fNLpfvYc/Q9EtFOI5I SYgOxLk3nD/whbUe9rmEQXLp8MB933Ij474gwwCPUpwv9mj2PMnXoc7mbrS22XUSeTwxCTP9bcmU dp4jmIoWfhQm7X9w/Zgddg+JZ/YnIHOwsGsaTUgj7fIvxqith7DoJC91WJ8Lce3CVJqb1XWeKIJ8 4F7YLXZN0oa7TktYgDdmQVxYkZo1c5noaDKH9Oq9cbm/vOYRUM1cWcef20Wkyk5S/GFyyPJwG0fR 1nRas3DqAf4cXxMiEKcff7PNa4M3RGTqH0pWR8p6EjCCBlQwggU8oAMCAQICEAf6KCF9+1doL2oE OTPysLwwDQYJKoZIhvcNAQELBQAwgZcxCzAJBgNVBAYTAkdCMRswGQYDVQQIExJHcmVhdGVyIE1h bmNoZXN0ZXIxEDAOBgNVBAcTB1NhbGZvcmQxGjAYBgNVBAoTEUNPTU9ETyBDQSBMaW1pdGVkMT0w OwYDVQQDEzRDT01PRE8gUlNBIENsaWVudCBBdXRoZW50aWNhdGlvbiBhbmQgU2VjdXJlIEVtYWls IENBMB4XDTE3MDUyMzAwMDAwMFoXDTIwMDUyMjIzNTk1OVowggFZMQswCQYDVQQGEwJTRTEPMA0G A1UEERMGNjgzIDMxMRIwEAYDVQQIEwlWYWVybWxhbmQxEDAOBgNVBAcTB0hhZ2ZvcnMxGDAWBgNV BAkTD05vcnJpbmdzIHZhZWcgMjEPMA0GA1UEEhMGQm94IDMwMSYwJAYDVQQKDB1LbGFyw6RsdmRh bGVucyBEYXRha29uc3VsdCBBQjEdMBsGA1UECxMUQSBLREFCIEdyb3VwIENvbXBhbnkxQzBBBgNV BAsMOklzc3VlZCB0aHJvdWdoIEtsYXLDpGx2ZGFsZW5zIERhdGFrb25zdWx0IEFCIEUtUEtJIE1h bmFnZXIxHzAdBgNVBAsTFkNvcnBvcmF0ZSBTZWN1cmUgRW1haWwxFTATBgNVBAMTDE1pbGlhbiBX b2xmZjEkMCIGCSqGSIb3DQEJARYVbWlsaWFuLndvbGZmQGtkYWIuY29tMIIBIjANBgkqhkiG9w0B AQEFAAOCAQ8AMIIBCgKCAQEAxrzfNBVvRbiAknuTBXuQnNm9sLIFLo0vbPB6kswk78A3tA++Zn5c lQUHhGlQq1cdYxagnUpqwvG3Sod15mPSOLkAPf/mabLN7p+lFbRaUP+97ZkTZtvb4BCC3osIEFI4 G393OSFWqc2qmIPE/SwSASbAA20Fcaa2M6P1lhOk/ttUh2jIurTPF0wUycIA7lBddrOgaOA8e2m6 iLTNHtlrfRbBaUX91D5ebY+UWmIjXSQ9+CtutMzBkwnF0rZKririvOkklg9VzEGNQVHrQfDF2s/U pOtmtuVSwElauGT/KALyCFuIrYC1pmaKH8S1xODJqiRaf6jH8E+KQzKjyM/ErwIDAQABo4IB1TCC AdEwHwYDVR0jBBgwFoAUgq9sjPjF/pZhfOgfPStxSF7Ei8AwHQYDVR0OBBYEFN+m99RtIuA1bSdw 6b1brOX7X3AJMA4GA1UdDwEB/wQEAwIFoDAMBgNVHRMBAf8EAjAAMB0GA1UdJQQWMBQGCCsGAQUF BwMEBggrBgEFBQcDAjBGBgNVHSAEPzA9MDsGDCsGAQQBsjEBAgEDBTArMCkGCCsGAQUFBwIBFh1o dHRwczovL3NlY3VyZS5jb21vZG8ubmV0L0NQUzBaBgNVHR8EUzBRME+gTaBLhklodHRwOi8vY3Js LmNvbW9kb2NhLmNvbS9DT01PRE9SU0FDbGllbnRBdXRoZW50aWNhdGlvbmFuZFNlY3VyZUVtYWls Q0EuY3JsMIGLBggrBgEFBQcBAQR/MH0wVQYIKwYBBQUHMAKGSWh0dHA6Ly9jcnQuY29tb2RvY2Eu Y29tL0NPTU9ET1JTQUNsaWVudEF1dGhlbnRpY2F0aW9uYW5kU2VjdXJlRW1haWxDQS5jcnQwJAYI KwYBBQUHMAGGGGh0dHA6Ly9vY3NwLmNvbW9kb2NhLmNvbTAgBgNVHREEGTAXgRVtaWxpYW4ud29s ZmZAa2RhYi5jb20wDQYJKoZIhvcNAQELBQADggEBABf47LSJADqH+ow9INv3QM1NC/qq2bjxGvsZ 68iD11VEUAFlsYfsVTgQqUirwPVTYenXtwVBELHZyywsui1JxL7HKQetLQegDDP/RyfjReVaWxhy 3OpuItsgLVbru9QVgPifnoBFPtfZcwjeJDmeSbLT8oj4Rd0KYBOIve7WKvsfNPsNwfbLwY2zILkE LjxZcVi2AwZHDyab+dzL/3YcLuJj1lSawBGn7ilpcdZydlv4aye51pD/MemLIYLcylt+ImrmjnTV y+QlAHRF3s5FE8yAr+W1MBD/1bKZCSgFt8VQoAlz3hiQh8QqZp4Zl8WuVL4+mP/mT6VDEWgq/0Bo cukxggJuMIICagIBATCBrDCBlzELMAkGA1UEBhMCR0IxGzAZBgNVBAgTEkdyZWF0ZXIgTWFuY2hl c3RlcjEQMA4GA1UEBxMHU2FsZm9yZDEaMBgGA1UEChMRQ09NT0RPIENBIExpbWl0ZWQxPTA7BgNV BAMTNENPTU9ETyBSU0EgQ2xpZW50IEF1dGhlbnRpY2F0aW9uIGFuZCBTZWN1cmUgRW1haWwgQ0EC EAf6KCF9+1doL2oEOTPysLwwDQYJYIZIAWUDBAIBBQCggZMwGAYJKoZIhvcNAQkDMQsGCSqGSIb3 DQEHATAcBgkqhkiG9w0BCQUxDxcNMTgxMDIxMjAzMjEwWjAoBgkqhkiG9w0BCQ8xGzAZMAsGCWCG SAFlAwQBAjAKBggqhkiG9w0DBzAvBgkqhkiG9w0BCQQxIgQgi6SnSiVUsVREGpMm185m2vfUTcMl Hjt7PZW/ERqHW8kwDQYJKoZIhvcNAQEBBQAEggEATsHQuc7KIzzFlSeWrfCKp4Eud223MWci3gww t29c/odb+uxFgcEOEJzCdk+yCll8y15BKOilgTpyv7vjDjfGy/oSaeXtZ75J/oG51zNPezKJd1U9 VocAUIrYq1LpefTblB9GM44EHeeNhszfl0oUKeXHzQLP0PU6qPOwfhygly3/s3url6n6KfyiWaVH j8dRNXXN4ZgXAUDZNUfFKvSr2nMx0hrUSA6DojjP9Gq9ejzfHefzsL6425afda5kmoT3gibtSmuf afcUmmUUFXiDdbYssaFThN/L+Fgj3buT+i0jG1ARRqmApoezvaZf+WEBg21S+CE32P+6npMa2w8d CQAAAAAAAA== --nextPart1699730.aQPqxBsjS8--