Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67;
From:   "Harris, Robert" <robert.harris@alertlogic.com>
To:     Mikael Pettersson <mikpelinux@gmail.com>
CC:     "tglx@linutronix.de" <tglx@linutronix.de>,
        "mingo@redhat.com" <mingo@redhat.com>,
        "peterz@infradead.org" <peterz@infradead.org>,
        "dvhart@infradead.org" <dvhart@infradead.org>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        "Harris, Robert" <robert.harris@alertlogic.com>
Subject: Re: Help requested: futex(..., FUTEX_WAIT_PRIVATE, ...) returns EPERM
Thread-Topic: Help requested: futex(..., FUTEX_WAIT_PRIVATE, ...) returns
 EPERM
Thread-Index: AQHVmYBWfg0+aL4WQEeFFOC3OWpiUqeJGdEAgAA7yYA=
Date:   Wed, 13 Nov 2019 17:03:44 +0000
Message-ID: <B5EC0909-54CE-47D6-8930-8C9CFC243180@alertlogic.com>
References: <E0332978-739B-4546-9C3F-975216C349D2@alertlogic.com>
 <CAM43=SP-CTHWdMCJwioUiEVSNnh-AgZj7YEK1i08TXHk3oCbLQ@mail.gmail.com>
In-Reply-To: <CAM43=SP-CTHWdMCJwioUiEVSNnh-AgZj7YEK1i08TXHk3oCbLQ@mail.gmail.com>
Accept-Language: en-GB, en-US
Content-Language: en-US
received-spf: None (protection.outlook.com: alertlogic.com does not designate
 permitted sender hosts)
Content-Type: text/plain; charset="us-ascii"
Content-ID: <44DAA7D80C967C44BD812C265C9D12AE@namprd20.prod.outlook.com>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-MS-Exchange-CrossTenant-Network-Message-Id: 5d148f9b-0b47-4f16-12fa-08d7685b7153
X-MS-Exchange-CrossTenant-originalarrivaltime: 13 Nov 2019 17:03:44.5368
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 04151827-cb2a-4231-9c24-1ef5ffc408eb
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: tWVgwPrYFwNcYZ/m2Sco9Waxx2vkrXSPO3MuWvnfDTPOW5saXGFF9I+7pCUrY1eONKVvN5d8EjLeNipE2Al7U1fjmMAbFBnOT4tV0p0T+Qk=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BYAPR20MB2664
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org


> On 13 Nov 2019, at 13:29, Mikael Pettersson <mikpelinux@gmail.com> wrote:
>
> On Tue, Nov 12, 2019 at 6:43 PM Harris, Robert
> <robert.harris@alertlogic.com> wrote:
>>
>> I am investigating an issue on 4.9.184 in which futex() returns EPERM
>> intermittently for
>>
>> futex(uaddr, FUTEX_WAIT_PRIVATE, val, &timeout, NULL, 0)
>>
>> The failure affects an application in an AWS lambda;  traditional
>> debugging approaches vary from difficult to impossible.  I cannot
>> reproduce the problem at will, instrument the kernel, install a new
>> kernel or get an application core dump.
>>
>> Understanding the circumstances under which EPERM can be returned for
>> FUTEX_WAIT_PRIVATE would be useful but it is not a documented failure
>> mode.  I have spent some time looking through futex.c but have not
>> found anything yet.  I would be grateful for a hint from someone more
>> knowledgeable.
>
>
> I just wanted to add that a colleague of mine reported the exact same
> issue to me two days ago: a highly threaded application (the Erlang
> VM) running in AWS lambda, futex wait calls occasionally failing with
> EPERM.  I don't have more specifics than that, I've asked for kernel
> version and the exact parameters in the failed futex call.

Thanks, that's a great data point.  One of my outstanding questions had
been "why does this happen to only us?"

When I look at the timings I can say with some confidence that the
problem stopped for us minutes after

2017 on 2019-10-23 in us-east-1
2030 on 2019-10-24 in eu-west-1
1817 on 2019-10-25 in us-west-2

(all times UTC).  I've logged a ticket with Amazon to find out what
changed.

Robert
Confidentiality Notice | This email and any included attachments may be pri=
vileged, confidential and/or otherwise protected from disclosure. Access to=
 this email by anyone other than the intended recipient is unauthorized. If=
 you believe you have received this email in error, please contact the send=
er immediately and delete all copies. If you are not the intended recipient=
, you are notified that disclosing, copying, distributing or taking any act=
ion in reliance on the contents of this information is strictly prohibited.