Received: by 2002:ab2:7855:0:b0:1f9:5764:f03e with SMTP id m21csp818148lqp; Thu, 23 May 2024 00:16:43 -0700 (PDT) X-Forwarded-Encrypted: i=3; AJvYcCWXWpo0OGEcXwMYdabN/LgHM0XJ0NbZqMPeM3/XjLAdppKOKULvcp2mY83rbgkdRmBWZwTcKpXFcA+aXRz5xNhK8QbDCffk39GsoWV6lA== X-Google-Smtp-Source: AGHT+IG2aVoHIVJi6q8v3aWi288E+nKCgf5Ydu9cVrqHk6ndw0M/d0d06JFSeegbVREn8y0kK/sK X-Received: by 2002:a05:6871:284:b0:22a:582a:9bcc with SMTP id 586e51a60fabf-24c68e102c7mr4496694fac.53.1716448602503; Thu, 23 May 2024 00:16:42 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1716448602; cv=pass; d=google.com; s=arc-20160816; b=xhBqrAPIkZMbb3cDw+CtL/HOnTylapeSuNF/5zemxu0q1iu+fHbU70I+P16GtOJtA8 b6bFq793VaxesohI9d3k6Petp6DqqU567em8071kQIJHdo64ExRXylpsi4W44XHXK61l gcKXV7wL5B6ms+twkPrB5J9Xc5Ku/Drz1cgIJBKvfg5LuadGXSavqUIp0otXxS2+agfK 1AdNCTFY4r+i7E/EFfcC9IzAIrKLSfv3tHHmR7kPTBbDqBT6vWLrnjbSxSLeKyGRdfQh OSSTzQqcMD5JIGM1oDRtCoOghIigHwHe6phiqwsSffbbqt9209UCPl08Sh1ARSR/dGzT IHow== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=message-id:subject:to:from:date:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:dkim-signature:dkim-signature; bh=dSXB6XVopQfpF06UChX8vsbXPHdH3jhN8uMYyjb2ldA=; fh=x36rUQw0ca73eBKj6n3hCnyrpYqMle/S3XU2tBuMpJE=; b=a/d3G1TrQq+7EKsoaVYWvfQBxSFyskiaLQUHPE6fpK3fR88fa9ZhSNvG3ayvBgufKb wcSgCYdC+HhyQJOwDpV22DvS1eW969xbhi7oEfY83L8THz43RZsW0uz9gFQY3gKHN2Cl OXydl3wcnXCnK68QCOVLf3FhMcrXrl2DzHJtpVNCwWbHr7TTc+GunG301Iv70Huj/97G H9WgaS5T1I5b9GXjptRA3AUKUrlmuk3e5IGlMNmHmV/CwyebBX+xd3vTfD3OOPOhMOlp rTAWk0TNgHoJqx+0VDLhE03tThVJDMbuVbrF1bMr/uXAcnIIoBbIHir0C44XgSqAcCK6 pXTA==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@kojedz.in header.s=s02 header.b=pJcA63yO; dkim=pass header.i=@kojedz.in header.s=s01 header.b="oRv/eOie"; arc=pass (i=1 spf=pass spfdomain=kojedz.in dkim=pass dkdomain=kojedz.in dkim=pass dkdomain=kojedz.in dmarc=pass fromdomain=kojedz.in); spf=pass (google.com: domain of linux-nfs+bounces-3348-linux.lists.archive=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) smtp.mailfrom="linux-nfs+bounces-3348-linux.lists.archive=gmail.com@vger.kernel.org"; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=kojedz.in Return-Path: Received: from sv.mirrors.kernel.org (sv.mirrors.kernel.org. [139.178.88.99]) by mx.google.com with ESMTPS id d2e1a72fcca58-6f4d2b4fa7asi4138890b3a.387.2024.05.23.00.16.42 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 23 May 2024 00:16:42 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-nfs+bounces-3348-linux.lists.archive=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) client-ip=139.178.88.99; Authentication-Results: mx.google.com; dkim=pass header.i=@kojedz.in header.s=s02 header.b=pJcA63yO; dkim=pass header.i=@kojedz.in header.s=s01 header.b="oRv/eOie"; arc=pass (i=1 spf=pass spfdomain=kojedz.in dkim=pass dkdomain=kojedz.in dkim=pass dkdomain=kojedz.in dmarc=pass fromdomain=kojedz.in); spf=pass (google.com: domain of linux-nfs+bounces-3348-linux.lists.archive=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) smtp.mailfrom="linux-nfs+bounces-3348-linux.lists.archive=gmail.com@vger.kernel.org"; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=kojedz.in Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sv.mirrors.kernel.org (Postfix) with ESMTPS id D936228101F for ; Thu, 23 May 2024 07:16:41 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 7AFD13B29D; Thu, 23 May 2024 07:16:39 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kojedz.in header.i=@kojedz.in header.b="pJcA63yO"; dkim=pass (3072-bit key) header.d=kojedz.in header.i=@kojedz.in header.b="oRv/eOie" X-Original-To: linux-nfs@vger.kernel.org Received: from fw.sz-a.kwebs.cloud (fw.sz-a.kwebs.cloud [109.61.102.40]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 953BF1E535 for ; Thu, 23 May 2024 07:16:30 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=109.61.102.40 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1716448599; cv=none; b=UfrIe+Jgbxn1fen3rb8Sivmfy9NvsGQ8C1exgjHC5CmoB1+Y1C/b6Ij8Cz3GY3D1PShQZgJQbOenniymcNTbN+sxkgXX9+d8pkuLTtRwXiIFswhQ86lqxcVg3xK96YD6w0hcBRBVIvMS6yiOGs97y3YxkKNXLJyid3JivCz4mF0= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1716448599; c=relaxed/simple; bh=roOI3dQFlQq1ia/T7ncad57FUGzOYdfa7E6ZNdxE3og=; h=MIME-Version:Date:From:To:Subject:Message-ID:Content-Type; b=O5UP33CJDO99nXJxys5fgINo+qqV7Pi65gQvbLjJV/4tRvHZyRvbGXr5NS/3/aa7iP3E2UPHnXlnkWPG8cUE2kjqjwpEWyF4qOoqkbzLCue24iZPPK05pcUDDuRah6pTFfSbCc8nNXi6i+WedWISwhEVOywo1+B4weq27KYD3do= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=kojedz.in; spf=pass smtp.mailfrom=kojedz.in; dkim=pass (2048-bit key) header.d=kojedz.in header.i=@kojedz.in header.b=pJcA63yO; dkim=pass (3072-bit key) header.d=kojedz.in header.i=@kojedz.in header.b=oRv/eOie; arc=none smtp.client-ip=109.61.102.40 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=kojedz.in Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kojedz.in Received: from webmail.srv.kwebs.cloud (172-16-36-102.prometheus-node-exporter.monitoring.svc.cluster.local [172.16.36.102]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1) server-digest SHA384) (No client certificate requested) (Authenticated sender: richard@kojedz.in) by mx.kwebs.cloud (Postfix) with ESMTPSA id BD373DED; Thu, 23 May 2024 07:10:33 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kojedz.in; s=s02; t=1716448233; bh=dSXB6XVopQfpF06UChX8vsbXPHdH3jhN8uMYyjb2ldA=; h=Date:From:To:Subject; b=pJcA63yOgf7gy39XUpBYeaMhvCd2nD5BJlioHN3kQ0jF/DbIrwuLt8W3D3KZthD2M VbS98AjaQnoybb+XmXvD4OgHibTpViXyDjMRiegA58euRYBubQ589ed6EzuooEhjcf TO3RIEpOqbRaFj+lVfVJ5iCZUYbbH0B6bi1CL+H0jEF0D0GcTpBG3lvRB2xbesYpjR 6Ol3QsDWw96QX2zOmMVNFl4jMCyb2honjDHyexPDcuAdYzD9bKGMfT2p+df0SRUExi aFsn2KKj7+/Eezlan2n4vZMD26bEeWGF/IpSbpk5VI6T4IR+nAvnFm7WAr5ywM+aAi uH+RPWzNJziPA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kojedz.in; s=s01; t=1716448233; bh=dSXB6XVopQfpF06UChX8vsbXPHdH3jhN8uMYyjb2ldA=; h=Date:From:To:Subject; b=oRv/eOiel+Ggv4Hmfp9cDXLnas9tmxxzFFyMI+VTHpDhQNn4DaNzmGnnrRqho7j3A jqJp5Wnj/s3zmeTDpChQBdsp4Hw7cU4cwjdJPlEbUcey1ACB3r6eIYfM4p+tZBMvdN vpirNupZ/CmX9Lvz0MDw/pXQ69qWmKI1c03U8JvPk91kKKhzRkE8b+vTfMNylox6HX axxJMl7fHMMxic4EkZlDP5TbC+UX5BNY5+LbxywvegPpH9pGkhV83tZopqUMEVyVE3 VJcSPoR2l2+TBcomSh9LQzwvAjUyU1j2shBlhRYFKQHpx9UZ5IR9i6wbD1qB4Z8B2V 55gLr6WjAqvUwfJ7dQ7tfTeOGvVZnbTFze2+NO3VIJHz9tmDyOsc2Z1PxBqW8yN0WH Szif6mCZtiggca3EqGiPJdYyLpnHr/+/0Hp5XRIomXjRgQCZnaNy0OTegp54t2z8m/ IcQQ/SmYMI1U/FoygGKTjtTuZq1xubBQlVJNUmOSJLgwX5zw+GG Precedence: bulk X-Mailing-List: linux-nfs@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Date: Thu, 23 May 2024 09:10:33 +0200 From: Richard Kojedzinszky To: linux-nfs@vger.kernel.org, 1071501@bugs.debian.org Subject: Linux NFS client hangs in nfs4_lookup_revalidate Message-ID: <0473c552b6fd8e96ef2ffbf0435a7552@kojedz.in> X-Sender: richard+debian+bugreport@kojedz.in Content-Type: multipart/mixed; boundary="=_32bfc5d800302542897cc12b3c8180fc" --=_32bfc5d800302542897cc12b3c8180fc Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset=US-ASCII; format=flowed Dear NFS developers, I am running multiple PODs on a Kubernetes node, they all mount different NFS shares from the same nfs server. I started to notice hangups in my dovecot process after I switched to Debian's kernel from upstream 5.15. You can find Debian bugreport at https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1071501. So, effectively I am running dovecot in Kubernetes, and dovecot's data directory is accessed over NFS. Eventually one dovecot process stucks in nfs4_lookup_revalidate(). From that point, that process cannot be killed, howewer, other processes can access NFS as normal. Also, another dovecot process running on the very same node accessing the same NFS share works too. Now, I am still in the process of bisecting, howewer, I cannot reliably trigger the bug. Originally it took a few days after I've noticed a hanging process. Now I am trying to mimic file operations what dovecot does in a faster way. Now it seems that it triggers the bug in a few hours, howewer, during bisects, I can still make mistakes. I've scheduled many of my applications which use NFS shares to the same node, to have more NFS load on that node. I am attaching my simple app which triggers the bug in a few hours, at least in my lab. I have two dedicated NFS shares for this test case, and I am running 3 instances of the applications for both shares. Also, I am running other production applications on the same node which also use NFS, howewer, I dont experience lockups with them. They are librenms, prometheus, and a docker private registry. This way I dont know if running the attached app only is enough to trigger the bug. Once I have a suspectible commit based on my bisecting process, I will report it here. My NFS server is a TrueNAS, based on FreeBSD 13.3. Thanks in advance, Richard --=_32bfc5d800302542897cc12b3c8180fc Content-Transfer-Encoding: base64 Content-Type: application/x-tar; name=ds.tar Content-Disposition: attachment; filename=ds.tar; size=10240 ZG92ZWNvdC1zaW11bGF0b3IvAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAADAwMDA3NTUAMDAwMTc1 MQAwMDAxNzUxADAwMDAwMDAwMDAwADE0NjIzNTY0NzM2ADAxNDEyNAAgNQAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAB1c3RhciAgAGtyaWNoeQAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAa3JpY2h5AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAABk b3ZlY290LXNpbXVsYXRvci9tYWluLmdvAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAMDAwMDY0NAAwMDAxNzUx ADAwMDE3NTEAMDAwMDAwMDIzNzEAMTQ2MjM1NjQ3MzYAMDE1NDAyACAwAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAHVzdGFyICAAa3JpY2h5AAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAABrcmljaHkAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAHBh Y2thZ2UgbWFpbgoKaW1wb3J0ICgKCSJjb250ZXh0IgoJImxvZyIKCSJuZXQvaHR0cCIKCSJvcyIK CSJvcy9zaWduYWwiCgkic3luYyIKCSJzeXNjYWxsIgoKCSJnaXRodWIuY29tL3Byb21ldGhldXMv Y2xpZW50X2dvbGFuZy9wcm9tZXRoZXVzIgoJImdpdGh1Yi5jb20vcHJvbWV0aGV1cy9jbGllbnRf Z29sYW5nL3Byb21ldGhldXMvcHJvbWh0dHAiCikKCmNvbnN0ICgKCWxvY2tGaWxlICAgID0gIm1h aWwubG9jayIKCWxpc3RGaWxlICAgID0gIm1haWwubGlzdCIKCWxpc3RGaWxlTmV3ID0gIm1haWwu bGlzdC5uZXciCikKCnZhciAoCgltZXRyaWNzID0gcHJvbWV0aGV1cy5OZXdIaXN0b2dyYW1WZWMo cHJvbWV0aGV1cy5IaXN0b2dyYW1PcHRzewoJCVN1YnN5c3RlbTogImRvdmVjb3RzaW0iLAoJCU5h bWU6ICAgICAgIm9wZXJhdGlvbnMiLAoJCUhlbHA6ICAgICAgIlNpbXVsYXRlZCBvcGVhdGlvbnMi LAoJfSwgW11zdHJpbmd7InBhdGgiLCAib3AifSkKCglybWV0cmljcyBwcm9tZXRoZXVzLk9ic2Vy dmVyCgl3bWV0cmljcyBwcm9tZXRoZXVzLk9ic2VydmVyCikKCmZ1bmMgbWFpbigpIHsKCXByb21l dGhldXMuTXVzdFJlZ2lzdGVyKG1ldHJpY3MpCgoJcGF0aCA6PSBvcy5BcmdzWzFdCglpZiBlcnIg Oj0gb3MuQ2hkaXIocGF0aCk7IGVyciAhPSBuaWwgewoJCWxvZy5GYXRhbChlcnIpCgl9CgoJcm1l dHJpY3MgPSBtZXRyaWNzLldpdGhMYWJlbFZhbHVlcyhwYXRoLCAicmVhZCIpCgl3bWV0cmljcyA9 IG1ldHJpY3MuV2l0aExhYmVsVmFsdWVzKHBhdGgsICJ3cml0ZSIpCgoJY3R4LCBjYW5jZWwgOj0g Y29udGV4dC5XaXRoQ2FuY2VsKGNvbnRleHQuQmFja2dyb3VuZCgpKQoJZGVmZXIgY2FuY2VsKCkK CglnbyBmdW5jKCkgewoJCXNpZ2NoYW4gOj0gbWFrZShjaGFuIG9zLlNpZ25hbCwgMSkKCQlzaWdu YWwuTm90aWZ5KHNpZ2NoYW4sIHN5c2NhbGwuU0lHSU5ULCBzeXNjYWxsLlNJR1RFUk0pCgoJCTwt c2lnY2hhbgoKCQlsb2cuUHJpbnQoIkV4aXRpbmciKQoKCQljYW5jZWwoKQoJfSgpCgoJd2cgOj0g JnN5bmMuV2FpdEdyb3Vwe30KCXdnLkFkZCgxKQoJZ28gZnVuYygpIHsKCQlkZWZlciB3Zy5Eb25l KCkKCgkJdyA6PSAmd3JpdGVye30KCgkJdy5ydW4oY3R4KQoJfSgpCgoJd2cuQWRkKDEpCglnbyBm dW5jKCkgewoJCWRlZmVyIHdnLkRvbmUoKQoKCQl3IDo9ICZyZWFkZXJ7fQoKCQl3LnJ1bihjdHgp Cgl9KCkKCglnbyBodHRwLkxpc3RlbkFuZFNlcnZlKCI6OTA5MCIsIHByb21odHRwLkhhbmRsZXIo KSkKCgl3Zy5XYWl0KCkKfQoAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAGRvdmVj b3Qtc2ltdWxhdG9yL3JlYWRlci5nbwAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAwMDAwNjQ0ADAwMDE3NTEAMDAw MTc1MQAwMDAwMDAwMjcxMgAxNDYyMzU2NDY3MwAwMTU3MTcAIDAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAdXN0YXIgIABrcmljaHkAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAGtyaWNoeQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAcGFja2Fn ZSBtYWluCgppbXBvcnQgKAoJImJ1ZmlvIgoJImNvbnRleHQiCgkibG9nIgoJIm1hdGgvcmFuZCIK CSJvcyIKCSJzdHJpbmdzIgoJInRpbWUiCikKCnR5cGUgcmVhZGVyIHN0cnVjdCB7CglsICAgICAg ICAgICAqbG9ja2ZpbGUKCWxhc3RNb2RUaW1lIHRpbWUuVGltZQp9CgpmdW5jIChyICpyZWFkZXIp IHJ1bihjdHggY29udGV4dC5Db250ZXh0KSB7CglyLmwgPSAmbG9ja2ZpbGV7ZmlsZTogbG9ja0Zp bGV9Cgl0aWNrZXIgOj0gdGltZS5OZXdUaWNrZXIoMTAgKiB0aW1lLk1pbGxpc2Vjb25kKQoJZGVm ZXIgdGlja2VyLlN0b3AoKQoKCWZvciB7CgkJc2VsZWN0IHsKCQljYXNlIDwtY3R4LkRvbmUoKToK CQkJcmV0dXJuCgkJY2FzZSA8LXRpY2tlci5DOgoJCX0KCgkJc3RhcnQgOj0gdGltZS5Ob3coKQoJ CWlmIGVyciA6PSByLmRvX3JlYWQoKTsgZXJyID09IG5pbCB7CgkJCXJtZXRyaWNzLk9ic2VydmUo dGltZS5TaW5jZShzdGFydCkuU2Vjb25kcygpKQoJCX0gZWxzZSB7CgkJCWxvZy5QcmludGYoImRv X3JlYWQ6ICUrdiIsIGVycikKCQl9Cgl9Cn0KCmZ1bmMgKHIgKnJlYWRlcikgZG9fcmVhZCgpIGVy cm9yIHsKCXN0YXQsIGVyciA6PSBvcy5TdGF0KGxpc3RGaWxlKQoJaWYgZXJyICE9IG5pbCB7CgkJ cmV0dXJuIGVycgoJfQoKCW1vZFRpbWUgOj0gc3RhdC5Nb2RUaW1lKCkKCWlmICFtb2RUaW1lLkFm dGVyKHIubGFzdE1vZFRpbWUpIHsKCQlyZXR1cm4gbmlsCgl9CgoJaWYgZXJyID0gci5sLkxvY2so KTsgZXJyICE9IG5pbCB7CgkJcmV0dXJuIGVycgoJfQoJZGVmZXIgci5sLlVubG9jaygpCgoJcmZo LCBlcnIgOj0gb3MuT3BlbihsaXN0RmlsZSkKCWlmIGVyciAhPSBuaWwgewoJCXJldHVybiBlcnIK CX0KCWRlZmVyIHJmaC5DbG9zZSgpCgoJd2ZoLCBlcnIgOj0gb3MuT3BlbkZpbGUobGlzdEZpbGVO ZXcsIG9zLk9fQ1JFQVRFfG9zLk9fV1JPTkxZLCAwbzYwMCkKCWlmIGVyciAhPSBuaWwgewoJCXJl dHVybiBlcnIKCX0KCWRlZmVyIHdmaC5DbG9zZSgpCgoJcnIgOj0gYnVmaW8uTmV3UmVhZGVyKHJm aCkKCXdyIDo9IGJ1ZmlvLk5ld1dyaXRlcih3ZmgpCgoJZm9yIHsKCQlmaWxlLCBfIDo9IHJyLlJl YWRTdHJpbmcoJ1xuJykKCQlpZiBmaWxlID09ICIiIHsKCQkJYnJlYWsKCQl9CgoJCWlmIHJhbmQu SW50bigxMCkgPCA0IHsgLy8ga2VlcCBmaWxlcyBhdCA0MCUgY2hhbmNlCgkJCXdyLldyaXRlU3Ry aW5nKGZpbGUpCgkJfSBlbHNlIHsKCQkJZmlsZSA9IHN0cmluZ3MuVHJpbVJpZ2h0KGZpbGUsICJc biIpCgkJCV8sIGVyciA6PSBvcy5TdGF0KGZpbGUpCgkJCWlmIGVyciA9PSBuaWwgewoJCQkJb3Mu UmVtb3ZlKGZpbGUpCgkJCX0gZWxzZSB7CgkJCQlsb2cuUHJpbnRmKCJkb19yZWFkOiBzdGF0KCVz KTogJSt2IiwgZmlsZSwgZXJyKQoJCQl9CgkJfQoJfQoKCXdyLkZsdXNoKCkKCXdmaC5DbG9zZSgp CgoJb3MuUmVuYW1lKGxpc3RGaWxlTmV3LCBsaXN0RmlsZSkKCglyZXR1cm4gbmlsCn0KAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAZG92ZWNvdC1z aW11bGF0b3Ivd3JpdGVyLmdvAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAADAwMDA2NDQAMDAwMTc1MQAwMDAxNzUx ADAwMDAwMDAyMDYwADE0NjIzNTY0NjczADAxNTc2NQAgMAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAB1c3RhciAgAGtyaWNoeQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAa3JpY2h5AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAABwYWNrYWdlIG1h aW4KCmltcG9ydCAoCgkiY29udGV4dCIKCSJmbXQiCgkibG9nIgoJIm1hdGgvcmFuZCIKCSJvcyIK CSJ0aW1lIgopCgp0eXBlIHdyaXRlciBzdHJ1Y3QgewoJbCAqbG9ja2ZpbGUKfQoKZnVuYyAodyAq d3JpdGVyKSBydW4oY3R4IGNvbnRleHQuQ29udGV4dCkgewoJdy5sID0gJmxvY2tmaWxle2ZpbGU6 IGxvY2tGaWxlfQoJdGlja2VyIDo9IHRpbWUuTmV3VGlja2VyKDEwICogdGltZS5NaWxsaXNlY29u ZCkKCWRlZmVyIHRpY2tlci5TdG9wKCkKCglmb3IgewoJCXNlbGVjdCB7CgkJY2FzZSA8LWN0eC5E b25lKCk6CgkJCXJldHVybgoJCWNhc2UgPC10aWNrZXIuQzoKCQl9CgoJCXN0YXJ0IDo9IHRpbWUu Tm93KCkKCQlpZiBlcnIgOj0gdy5kb193cml0ZSgpOyBlcnIgPT0gbmlsIHsKCQkJd21ldHJpY3Mu T2JzZXJ2ZSh0aW1lLlNpbmNlKHN0YXJ0KS5TZWNvbmRzKCkpCgkJfSBlbHNlIHsKCQkJbG9nLlBy aW50ZigiZG9fd3JpdGU6ICUrdiIsIGVycikKCQl9Cgl9Cn0KCmZ1bmMgKHcgKndyaXRlcikgZG9f d3JpdGUoKSBlcnJvciB7CgluZmlsZXMgOj0gMSArIHJhbmQuSW50bigxMCkKCglpZiBlcnIgOj0g dy5sLkxvY2soKTsgZXJyICE9IG5pbCB7CgkJcmV0dXJuIGVycgoJfQoJZGVmZXIgdy5sLlVubG9j aygpCgoJZmgsIGVyciA6PSBvcy5PcGVuRmlsZShsaXN0RmlsZSwgb3MuT19DUkVBVEV8b3MuT19X Uk9OTFl8b3MuT19BUFBFTkQsIDBvNjAwKQoJaWYgZXJyICE9IG5pbCB7CgkJcmV0dXJuIGVycgoJ fQoJZGVmZXIgZmguQ2xvc2UoKQoKCWZvciBpIDo9IDA7IGkgPCBuZmlsZXM7IGkrKyB7CgkJZm5h bWUgOj0gZm10LlNwcmludGYoImZpbGUtJWQuJWQiLCB0aW1lLk5vdygpLk5hbm9zZWNvbmQoKSwg cmFuZC5JbnQoKSkKCQlpZiBlcnIgPSBvcy5Xcml0ZUZpbGUoZm5hbWUsIFtdYnl0ZShmbmFtZSks IDBvNjAwKTsgZXJyID09IG5pbCB7CgkJCWZoLldyaXRlKFtdYnl0ZShmbXQuU3ByaW50ZigiJXNc biIsIGZuYW1lKSkpCgkJfSBlbHNlIHsKCQkJbG9nLlByaW50ZigiZG9fd3JpdGU6IHdyaXRlZmls ZTogJSt2IiwgZXJyKQoJCX0KCX0KCglyZXR1cm4gbmlsCn0KAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAABkb3ZlY290LXNpbXVs YXRvci9sb2NrZmlsZS5nbwAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAMDAwMDY0NAAwMDAxNzUxADAwMDE3NTEAMDAw MDAwMDEyMDIAMTQ2MjM1NjQ2NzMAMDE2MjM2ACAwAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAHVzdGFyICAAa3JpY2h5AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAABr cmljaHkAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAHBhY2thZ2UgbWFpbgoK aW1wb3J0ICgKCSJtYXRoL3JhbmQiCgkib3MiCgkidGltZSIKKQoKdHlwZSBsb2NrZmlsZSBzdHJ1 Y3QgewoJZmlsZSBzdHJpbmcKCWZoICAgKm9zLkZpbGUKfQoKZnVuYyAobCAqbG9ja2ZpbGUpIExv Y2soKSAoZXJyIGVycm9yKSB7CglpIDo9IDAKCWQgOj0gdGltZS5NaWxsaXNlY29uZCAqIHRpbWUu RHVyYXRpb24oKDEwICsgcmFuZC5JbnRuKDEwKSkpCgoJZm9yIHsKCQlpZiBfLCBlcnIgPSBvcy5T dGF0KGwuZmlsZSk7IGVyciAhPSBuaWwgJiYgb3MuSXNOb3RFeGlzdChlcnIpIHsKCQkJbC5maCwg ZXJyID0gb3MuT3BlbkZpbGUobC5maWxlLCBvcy5PX0NSRUFURXxvcy5PX1JEV1J8b3MuT19FWENM LCAwbzY0NCkKCQkJaWYgZXJyID09IG5pbCB7CgkJCQlyZXR1cm4KCQkJfQoJCX0gZWxzZSB7CgkJ CWVyciA9IG9zLkVyckV4aXN0CgkJfQoKCQlpKysKCQlpZiBpID09IDE1IHsKCQkJcmV0dXJuCgkJ fQoKCQl0aW1lLlNsZWVwKGQpCgkJZCArPSAxMCAqIHRpbWUuTWlsbGlzZWNvbmQKCX0KfQoKZnVu YyAobCAqbG9ja2ZpbGUpIFVubG9jaygpIChlcnIgZXJyb3IpIHsKCWlmIGwuZmggPT0gbmlsIHsK CQlyZXR1cm4KCX0KCglsLmZoLkNsb3NlKCkKCglvcy5SZW1vdmUobC5maWxlKQoKCXJldHVybgp9 CgAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA== --=_32bfc5d800302542897cc12b3c8180fc--