Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1759078AbcLPTug (ORCPT ); Fri, 16 Dec 2016 14:50:36 -0500 Received: from mx0b-00082601.pphosted.com ([67.231.153.30]:52096 "EHLO mx0a-00082601.pphosted.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1754556AbcLPTu3 (ORCPT ); Fri, 16 Dec 2016 14:50:29 -0500 Subject: Re: OOM: Better, but still there on 4.9 To: Michal Hocko , Nils Holland References: <20161215225702.GA27944@boerne.fritz.box> <20161216073941.GA26976@dhcp22.suse.cz> CC: , , David Sterba , From: Chris Mason Message-ID: <1e7af6ae-ff31-2678-11e2-aa22cf554d8d@fb.com> Date: Fri, 16 Dec 2016 14:50:07 -0500 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Thunderbird/45.3.0 MIME-Version: 1.0 In-Reply-To: <20161216073941.GA26976@dhcp22.suse.cz> Content-Type: text/plain; charset="windows-1252"; format=flowed Content-Transfer-Encoding: 7bit X-Originating-IP: [172.101.197.43] X-ClientProxiedBy: BN6PR04CA0014.namprd04.prod.outlook.com (10.172.194.24) To MWHPR15MB1245.namprd15.prod.outlook.com (10.175.3.7) X-Microsoft-Exchange-Diagnostics: 1;MWHPR15MB1245;2:v1IsMZnqJUveG9vj5rByVh8TsSWn0IwOXm2IP9zEM0+uNo0qIh6VEP1PYqctLsCFhM5/a4ctstgGF8ZM6HT0b9S1FUqEbyVmI5tuSR0jW9QVR8pjCHIO9XhCq6l0yQAAMOj5Vy68IKVrq0ZraxoUUaGX6mCCoN/VwcFQJFREf9U=;3:cpdW6b1XqkQEMvtyyV4D1Fs5iPvQuz2h1qx1LhoS7XH+diakhFkfkvpeXRMiG/IRt435WmqOdCV50QUbaslPMj0hp96NqmC8zilbs9MHr/pKIpa0HhzJkLJR/tqBcnffcm/Kmmr7TF7sR/s4pMtj2QArjifmobJfHaSuaMNdWAo= X-MS-Office365-Filtering-Correlation-Id: a1386c7f-3aae-4a35-5a5b-08d425ecbfd9 X-Microsoft-Antispam: UriScan:;BCL:0;PCL:0;RULEID:(22001);SRVR:MWHPR15MB1245; X-Microsoft-Exchange-Diagnostics: 1;MWHPR15MB1245;25:/emkQMj/0xT2zXGKRNjYxx7bGXpPpfOW/OEHMa2Cb5FCirCtXDGXG/B/VMX9xSoCins9GjE3IOfXEz5AXUsoenehk+cYuRsjM/bhMi1PRHwzTkAC7FoZU0U4ZBK9k2u8HZ0+9dQ4BYXEgC46xPhYMCwC5WOK+x2zuNGefcPNmULJbIN/NW0HwNJ+pGDI5pAfaviJvSE94fdlNwdZmQZRil0bZzuwXkoDxj/7YDZNY9JNAqFphhneeRAvXFo7t1GJpbfgtKMVqb8ISOUr/HJavAQZlTjz/KLZC9MRrtJN7WZnt6EppxVErTV7mT5znFU/j3XnT7zGSb0mjGLbBsHMOr2OzM9iNxbxL/jdOFAW4xGfOH/Ch2Evp5brmZIdnrlk5Z6qgKbQ7Kq5vQVKuzoeqn0a6LZt2fzPYNCvmvxzUjNlJjwLiufOqSer4cfMUMoZhvcVveoZW+uUX6UDOD1XcEA7jUW+YgsD4rrcksbenSmwFyw78sUrSElYZ66mN8KBTZNHjhB+9wyaQUp16Wquur6MRSyQYK7xjIzWUvTnKQZEHVOKtWyR8P6QHRj/KS/RgRrMlndKNOSHPVN9PRjUJZIwnrH7zXYyGg7jrJftgi7IwHQKItsv3Os5ReYcn2cSbehH3UX6GxEIfVGra+t9r3V/5YnHFH0CC8Ia1Dr4bujCssbIsfykmht98TPFmYzDbO52zdiuwBb+G0ZAnc0mmZvDpM0wO2ZGQXxhUyxWdkHTeWc245n/LnlCoVG4HsOwP6GVD30hWh16i3uCcHfnPg== X-Microsoft-Exchange-Diagnostics: 1;MWHPR15MB1245;31:07S2Pdz68YVbGPaqV3tvKhg/SpdAiecFOUNGR2n+ELllb7IbLTglEHEBYT87NNml4VKc2fqlr/DsbE/valJGqry/h37b3eCdCmYigud6h8kxLPDBEQWmq+RFoCowbUrKcaSfAsZOziBaRih0zDbepQ/VqE0xnouCmJpqJvEHkhHs2dShgQZhWAcScKVJVV2SRRo6bs34lhn4pkRazvUr2DiI/LeJCspVB4oqsAc2WNpQ34GYssQaSYUkmPsTpOi1khtuTa0EBtGhEWRs1pxBRA==;20:4OEX3IhjNSsvuMlg4z9H73qlwU5abBi1pePZEX+ncBJNBQXGxzsvnSsqvrIIFLMWps9TeL7gRxL83kwLYju0U/0zezf6748JVxuDi9ZaX4w5ZSGeBR5N23+nGQbbcevUWqR4DQ7BKbKSFcCpG40piR6ttIMnw/z2Nx+Xa0LYSvDqhF6FgSCTJIJQKs6aGbWa0pjO4dpsuganiLtqE5OOLyrpEnbagf7gSoF7UZpN9oVzKFnsoW4sRMJv/Pmc5/JYnfBFma35Xb5CS05jLuVUB4LJ4XeBXwpNJ8rbjGkzoIKd9+ki1oIdDJZLJmow8NF0ZjK/AmfJI2NDSnWC63yl1dK8p/YHutFBKbmm6ykQG7+hNsYb7fcNyhreoEa1tcKUbkOt50AahY7pXZxNgSUqLZ9oYY9L0Qoo+OGarwZlzrtWxEKyCNE22eISOcAE8MSBFYut1AGRoe06jG0Ol2dV0D563gkA57B4GeXI/NR5NV7qg8CqwtJw8qMlNuY85mAr X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:; X-Exchange-Antispam-Report-CFA-Test: BCL:0;PCL:0;RULEID:(6040375)(601004)(2401047)(5005006)(8121501046)(10201501046)(3002001)(6041248)(20161123564025)(20161123555025)(20161123560025)(20161123562025)(6072148);SRVR:MWHPR15MB1245;BCL:0;PCL:0;RULEID:;SRVR:MWHPR15MB1245; X-Microsoft-Exchange-Diagnostics: 1;MWHPR15MB1245;4:Q4naL8PgK9/Dn8CCHKs+HyQ4GlF2cBLmW4mTtJ8HULuw9Un1cVMLtgZAfSjz5ZNqLSKaA1kKsKzXVKBdDWVeXioQRgGujYkO/Yqz3XOdXrBLTERsFIVwR1AeYjVGoHJmNG8kL5otn8IQcz+uNNXWw6QsjaRLlmb6Dad1v8O6T0+unIXHax35ay+uVf1KU+P8yNsccvDyKg4MzpNjZwEpHiz0bvfeYBGPxpgy5RNn5WvqhqnamifpQ3ebp4TCvJ3iQ7l19F/5N+b2sLU8ks70F7uY6Lu8TxAQ8vUeG661aGqPzR6iWEgE4OlZRYLBzyqAU93e/5LwgaMqKScYm6t7cr1Lmvt+XvhaXTKMtDK7KOHkiJgwa0BMsY9xNtbbds3NKDtY5pwGSzPNdpaAp3pOdHqwj2Rr7n0E2UnauSVS0JE069F7sn18buFV3mlxI/DK8sbR3U7DT/VJ5Tf9+ahSHJQvHHueOsh9H/zB44gCatqwd4KDEg0pcBtZShbvsWG2f6VcI9jQ5bfUZFpsMOssSOfLj19NggLayVRo/JaWWDPEUUMtVYTmrAF2Bz0HnU3l X-Forefront-PRVS: 01583E185C X-Forefront-Antispam-Report: SFV:NSPM;SFS:(10019020)(4630300001)(6009001)(6049001)(7916002)(39840400002)(39410400002)(39450400003)(39850400002)(199003)(51234002)(377454003)(377424004)(189002)(24454002)(65826007)(6486002)(5001770100001)(92566002)(81166006)(54356999)(23746002)(551934003)(68736007)(101416001)(106356001)(42186005)(229853002)(105586002)(31686004)(7736002)(76176999)(4001350100001)(50466002)(97736004)(3846002)(31696002)(65956001)(189998001)(25786008)(90366009)(66066001)(8676002)(305945005)(5660300001)(2950100002)(65806001)(575784001)(47776003)(50986999)(230700001)(2906002)(6116002)(64126003)(36756003)(86362001)(38730400001)(81156014)(77096006)(117156001)(4326007)(83506001)(6666003)(33646002)(42262002);DIR:OUT;SFP:1102;SCL:1;SRVR:MWHPR15MB1245;H:[192.168.1.14];FPR:;SPF:None;PTR:InfoNoRecords;A:1;MX:1;LANG:en; X-Microsoft-Exchange-Diagnostics: =?Windows-1252?Q?1;MWHPR15MB1245;23:qeokXaVhVyb9X7znKWfCZiApvpoK7pRrLjT+v?= =?Windows-1252?Q?cZJiMTr6czwUqZQmxS9BuAWYU8fqsWPpJ9N1PIxLfkxuwSJMbkIr1QFS?= =?Windows-1252?Q?Xg0fG0zTpCfmY6WDmYJJg/2ND0HonRj0fhMms6VSgkW9vm6WGO3Rp9S0?= =?Windows-1252?Q?Ycquk48kjBy9Yx+g0SvUFKMJcaBbpk2VkkP8P7zcP3ZL7awp26aNE3cu?= =?Windows-1252?Q?I8hCiKFmeCicN3a+g4i5BND7VBTA2FC5i+7n2up+R/U8VRdORVjfteNQ?= =?Windows-1252?Q?KUi6SOcupkeoj6IZgV+vPTZ/vH7iE8K9maAl5X4fY7lnVudrbQ4SRdLf?= =?Windows-1252?Q?2eAnLot36fk49R0aS4tyWrqUzVjLK9TXB0Ikn6Tjq8FL1njMbMex74no?= =?Windows-1252?Q?KMrLP1E4MwNhl0vbBafyhBmllb4oOn751JrhPGx/4LtM1aX/kPjch4EZ?= =?Windows-1252?Q?Wk/x7CnezUieP1O1ASREuj3Nj22M7gxz7Dp83W7d05VfgghGIa4bPXqO?= =?Windows-1252?Q?QL7on1Se3Nbs+wg0JPbStMg+rhMHwWU8mFSfzYQRtGq1clqzRoJklrDi?= =?Windows-1252?Q?r4TV0kXT1lPGdLO1mMnOMPIgYYg9403uGs6hlMZUnPv7Jv2NIyG01mQF?= =?Windows-1252?Q?RqUADqxSUL5WKaQFSu8ERdoZ5YuK0MT49qN06LeO/SYhsE1Y/E4ecbGv?= =?Windows-1252?Q?z7yHA7TFu2GIfoyqwVFN8Oaa1sPX0lPsdVj0rhm7Vt7WUk3f9Iqpu4Yx?= =?Windows-1252?Q?I5wfdHeLJbFgswpTCkg1LghvvkIZGWmMzYBRNMdtAPvfze8zMc6gMKKn?= =?Windows-1252?Q?3UDhOzcZE+Q+7PPSyTLbp94YVfCtBZfww/kHreyB5Dvgunre9gKg8qrN?= =?Windows-1252?Q?CC9q7tdQKH/lEp/cH37jfT+yMItq3mae64Szq6P+gZFprNK4SMJRCTVF?= =?Windows-1252?Q?ML1faS/FXzTSzAg0Bx2iWJ059zMm5swXzfSZcUK5MxBmsyEve7Ef6kX4?= =?Windows-1252?Q?VXXHaIfURj44YI7bwhzUUgKXZFwm4tHIOPYwF4GypA06TCZdhasGtVwK?= =?Windows-1252?Q?2jJzt5SGv3VTutIbgXGfqXENpFgfBneu7JbpsEuKSrQrk2t2nIxG3JEB?= =?Windows-1252?Q?mQDNbEuIYcnNpqmJ6FOqC9RJvFXa7FfE4nvNKwMsgnKEjyHxTyWnze14?= =?Windows-1252?Q?irYQwXL2YdAO+zkFWkJ+8hzNQ92klMBrG8E140OgAaXJL+kiiCihINvl?= =?Windows-1252?Q?+ERYxGn/UuvYknmPKmIbHCC8s5Jl3TYLrVi+lsGl6QA8pooB5m48zzuD?= =?Windows-1252?Q?kRrqyxqJ0re9BMXJgBedgIyZj33L+OfKXbyxLdY3xwPctEdMNas9M3EC?= =?Windows-1252?Q?ujNN4/BRiIg+tGpgltEv4gx8UIqiT/lZRsvXqc0pgz5LJgCIrpmemFe2?= =?Windows-1252?Q?a/8VIGciuBBhEqIsXSjIuXMyqjXUIK7TPXqyykSLhdsmDYaP8VV5FWtl?= =?Windows-1252?Q?sDP1NPHP7Vvo6PObMj9OR02nPTKpwabks/pktp3kjhnUJsjsJSeNFK48?= =?Windows-1252?Q?feqJCJfB7gDvFY1q78yBU90DtLNfZblre2RjrkCC86zBEmRnuRMZKM+F?= =?Windows-1252?Q?wYQcrGyNFk9sZJMxKi07SEKa9CY2hbUuVjA+1KEeMR72RIzNF/ffL0iv?= =?Windows-1252?Q?oheD9tKXnldQag7rhje36GdtM+maWnTkPI/a0Z5rBlE1VuAuoGB?= X-Microsoft-Exchange-Diagnostics: 1;MWHPR15MB1245;6:hkijjX56DfiT8gZaUjJwh9iD0zSZDWBiIS4YNohUnyP8gEIYdJCfWvI/4s1PM6+y3EjtZ0oUUH0L9UF8Ge96iO3i9nLnQQI5BFc9Sx6z7ADWBBJbJ0YKNq4pPTpHvvPcqZlZAlxtNhNAWpcmBb9Z6yKAkk/ikByDwcub6t4CcDVr1M+faAaD2F/xnzHQEZksWjaNfTuoOjoaHCYG3kB1MQifridnPI9kafZ+j2CLMC8015dyK/8Lyf90wbGOc5rRXNJYjfB1GrM+4yerFWT/bOkIZIOKH5UeBKNaQo9aWVsxz58hFqlivG6YNqjfgYUtxKP4swUC3FdwlvK1/CGVbLlAuegT51o6TMvv4OcCMUg=;5:nYgUYYv8LYwProSCJ+C8Wp1Sky+QYFJ7qFP92Mm/wiLLHmUZfbJZmBHkLtt+zkDv5pyq9TKFrEANlUdekttw9MzIIIxqrLXBo/nEOFWbbbE4KHPL6YAiGA5KFlC0KM74EJl8IT56l0gbZjRWoUufOQ==;24:iU5oMfZLPthDzxqn8OSMBQ5/mcpqre+vtJ3TeiNOtwPs/AYA3tsvHxeFRoXwAfRAn9E7eIxvGCIWJHWHCIeKcVJGHE7AUXz9Mx5K31BItbs= SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-Microsoft-Exchange-Diagnostics: 1;MWHPR15MB1245;7:AYjp7Zp+HSL2pQ4rvfVmcngmvk5C/CpTU7Z9PHBY2F5DZ9o780VVAs23X2ukkoPsSXx180cFpq8visQ6lFJgwp9rtBJ1r4s9uH5159hhZmSvSh84hKnObBtX3UR/88Md2ff3QINxVc1Xno/evxeJYFRwlgkYJLnAXskHss9bnET4J6MFD+6Fk059kG70Q+aXZHi5rPJcBLW8LOleBhkz6BRgrZYQqNnnkwvAUbmLqZosZmBb+aFpArtnSL/62kQK61WgYUUeZlFyWep4DO2AHjRd29QPD5elcEjR+0xZgjDIHcd28rcfJUHq7DFS3t4MqgfVCknZd9Ls4udGSlIYzLt8CWsZwl1jf/AG8SIMEP8=;20:13B9CwKKVtF4w67DZOYLSREdMAFvKwgEClYbni32WnZKH3hgl24OpNoiXaRP2KuRejCFI3melChPyAwLOYLQJgOQAKcleMeWWpDoeyIX17gXuKM1Tv/mqcs+QG2ZvL6u1JbL7ioLrv4gZSJkfDjG66wsAE8+5x3+kN+flB6h5xI= X-MS-Exchange-CrossTenant-OriginalArrivalTime: 16 Dec 2016 19:50:11.7816 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-Transport-CrossTenantHeadersStamped: MWHPR15MB1245 X-OriginatorOrg: fb.com X-Proofpoint-Spam-Reason: safe X-FB-Internal: Safe X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:,, definitions=2016-12-16_13:,, signatures=0 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 5054 Lines: 74 On 12/16/2016 02:39 AM, Michal Hocko wrote: > [CC linux-mm and btrfs guys] > > On Thu 15-12-16 23:57:04, Nils Holland wrote: > [...] >> Of course, none of this are workloads that are new / special in any >> way - prior to 4.8, I never experienced any issues doing the exact >> same things. >> >> Dec 15 19:02:16 teela kernel: kworker/u4:5 invoked oom-killer: gfp_mask=0x2400840(GFP_NOFS|__GFP_NOFAIL), nodemask=0, order=0, oom_score_adj=0 >> Dec 15 19:02:18 teela kernel: kworker/u4:5 cpuset=/ mems_allowed=0 >> Dec 15 19:02:18 teela kernel: CPU: 1 PID: 2603 Comm: kworker/u4:5 Not tainted 4.9.0-gentoo #2 >> Dec 15 19:02:18 teela kernel: Hardware name: Hewlett-Packard Compaq 15 Notebook PC/21F7, BIOS F.22 08/06/2014 >> Dec 15 19:02:18 teela kernel: Workqueue: writeback wb_workfn (flush-btrfs-1) >> Dec 15 19:02:18 teela kernel: eff0b604 c142bcce eff0b734 00000000 eff0b634 c1163332 00000000 00000292 >> Dec 15 19:02:18 teela kernel: eff0b634 c1431876 eff0b638 e7fb0b00 e7fa2900 e7fa2900 c1b58785 eff0b734 >> Dec 15 19:02:18 teela kernel: eff0b678 c110795f c1043895 eff0b664 c11075c7 00000007 00000000 00000000 >> Dec 15 19:02:18 teela kernel: Call Trace: >> Dec 15 19:02:18 teela kernel: [] dump_stack+0x47/0x69 >> Dec 15 19:02:18 teela kernel: [] dump_header+0x60/0x178 >> Dec 15 19:02:18 teela kernel: [] ? ___ratelimit+0x86/0xe0 >> Dec 15 19:02:18 teela kernel: [] oom_kill_process+0x20f/0x3d0 >> Dec 15 19:02:18 teela kernel: [] ? has_capability_noaudit+0x15/0x20 >> Dec 15 19:02:18 teela kernel: [] ? oom_badness.part.13+0xb7/0x130 >> Dec 15 19:02:18 teela kernel: [] out_of_memory+0xd9/0x260 >> Dec 15 19:02:18 teela kernel: [] __alloc_pages_nodemask+0xbfb/0xc80 >> Dec 15 19:02:18 teela kernel: [] pagecache_get_page+0xad/0x270 >> Dec 15 19:02:18 teela kernel: [] alloc_extent_buffer+0x116/0x3e0 >> Dec 15 19:02:18 teela kernel: [] btrfs_find_create_tree_block+0xe/0x10 >> Dec 15 19:02:18 teela kernel: [] btrfs_alloc_tree_block+0x1ef/0x5f0 >> Dec 15 19:02:18 teela kernel: [] __btrfs_cow_block+0x143/0x5f0 >> Dec 15 19:02:18 teela kernel: [] btrfs_cow_block+0x13a/0x220 >> Dec 15 19:02:18 teela kernel: [] btrfs_search_slot+0x1d1/0x870 >> Dec 15 19:02:18 teela kernel: [] btrfs_lookup_file_extent+0x4d/0x60 >> Dec 15 19:02:18 teela kernel: [] __btrfs_drop_extents+0x176/0x1070 >> Dec 15 19:02:18 teela kernel: [] ? kmem_cache_alloc+0xb7/0x190 >> Dec 15 19:02:18 teela kernel: [] ? start_transaction+0x65/0x4b0 >> Dec 15 19:02:18 teela kernel: [] ? __kmalloc+0x147/0x1e0 >> Dec 15 19:02:18 teela kernel: [] cow_file_range_inline+0x215/0x6b0 >> Dec 15 19:02:18 teela kernel: [] cow_file_range.isra.49+0x55c/0x6d0 >> Dec 15 19:02:18 teela kernel: [] ? lock_extent_bits+0x75/0x1e0 >> Dec 15 19:02:18 teela kernel: [] run_delalloc_range+0x441/0x470 >> Dec 15 19:02:18 teela kernel: [] writepage_delalloc.isra.47+0x144/0x1e0 >> Dec 15 19:02:18 teela kernel: [] __extent_writepage+0xd8/0x2b0 >> Dec 15 19:02:18 teela kernel: [] extent_writepages+0x25c/0x380 >> Dec 15 19:02:18 teela kernel: [] ? btrfs_real_readdir+0x610/0x610 >> Dec 15 19:02:18 teela kernel: [] btrfs_writepages+0x1f/0x30 >> Dec 15 19:02:18 teela kernel: [] do_writepages+0x15/0x40 >> Dec 15 19:02:18 teela kernel: [] __writeback_single_inode+0x35/0x2f0 >> Dec 15 19:02:18 teela kernel: [] writeback_sb_inodes+0x16e/0x340 >> Dec 15 19:02:18 teela kernel: [] wb_writeback+0xaa/0x280 >> Dec 15 19:02:18 teela kernel: [] wb_workfn+0xd8/0x3e0 >> Dec 15 19:02:18 teela kernel: [] process_one_work+0x114/0x3e0 >> Dec 15 19:02:18 teela kernel: [] worker_thread+0x2f/0x4b0 >> Dec 15 19:02:18 teela kernel: [] ? create_worker+0x180/0x180 >> Dec 15 19:02:18 teela kernel: [] kthread+0x97/0xb0 >> Dec 15 19:02:18 teela kernel: [] ? __kthread_parkme+0x60/0x60 >> Dec 15 19:02:18 teela kernel: [] ret_from_fork+0x1b/0x28 >> Dec 15 19:02:18 teela kernel: Mem-Info: >> Dec 15 19:02:18 teela kernel: active_anon:58685 inactive_anon:90 isolated_anon:0 >> active_file:274324 inactive_file:281962 isolated_file:0 > > OK, so there is still some anonymous memory that could be swapped out > and quite a lot of page cache. This might be harder to reclaim because > the allocation is a GFP_NOFS request which is limited in its reclaim > capabilities. It might be possible that those pagecache pages are pinned > in some way by the the filesystem. Reading harder, its possible those pagecache pages are all from the btree inode. They shouldn't be pinned by btrfs, kswapd should be able to wander in and free a good chunk. What btrfs wants to happen is for this allocation to sit and wait for kswapd to make progress. -chris