Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67;
Date:   Thu, 7 Jun 2018 17:14:01 -0400
From:   Mike Snitzer <snitzer@redhat.com>
To:     valdis.kletnieks@vt.edu
Cc:     Alasdair Kergon <agk@redhat.com>, linux-kernel@vger.kernel.org,
        dm-devel@redhat.com, axboe@kernel.dk
Subject: Re: next-20180605 - kernel tried to execute NX-protected page -
 exploit attempt?
Message-ID: <20180607211401.GD4481@redhat.com>
References: <21051.1528404057@turing-police.cc.vt.edu>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <21051.1528404057@turing-police.cc.vt.edu>
User-Agent: Mutt/1.5.21 (2010-09-15)
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk

On Thu, Jun 07 2018 at  4:40pm -0400,
valdis.kletnieks@vt.edu <valdis.kletnieks@vt.edu> wrote:

> I've hit this one twice today with pretty much the same traceback.
> The disk has 3 partitions - one for EFI, one for /boot, and then the rest of
> the disk is a cryptluks partition that contains a dozen or so LVM logical
> volumes.
> 
> 'git log -- drivers/md' didn't show any obvious suspects since next-20180529, which worked
> for me just fine....

I just bounced 2 patches to you that Jens sent out that will hopefully
fix the issue.

Can you please share what you test is?  We've gotten lots of reports
with failure following wake_up but I don't have a canned test to trigger
this.  And my testbed has so much memory that I think I'm never
exhausting the mempool limits.

Mike


> 
> [ 6090.781839] kernel tried to execute NX-protected page - exploit attempt? (uid: 0)
> [ 6090.781847] BUG: unable to handle kernel paging request at ffff9d4bc8b766c0
> [ 6090.781856] PGD 17b7a067 P4D 17b7a067 PUD 17b7e067 PMD 408b9d063 PTE 8000000408b76063
> [ 6090.781872] Oops: 0011 [#1] PREEMPT SMP PTI
> 
> [ 6090.781893] Workqueue: kcryptd kcryptd_crypt
> [ 6090.781901] RIP: 0010:0xffff9d4bc8b766c0
> [ 6090.781905] Code: ff ff ff f9 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
> ff ff ff ff ff ff ff ff ff <ff> ff ff ff ff ff 9f ff ff ff ff f9 ff ff bf ff ff ff ff ff ff 7f
> [ 6090.782012] RSP: 0018:ffff9d4bdd2039d8 EFLAGS: 00010046
> [ 6090.782018] RAX: ffff9d4bc8b766c0 RBX: ffff9d4bd53744e8 RCX: 0000000000000000
> [ 6090.782023] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9d4bd31e5c90
> [ 6090.782027] RBP: ffff9d4bdd203a40 R08: 0000000000000000 R09: ffff9d4bd31e5c90
> [ 6090.782030] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
> [ 6090.782034] R13: ffff9d4bd7860228 R14: 00000000d31a2b40 R15: ffff9d4bdd203a58
> [ 6090.782038] FS:  0000000000000000(0000) GS:ffff9d4bdd200000(0000) knlGS:0000000000000000
> [ 6090.782042] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [ 6090.782046] CR2: ffff9d4bc8b766c0 CR3: 0000000015e24003 CR4: 00000000001606e0
> [ 6090.782050] Call Trace:
> [ 6090.782054]  <IRQ>
> [ 6090.782061]  ? __wake_up_common+0xb7/0x3d0
> [ 6090.782071]  __wake_up_common_lock+0x87/0xe0
> [ 6090.782080]  __wake_up+0x13/0x20
> [ 6090.782087]  mempool_free+0x122/0x190
> [ 6090.782095]  bio_free+0x59/0x80
> [ 6090.782101]  bio_put+0x50/0x90
> [ 6090.782107]  dec_pending+0x1b0/0x560
> [ 6090.782117]  clone_endio+0xd5/0x2e0
> [ 6090.782125]  bio_endio+0x22e/0x4b0
> [ 6090.782132]  crypt_dec_pending+0x92/0xf0
> [ 6090.782139]  crypt_endio+0x9b/0xe0
> [ 6090.782146]  bio_endio+0x22e/0x4b0
> [ 6090.782153]  blk_update_request+0x145/0x680
> [ 6090.782162]  scsi_end_request+0x56/0x440
> [ 6090.782169]  scsi_io_completion+0x462/0x9b0
> [ 6090.782178]  scsi_finish_command+0x189/0x2a0
> [ 6090.782185]  scsi_softirq_done+0x17e/0x1f0
> [ 6090.782193]  blk_done_softirq+0x229/0x410
> [ 6090.782198]  ? __do_softirq+0xfb/0x914
> [ 6090.782207]  __do_softirq+0x13a/0x914
> [ 6090.782219]  irq_exit+0xea/0x140
> [ 6090.782224]  do_IRQ+0xcc/0x1c0
> [ 6090.782232]  common_interrupt+0xf/0xf
> [ 6090.782237]  </IRQ>
> [ 6090.782241] RIP: 0010:memset_erms+0x9/0x10
> 
> The other traceback was about the same, with the following
> interleaved:
> 
> [27847.571250] list_add corruption. next->prev should be prev (ffff9e2c1347a4e8), but was 0000000000000000. (next=ffff9e2c13cde4a8).
> [27847.571278] kernel BUG at lib/list_debug.c:25!
> [27847.571685] invalid opcode: 0000 [#2] PREEMPT SMP PTI
> [27847.571689] CPU: 0 PID: 55 Comm: kswapd0 Tainted: G      D    O    T 4.17.0-next-20180605-dirty #586
> [27847.573947] Call Trace:
> [27847.573958]  prepare_to_wait+0x133/0x210
> [27847.573966]  ? mempool_alloc+0xe9/0x200
> [27847.573975]  mempool_alloc+0x17e/0x200
> [27847.573983]  ? remove_wait_queue+0x170/0x170
> [27847.573994]  bio_alloc_bioset+0x122/0x3f0
> [27847.574000]  ? bio_advance+0xbf/0x240
> [27847.574006]  ? bio_clone_blkcg_association+0x5b/0x80
> [27847.574015]  alloc_io+0x48/0x320
> [27847.574021]  ? dm_get_live_table+0x3a/0x140
> [27847.574030]  ? __split_and_process_non_flush+0x420/0x420
> [27847.574035]  __split_and_process_bio+0x5d/0x2b0
> [27847.574042]  ? __split_and_process_non_flush+0x420/0x420
> [27847.574048]  ? dm_get_live_table+0x5d/0x140
> [27847.574053]  ? dm_get_live_table+0x84/0x140
> [27847.574061]  __dm_make_request+0xaf/0x1f0
> [27847.574071]  dm_make_request+0x15/0x20
> [27847.574078]  generic_make_request+0x3b9/0x7c0
> [27847.574091]  submit_bio+0xb9/0x240
> [27847.574097]  ? submit_bio+0xb9/0x240
> [27847.574104]  ? __test_set_page_writeback+0x402/0xd30
> [27847.574111]  ? get_swap_bio+0x106/0x180
> [27847.574121]  __swap_writepage+0x153/0x8d0
> [27847.574128]  ? page_swapcount+0xbf/0x140
> [27847.574139]  ? __frontswap_store+0x8d/0x142
> [27847.574147]  swap_writepage+0x4d/0xc0
> [27847.574155]  pageout.isra.29+0x304/0x980
> [27847.574171]  shrink_page_list+0x11e9/0x2020
> [27847.574189]  shrink_inactive_list+0x291/0xdb0
> [27847.574204]  shrink_node_memcg+0x38a/0x1530
> [27847.574211]  ? percpu_ref_get_many+0x200/0x200
> [27847.574233]  shrink_node+0xdc/0x920
> [27847.574246]  balance_pgdat+0x288/0x680
> [27847.574262]  kswapd+0x2ca/0x990
> [27847.574271]  ? remove_wait_queue+0x170/0x170
> [27847.574282]  kthread+0x1d3/0x2a0
> [27847.574288]  ? balance_pgdat+0x680/0x680
> [27847.574294]  ? kthread_create_worker_on_cpu+0x70/0x70
> [27847.574304]  ret_from_fork+0x3a/0x50
>