Received: by 2002:a05:7412:b10a:b0:f3:1519:9f41 with SMTP id az10csp2298799rdb; Sun, 3 Dec 2023 10:33:36 -0800 (PST) X-Google-Smtp-Source: AGHT+IFSXq6ac7BrR5RcT5R3blD8NPJkPZ356lJaw69nGiWYXPDgsxPtJYaGLAcHIT82qz1L150R X-Received: by 2002:a17:90b:104:b0:277:68c3:64b9 with SMTP id p4-20020a17090b010400b0027768c364b9mr3045001pjz.5.1701628416251; Sun, 03 Dec 2023 10:33:36 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1701628416; cv=none; d=google.com; s=arc-20160816; b=Wmgd6MIWCwF6zWKmGygWrLCE2udFX+TpCisrbQYTM5YgScXjqrVElTUX3pnhhaLPzU p1LhTWwfrJEq4NUDzX5MaHIp32r68metxmqgGstrdMDuRV0tya2kkj9HtasrV2x4HEcx 04SQn52HU2VXYvFRgJGngedfuHW71cootUjNaZJgDNkLzHXeolSY6jb9PUNjkkl5F+ge eoXOis3GrV27h+9tAKgB/CBliHaD4CgAmAWJMgUhNjbnwxTvR4Wc/zzeCpd4g70XnnXc 5dH00U+l+7OThWWyHxTNV9x0PZ7JyLgAQ6qPXTVQCv4HZYoXy69r0DZ/1Nv5PSLYYktJ EPOw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:list-unsubscribe:list-subscribe :list-id:precedence:dkim-signature; bh=jzuNvt1M83C1nZ2ailqYr8YmM+HGUOe16GkYorpbLx0=; fh=JBV+SmA05k12BmnkvNrYeJGQ5m3DPnyHstdyt9/6DB4=; b=SqUW5EqVFr9cVsVW/l9+fWYDNJTY9x8t9/lq9gDijQVr499mVqI9icHAGFeWfqvqjz TyDz5WH8dVtwQSTLi54PflQdL50zl4vghy8r6RO8E3pGJwH0L3f6PZYrmpCNo2LGkHyY 1DQ2UzRg+NxmJdPtJ7b8I8f6ef12yaKDldhgZOAzKBeoF8LJTSQRDqxWySAv1SOGXRve IVZ/5loORDeNlxIBN6RsS16Yo+TjQ8IRSvgXXBghdLXhQVT7tZMgQlTVho2CoqKSlXwT sz2mKJjRMzEueX1RiJwSh8xzw4+85Tb6VKpydK2lK97stycu1Au5Bo/Hdk5DiCx2TNWN g4FQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20230601 header.b=LWjOgyq5; spf=pass (google.com: domain of linux-crypto+bounces-512-linux.lists.archive=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) smtp.mailfrom="linux-crypto+bounces-512-linux.lists.archive=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from sv.mirrors.kernel.org (sv.mirrors.kernel.org. [139.178.88.99]) by mx.google.com with ESMTPS id b7-20020a655cc7000000b005b92edaa151si6714764pgt.739.2023.12.03.10.33.36 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 03 Dec 2023 10:33:36 -0800 (PST) Received-SPF: pass (google.com: domain of linux-crypto+bounces-512-linux.lists.archive=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) client-ip=139.178.88.99; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20230601 header.b=LWjOgyq5; spf=pass (google.com: domain of linux-crypto+bounces-512-linux.lists.archive=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) smtp.mailfrom="linux-crypto+bounces-512-linux.lists.archive=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sv.mirrors.kernel.org (Postfix) with ESMTPS id B8A13280D68 for ; Sun, 3 Dec 2023 18:33:35 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 70BD318030 for ; Sun, 3 Dec 2023 18:33:35 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="LWjOgyq5" X-Original-To: linux-crypto@vger.kernel.org Received: from mail-ej1-x629.google.com (mail-ej1-x629.google.com [IPv6:2a00:1450:4864:20::629]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D2169D7 for ; Sun, 3 Dec 2023 09:01:39 -0800 (PST) Received: by mail-ej1-x629.google.com with SMTP id a640c23a62f3a-a1b03886fd7so93993466b.2 for ; Sun, 03 Dec 2023 09:01:39 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1701622898; x=1702227698; darn=vger.kernel.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=jzuNvt1M83C1nZ2ailqYr8YmM+HGUOe16GkYorpbLx0=; b=LWjOgyq5bDbzyM4enK+/trrVEXOM5NMrZB36+8UvpyW3sUKCxB3R8SO6hKjaENVbGa Hv+8HTICx/JSkh3y0P4eY5bfHwafK7KiekmSOOp0nSlFhRXFATxTuhzbewCIM3KvHUPr eManQ7k1zP3uidmuoF4dyJHxbqP01Z+g3kX9B3ZXhoiVX8xP+9l46PJNtnR3vk6QPP+o +x44Am7g1VeXa2dwV94lsP7BHFYfzWTfHyFnRbUuksnpfd+N3TPRHYszNg0ldbbgisiK 0mhHaJnRKysczxJKYOr/qqFciwrozpfuoh2KRF+wP1PdEI66ZXD549o43SEBCc0Ud6/5 94uQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1701622898; x=1702227698; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=jzuNvt1M83C1nZ2ailqYr8YmM+HGUOe16GkYorpbLx0=; b=On7E2deXC0krGE881oKjNlOgdYwWVjIcZj6NN34YT8gT/f/BV7tk7oEzSW0RtCWP2A HfsP09/4QZbzJ9wyHst1sXK256INCNHGvnScwkwUm8GREtEUw5y1XIm4kOFwF+UUgZLt LSNzuK0zeNc4cqoj4UA/Nv/5xM9la72PnUd5hyglhpGbBv0lBIaoh5T41wRlRzSNhsbl 4McaTKrS0ZM15mkbOqWWVb+AGo2LWilVKi5JQhu0wb1HBMHq8AjV5zv/kmvyPv+hpvgT x7+SDOOd4Z33jj2pof56CKu045JG6/lm8uklKfCXdGtB3ujyY3oGL0LeCggEoaleWT0L BvYQ== X-Gm-Message-State: AOJu0Yx0MHAwWm+kpsmxHNihJnObRN64gMIe0p4gsPZI59tOsxjXvbvk ad86+6U924Xh4g0YmCFne1yUjGDf9v7XKBsft5U= X-Received: by 2002:a17:907:12c6:b0:a19:a1ba:da55 with SMTP id vp6-20020a17090712c600b00a19a1bada55mr2035935ejb.124.1701622897981; Sun, 03 Dec 2023 09:01:37 -0800 (PST) Precedence: bulk X-Mailing-List: linux-crypto@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 References: <5a0e8b44-6feb-b489-cdea-e3be3811804a@linux.alibaba.com> In-Reply-To: <5a0e8b44-6feb-b489-cdea-e3be3811804a@linux.alibaba.com> From: Juhyung Park Date: Mon, 4 Dec 2023 02:01:27 +0900 Message-ID: Subject: Re: Weird EROFS data corruption To: Gao Xiang Cc: Gao Xiang , linux-erofs@lists.ozlabs.org, linux-f2fs-devel@lists.sourceforge.net, linux-crypto@vger.kernel.org, Yann Collet Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Hi Gao, On Mon, Dec 4, 2023 at 1:52=E2=80=AFAM Gao Xiang wrote: > > Hi Juhyung, > > On 2023/12/4 00:22, Juhyung Park wrote: > > (Cc'ing f2fs and crypto as I've noticed something similar with f2fs a > > while ago, which may mean that this is not specific to EROFS: > > https://lore.kernel.org/all/CAD14+f2nBZtLfLC6CwNjgCOuRRRjwzttp3D3iK4Of+= 1EEjK+cw@mail.gmail.com/ > > ) > > > > Hi. > > > > I'm encountering a very weird EROFS data corruption. > > > > I noticed when I build an EROFS image for AOSP development, the device > > would randomly not boot from a certain build. > > After inspecting the log, I noticed that a file got corrupted. > > Is it observed on your laptop (i7-1185G7), yes? or some other arm64 > device? Yes, only on my laptop. The arm64 device seems fine. The reason that it would not boot was that the host machine (my laptop) was repacking the EROFS image wrongfully. The workflow is something like this: Server-built EROFS AOSP image -> Image copied to laptop -> Laptop mounts the EROFS image -> Copies the entire content to a scratch directory (CORRUPT!) -> Changes some files -> mkfs.erofs So the device is not responsible for the corruption, the laptop is. > > > > > After adding a hash check during the build flow, I noticed that EROFS > > would randomly read data wrong. > > > > I now have a reliable method of reproducing the issue, but here's the > > funny/weird part: it's only happening on my laptop (i7-1185G7). This > > is not happening with my 128 cores buildfarm machine (Threadripper > > 3990X).> > > I first suspected a hardware issue, but: > > a. The laptop had its motherboard replaced recently (due to a failing > > physical Type-C port). > > b. The laptop passes memory test (memtest86). > > c. This happens on all kernel versions from v5.4 to the latest v6.6 > > including my personal custom builds and Canonical's official Ubuntu > > kernels. > > d. This happens on different host SSDs and file-system combinations. > > e. This only happens on LZ4. LZ4HC doesn't trigger the issue. > > f. This only happens when mounting the image natively by the kernel. > > Using fuse with erofsfuse is fine. > > I think it's a weird issue with inplace decompression because you said > it depends on the hardware. In addition, with your dataset sadly I > cannot reproduce on my local server (Xeon(R) CPU E5-2682 v4). As I feared. Bummer :( > > What is the difference between these two machines? just different CPU or > they have some other difference like different compliers? I fully and exclusively control both devices, and the setup is almost the s= ame. Same Ubuntu version, kernel/compiler version. But as I said, on my laptop, the issue happens on kernels that someone else (Canonical) built, so I don't think it matters. > > Thanks, > Gao Xiang