Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <dc61408d-8fc7-ca29-d284-0c92c2e1828c@huawei.com>
Date:   Mon, 24 Oct 2022 13:44:56 +0100
MIME-Version: 1.0
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:91.0) Gecko/20100101
 Thunderbird/91.6.1
Subject: Re: [PATCH v5 0/7] libsas and drivers: NCQ error handling
To:     Niklas Cassel <Niklas.Cassel@wdc.com>
CC:     Damien Le Moal <damien.lemoal@opensource.wdc.com>,
        "jejb@linux.ibm.com" <jejb@linux.ibm.com>,
        "martin.petersen@oracle.com" <martin.petersen@oracle.com>,
        "jinpu.wang@cloud.ionos.com" <jinpu.wang@cloud.ionos.com>,
        "linux-scsi@vger.kernel.org" <linux-scsi@vger.kernel.org>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        Linuxarm <linuxarm@huawei.com>,
        yangxingui <yangxingui@huawei.com>,
        yanaijie <yanaijie@huawei.com>
References: <YzwvpUUftX6Ziurt@x1-carbon>
 <cfa52b91-db81-a179-76c2-8a61266c099d@huawei.com>
 <27148ec5-d1ae-d9a2-1b00-a4c34d2da198@huawei.com>
 <Yz33FGwd3YvQUAqT@x1-carbon>
 <5db6a7bc-dfeb-76e1-6899-7041daa934cf@opensource.wdc.com>
 <Yz4BLTPkXqyjW4a4@x1-carbon>
 <64ab35a7-f1ff-92ee-890e-89a5aee935a4@opensource.wdc.com>
 <f190f19e-34b2-611c-1cf4-f8f34d12fe74@huawei.com>
 <Yz7qD+gpmI1bdw16@x1-carbon>
 <bc7d74dc-5aaa-1ad3-626f-df89955b1380@huawei.com>
 <Y1aEGW2BtdIRJy7s@x1-carbon>
From:   John Garry <john.garry@huawei.com>
In-Reply-To: <Y1aEGW2BtdIRJy7s@x1-carbon>
Content-Type: text/plain; charset="UTF-8"; format=flowed
Content-Transfer-Encoding: 7bit
Precedence: bulk

Hi Niklas,

> 
> For the record, I tested the pm80xx driver on a HoneyComb LX2 board
> (an arm64 board using ACPI).
> 
> I tried v6.1-rc1 both with and without your series in $subject.
> 
> I couldn't see any issues.

ok, thanks for the effort.

> 
> 
> What I tried:
> -Running fio:
> fio --name=test --filename=/dev/sdc --ioengine=io_uring --rw=randrw --direct=1 --iodepth=32 --bs=1M
> on three different HDDs simultaneously for 15+ minutes,
> without any errors in fio or dmesg.
> 
> -Creating and mounting a btrfs volume, doing a huge dd to the filesystem
> without issues.
> 
> -sg_sat_read_gplog -d --log=0x10 /dev/sda
> which successfully returned the log.
> 
> 
> It is worth mentioning that this arm64 board has reserved memory regions,
> but does not yet have a firmware that supplies a IORT RMR (reserved memory
> regions) revision E.d node, which means that in order to get this board to
> boot successfully, we need to supply:
> "arm-smmu.disable_bypass=0 iommu.passthrough=1"
> on the kernel command line.

hmmm... that's interesting. I can try again with the IOMMU turned off, 
but, as I recall, it did not make a difference before. I think that 
requiring reserved memory regions would totally bust the driver (if not 
present) with IOMMU enabled. As I recall, sas 3008 card would not work 
without RMR for us.

It's also interesting that this LX2 board has A72 cores. For my system, 
we have newer custom arm v8 cores with quite weak memory ordering 
implementation. With that same system, I have detected a couple of other 
driver memory ordering bugs which we did not see on our A72-based platforms.

I always suspected that this issue was a memory ordering issue, but 
since the hang so reliably occurs I ruled it out. Maybe it is...

thanks,
John