Message-ID: <8f17a884-09d2-b6a1-7ced-77dd4b4f21a5@linux.ibm.com>
Date:   Fri, 17 Mar 2023 17:52:40 +0100
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101
 Thunderbird/102.8.0
Subject: Re: [PATCH for-next] scsi: Implement host state statistics
Content-Language: en-US
To:     "Seymour, Shane M" <shane.seymour@hpe.com>,
        Greg KH <gregkh@linuxfoundation.org>
Cc:     "Martin K. Petersen" <martin.petersen@oracle.com>,
        "jejb@linux.ibm.com" <jejb@linux.ibm.com>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        "linux-api@vger.kernel.org" <linux-api@vger.kernel.org>,
        "linux-scsi@vger.kernel.org" <linux-scsi@vger.kernel.org>
References: <DM4PR84MB1373DCD07EABD28D88129C50FDBF9@DM4PR84MB1373.NAMPRD84.PROD.OUTLOOK.COM>
 <ZBFnYwtr+2bfjvcY@kroah.com>
 <DM4PR84MB13737BE099BF599DF83617DBFDBF9@DM4PR84MB1373.NAMPRD84.PROD.OUTLOOK.COM>
From:   Steffen Maier <maier@linux.ibm.com>
In-Reply-To: <DM4PR84MB13737BE099BF599DF83617DBFDBF9@DM4PR84MB1373.NAMPRD84.PROD.OUTLOOK.COM>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
Precedence: bulk

On 3/15/23 23:41, Seymour, Shane M wrote:
>> On Wed, Mar 15, 2023 at 06:08:19AM +0000, Seymour, Shane M wrote:
>>> The following patch implements host state statistics via sysfs. The intent
>>> is to allow user space to see the state changes and be able to report when
>>> a host changes state. The files do not separate out the time spent into
>>> each state but only into three:
>>
>> Why does userspace care about these things at all?  What tool needs them
>> and what can userspace do with the information?
>>
> 
> In enterprise setups you may a significant number of LUNs presented to a
> system (100s to 1000s) via a single HBA (usually via FC). Having a HBA going
> into error handling causes issues. Every time a host goes into SCSI EH all
> I/O to that host is blocked until SCSI EH completes. That means waiting for
> every I/O to either complete or timeout before starting any recovery
> processing.
> 
> At this time there is no way for anything outside of the kernel to know if a
> HBA is having any issues. The cause of those issues can vary significantly,
> just two examples:
> 
> 1) Storage end point issues
> 2) SAN issues (e.g. laser transmit power at any point in the SAN)
> 
> My experience with downstream distros is that nobody seems to notice the
> noise that SCSI EH produces (LUN, device, bus, host resets) and we see it
> when we get a vmcore and have to try and work out what caused an I/O hang.

I hear you. Especially, the fact that the very desirable asynchronous aborts 
and even eh with escalations seems pretty much silent as long as a SCSI command 
succeeds within one of the allowed retries. I suspect this was done in order 
not to unsettle users by showing intermediate recovery, which can still lead to 
successful I/O eventually.

FWIW, at some point we figured out a nice scsi_logging_level of 4605, in order 
to see any problems with lun probing ("why don't I get my volume in Linux?") or 
timeouts/aborts/eh ("why are things so slow?") without producing kernel 
messages for regular good I/O. Of course, it's not set by default, but can be 
dynamically set if one suspects such problems.

> I wanted to be more proactive in warning users that you've got a potential
> storage issue you need to look at. It won't help when you have a sudden
> massive issue but if you have an issue that is slowly getting worse over
> a period of time you will at least get some warning.
> 
>>>
>>> A (GPLv2) program called hostmond will be released in a few months that
>>> will monitor these interfaces and report (local host only via syslog(3C))
>>> when hosts change state.
>>
>> We kind of need to see this before the kernel changes can be accepted
>> for obvious reasons, what is preventing that from happening now?
> 
> If you don't mind I'll answer this in my reply to James' email soon since
> he commented about this.
> 
>>
>> Please always use sysfs_emit() instead of the crazy scnprintf() for
>> sysfs entries.
> 
> No problem I can make that change.
> 
>>
>> u32 is a kernel type, not uint32_t please, but I don't know what the
>> scsi layer is used to.
> 
> No problem I can make that change.
> 
>>
>> thanks,
>>
>> greg k-h
> 
> Thank you for your willingness to provide feedback.
> 
> Shane

-- 
Mit freundlichen Gruessen / Kind regards
Steffen Maier

Linux on IBM Z and LinuxONE

https://www.ibm.com/privacy/us/en/
IBM Deutschland Research & Development GmbH
Vorsitzender des Aufsichtsrats: Gregor Pillen
Geschaeftsfuehrung: David Faller
Sitz der Gesellschaft: Boeblingen
Registergericht: Amtsgericht Stuttgart, HRB 243294