Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
Subject: Re: [PATCH net-next RFC 00/13] Add devlink reload level option
To:     Vasundhara Volam <vasundhara-v.volam@broadcom.com>
Cc:     Jacob Keller <jacob.e.keller@intel.com>,
        "David S. Miller" <davem@davemloft.net>,
        Jiri Pirko <jiri@mellanox.com>,
        Netdev <netdev@vger.kernel.org>,
        open list <linux-kernel@vger.kernel.org>
References: <1595847753-2234-1-git-send-email-moshe@mellanox.com>
 <CAACQVJqNXh0B=oe5W7psiMGc6LzNPujNe2sypWi_SvH5sY=F3Q@mail.gmail.com>
 <a3e20b44-9399-93c1-210f-e3c1172bf60d@intel.com>
 <CAACQVJo+bAr_k=LjgdTKbOxFEkpbYAsaWbkSDjUepgO7_XQfNA@mail.gmail.com>
 <7a9c315f-fa29-7bd5-31be-3748b8841b29@mellanox.com>
 <CAACQVJpZZPfiWszZ36E0Awuo2Ad1w5=4C1rgG=d4qPiWVP609Q@mail.gmail.com>
 <7fd63d16-f9fa-9d55-0b30-fe190d0fb1cb@mellanox.com>
 <CAACQVJqXa-8v4TU+M1DWA2Tfv3ayrAobiH9Fajd=5MCgsfAA6A@mail.gmail.com>
 <da0e4997-73d7-9f3c-d877-f2d3bcc718b9@mellanox.com>
 <CAACQVJofS2B3y40H=QxBzNaccsa+gNnSqfmoATyML_S686ykfw@mail.gmail.com>
From:   Moshe Shemesh <moshe@mellanox.com>
Message-ID: <da7a2f2d-3ff5-0cd1-f166-79d7355f3df0@mellanox.com>
Date:   Wed, 5 Aug 2020 11:20:59 +0300
User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:68.0) Gecko/20100101
 Thunderbird/68.10.0
In-Reply-To: <CAACQVJofS2B3y40H=QxBzNaccsa+gNnSqfmoATyML_S686ykfw@mail.gmail.com>
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 7bit
Content-Language: en-US
MIME-Version: 1.0
X-MS-Exchange-MessageSentRepresentingType: 1
Received: from [192.168.0.105] (31.210.180.3) by FR2P281CA0011.DEUP281.PROD.OUTLOOK.COM (2603:10a6:d10:a::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3261.13 via Frontend Transport; Wed, 5 Aug 2020 08:21:02 +0000
X-Originating-IP: [31.210.180.3]
X-MS-PublicTrafficType: Email
X-MS-Office365-Filtering-HT: Tenant
X-MS-Office365-Filtering-Correlation-Id: a1c799ae-bf62-4db6-d28e-08d839187e08
X-MS-TrafficTypeDiagnostic: AM0PR05MB5987:
X-MS-Exchange-Transport-Forked: True
X-Microsoft-Antispam-PRVS: <AM0PR05MB5987C1A9CFBE77863B72DF58D94B0@AM0PR05MB5987.eurprd05.prod.outlook.com>
X-MS-Oob-TLC-OOBClassifiers: OLM:9508;
X-MS-Exchange-SenderADCheck: 1
X-Microsoft-Antispam: BCL:0;
X-Microsoft-Antispam-Message-Info: xcEBMhib/B5BZx3KqCZuktqYGlcLFUF/+jqNoFl483DCi+xQVEgSGWgxjaTKcbjpH42ddV7dYqacZSfAKo4H6c7VVnTCNqRZbpChL6vfo2lolUT00BJiXADEkw4LsjmWDv+qkoA4/OWKmtsYenaQRyMxPYYeWjhCHSG8Kb0NO2bRmFw6adrueEsbcO4kkJVD/aoKWPQjoSYq7lOeYGFuk7vYn37eOhNXgXLjkQd8Po/rt8dtH9u0WssA0E0TdZ+kZiFQB65NMRTwnAMgPM+jhqByw7F7sfhiysU7f1j0n1SNGjVmEotpupLRDf+MwOuN+sh9frQc+jwZ2sv8y+hXtm1TVSecNzYYnOEDbIorXs+m3RJZd0tkyV+VvuvKbCgF
X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:AM0PR05MB4290.eurprd05.prod.outlook.com;PTR:;CAT:NONE;SFTY:;SFS:(4636009)(136003)(39860400002)(366004)(346002)(396003)(376002)(956004)(2906002)(54906003)(2616005)(26005)(83380400001)(186003)(53546011)(6916009)(86362001)(31696002)(16526019)(316002)(16576012)(6486002)(31686004)(66476007)(66556008)(8936002)(4326008)(478600001)(52116002)(66946007)(8676002)(6666004)(36756003)(5660300002)(43740500002);DIR:OUT;SFP:1101;
X-MS-Exchange-AntiSpam-MessageData: o4A7L6DGHar6dWcTWA+N1olseQ4w3n66ucyUIsgUqYCoylssnNNbIUDYWXU/f9nul0XopOPSviqN2nNfr2NFpVqcQsk5TIjRMZsLqs3+81n2ydqqDhNy1n0CsYMs7+DzI3yK8D8PV/tCns2ZVhufXLh4P9IkBmdc/aKF7QhzST0euoyw51d5SAbV0l5VxzgRxQNLq0ZeugF7LeQgCeTRK/IfTNS/5qYHozkPf6ZyC2PepnnXngARKx6wHu7E8+tRW0TvzLEZGc2syz37zXAvfQToNUgczHzOWfARk1N9p/rfmQLmr76u0WLtzaEUBu0WtDtbA8KO9/5BHIeF9pIBM0pMdZYtwokilj9nI8jsOW95H1Z58spVVNQ3Eeu4QcX66xQP0z5kKbFoyLABYVG+U0LtEBxFudh/YRGAOt5GpZTqkqiyd1ED7bEtDm5quzdy40V6YGl5QcMW0tyZXOJc6Aqygp24jLHZJVAkCWbDbJYTO6+UMRbJmJAWXwCLiccE8U/JFeys+unV2+Wxsbh7yH4rpKAL7/l83qN5p2FYs+yvQ9pnTN02hnbff3QO1w4d/iesQgSjtMTcfEeBjhyg4QK6AxyeJgsq5Ry/cHWuGydaLus3cF5V6mc+YRIEwLuHmyrrTAHv6cbbNMp+Jns2UQ==
X-OriginatorOrg: Mellanox.com
X-MS-Exchange-CrossTenant-Network-Message-Id: a1c799ae-bf62-4db6-d28e-08d839187e08
X-MS-Exchange-CrossTenant-AuthSource: AM0PR05MB4290.eurprd05.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 05 Aug 2020 08:21:03.1539
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: a652971c-7d2e-4d9b-a6a4-d149256f461b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: FyTUCNp+0pcVgLOBXPzxUl35BlXPqAXpZOZDe3PQ/ktzRqFRRnVLKkOcveJnLlVwmAV/tUnVAU+ArY8PGe4QoQ==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM0PR05MB5987
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org


On 8/5/2020 9:55 AM, Vasundhara Volam wrote:
> On Wed, Aug 5, 2020 at 12:02 PM Moshe Shemesh <moshe@mellanox.com> wrote:
>>
>> On 8/4/2020 1:13 PM, Vasundhara Volam wrote:
>>> On Mon, Aug 3, 2020 at 7:23 PM Moshe Shemesh <moshe@mellanox.com> wrote:
>>>> On 8/3/2020 3:47 PM, Vasundhara Volam wrote:
>>>>> On Mon, Aug 3, 2020 at 5:47 PM Moshe Shemesh <moshe@mellanox.com> wrote:
>>>>>> On 8/3/2020 1:24 PM, Vasundhara Volam wrote:
>>>>>>> On Tue, Jul 28, 2020 at 10:13 PM Jacob Keller <jacob.e.keller@intel.com> wrote:
>>>>>>>> On 7/27/2020 10:25 PM, Vasundhara Volam wrote:
>>>>>>>>> On Mon, Jul 27, 2020 at 4:36 PM Moshe Shemesh <moshe@mellanox.com> wrote:
>>>>>>>>>> Introduce new option on devlink reload API to enable the user to select the
>>>>>>>>>> reload level required. Complete support for all levels in mlx5.
>>>>>>>>>> The following reload levels are supported:
>>>>>>>>>>       driver: Driver entities re-instantiation only.
>>>>>>>>>>       fw_reset: Firmware reset and driver entities re-instantiation.
>>>>>>>>> The Name is a little confusing. I think it should be renamed to
>>>>>>>>> fw_live_reset (in which both firmware and driver entities are
>>>>>>>>> re-instantiated).  For only fw_reset, the driver should not undergo
>>>>>>>>> reset (it requires a driver reload for firmware to undergo reset).
>>>>>>>>>
>>>>>>>> So, I think the differentiation here is that "live_patch" doesn't reset
>>>>>>>> anything.
>>>>>>> This seems similar to flashing the firmware and does not reset anything.
>>>>>> The live patch is activating fw change without reset.
>>>>>>
>>>>>> It is not suitable for any fw change but fw gaps which don't require reset.
>>>>>>
>>>>>> I can query the fw to check if the pending image change is suitable or
>>>>>> require fw reset.
>>>>> Okay.
>>>>>>>>>>       fw_live_patch: Firmware live patching only.
>>>>>>>>> This level is not clear. Is this similar to flashing??
>>>>>>>>>
>>>>>>>>> Also I have a basic query. The reload command is split into
>>>>>>>>> reload_up/reload_down handlers (Please correct me if this behaviour is
>>>>>>>>> changed with this patchset). What if the vendor specific driver does
>>>>>>>>> not support up/down and needs only a single handler to fire a firmware
>>>>>>>>> reset or firmware live reset command?
>>>>>>>> In the "reload_down" handler, they would trigger the appropriate reset,
>>>>>>>> and quiesce anything that needs to be done. Then on reload up, it would
>>>>>>>> restore and bring up anything quiesced in the first stage.
>>>>>>> Yes, I got the "reload_down" and "reload_up". Similar to the device
>>>>>>> "remove" and "re-probe" respectively.
>>>>>>>
>>>>>>> But our requirement is a similar "ethtool reset" command, where
>>>>>>> ethtool calls a single callback in driver and driver just sends a
>>>>>>> firmware command for doing the reset. Once firmware receives the
>>>>>>> command, it will initiate the reset of driver and firmware entities
>>>>>>> asynchronously.
>>>>>> It is similar to mlx5 case here for fw_reset. The driver triggers the fw
>>>>>> command to reset and all PFs drivers gets events to handle and do
>>>>>> re-initialization.  To fit it to the devlink reload_down and reload_up,
>>>>>> I wait for the event handler to complete and it stops at driver unload
>>>>>> to have the driver up by devlink reload_up. See patch 8 in this patchset.
>>>>>>
>>>>> Yes, I see reload_down is triggering the reset. In our driver, after
>>>>> triggering the reset through a firmware command, reset is done in
>>>>> another context as the driver initiates the reset only after receiving
>>>>> an ASYNC event from the firmware.
>>>> Same here.
>>>>
>>>>> Probably, we have to use reload_down() to send firmware command to
>>>>> trigger reset and do nothing in reload_up.
>>>> I had that in previous version, but its wrong to use devlink reload this
>>>> way, so I added wait with timeout for the event handling to complete
>>>> before unload_down function ends. See mlx5_fw_wait_fw_reset_done(). Also
>>>> the event handler stops before load back to have that done by devlink
>>>> reload_up.
>>> But "devlink dev reload" will be invoked by the user only on a single
>>> dev handler and all function drivers will be re-instantiated upon the
>>> ASYNC event. reload_down and reload_up are invoked only the function
>>> which the user invoked.
>>>
>>> Take an example of a 2-port (PF0 and PF1) adapter on a single host and
>>> with some VFs loaded on the device. User invokes "devlink dev reload"
>>> on PF0, ASYNC event is received on 2 PFs and VFs for reset. All the
>>> function drivers will be re-instantiated including PF0.
>>>
>>> If we wait for some time in reload_down() of PF0 and then call load in
>>> reload_up(), this code will be different from other function drivers.
>>
>> I see your point here, but the user run devlink reload command on one
>> PF, in this case of fw-reset it will influence other PFs, but that's a
>> result of the fw-reset, the user if asked for params change or namespace
>> change that was for this PF.
> Right, if any driver is implementing only fw-reset have to leave
> reload_up as an empty function.


No, its not only up the driver. The netns option is implemented by 
devlink and its running between reload_down and reload_up.

>>>>>     And returning from reload
>>>>> does not mean that reset is complete as it is done in another context
>>>>> and the driver notifies the health reporter once the reset is
>>>>> complete. devlink framework may have to allow drivers to implement
>>>>> reload_down only to look more clean or call reload_up only if the
>>>>> driver notifies the devlink once reset is completed from another
>>>>> context. Please suggest.