Received: by 2002:a05:6a10:d5a5:0:0:0:0 with SMTP id gn37csp484014pxb; Thu, 30 Sep 2021 10:07:34 -0700 (PDT) X-Google-Smtp-Source: ABdhPJywK2CG1RK4b/0PyREcLf1DmLrJRzk8qQ3TTFHb9MadZHC4g4vppRR7MbZVmpOAwCKES8CI X-Received: by 2002:a17:906:7007:: with SMTP id n7mr332112ejj.275.1633021654070; Thu, 30 Sep 2021 10:07:34 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1633021654; cv=none; d=google.com; s=arc-20160816; b=KLpUItpG13xHjwtxEZ31b/NboaONCLn3lJmLbQzuEwzjRjIthOFnwUy1SfVtkLQsYW SuCz5DPr/Fskjef5LuDySkaWpewrkxc5v56nAqy60CbK58QjfGQo9DC4CSaER3qbJJbW YLhAT31njMH8daS4ikyTbtHRxmXgRh8OscnxAueWyPmQmmMQNFqRsBDKLHmJKRf8vFoW PTyTNEWWHEwXiiE/kGd4q4v6uFbyx3ZgFkhdTAEDwZnhQ1wgOK3V+Seivz2IcgsdIMtr LxYZg1q9MsgDvrrZuQGADLQ46RLy6Qo5WkCAFYpy+Xz7DobOhNPyEHTmySMR5iRigjEI gvtQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:in-reply-to:content-disposition:mime-version :references:message-id:subject:cc:to:from:date:dkim-signature; bh=4vHHMFNBH4hAm5F563BvE8+3o+7NJnRDr/RUjwsUT7o=; b=VIj+pVLGMdMhKHCmvjUONRsns3yEvx8pkDtPfVgQf8WAdIW96zP3A8Evl3o+e+3fnu jb2SRskJLAXOEZe8JAQZSlctjbSZIiiVzWH8QxxnTrcp8+lBX2ZqWJX3eUtGBRwmRp3O rgEYlQQ689pqLwGIjy3AGxI36yGCAAGuitFwqYC+B8u8xq8milACEzeZC+EJD+GY2tu+ eDLfqhF91YgyR59XVPS3PhYphonTrtUd/kpI6ymU9QOnNtBj5AWvOcKZ7gq4+r+SrlIT Oilr2DmWeujkosDvbFcIJx2DdLceYGX15XxYMSPKI47M0Spb9IpBysrZS+TSN3mVZ9G4 HagA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@ziepe.ca header.s=google header.b=QGIRkmLf; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [23.128.96.18]) by mx.google.com with ESMTP id m19si3890496ejd.372.2021.09.30.10.07.07; Thu, 30 Sep 2021 10:07:34 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18; Authentication-Results: mx.google.com; dkim=pass header.i=@ziepe.ca header.s=google header.b=QGIRkmLf; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1352471AbhI3RDa (ORCPT + 99 others); Thu, 30 Sep 2021 13:03:30 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44236 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1351965AbhI3RD3 (ORCPT ); Thu, 30 Sep 2021 13:03:29 -0400 Received: from mail-qt1-x829.google.com (mail-qt1-x829.google.com [IPv6:2607:f8b0:4864:20::829]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 792B5C06176F for ; Thu, 30 Sep 2021 10:01:46 -0700 (PDT) Received: by mail-qt1-x829.google.com with SMTP id d8so6358254qtd.5 for ; Thu, 30 Sep 2021 10:01:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ziepe.ca; s=google; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=4vHHMFNBH4hAm5F563BvE8+3o+7NJnRDr/RUjwsUT7o=; b=QGIRkmLfQXIeyUEAHPlvDndXmMMIAMxSXtZXhhRKYvklIE9s9V0mxKkMNZc361fu2+ /oV28Ti2mLOszUCQtAVFm+lO7hdx7RISinOyXEgG679jauOsPEZJsDn97BSKI/Y+XZGn nvKzE18TgoAr5yYwzqF0BLBR/d2XlDNYCnfE8BWTl7rPF2hP87rXsdSJGEEz4xNJo7IA XD/7Y7rdo0+4GJu9Kcd/XwUsO2juMuUKx3MqXsyQc6zYkSOTMavlqDYBTbIG5C39TIF8 RnCcanamumbSy2k6pKQ2iu96IWFTPNHP8Dtw3MHS8f+D1FklgeJFEug0i3XEm+5ai9kb OSaA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=4vHHMFNBH4hAm5F563BvE8+3o+7NJnRDr/RUjwsUT7o=; b=6uc/FtxVYLNFRl1zCoONnf9P1s2H4xqMq/6lVLPeITu7q53DvUgVQole+eQBI/272/ 5b5W2URNtan3WEZn0CPIsqyoJKPit/xPfLkJmmODvNTmquVw4ZO0Si9aZEGOl2lIYhpL oCu0Bb+1faw8irouSeG3uM94qrdsLLTEaSsQHu+ZIa/YQlPhvGdVogPLiWtguHfMDkS7 BXgn1hJn52AayFXzZBNUrP9GyV0N4gFMlRvvUWUXafs4KIA8X2++qOL04tK8s0vvDQAa E3knLYMDUGTKUigEXbjsD8Ouw5BIb6rS7DISkiMN1r81O9WD1n1CpYT6oNjN6gWBuEUs l4Mw== X-Gm-Message-State: AOAM532SYHDVhp172XJeZe9Urp91zo75Ohhvn+9gaMbWrv+PCVzZrjXf IMhl9B5LfQQZv0jdeAVWUT926A== X-Received: by 2002:ac8:4084:: with SMTP id p4mr7765655qtl.306.1633021305549; Thu, 30 Sep 2021 10:01:45 -0700 (PDT) Received: from ziepe.ca (hlfxns017vw-142-162-113-129.dhcp-dynamic.fibreop.ns.bellaliant.net. [142.162.113.129]) by smtp.gmail.com with ESMTPSA id l7sm1852806qth.81.2021.09.30.10.01.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 30 Sep 2021 10:01:44 -0700 (PDT) Received: from jgg by jggl with local (Exim 4.94) (envelope-from ) id 1mVzRH-000I2E-Ia; Thu, 30 Sep 2021 14:01:43 -0300 Date: Thu, 30 Sep 2021 14:01:43 -0300 From: Jason Gunthorpe To: Max Gurtovoy Cc: Alex Williamson , Leon Romanovsky , Doug Ledford , Yishai Hadas , Bjorn Helgaas , "David S. Miller" , Jakub Kicinski , Kirti Wankhede , kvm@vger.kernel.org, linux-kernel@vger.kernel.org, linux-pci@vger.kernel.org, linux-rdma@vger.kernel.org, netdev@vger.kernel.org, Saeed Mahameed , Cornelia Huck Subject: Re: [PATCH mlx5-next 2/7] vfio: Add an API to check migration state transition validity Message-ID: <20210930170143.GB69218@ziepe.ca> References: <20210929091712.6390141c.alex.williamson@redhat.com> <20210929161433.GA1808627@ziepe.ca> <29835bf4-d094-ae6d-1a32-08e65847b52c@nvidia.com> <20210929232109.GC3544071@ziepe.ca> <20210930144752.GA67618@ziepe.ca> <20210930162442.GB67618@ziepe.ca> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Thu, Sep 30, 2021 at 07:51:22PM +0300, Max Gurtovoy wrote: > > On 9/30/2021 7:24 PM, Jason Gunthorpe wrote: > > On Thu, Sep 30, 2021 at 06:32:07PM +0300, Max Gurtovoy wrote: > > > > Just prior to open device the vfio pci layer will generate a FLR to > > > > the function so we expect that post open_device has a fresh from reset > > > > fully running device state. > > > running also mean that the device doesn't have a clue on its internal state > > > ? or running means unfreezed and unquiesced ? > > The device just got FLR'd and it should be in a clean state and > > operating. Think the VM is booting for the first time. > > During the resume phase in the dst, the VM is paused and not booting. > Migration SW is waiting to get memory and state from SRC. The device will > start from the exact point that was in the src. > > it's exactly "000b => Device Stopped, not saving or resuming" For this case qmeu should open the VFIO device and immediately issue a command to go to resuming. The kernel cannot know at open_device time which case userspace is trying to do. Due to backwards compat we assume userspace is going to boot a fresh VM. > Well, this is your design for the driver implementation. Nobody is > preventing other drivers to start deserializing device state into the device > during RESUMING bit on. It is a logical model. Devices can stream the migration data directly into the internal state if they like. It just creates more conditions where they have report an error state. > So if we moved from 100b to 010b somehow, one should deserialized its buffer > to the device, and then serialize it to migration region again ? Yes. > I guess its doable since the device is freeze and quiesced. But moving from > 100b to 011b is not possible, right ? Why not? 100b to 011b is no different than going indirectly 100b -> 001b -> 011b The time spent in 001b is just negligable. Jason