Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S932070AbbLAPFV (ORCPT ); Tue, 1 Dec 2015 10:05:21 -0500 Received: from mga03.intel.com ([134.134.136.65]:9010 "EHLO mga03.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1756212AbbLAPFQ (ORCPT ); Tue, 1 Dec 2015 10:05:16 -0500 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.20,369,1444719600"; d="scan'208";a="862212822" Subject: Re: [RFC PATCH V2 0/3] IXGBE/VFIO: Add live migration support for SRIOV NIC To: Alexander Duyck References: <1448372298-28386-1-git-send-email-tianyu.lan@intel.com> <5654722D.4010409@gmail.com> <56552888.90108@intel.com> <56556F98.5060507@intel.com> <565BF285.4040507@intel.com> Cc: "Dong, Eddie" , "a.motakis@virtualopensystems.com" , Alex Williamson , "b.reynal@virtualopensystems.com" , Bjorn Helgaas , "Wyborny, Carolyn" , "Skidmore, Donald C" , "Jani, Nrupal" , Alexander Graf , "kvm@vger.kernel.org" , Paolo Bonzini , "qemu-devel@nongnu.org" , "Tantilov, Emil S" , Or Gerlitz , "Rustad, Mark D" , "Michael S. Tsirkin" , Eric Auger , intel-wired-lan , "Kirsher, Jeffrey T" , "Brandeburg, Jesse" , "Ronciak, John" , "linux-api@vger.kernel.org" , "linux-kernel@vger.kernel.org" , "Williams, Mitch A" , Netdev , "Nelson, Shannon" , Wei Yang , "zajec5@gmail.com" From: "Lan, Tianyu" Message-ID: <565DB6FF.1050602@intel.com> Date: Tue, 1 Dec 2015 23:04:31 +0800 User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:38.0) Gecko/20100101 Thunderbird/38.3.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 2286 Lines: 40 On 12/1/2015 12:07 AM, Alexander Duyck wrote: > They can only be corrected if the underlying assumptions are correct > and they aren't. Your solution would have never worked correctly. > The problem is you assume you can keep the device running when you are > migrating and you simply cannot. At some point you will always have > to stop the device in order to complete the migration, and you cannot > stop it before you have stopped your page tracking mechanism. So > unless the platform has an IOMMU that is somehow taking part in the > dirty page tracking you will not be able to stop the guest and then > the device, it will have to be the device and then the guest. > >> >Doing suspend and resume() may help to do migration easily but some >> >devices requires low service down time. Especially network and I got >> >that some cloud company promised less than 500ms network service downtime. > Honestly focusing on the downtime is getting the cart ahead of the > horse. First you need to be able to do this without corrupting system > memory and regardless of the state of the device. You haven't even > gotten to that state yet. Last I knew the device had to be up in > order for your migration to even work. I think the issue is that the content of rx package delivered to stack maybe changed during migration because the piece of memory won't be migrated to new machine. This may confuse applications or stack. Current dummy write solution can ensure the content of package won't change after doing dummy write while the content maybe not received data if migration happens before that point. We can recheck the content via checksum or crc in the protocol after dummy write to ensure the content is what VF received. I think stack has already done such checks and the package will be abandoned if failed to pass through the check. Another way is to tell all memory driver are using to Qemu and let Qemu to migrate these memory after stopping VCPU and the device. This seems safe but implementation maybe complex. -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/