Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.37 as permitted sender) client-ip=23.128.96.37;
Message-ID: <91e0842c-1ba1-449d-b2d8-a3e2fc5ac95a@amd.com>
Date:   Mon, 18 Sep 2023 17:13:50 -0700
User-Agent: Mozilla Thunderbird
Subject: Re: [PATCH vfio] vfio/pci: remove msi domain on msi disable
To:     Jason Gunthorpe <jgg@ziepe.ca>
Cc:     alex.williamson@redhat.com, kevin.tian@intel.com,
        reinette.chatre@intel.com, tglx@linutronix.de, kvm@vger.kernel.org,
        brett.creeley@amd.com, linux-kernel@vger.kernel.org,
        Leon Romanovsky <leonro@nvidia.com>
References: <20230914191406.54656-1-shannon.nelson@amd.com>
 <20230918141705.GE13795@ziepe.ca>
 <acfa5d59-242b-4b31-a3ef-b4163972f26b@amd.com>
 <20230918233219.GO13795@ziepe.ca>
Content-Language: en-US
From:   "Nelson, Shannon" <shannon.nelson@amd.com>
In-Reply-To: <20230918233219.GO13795@ziepe.ca>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?eUVST2o0blZBZzd1YUpqRXR0YmplTWxWTEVab3E4K2JaaFFjRkNCb1FDTk5w?=
 =?utf-8?B?WXMremZtdmxIUDFWSmo4WlA3VGxNSkRPcGRsc1F6ZmV1YmxldnF2UEZWMmo4?=
 =?utf-8?B?Q0pCSmpmY3JWU250YzZRVkwwVHFUV2xkTXlLRzkyVVhyOXJwYm9RREJyN0Iw?=
 =?utf-8?B?aFcvNmJQNDZRTmZXNGNQUm1OMmtLUXBVLyt2VVBhVDloL2tiMUZtRlRpaVZK?=
 =?utf-8?B?THdubFAwV3RxL0cxZEoxcUQ5bUx1ckVJL0tKRy80NmQ0djBMT0x5c1hqNGwx?=
 =?utf-8?B?WUloQ2ZWd1JyelZ4Q0Jvcno1VitiNW1QV3p4c2s0elU1WG5EU2tuWk13TEZr?=
 =?utf-8?B?SjJHUjRWbG5vbDdpNjBzeWtBUFMydDVqcEt3ZVo2cEJuaGpLdWliL1VaeTg5?=
 =?utf-8?B?dTZuR1lpNGNNRHE0N0RxR1hVcUVFdTdKSzczUlZYY1RzdlhJTHFzUW8yRlN4?=
 =?utf-8?B?T3p5bE14R25DdzdsTzdrQWdkdHZQazlvemRaLzBwdGtNd2IyVzFwSmp4MGF4?=
 =?utf-8?B?Y2JzRTRpR1pSVzNoS3RHVE00OHNPSWFGWm9YeFVkNDdDS1FSS0prYnVqWDB4?=
 =?utf-8?B?N0Z0NDlqV3N1TTA0MklNOXFUWExHUnhrNlRLcUVKcDJjbnpsbFBqZzZ2YlNu?=
 =?utf-8?B?b25GTE55bXgxNXF1YkU4Uy9zU1ExVTl0Q0N2QlpWR2RjNlhuWDRaSzM1SGNT?=
 =?utf-8?B?QUpIU2lSK1JzV2pmY29rWjJPZUdMcFdaT2IzSmNWTzhZZWRDdjladUFNZEx0?=
 =?utf-8?B?VHNSTnNGTm84b3hqbHdhNVkwSmdkZzY3cWUwMVZFNWRIMGt3TE1WT2dPYkFH?=
 =?utf-8?B?MGR4TlQ0ZzVwbmVCZDBvdVVlazBTczk3UWN5WVV0QlNxbnQvL3ltY3ZLM1Br?=
 =?utf-8?B?c21kOTVGUHVlYm45SFFuQmZKa2Y2RGk0UXNGclhtemhERlNwdUFOK0NSOUpo?=
 =?utf-8?B?Wm9xL2FXUkd0bkwrVTczSDdHYnZOOHNXSmt0QVhpZW12ckdibC9vU1VWc1Ni?=
 =?utf-8?B?RXlqOE8vemVJMDhya1ZTYTh1QzBoZ0UrTHdNYzBwNkg2K0JHaWJoRksyT2Ex?=
 =?utf-8?B?L0cwZmxqQk1ISFFoZ0trYmdZODVhVzMxNW5odEdDV1B1TTlLbXJvQ0lsclVB?=
 =?utf-8?B?UnFKNUtrT2VKTWM2bUUxTmRCOHNsNGgxcFBSR1JEWm10Y2c2SDJHS01TZTRV?=
 =?utf-8?B?RVpTNjNZcThOek9ZTkF6NE9yMXFXWTUwQXZIT1pxMEhScVBsRkdDVTljSThX?=
 =?utf-8?B?OWdLWkllVXorU1o2OG1PbU50NzVxYXo2eG8rSTAzQVgvSGpNL3hZSDlXaWJl?=
 =?utf-8?B?Y3RsdnRoQkJ5aUtWcDFjTEJ4NHFkNXRrY2dDbG1ON2dEK3huZkRUdFUxMzBE?=
 =?utf-8?B?MHh0d0VMeHRmTDBEZis0R0lQQ1hXZ2huQUdkWlovUTROZGJxVFhSaXZQOUx3?=
 =?utf-8?B?WUZWUFJ2UGthS01EQ2w1QXczZkZ4SGlaMlp3aHJRVng1NVNwWWxTYmNONTd4?=
 =?utf-8?B?bTB1dkdxV0tyWjRORlFsbVBFWnRzbTRCeXFVd3NTYTRkSEs5S1Z6N0RIeks4?=
 =?utf-8?B?REw3VE43MEJybzZDckU2YThBcXduWDdBakJOQW1xYVNSVXUrN1BlQnRXZUJv?=
 =?utf-8?B?WkFJOGlseCtsM2s3UnJSd2s4cEpzZ2UyWDBUNzFDT3VxOEdnb2l6VHVITWk0?=
 =?utf-8?B?eUxiTnMwOGx0NmQ2TnRNOGhLN3ZwVE1ZTXd2bzVicmlLM08rVnNGQ0Myc2gz?=
 =?utf-8?B?cGtwRlVxQU1aQVAxWk10MGFucUlkMFJmMjlNOWJZd2lTMjAzOEhrZXlTTHRX?=
 =?utf-8?B?UXRPQ0JCZXZoR0o5VmdvSHh5Q0k4ckpPc2tSUmk0VS80cWdvOW8xUENqRHBw?=
 =?utf-8?B?cVhSbzJYOC8zRFNNN2gxUzliU1JLWTdBYllHTGRqUGtrSXRWdEExYkwvbmQw?=
 =?utf-8?B?SEJsbzJXWnZxb1pMWXdnU0cwSDBMYVh6SXpReUlzaHJmVElqaWJUdlNsUGVJ?=
 =?utf-8?B?LzBFVHBGQkZqRGNrTDFwWFdoajVGcGNxbU14WG91QVN2R2ZlTUdXNkJYVFUx?=
 =?utf-8?B?bXozVEVEWVBSc2dzYW9PQ1FLK09icStBcHdXbkZTdGZtb3k1THRXUk4rajFZ?=
 =?utf-8?Q?on8Dwv3w1FU8SC96mjXvcp5nT?=
X-OriginatorOrg: amd.com
X-MS-Exchange-CrossTenant-Network-Message-Id: cdcfc84a-a579-4c60-e539-08dbb8a54ebb
X-MS-Exchange-CrossTenant-AuthSource: DS0PR12MB6583.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 19 Sep 2023 00:13:53.6771
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: R5dDQXIdX8/gEw0a4aZimhLaTg68ur+S3drRTBAcurzXgkPF25gsEv1wRGuGEDkV3Mr8aqRjCStJlNwdwD/GOQ==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN0PR12MB5788
X-Spam-Status: No, score=-1.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FORGED_SPF_HELO,
        RCVD_IN_DNSWL_NONE,RCVD_IN_MSPIKE_H2,SPF_HELO_PASS,SPF_NONE
        autolearn=no autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org
X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (snail.vger.email [0.0.0.0]); Mon, 18 Sep 2023 17:14:06 -0700 (PDT)

On 9/18/2023 4:32 PM, Jason Gunthorpe wrote:
> 
> On Mon, Sep 18, 2023 at 10:48:54AM -0700, Nelson, Shannon wrote:
> 
>> In our case, the VF device's msix count value found in PCI config space is
>> changed by device configuration management outside of the baremetal host and
>> read by the QEMU instance when it starts up, and then read by the vfio PCI
>> core when QEMU requests the first IRQ.
> 
> Oh, you definitely can't do that!
> 
> PCI config space is not allowed to change outside the OS's view and we
> added sriov_set_msix_vec_count() specifically as a way to provide the
> necessary synchronization between all the parts.
> 
> Randomly changing, what should be immutable, parts of the config space
> from under a running OS is just non-compliant PCI behavior.

Hmmm... I guess I need to have a little chat with my friendly HW/FW folks.

Thanks,
sln