Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <4ddb5909-0f1e-d701-5678-ce7f3a57446b@gmx.ch>
Date:   Mon, 2 Jan 2023 11:33:59 +0100
MIME-Version: 1.0
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101
 Thunderbird/102.6.1
Subject: Re: [PATCH 2/2] vfio/pci: Remove console drivers
Content-Language: en-US
To:     "mb@lab.how" <mb@lab.how>, Thomas Zimmermann <tzimmermann@suse.de>
Cc:     kvm@vger.kernel.org, airlied@linux.ie,
        linux-kernel@vger.kernel.org, dri-devel@lists.freedesktop.org,
        Alex Williamson <alex.williamson@redhat.com>,
        kraxel@redhat.com, lersek@redhat.com
References: <CAEdEoBYXHq9cCzsbMYTpG1B41Yz=-QAjFx7bJDOnPanN5Tmo7A@mail.gmail.com>
 <20221204175142.658d5c37.alex.williamson@redhat.com>
 <1e4d62cf-8893-0bff-51f5-5a2e419ed5a0@suse.de>
 <CAEdEoBYZa9cg0nq=P7EDsDS9m2EKYrd8M8ucqi8U0Csj0mtjDg@mail.gmail.com>
 <c1c8bfa5-8ba4-c365-1663-535f656ca353@suse.de>
 <CAEdEoBaixaTEuNfQGv1das7TwHKV9MiRMKQM0kLspveJmipzyg@mail.gmail.com>
From:   Shawn Michaels <shawn_michaels@gmx.ch>
In-Reply-To: <CAEdEoBaixaTEuNfQGv1das7TwHKV9MiRMKQM0kLspveJmipzyg@mail.gmail.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
UI-OutboundReport: notjunk:1;M01:P0:kXnMy45TwZM=;YdKfddT3ITGr9r4YtRKUBXCg8hC
 K8yx3Hjh2XQPqWHG8zbACmM6bpStxyNsiPFS3Qh3J6soI3qY1s9mGOwiner5QOCvyGns0CHHp
 YoqB4UjaXb5UKkWNIJRXE14E2Kf5WBBT+7C7l4PhQX4xCw2diiPXA9l2pYniWBbRuAATitj0K
 f0ocDZxIHMEh1YtCInbGogPim82/8gfIzIi0pYE14dwrVyDlNZvuhAnYlCG4MyAEDVlraoVtv
 gEQCon3XLHoE6iFwbV49ZOLo64nEpSF0InDGBW/ToTTF35QLtVevXctt0auXQQohsbilPoom2
 7vmQkQwblFjLniufyVRmeU6vMF6NQt4zBWd6Z+UHGQnw3MCmf5+ARMHf/aXae2C8Uyc1TKOm3
 RI8Ey514q2k6kd9jBtYsD4eZN9UI/T+oBWSV4+lQTeblzrQ7CPYfAGkbTxtbWEoraOk10Ysyy
 cFOhqB6rgs/QiTGv6HaPXiO3heKnQblzIimQEgEZCML6M6uHlTIGTelsWTGMxY4iEDnboVauU
 jdleYp1IE7M+E7dUtmwT6VsieTuHMKz6kVDLNGS5QSG2b/KtFyyzjIowYbUjYq22evBnnO9qL
 A5qatYU4hULENSLIl0dSMsEdAA7hTgMUxCWGZqy5To6nRNlflK1t4/9Fs6iieeoeaD7se2vcX
 jYBK6SQ63K0ceGk+312Hcg8NTmfKMc5ayuiWvGUNNt/4EcPNGU4YtuA7LiTEFuyZ1QAJx02sY
 rb6G9KHBLzex7dewBVFQwB4QHZK1zakAhquXNwQ6aWgLM8/CGzU/xW5sEXrM18S5r42aGL1rd
 7Fmhe226jqLtj1hALbGaF+nYd0C+M5ArnRdEDInVqBHN2CbeqF1JXPnlK54fxMMQ8yevrWfdj
 Ao2tTRnuNhmLNAfy4QjLTVUvCWTifNFSM1QpGtkx9BcZudwKAtSSrCD+rSzL4gbtgOBrHiVVc
 kLuXIQ4o9qqiBn1Hcm5oVtY+kBw=
Precedence: bulk

Hi,

I just upgraded my system (after 7 months) and I also lost my framebuffer on boot.
At first, I thought that my computer was freezing on startup, but it turns out
that it is running fine (I can SSH to it and even startx remotely), only the
framebuffer stops working very early on boot:

	:: running early hook [udev]
	Starting systemd-udevd version 252.4-2-arch
	[3.400568] VFIO - User Level meta-driver version: 0.3
	*LOSING FRAMEBUFFER HERE*

Running startx from an SSH session starts my X server and display works again.

I have two identical GPU cards (nVidia GTX 1070). I have been using the method
from [1] for years in order to prevent the nvidia driver from grabbing my guest
GPU. As mb already pointed out, vfio_pci now kills the framebuffer of the host
GPU even though it is assigned to the guest GPU. I could isolate it to the
following line from [1]:

	echo "vfio-pci" > "$GPU/driver_override"

I also double checked and this is correctly written to the guest GPU, and not
to the host GPU. My kernel version is:

	Linux cc 6.1.1-arch1-1 #1 SMP PREEMPT_DYNAMIC Wed, 21 Dec 2022 22:27:55 +0000 x86_64 GNU/Linux

You can find people having the same issue in threads from [2] and [3]. There are
also some reports on the VFIO discord server.

This is a problem for people using an encrypted filesystem (password needs to be
typed blindly) or in case boot fails for some reason (and you cannot see console
logs). In my case, I switched from manually starting X with startx to using a
graphical login manager. Blacklisting the vfio_pci module by passing the following
kernel parameter brings the boot console back:

	module_blacklist=vfio_pci

Happy new year everyone

Shawn

[1] https://wiki.archlinux.org/title/PCI_passthrough_via_OVMF#Passthrough_all_GPUs_but_the_boot_GPU
[2] https://bbs.archlinux.org/viewtopic.php?pid=2063423
[3] https://forum.level1techs.com/t/linux-kernel-6-seems-to-be-incompatible-with-the-vfio-pci-module-needed-for-pci-passthrough/190039/11