[Cc: folks from IPU6 external driver, IPU3 maintainers, Dell, Lenovo,
Chromium, Canonical/Ubuntu, Red Hat]
Dear Greg,
I am writing you, because I remember you – at least – were active
several years ago in getting good upstream driver support into the Linux
kernel.
It looks like the driver for the MIPI IPU6 camera of Intel Alder Lake
based laptops, for example, Lenovo ThinkPad X1 Carbon and Dell XPS 13
9315/9320, is not in the upstream Linux kernel.
With Ubuntu’s OEM image pre-installed by Dell, and on Google Chromebooks
the camera is supposedly working [1]. There is a GitHub project with a
repository for the Linux kernel patches [2], but the patches – as
expected, and known from other external drivers, and Android drivers –
do not build against all Linux kernel versions [3]. A request to
upstream the drivers was opened at the end of March [4], and got the
reply below in July:
> Intel is not ready yet to provide technical support to various Linux
> distro and all OEM devices at this moment with Intel IPU6 github.
>
> Intel is supporting Dell laptop and their MIPI camera functionality
> on Ubuntu with release/integration support from Canonical. If you are
> using Dell devices, please refer to the devices listed in this link.
> (https://wiki.ubuntu.com/Dell) You can learn the latest updates from
> the installation guide.
The patches were integrated into ChromiumOS’ Linux kernel tree – for
specific version, like 5.15 [5], and the same for Ubuntu 22.04 with
Linux 5.15 – only in June [6]. A lot of that (redundant) work seems to
be done by the Intel developers involved also in the GitHub project.
In the last seven or so years Linux upstream driver support worked well
for me – especially with Intel hardware –, so I wanted to make you (and
others) aware of the situation, and hope, that you could do something
about the situation. With the current situation I can only recommend to
FLOSS users to *not* buy these devices.
Kind regards,
Paul
PS: The proprietary firmware is also not in the linux-firmware
repository [7], and the user space packages [8] are also not packaged
yet for some distributions like Debian.
PPS: VA-API support also does not work [9].
[1]: https://ubuntu.com/certified/202203-30070
[2]: https://github.com/intel/ipu6-drivers
[3]: https://github.com/intel/ipu6-drivers/issues/13
[4]: https://github.com/intel/ipu6-drivers/issues/22
[5]:
https://chromium-review.googlesource.com/c/chromiumos/third_party/kernel/+/3034663
[6]:
https://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/jammy/log/drivers/media/pci/intel
[7]: https://bugs.launchpad.net/ubuntu/+source/linux-firmware/+bug/1955383
[8]:
https://launchpad.net/~oem-solutions-group/+archive/ubuntu/intel-ipu6/+packages
[9]: https://github.com/intel/media-driver/issues/1371
Hi Paul,
(Adding myself on the CC list, to keep track of the discussion on behalf
of the libcamera project)
On Thu, Aug 11, 2022 at 10:23:26AM +0200, Paul Menzel wrote:
> [Cc: folks from IPU6 external driver, IPU3 maintainers, Dell, Lenovo,
> Chromium, Canonical/Ubuntu, Red Hat]
>
> Dear Greg,
>
>
> I am writing you, because I remember you – at least – were active
> several years ago in getting good upstream driver support into the Linux
> kernel.
>
> It looks like the driver for the MIPI IPU6 camera of Intel Alder Lake
> based laptops, for example, Lenovo ThinkPad X1 Carbon and Dell XPS 13
> 9315/9320, is not in the upstream Linux kernel.
>
> With Ubuntu’s OEM image pre-installed by Dell, and on Google Chromebooks
> the camera is supposedly working [1]. There is a GitHub project with a
> repository for the Linux kernel patches [2], but the patches – as
> expected, and known from other external drivers, and Android drivers –
> do not build against all Linux kernel versions [3]. A request to
> upstream the drivers was opened at the end of March [4], and got the
> reply below in July:
>
> > Intel is not ready yet to provide technical support to various Linux
> > distro and all OEM devices at this moment with Intel IPU6 github.
> >
> > Intel is supporting Dell laptop and their MIPI camera functionality
> > on Ubuntu with release/integration support from Canonical. If you are
> > using Dell devices, please refer to the devices listed in this link.
> > (https://wiki.ubuntu.com/Dell) You can learn the latest updates from
> > the installation guide.
>
> The patches were integrated into ChromiumOS’ Linux kernel tree – for
> specific version, like 5.15 [5], and the same for Ubuntu 22.04 with
> Linux 5.15 – only in June [6]. A lot of that (redundant) work seems to
> be done by the Intel developers involved also in the GitHub project.
>
> In the last seven or so years Linux upstream driver support worked well
> for me – especially with Intel hardware –, so I wanted to make you (and
> others) aware of the situation, and hope, that you could do something
> about the situation. With the current situation I can only recommend to
> FLOSS users to *not* buy these devices.
>
>
> Kind regards,
>
> Paul
>
>
> PS: The proprietary firmware is also not in the linux-firmware
> repository [7], and the user space packages [8] are also not packaged
> yet for some distributions like Debian.
A few more things to consider.
Support for the IPU6 on Ubuntu is very much of a stop-gap measure as far
as I can tell. As you correctly stated, the kernel drivers are not
upstream, and there is a long way to go before that can be fixed. The
drivers use a vendor-specific userspace API and are thus not fit for
upstream as-is. They will need to be rewritten to use either the V4L2
API (the only existing camera API upstream at this point), or a new
kernel API will need to be developed and upstreamed should V4L2 deemed
to be unsuitable. I can't comment on whether V4L2 would be a good
option, as I'm not aware of the detailed architecture of the device, but
I've heard it's significantly more complex than its predecessor, the
IPU3 found in selected Sky Lake and Kaby Lake SoCs (I'm pretending here
not to have heard of the IPU4).
On the userspace side, the stack is based on proprietary software, with
a userspace daemon and a V4L2 loopback driver to emulate the V4L2 API.
This complex architecture is due to the fact that this kind of device
requires complex imaging algorithms running in userspace, and a pure
V4L2 camera stack has no support for this. This isn't specific to the
IPU6, the IPU3 has similar requirements, and so do lots of embedded
SoCs. Given the direction the industry is taking, this situation will
become increasingly common in the future. With the notable exception of
Raspberry Pi who is leading the way in open-source camera support, no
SoC vendor is willing today to open their imaging algorithms. The issue
will thus not be solved simply by packaging the userspace software
provided by Intel for Ubuntu in other distributions, it will never make
it into pure FLOSS distributions, and is unmaintainable by the
community.
For a piece of good news, this is a problem we have been aware of for
years, which led to the creation of the libcamera project ([10]).
libcamera is an open-source userspace camera framework for Linux that
aims at supporting this exact type of hardware architecture. It already
supports the Intel IPU3, both on machines that have been designed for
Chrome OS and for Windows (sparing you the details, the latter describe
the camera hardware in their ACPI tables in a very different way than
the former, making them much more difficult to support). We have
integrated, as a proof of concept, the IPU3 closed-source imaging
libraries from Intel with libcamera ([11]), and have also developed a
fully open-source implementation ([12]). The open-source version is not
as feature-complete as the closed-source library, but it showcases how
both options are viable, and quite importantly, how multiple actors can
collaborate on open-source implementations of camera algorithms (the
auto-focus support for the IPU3 has been developed and contributed to
libcamera by Kate Hsuan from Red Hat for instance).
Now, the question is how we can get IPU6 support in libcamera. The first
problem to fix will be the availability of kernel drivers. Given the
issues listed above, this will require a large effort and a clear
commitment from vendors to happen. I wouldn't bet on a solution being
available before at least several years.
For the time being, I agree with your recommendation to not buy these
devices if you care about camera support.
[10] https://libcamera.org
[11] https://git.libcamera.org/libcamera/ipu3-ipa.git/
[12] https://git.libcamera.org/libcamera/libcamera.git/tree/src/ipa/ipu3
> PPS: VA-API support also does not work [9].
>
> [1]: https://ubuntu.com/certified/202203-30070
> [2]: https://github.com/intel/ipu6-drivers
> [3]: https://github.com/intel/ipu6-drivers/issues/13
> [4]: https://github.com/intel/ipu6-drivers/issues/22
> [5]: https://chromium-review.googlesource.com/c/chromiumos/third_party/kernel/+/3034663
> [6]: https://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/jammy/log/drivers/media/pci/intel
> [7]: https://bugs.launchpad.net/ubuntu/+source/linux-firmware/+bug/1955383
> [8]: https://launchpad.net/~oem-solutions-group/+archive/ubuntu/intel-ipu6/+packages
> [9]: https://github.com/intel/media-driver/issues/1371
> $ lspci -nn
> 00:00.0 Host bridge [0600]: Intel Corporation Device [8086:4602] (rev 06)
> 00:02.0 VGA compatible controller [0300]: Intel Corporation Alder Lake-UP4 GT2 [Iris Xe Graphics] [8086:46aa] (rev 0c)
> 00:04.0 Signal processing controller [1180]: Intel Corporation Alder Lake Innovation Platform Framework Processor Participant [8086:461d] (rev 06)
> 00:05.0 Multimedia controller [0480]: Intel Corporation Device [8086:465d] (rev 06)
> 00:06.0 PCI bridge [0604]: Intel Corporation 12th Gen Core Processor PCI Express x4 Controller #0 [8086:464d] (rev 06)
> 00:07.0 PCI bridge [0604]: Intel Corporation Alder Lake-P Thunderbolt 4 PCI Express Root Port #0 [8086:466e] (rev 06)
> 00:07.1 PCI bridge [0604]: Intel Corporation Alder Lake-P Thunderbolt 4 PCI Express Root Port #1 [8086:463f] (rev 06)
> 00:08.0 System peripheral [0880]: Intel Corporation 12th Gen Core Processor Gaussian & Neural Accelerator [8086:464f] (rev 06)
> 00:0d.0 USB controller [0c03]: Intel Corporation Alder Lake-P Thunderbolt 4 USB Controller [8086:461e] (rev 06)
> 00:0d.2 USB controller [0c03]: Intel Corporation Alder Lake-P Thunderbolt 4 NHI #0 [8086:463e] (rev 06)
> 00:12.0 Serial controller [0700]: Intel Corporation Device [8086:51fc] (rev 01)
> 00:14.0 USB controller [0c03]: Intel Corporation Alder Lake PCH USB 3.2 xHCI Host Controller [8086:51ed] (rev 01)
> 00:14.2 RAM memory [0500]: Intel Corporation Alder Lake PCH Shared SRAM [8086:51ef] (rev 01)
> 00:14.3 Network controller [0280]: Intel Corporation Alder Lake-P PCH CNVi WiFi [8086:51f0] (rev 01)
> 00:15.0 Serial bus controller [0c80]: Intel Corporation Alder Lake PCH Serial IO I2C Controller #0 [8086:51e8] (rev 01)
> 00:15.1 Serial bus controller [0c80]: Intel Corporation Alder Lake PCH Serial IO I2C Controller #1 [8086:51e9] (rev 01)
> 00:16.0 Communication controller [0780]: Intel Corporation Alder Lake PCH HECI Controller [8086:51e0] (rev 01)
> 00:1e.0 Communication controller [0780]: Intel Corporation Alder Lake PCH UART #0 [8086:51a8] (rev 01)
> 00:1e.2 Serial bus controller [0c80]: Intel Corporation Device [8086:51aa] (rev 01)
> 00:1e.3 Serial bus controller [0c80]: Intel Corporation Device [8086:51ab] (rev 01)
> 00:1f.0 ISA bridge [0601]: Intel Corporation Device [8086:5187] (rev 01)
> 00:1f.3 Multimedia audio controller [0401]: Intel Corporation Device [8086:51cc] (rev 01)
> 00:1f.4 SMBus [0c05]: Intel Corporation Alder Lake PCH-P SMBus Host Controller [8086:51a3] (rev 01)
> 00:1f.5 Serial bus controller [0c80]: Intel Corporation Alder Lake-P PCH SPI Controller [8086:51a4] (rev 01)
> 01:00.0 Non-Volatile memory controller [0108]: Phison Electronics Corporation Device [1987:5019] (rev 01)
--
Regards,
Laurent Pinchart
On Thu, Aug 11, 2022 at 05:02:40PM +0200, Greg KH wrote:
> On Thu, Aug 11, 2022 at 04:54:53PM +0300, Laurent Pinchart wrote:
> > For the time being, I agree with your recommendation to not buy these
> > devices if you care about camera support.
>
> I second this, don't buy these devices if the vendor is not willing to
> get their drivers upstreamed properly.
"Not willing" may be a bit too harsh here. I wouldn't just blame Intel
for not upstreaming a driver if it turns out that the V4L2 API isn't a
good match and we have no proposal to provide an alternative.
This being said, I agree it's the vendor's responsibility to at least
engage with the upstream community to find a solution, and collaborate
on the implementation. I'm sure there have been way more private
discussions than I'm aware of, maybe now is the right time to extend
this to a wider audience and create a realistic plan on how to move
forward. You all know my e-mail address :-)
--
Regards,
Laurent Pinchart
On Thu, Aug 11, 2022 at 06:08:55PM +0300, Laurent Pinchart wrote:
> On Thu, Aug 11, 2022 at 05:02:40PM +0200, Greg KH wrote:
> > On Thu, Aug 11, 2022 at 04:54:53PM +0300, Laurent Pinchart wrote:
> > > For the time being, I agree with your recommendation to not buy these
> > > devices if you care about camera support.
> >
> > I second this, don't buy these devices if the vendor is not willing to
> > get their drivers upstreamed properly.
>
> "Not willing" may be a bit too harsh here. I wouldn't just blame Intel
> for not upstreaming a driver if it turns out that the V4L2 API isn't a
> good match and we have no proposal to provide an alternative.
Did they propose an alternative? From what I saw here they didn't even
attempt it, or did I miss that?
Not even attempting is just not ok, as you say, they know where we are,
it's not like we can hide...
thanks,
greg k-h
On Thu, Aug 11, 2022 at 04:54:53PM +0300, Laurent Pinchart wrote:
> For the time being, I agree with your recommendation to not buy these
> devices if you care about camera support.
I second this, don't buy these devices if the vendor is not willing to
get their drivers upstreamed properly.
thanks,
greg k-h
On (22/08/11 17:30), Greg KH wrote:
> On Thu, Aug 11, 2022 at 06:08:55PM +0300, Laurent Pinchart wrote:
> > On Thu, Aug 11, 2022 at 05:02:40PM +0200, Greg KH wrote:
> > > On Thu, Aug 11, 2022 at 04:54:53PM +0300, Laurent Pinchart wrote:
> > > > For the time being, I agree with your recommendation to not buy these
> > > > devices if you care about camera support.
> > >
> > > I second this, don't buy these devices if the vendor is not willing to
> > > get their drivers upstreamed properly.
> >
> > "Not willing" may be a bit too harsh here. I wouldn't just blame Intel
> > for not upstreaming a driver if it turns out that the V4L2 API isn't a
> > good match and we have no proposal to provide an alternative.
>
> Did they propose an alternative? From what I saw here they didn't even
> attempt it, or did I miss that?
The plan here is to land CAM kernel API first and then switch IPU
(driver and user-space) to it so that upstreaming for Intel will
be easier.
Hi Sergei,
On Mon, Aug 15, 2022 at 11:06:39AM +0900, Sergey Senozhatsky wrote:
> On (22/08/11 17:30), Greg KH wrote:
> > On Thu, Aug 11, 2022 at 06:08:55PM +0300, Laurent Pinchart wrote:
> > > On Thu, Aug 11, 2022 at 05:02:40PM +0200, Greg KH wrote:
> > > > On Thu, Aug 11, 2022 at 04:54:53PM +0300, Laurent Pinchart wrote:
> > > > > For the time being, I agree with your recommendation to not buy these
> > > > > devices if you care about camera support.
> > > >
> > > > I second this, don't buy these devices if the vendor is not willing to
> > > > get their drivers upstreamed properly.
> > >
> > > "Not willing" may be a bit too harsh here. I wouldn't just blame Intel
> > > for not upstreaming a driver if it turns out that the V4L2 API isn't a
> > > good match and we have no proposal to provide an alternative.
> >
> > Did they propose an alternative? From what I saw here they didn't even
> > attempt it, or did I miss that?
>
> The plan here is to land CAM kernel API first and then switch IPU
> (driver and user-space) to it so that upstreaming for Intel will
> be easier.
In its current form, that API will likely have trouble going upstream.
Reception wasn't exactly positive in the mainline kernel community. I
would like to restate my recommendation of sitting down with the
upstream community around the same (virtual) table to design an that
will be suitable for the upstream kernel.
--
Regards,
Laurent Pinchart
On Mon, Aug 15, 2022 at 11:06:39AM +0900, Sergey Senozhatsky wrote:
> On (22/08/11 17:30), Greg KH wrote:
> > On Thu, Aug 11, 2022 at 06:08:55PM +0300, Laurent Pinchart wrote:
> > > On Thu, Aug 11, 2022 at 05:02:40PM +0200, Greg KH wrote:
> > > > On Thu, Aug 11, 2022 at 04:54:53PM +0300, Laurent Pinchart wrote:
> > > > > For the time being, I agree with your recommendation to not buy these
> > > > > devices if you care about camera support.
> > > >
> > > > I second this, don't buy these devices if the vendor is not willing to
> > > > get their drivers upstreamed properly.
> > >
> > > "Not willing" may be a bit too harsh here. I wouldn't just blame Intel
> > > for not upstreaming a driver if it turns out that the V4L2 API isn't a
> > > good match and we have no proposal to provide an alternative.
> >
> > Did they propose an alternative? From what I saw here they didn't even
> > attempt it, or did I miss that?
>
> The plan here is to land CAM kernel API first and then switch IPU
> (driver and user-space) to it so that upstreaming for Intel will
> be easier.
And what is the timeframe on the plan? Where will these changes be sent
to for review? I'm guessing they are already in a shipping device so
what's the delay?
thanks,
greg k-h
On Mon, Aug 15, 2022 at 03:43:48PM +0900, Sergey Senozhatsky wrote:
> On (22/08/15 08:36), Greg KH wrote:
> > On Mon, Aug 15, 2022 at 11:06:39AM +0900, Sergey Senozhatsky wrote:
> > > On (22/08/11 17:30), Greg KH wrote:
> > > > On Thu, Aug 11, 2022 at 06:08:55PM +0300, Laurent Pinchart wrote:
> > > > > On Thu, Aug 11, 2022 at 05:02:40PM +0200, Greg KH wrote:
> > > > > > On Thu, Aug 11, 2022 at 04:54:53PM +0300, Laurent Pinchart wrote:
> > > > > > > For the time being, I agree with your recommendation to not buy these
> > > > > > > devices if you care about camera support.
> > > > > >
> > > > > > I second this, don't buy these devices if the vendor is not willing to
> > > > > > get their drivers upstreamed properly.
> > > > >
> > > > > "Not willing" may be a bit too harsh here. I wouldn't just blame Intel
> > > > > for not upstreaming a driver if it turns out that the V4L2 API isn't a
> > > > > good match and we have no proposal to provide an alternative.
> > > >
> > > > Did they propose an alternative? From what I saw here they didn't even
> > > > attempt it, or did I miss that?
> > >
> > > The plan here is to land CAM kernel API first and then switch IPU
> > > (driver and user-space) to it so that upstreaming for Intel will
> > > be easier.
> >
> > And what is the timeframe on the plan? Where will these changes be sent
> > to for review? I'm guessing they are already in a shipping device so
> > what's the delay?
>
> We haven't sent out KCAM for upstream review yet. It's open sourced,
> as of this moment [1], but we still need some time and wanted to convert
> one of the previous generations of IPU drivers (IPU3) to KCAM first to
> see if everything is working as we wanted it to.
That didn't answer my question on when you were planning to actually
submit this :)
thanks,
greg k-h
On (22/08/15 09:02), Greg KH wrote:
[..]
> > We haven't sent out KCAM for upstream review yet. It's open sourced,
> > as of this moment [1], but we still need some time and wanted to convert
> > one of the previous generations of IPU drivers (IPU3) to KCAM first to
> > see if everything is working as we wanted it to.
>
> That didn't answer my question on when you were planning to actually
> submit this :)
Definitely not today. Someday, for sure :)
I don't want to promise any timelines. But we are certainly not
talking "weeks", we are talking "months". Several months is a
realistic timeline.
On Mon, Aug 15, 2022 at 04:07:32PM +0900, Sergey Senozhatsky wrote:
> On (22/08/15 09:02), Greg KH wrote:
> [..]
> > > We haven't sent out KCAM for upstream review yet. It's open sourced,
> > > as of this moment [1], but we still need some time and wanted to convert
> > > one of the previous generations of IPU drivers (IPU3) to KCAM first to
> > > see if everything is working as we wanted it to.
> >
> > That didn't answer my question on when you were planning to actually
> > submit this :)
>
> Definitely not today. Someday, for sure :)
>
> I don't want to promise any timelines. But we are certainly not
> talking "weeks", we are talking "months". Several months is a
> realistic timeline.
If I were to bet, I'd say no less than a year :-) And even then, as
stated separately, you'll have to convince the upstream community that
the API is suitable. It would be best to work with upstream today on a
proposal that please all parties, and not wait a year for a full
implementation to be ready, submitted, and rejected.
--
Regards,
Laurent Pinchart
On (22/08/15 08:36), Greg KH wrote:
> On Mon, Aug 15, 2022 at 11:06:39AM +0900, Sergey Senozhatsky wrote:
> > On (22/08/11 17:30), Greg KH wrote:
> > > On Thu, Aug 11, 2022 at 06:08:55PM +0300, Laurent Pinchart wrote:
> > > > On Thu, Aug 11, 2022 at 05:02:40PM +0200, Greg KH wrote:
> > > > > On Thu, Aug 11, 2022 at 04:54:53PM +0300, Laurent Pinchart wrote:
> > > > > > For the time being, I agree with your recommendation to not buy these
> > > > > > devices if you care about camera support.
> > > > >
> > > > > I second this, don't buy these devices if the vendor is not willing to
> > > > > get their drivers upstreamed properly.
> > > >
> > > > "Not willing" may be a bit too harsh here. I wouldn't just blame Intel
> > > > for not upstreaming a driver if it turns out that the V4L2 API isn't a
> > > > good match and we have no proposal to provide an alternative.
> > >
> > > Did they propose an alternative? From what I saw here they didn't even
> > > attempt it, or did I miss that?
> >
> > The plan here is to land CAM kernel API first and then switch IPU
> > (driver and user-space) to it so that upstreaming for Intel will
> > be easier.
>
> And what is the timeframe on the plan? Where will these changes be sent
> to for review? I'm guessing they are already in a shipping device so
> what's the delay?
We haven't sent out KCAM for upstream review yet. It's open sourced,
as of this moment [1], but we still need some time and wanted to convert
one of the previous generations of IPU drivers (IPU3) to KCAM first to
see if everything is working as we wanted it to.
[1] For everyone who is curious, our first public KCAM version can
be found here:
https://chromium-review.googlesource.com/c/chromiumos/third_party/kernel/+/3668500/
On Mon, Aug 15, 2022 at 04:07:32PM +0900, Sergey Senozhatsky wrote:
> On (22/08/15 09:02), Greg KH wrote:
> [..]
> > > We haven't sent out KCAM for upstream review yet. It's open sourced,
> > > as of this moment [1], but we still need some time and wanted to convert
> > > one of the previous generations of IPU drivers (IPU3) to KCAM first to
> > > see if everything is working as we wanted it to.
> >
> > That didn't answer my question on when you were planning to actually
> > submit this :)
>
> Definitely not today. Someday, for sure :)
>
> I don't want to promise any timelines. But we are certainly not
> talking "weeks", we are talking "months". Several months is a
> realistic timeline.
Ok, so getting this merged is a good year out at the best, realisticly 2
years given that once you submit the first version for review, the real
work will start happening.
So I'll stick with my original statement, don't buy this hardware as the
vendors don't seem to want to upstream the drivers any time soon :(
thanks,
greg k-h
On (22/08/15 09:14), Greg KH wrote:
> > Definitely not today. Someday, for sure :)
> >
> > I don't want to promise any timelines. But we are certainly not
> > talking "weeks", we are talking "months". Several months is a
> > realistic timeline.
>
> Ok, so getting this merged is a good year out at the best, realisticly 2
> years given that once you submit the first version for review, the real
> work will start happening.
Well, nothing is really set in stone here. We may revisit our strategy,
say turn it upside down, and hit upstream mailing list as soon as possible
instead.
> So I'll stick with my original statement, don't buy this hardware
... if you plan on running vanilla kernel. Otherwise do buy :)
On Mon, Aug 15, 2022 at 04:24:19PM +0900, Sergey Senozhatsky wrote:
> On (22/08/15 09:14), Greg KH wrote:
> > > Definitely not today. Someday, for sure :)
> > >
> > > I don't want to promise any timelines. But we are certainly not
> > > talking "weeks", we are talking "months". Several months is a
> > > realistic timeline.
> >
> > Ok, so getting this merged is a good year out at the best, realisticly 2
> > years given that once you submit the first version for review, the real
> > work will start happening.
>
> Well, nothing is really set in stone here. We may revisit our strategy,
> say turn it upside down, and hit upstream mailing list as soon as possible
> instead.
>
> > So I'll stick with my original statement, don't buy this hardware
> ... if you plan on running vanilla kernel. Otherwise do buy :)
If you want to run a proprietary OS.
--
Regards,
Laurent Pinchart
On Mon, Aug 15, 2022 at 04:24:19PM +0900, Sergey Senozhatsky wrote:
> On (22/08/15 09:14), Greg KH wrote:
> > > Definitely not today. Someday, for sure :)
> > >
> > > I don't want to promise any timelines. But we are certainly not
> > > talking "weeks", we are talking "months". Several months is a
> > > realistic timeline.
> >
> > Ok, so getting this merged is a good year out at the best, realisticly 2
> > years given that once you submit the first version for review, the real
> > work will start happening.
>
> Well, nothing is really set in stone here. We may revisit our strategy,
> say turn it upside down, and hit upstream mailing list as soon as possible
> instead.
I would recommend this, as you all have been talking about this for a
very long time now with no actual code submissions anywhere (which is
what actually matters). Waiting even longer feels like you are just
delaying the real work that has to be done, and it prevents hardware
that people have now from ever being supported in the device's lifetime.
But hey, what do I know about getting code upstream :)
good luck!
greg k-h
Hi Sergey,
On Mon, Aug 15, 2022 at 03:43:48PM +0900, Sergey Senozhatsky wrote:
> On (22/08/15 08:36), Greg KH wrote:
> > On Mon, Aug 15, 2022 at 11:06:39AM +0900, Sergey Senozhatsky wrote:
> > > On (22/08/11 17:30), Greg KH wrote:
> > > > On Thu, Aug 11, 2022 at 06:08:55PM +0300, Laurent Pinchart wrote:
> > > > > On Thu, Aug 11, 2022 at 05:02:40PM +0200, Greg KH wrote:
> > > > > > On Thu, Aug 11, 2022 at 04:54:53PM +0300, Laurent Pinchart wrote:
> > > > > > > For the time being, I agree with your recommendation to not buy these
> > > > > > > devices if you care about camera support.
> > > > > >
> > > > > > I second this, don't buy these devices if the vendor is not willing to
> > > > > > get their drivers upstreamed properly.
> > > > >
> > > > > "Not willing" may be a bit too harsh here. I wouldn't just blame Intel
> > > > > for not upstreaming a driver if it turns out that the V4L2 API isn't a
> > > > > good match and we have no proposal to provide an alternative.
> > > >
> > > > Did they propose an alternative? From what I saw here they didn't even
> > > > attempt it, or did I miss that?
> > >
> > > The plan here is to land CAM kernel API first and then switch IPU
> > > (driver and user-space) to it so that upstreaming for Intel will
> > > be easier.
> >
> > And what is the timeframe on the plan? Where will these changes be sent
> > to for review? I'm guessing they are already in a shipping device so
> > what's the delay?
>
> We haven't sent out KCAM for upstream review yet. It's open sourced,
> as of this moment [1], but we still need some time and wanted to convert
> one of the previous generations of IPU drivers (IPU3) to KCAM first to
> see if everything is working as we wanted it to.
>
> [1] For everyone who is curious, our first public KCAM version can
> be found here:
> https://chromium-review.googlesource.com/c/chromiumos/third_party/kernel/+/3668500/
Thanks for the URL.
Looking at the code, I wouldn't have guessed this is an API for cameras if
I hadn't been told so. I expect changes, possibly major changes, in the API
as well as the underlying framework will be needed for upstreaming. The
exact API toward user space that is required to make use of the camera
related hardware doesn't seem to be there, nor there seem to be any
in-kernel API for exchanging information between drivers.
Having said that, I don't want to judge this too harshly (e.g. without
documentation that supposedly will be written?) at this point as it is not
meant for upstream review yet. Drivers for actual hardware would tell more,
too.
--
Regards,
Sakari Ailus
On 8/15/22 03:40, Laurent Pinchart wrote:
> On Mon, Aug 15, 2022 at 04:24:19PM +0900, Sergey Senozhatsky wrote:
>> On (22/08/15 09:14), Greg KH wrote:
>>>> Definitely not today. Someday, for sure :)
>>>>
>>>> I don't want to promise any timelines. But we are certainly not
>>>> talking "weeks", we are talking "months". Several months is a
>>>> realistic timeline.
>>>
>>> Ok, so getting this merged is a good year out at the best, realisticly 2
>>> years given that once you submit the first version for review, the real
>>> work will start happening.
>>
>> Well, nothing is really set in stone here. We may revisit our strategy,
>> say turn it upside down, and hit upstream mailing list as soon as possible
>> instead.
>>
>>> So I'll stick with my original statement, don't buy this hardware
>> ... if you plan on running vanilla kernel. Otherwise do buy :)
>
> If you want to run a proprietary OS.
>
Not sure where best to jump in...but just to add some perspective from a
Lenovo perspective.
100% agreed - don't buy the platforms (X1 Carbon G10, X1 Yoga G7, X1
Nano G2) with the MIPI camera (called 'computer vision' on the Lenovo
site) if you plan to run Linux. We are disabling that config on our
Linux preload.
From the 'reaching out to the community' perspective that was flagged
earlier in the thread - I've been discussing MIPI with Red Hat,
Canonical and Intel for a year now. We knew it was going to be a problem
and in fairness to Intel they've been very clear that initially the
'workaround' was going to be available and that the full solution will
be coming later. I've had some disappointments with progress but I also
don't have the expertise to be able to solve it myself and fully
appreciate these things don't happen overnight. Intel have been working
with us on this and I appreciate their support.
If there are people I should be engaged with let me know and we can
discuss offline. Going forward - I guess the good news is we have some
platforms with MIPI on to work with....having HW usually helps. I'm also
very open to any ideas of suggestions for accelerating getting this done
and where Lenovo can help. We're limited on a technical resource side of
things, but we do have HW :)
Mark
Hi Greg,
On Mon, Aug 15, 2022 at 09:14:52AM +0200, Greg KH wrote:
> On Mon, Aug 15, 2022 at 04:07:32PM +0900, Sergey Senozhatsky wrote:
> > On (22/08/15 09:02), Greg KH wrote:
> > [..]
> > > > We haven't sent out KCAM for upstream review yet. It's open sourced,
> > > > as of this moment [1], but we still need some time and wanted to convert
> > > > one of the previous generations of IPU drivers (IPU3) to KCAM first to
> > > > see if everything is working as we wanted it to.
> > >
> > > That didn't answer my question on when you were planning to actually
> > > submit this :)
> >
> > Definitely not today. Someday, for sure :)
> >
> > I don't want to promise any timelines. But we are certainly not
> > talking "weeks", we are talking "months". Several months is a
> > realistic timeline.
>
> Ok, so getting this merged is a good year out at the best, realisticly 2
> years given that once you submit the first version for review, the real
> work will start happening.
>
> So I'll stick with my original statement, don't buy this hardware as the
> vendors don't seem to want to upstream the drivers any time soon :(
Do note that USB webcams in Alder Lake laptops still work as usual, with
the uvcvideo driver.
V4L2 + MC are not great APIs for supporting hardware such as IPU6 and
coming up with an alternative is a major and risky endeavour. I expect many
developers of drivers for similar hardware are in the same situation.
The hardware is getting increasingly complex and while there is some
standardisation in the industry, it's mainly focussed on interoperatibility
on hardware level rather than control interfaces or defining how a given
feature is to be implemented.
As the camera is no longer a single, integrated device in this context but
multiple devices from different vendors, there still remains a larger
requirement for interoperability between, at the very least, cameras and
CSI-2 receivers --- that in turn increasingly often are integrated in Image
Signal Processors such as IPU6. This further raises the bar for an
interface that would better support these devices.
This of course does not help the owners of such hardware but perhaps
explains the current state of affairs a little. Eventually we will need a
new kernel interface for this but at this point I can't tell whether it
will be based on KCAM Sergey mentioned, or not.
--
Regards,
Sakari Ailus
On Wed, Aug 17, 2022 at 08:09:16AM +0000, Sakari Ailus wrote:
> V4L2 + MC are not great APIs for supporting hardware such as IPU6 and
> coming up with an alternative is a major and risky endeavour. I expect many
> developers of drivers for similar hardware are in the same situation.
What has been done to help maybe make those apis work properly for this
hardware? The hardware design pipeline is very long, this isn't
anything new that people just suddenly realized was going to need to
have full support on Linux.
> The hardware is getting increasingly complex and while there is some
> standardisation in the industry, it's mainly focussed on interoperatibility
> on hardware level rather than control interfaces or defining how a given
> feature is to be implemented.
Hardware standardization isn't an issue here, it's having working Linux
code to support all the crazy devices.
> As the camera is no longer a single, integrated device in this context but
> multiple devices from different vendors, there still remains a larger
> requirement for interoperability between, at the very least, cameras and
> CSI-2 receivers --- that in turn increasingly often are integrated in Image
> Signal Processors such as IPU6. This further raises the bar for an
> interface that would better support these devices.
Ok, great, what type of api would work for this hardware? You all have
access to the hardware, know how it works, and know how userspace needs
to consume it. So why not work to create the needed kernel support for
this now? What is the causing the delay?
thanks,
greg k-h
Hi Mark,
On Mon, Aug 15, 2022 at 11:14:52AM -0400, Mark Pearson wrote:
> On 8/15/22 03:40, Laurent Pinchart wrote:
> > On Mon, Aug 15, 2022 at 04:24:19PM +0900, Sergey Senozhatsky wrote:
> >> On (22/08/15 09:14), Greg KH wrote:
> >>>> Definitely not today. Someday, for sure :)
> >>>>
> >>>> I don't want to promise any timelines. But we are certainly not
> >>>> talking "weeks", we are talking "months". Several months is a
> >>>> realistic timeline.
> >>>
> >>> Ok, so getting this merged is a good year out at the best, realisticly 2
> >>> years given that once you submit the first version for review, the real
> >>> work will start happening.
> >>
> >> Well, nothing is really set in stone here. We may revisit our strategy,
> >> say turn it upside down, and hit upstream mailing list as soon as possible
> >> instead.
> >>
> >>> So I'll stick with my original statement, don't buy this hardware
> >> ... if you plan on running vanilla kernel. Otherwise do buy :)
> >
> > If you want to run a proprietary OS.
> >
> Not sure where best to jump in...but just to add some perspective from a
> Lenovo perspective.
>
> 100% agreed - don't buy the platforms (X1 Carbon G10, X1 Yoga G7, X1
> Nano G2) with the MIPI camera (called 'computer vision' on the Lenovo
> site) if you plan to run Linux. We are disabling that config on our
> Linux preload.
>
> From the 'reaching out to the community' perspective that was flagged
> earlier in the thread - I've been discussing MIPI with Red Hat,
> Canonical and Intel for a year now. We knew it was going to be a problem
> and in fairness to Intel they've been very clear that initially the
> 'workaround' was going to be available and that the full solution will
> be coming later. I've had some disappointments with progress but I also
> don't have the expertise to be able to solve it myself and fully
> appreciate these things don't happen overnight. Intel have been working
> with us on this and I appreciate their support.
>
> If there are people I should be engaged with let me know and we can
> discuss offline. Going forward - I guess the good news is we have some
> platforms with MIPI on to work with....having HW usually helps. I'm also
> very open to any ideas of suggestions for accelerating getting this done
> and where Lenovo can help. We're limited on a technical resource side of
> things, but we do have HW :)
Thank you for the insight.
Focussing on the IPU6, we need two things: an upstream kernel driver,
and corresponding support in the libcamera project (if you're familiar
with the Linux display stack, that would be the equivalent of a DRM
driver and corresponding support in Mesa for a GPU).
The kernel driver likely needs to come first, and there we have two
options. The first one is implementing a V4L2 driver, as that's the
camera API in the Linux kernel today. As mentioned by Sakari, it can be
done, but isn't a great match due to the IPU6 hardware complexity. V4L2
is generally considered as a dead end for recent ISPs, it may still be
able to cover that space for some time, but its days are counted (in
this area only, the API works fine for lots of other use cases).
The second option is to create a better kernel API, and then implement
an IPU6 driver for it. Ideas have been brainstormed for a few years now,
and the Google Chrome OS proposal that was unveiled by Ricardo at the
Kernel Recipes conference in June is one particular embodiment of this.
There is however no consensus on that proposal (that may be the biggest
euphemism I've ever made).
--
Regards,
Laurent Pinchart
On (22/08/17 08:09), Sakari Ailus wrote:
>
> Do note that USB webcams in Alder Lake laptops still work as usual, with
> the uvcvideo driver.
A massive +1 to this.
> This of course does not help the owners of such hardware but perhaps
> explains the current state of affairs a little. Eventually we will need a
> new kernel interface for this but at this point I can't tell whether it
> will be based on KCAM Sergey mentioned, or not.
We don't consider KCAM ready and this is why we have not published it
to the mailing list. We haven't really started with KCAM yet, but we
are about to do so. And I'm sure that we will find out a thing or two
missing or not working properly while we convert a real H/W driver to
KCAM. We will sit down and write some code to address those problems.
Hi Greg, others,
On Thu, Aug 11, 2022 at 05:02:40PM +0200, Greg KH wrote:
> On Thu, Aug 11, 2022 at 04:54:53PM +0300, Laurent Pinchart wrote:
> > For the time being, I agree with your recommendation to not buy these
> > devices if you care about camera support.
>
> I second this, don't buy these devices if the vendor is not willing to
> get their drivers upstreamed properly.
I can now confirm that IPU6 driver upstreaming is now planned, with IPU6
input system driver to be upstreamed first. The intent is that we would
have patches for review on LMML around the end of the year.
The processing system driver will need more work to replace the custom
interface. For a fully functional camera stack, also libcamera support for
IPU6 will be needed.
I think I can say a major factor for why we're here is that the original
intent was to use a different interface for upstream but as we don't have
one yet and probably won't for quite some time, this doesn't really seem
like a viable option anymore.
So I'm afraid I can't promise a quick fix but at the same time I can say
work is being done to address this.
--
Kind regards,
Sakari Ailus
Hi Sakari,
On 9/22/22 6:06 PM, Sakari Ailus wrote:
> Hi Greg, others,
>
> On Thu, Aug 11, 2022 at 05:02:40PM +0200, Greg KH wrote:
>> On Thu, Aug 11, 2022 at 04:54:53PM +0300, Laurent Pinchart wrote:
>>> For the time being, I agree with your recommendation to not buy these
>>> devices if you care about camera support.
>>
>> I second this, don't buy these devices if the vendor is not willing to
>> get their drivers upstreamed properly.
>
> I can now confirm that IPU6 driver upstreaming is now planned, with IPU6
> input system driver to be upstreamed first. The intent is that we would
> have patches for review on LMML around the end of the year.
Is there any update on IPU6 driver? Probably it hasn't been sent upstream yet?
>
> The processing system driver will need more work to replace the custom
> interface. For a fully functional camera stack, also libcamera support for
> IPU6 will be needed.
>
> I think I can say a major factor for why we're here is that the original
> intent was to use a different interface for upstream but as we don't have
> one yet and probably won't for quite some time, this doesn't really seem
> like a viable option anymore.
>
> So I'm afraid I can't promise a quick fix but at the same time I can say
> work is being done to address this.
>
--
BR,
Muhammad Usama Anjum
On Fri, May 26, 2023 at 03:51:36PM +0500, Muhammad Usama Anjum wrote:
> On 9/22/22 6:06 PM, Sakari Ailus wrote:
> > On Thu, Aug 11, 2022 at 05:02:40PM +0200, Greg KH wrote:
> >> On Thu, Aug 11, 2022 at 04:54:53PM +0300, Laurent Pinchart wrote:
> >>> For the time being, I agree with your recommendation to not buy these
> >>> devices if you care about camera support.
> >>
> >> I second this, don't buy these devices if the vendor is not willing to
> >> get their drivers upstreamed properly.
> >
> > I can now confirm that IPU6 driver upstreaming is now planned, with IPU6
> > input system driver to be upstreamed first. The intent is that we would
> > have patches for review on LMML around the end of the year.
>
> Is there any update on IPU6 driver? Probably it hasn't been sent upstream yet?
The IPU6 requires multiple drivers. The "easy" part (the IPU6 input
system) has been submitted to the linux-media mailing list (see [1]).
That driver will allow capturing raw frames from the camera sensors
(assuming drivers for the sensors in your particular device are
available). Another driver has also be submitted to setup the IVSC (a
chip present in some systems between the camera sensor and the IPU6),
see [2].
The hard part (the IPU6 processing system) will still take time, and
once a driver for that is available, the userspace implementation in
libcamera will also take time.
[1] https://lore.kernel.org/linux-media/[email protected]/
[2] https://lore.kernel.org/linux-media/[email protected]/
> > The processing system driver will need more work to replace the custom
> > interface. For a fully functional camera stack, also libcamera support for
> > IPU6 will be needed.
> >
> > I think I can say a major factor for why we're here is that the original
> > intent was to use a different interface for upstream but as we don't have
> > one yet and probably won't for quite some time, this doesn't really seem
> > like a viable option anymore.
> >
> > So I'm afraid I can't promise a quick fix but at the same time I can say
> > work is being done to address this.
--
Regards,
Laurent Pinchart
Hi Muhammad,
On Fri, May 26, 2023 at 03:51:36PM +0500, Muhammad Usama Anjum wrote:
> Hi Sakari,
>
> On 9/22/22 6:06 PM, Sakari Ailus wrote:
> > Hi Greg, others,
> >
> > On Thu, Aug 11, 2022 at 05:02:40PM +0200, Greg KH wrote:
> >> On Thu, Aug 11, 2022 at 04:54:53PM +0300, Laurent Pinchart wrote:
> >>> For the time being, I agree with your recommendation to not buy these
> >>> devices if you care about camera support.
> >>
> >> I second this, don't buy these devices if the vendor is not willing to
> >> get their drivers upstreamed properly.
> >
> > I can now confirm that IPU6 driver upstreaming is now planned, with IPU6
> > input system driver to be upstreamed first. The intent is that we would
> > have patches for review on LMML around the end of the year.
> Is there any update on IPU6 driver? Probably it hasn't been sent upstream
> yet?
I can confirm work is ongoing to upstream IPU6 ISYS driver. I'd hope to
have that in upstream some time this summer.
--
Kind regards,
Sakari Ailus