Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
Subject: Re: [PATCH v2 0/3] media: vimc: Allow multiple capture devices to use
 the same sensor
To:     kieran.bingham@ideasonboard.com,
        Kaaira Gupta <kgupta@es.iitr.ac.in>,
        Helen Koike <helen.koike@collabora.com>
Cc:     =?UTF-8?Q?Niklas_S=c3=b6derlund?= <niklas.soderlund@ragnatech.se>,
        Shuah Khan <skhan@linuxfoundation.org>,
        Mauro Carvalho Chehab <mchehab@kernel.org>,
        linux-media@vger.kernel.org, linux-kernel@vger.kernel.org,
        Hans Verkuil <hverkuil@xs4all.nl>,
        Laurent Pinchart <laurent.pinchart@ideasonboard.com>,
        Ezequiel Garcia <ezequiel@collabora.com>
References: <20200724120213.17119-1-kgupta@es.iitr.ac.in>
 <20200724121521.GA2705690@oden.dyn.berto.se>
 <20200724122104.GA18482@kaaira-HP-Pavilion-Notebook>
 <a6f4eabf-6cd5-950b-f2e3-853370c77629@ideasonboard.com>
 <2a6cb067-283d-ca65-2698-1fae66a17d02@collabora.com>
 <20200728113959.GA6350@kaaira-HP-Pavilion-Notebook>
 <3a9ac970-77b8-1bc5-536a-5b4f2bd60745@collabora.com>
 <b5fd3811-2f0e-7563-13fa-bb1e32189814@collabora.com>
 <f2fc6d81-16a6-206b-6bb2-fb99d5a84f65@ideasonboard.com>
 <cd1d18ef-d4ff-8240-e9dd-2d9f9c7c5a45@collabora.com>
 <a2ae2dba-6076-0abc-f620-9047a52b513c@ideasonboard.com>
From:   Dafna Hirschfeld <dafna.hirschfeld@collabora.com>
Message-ID: <5faeed28-75b2-48d8-4a48-c38418fd89f2@collabora.com>
Date:   Wed, 29 Jul 2020 17:24:25 +0200
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101
 Thunderbird/68.10.0
MIME-Version: 1.0
In-Reply-To: <a2ae2dba-6076-0abc-f620-9047a52b513c@ideasonboard.com>
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Language: en-US
Content-Transfer-Encoding: 8bit
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk


On 29.07.20 15:27, Kieran Bingham wrote:
> Hi Dafna, Kaaira,
> 
> On 29/07/2020 14:16, Dafna Hirschfeld wrote:
>>
>>
>> On 29.07.20 15:05, Kieran Bingham wrote:
>>> Hi Dafna,
>>>
>>> On 28/07/2020 15:00, Dafna Hirschfeld wrote:
>>>>
>>>>
>>>> On 28.07.20 14:07, Dafna Hirschfeld wrote:
>>>>> Hi
>>>>>
>>>>> On 28.07.20 13:39, Kaaira Gupta wrote:
>>>>>> On Mon, Jul 27, 2020 at 02:54:30PM -0300, Helen Koike wrote:
>>>>>>> Hi,
>>>>>>>
>>>>>>> On 7/27/20 11:31 AM, Kieran Bingham wrote:
>>>>>>>> Hi all,
>>>>>>>>
>>>>>>>> +Dafna for the thread discussion, as she's missed from the to/cc
>>>>>>>> list.
>>>>>>>>
>>>>>>>>
>>>>>>>> On 24/07/2020 13:21, Kaaira Gupta wrote:
>>>>>>>>> On Fri, Jul 24, 2020 at 02:15:21PM +0200, Niklas Söderlund wrote:
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>>> Hi Kaaira,
>>>>>>>>>>
>>>>>>>>>> Thanks for your work.
>>>>>>>>>
>>>>>>>>> Thanks for yours :D
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On 2020-07-24 17:32:10 +0530, Kaaira Gupta wrote:
>>>>>>>>>>> This is version 2 of the patch series posted by Niklas for
>>>>>>>>>>> allowing
>>>>>>>>>>> multiple streams in VIMC.
>>>>>>>>>>> The original series can be found here:
>>>>>>>>>>> https://patchwork.kernel.org/cover/10948831/
>>>>>>>>>>>
>>>>>>>>>>> This series adds support for two (or more) capture devices to be
>>>>>>>>>>> connected to the same sensors and run simultaneously. Each
>>>>>>>>>>> capture device
>>>>>>>>>>> can be started and stopped independent of each other.
>>>>>>>>>>>
>>>>>>>>>>> Patch 1/3 and 2/3 deals with solving the issues that arises once
>>>>>>>>>>> two
>>>>>>>>>>> capture devices can be part of the same pipeline. While 3/3
>>>>>>>>>>> allows for
>>>>>>>>>>> two capture devices to be part of the same pipeline and thus
>>>>>>>>>>> allows for
>>>>>>>>>>> simultaneously use.
>>>>>
>>>>> I wonder if these two patches are enough, since each vimc entity also
>>>>> have
>>>>> a 'process_frame' callback, but only one allocated frame. That means
>>>>> that the 'process_frame' can be called concurrently by two different
>>>>> streams
>>>>> on the same frame and cause corruption.
>>>>>
>>>>
>>>> I think we should somehow change the vimc-stream.c code so that we have
>>>> only
>>>> one stream process per pipe. So if one capture is already streaming,
>>>> then the new
>>>> capture that wants to stream uses the same thread so we don't have two
>>>> threads
>>>> both calling 'process_frame'.
>>>
>>>
>>> Yes, I think it looks and sounds like there are two threads running when
>>> there are two streams.
>>>
>>> so in effect, although they 'share a pipe', aren't they in effect just
>>> sending two separate buffers through their stream-path?
>>>
>>> If that's the case, then I don't think there's any frame corruption,
>>> because they would both have grabbed their own frame separately.
>>
>> But each entity allocates just one buffer. So the same buffer is used for
>> both stream.
> 
> Aha, ok, I hadn't realised there was only a single buffer available in
> the pipeline for each entity. Indeed there is a risk of corruption in
> that case.
> 
>> What for example can happen is that the debayer of one stream can read the
>> sensor's buffer while the sensor itself writes to the buffer for the other
>> stream.
> 
> 
> So, In that case, we have currently got a scenario where each 'stream'
> really is operating it's own pipe (even though all components are reused).
> 
> Two questions:
> 
> Is this acceptable, and we should just use a mutex to ensure the buffers
> are not corrupted, but essentially each stream is a separate temporal
> capture?
> 
> 
> Or B:
> 
> Should we refactor to make sure that there is a single thread, and the
> code which calls process_frame on each entity should become aware of the
> potential for multiple paths at the point of the sensor.
> 
> 
> I suspect option B is really the 'right' path to take, but it is more
> complicated of course.

I also think option B is preferable.

Maybe we can add a bool field 'is_streaming' to struct 'vimc_ent_device'
The stream thread can do a BFS scan from the sensor up to the captures
and call the 'process_frame' for each entity if 'is_streaming == true'.
When a new capture wants to stream it sets 'is_streaming = true'
on the entities on his streaming path.

Thanks,
Dafna


> 
> --
> Kieran
> 
> 
> 
> 
>> Thanks,
>> Dafna
>>
>>>
>>>
>>> I don't think that's a good example of the hardware though, as that
>>> doesn't reflect what 'should' happen where the TPG runs once to generate
>>> a frame at the sensor, which is then read by both the debayer entity and
>>> the RAW capture device when there are two streams...
>>>
>>>
>>> So I suspect trying to move to a single thread is desirable, but that
>>> might be a fair bit of work also.
>>>
>>> -- 
>>> Kieran
>>>
>>>
>>>
>>>> The second capture that wants to stream should iterate the topology
>>>> downwards until
>>>> reaching an entity that already belong to the stream path of the other
>>>> streaming capture
>>>> and tell the streamer it wants to read the frames this entity
>>>> produces.
>>>>
>>>> Thanks,
>>>> Dafna
>>>>
>>>>> Thanks,
>>>>> Dafna
>>>>>
>>>>>>>>>>
>>>>>>>>>> I'm just curious if you are aware of this series? It would
>>>>>>>>>> replace the
>>>>>>>>>> need for 1/3 and 2/3 of this series right?
>>>>>>>>>
>>>>>>>>> v3 of this series replaces the need for 1/3, but not the current
>>>>>>>>> version
>>>>>>>>> (ie v4). v4 of patch 2/5 removes the stream_counter that is
>>>>>>>>> needed to
>>>>>>>>> keep count of the calls to s_stream. Hence 1/3 becomes relevant
>>>>>>>>> again.
>>>>>>>>
>>>>>>>> So the question really is, how do we best make use of the two
>>>>>>>> current
>>>>>>>> series, to achieve our goal of supporting multiple streams.
>>>>>>>>
>>>>>>>> Having not parsed Dafna's series yet, do we need to combine
>>>>>>>> elements of
>>>>>>>> both ? Or should we work towards starting with this series and get
>>>>>>>> dafna's patches built on top ?
>>>>>>>>
>>>>>>>> Or should patch 1/3 and 3/3 of this series be on top of Dafna's v4 ?
>>>>>>>>
>>>>>>>> (It might be noteworthy to say that Kaaira has reported successful
>>>>>>>> multiple stream operation from /this/ series and her development
>>>>>>>> branch
>>>>>>>> on libcamera).
>>>>>>>
>>>>>>> Dafna's patch seems still under discussion, but I don't want to
>>>>>>> block progress in Vimc either.
>>>>>>>
>>>>>>> So I was wondering if we can move forward with Vimc support for
>>>>>>> multistreaming,
>>>>>>> without considering Dafna's patchset, and we can do the clean up
>>>>>>> later once we solve that.
>>>>>>>
>>>>>>> What do you think?
>>>>>>
>>>>>> I agree with supporting multiple streams with VIMC with this patchset,
>>>>>> and then we can refactor the counters for s_stream in VIMC later (over
>>>>>> this series) if dafna includes them in subsequent version of her
>>>>>> patchset.
>>>>>>
>>>>>
>>>>> I also think that adding support in the code will take much longer and
>>>>> should not
>>>>> stop us from supporting vimc independently.
>>>>>
>>>>> Thanks,
>>>>> Dafna
>>>>>
>>>>>>>
>>>>>>> Regards,
>>>>>>> Helen
>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>>> 1.
>>>>>>>>>> https://lore.kernel.org/linux-media/20200522075522.6190-1-dafna.hirschfeld@collabora.com/
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Changes since v1:
>>>>>>>>>>>       - All three patches rebased on latest media-tree.
>>>>>>>>>>>       Patch 3:
>>>>>>>>>>>       - Search for an entity with a non-NULL pipe instead of
>>>>>>>>>>> searching
>>>>>>>>>>>         for sensor. This terminates the search at output itself.
>>>>>>>>>>>
>>>>>>>>>>> Kaaira Gupta (3):
>>>>>>>>>>>      media: vimc: Add usage count to subdevices
>>>>>>>>>>>      media: vimc: Serialize vimc_streamer_s_stream()
>>>>>>>>>>>      media: vimc: Join pipeline if one already exists
>>>>>>>>>>>
>>>>>>>>>>>     .../media/test-drivers/vimc/vimc-capture.c    | 35
>>>>>>>>>>> ++++++++++++++++++-
>>>>>>>>>>>     .../media/test-drivers/vimc/vimc-debayer.c    |  8 +++++
>>>>>>>>>>>     drivers/media/test-drivers/vimc/vimc-scaler.c |  8 +++++
>>>>>>>>>>>     drivers/media/test-drivers/vimc/vimc-sensor.c |  9 ++++-
>>>>>>>>>>>     .../media/test-drivers/vimc/vimc-streamer.c   | 23
>>>>>>>>>>> +++++++-----
>>>>>>>>>>>     5 files changed, 73 insertions(+), 10 deletions(-)
>>>>>>>>>>>
>>>>>>>>>>> -- 
>>>>>>>>>>> 2.17.1
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> -- 
>>>>>>>>>> Regards,
>>>>>>>>>> Niklas Söderlund
>>>>>>>>
>>>
>