2. Architecture¶
video-pipeline
consists of four high level components that describe how the
flow of video data progresses through the system.
- VideoStream - Stream video frames from a video source and buffers the frames into memory.
- VideoProcessor - Processes image frames in parallel
with a specified
FrameFilter
. - FrameFilter - Transforms one image frame of video at a time.
Used in the
VideoProcessor
. - VideoTransport - Transport handler to stream content to a destination (i.e. disk/network).
These components and this document should give you a high level understanding
of how video-pipeline
works. Be aware that the core purpose of
video-pipeline
is to make it easier to understand how video processing
works providing some deeper level of insight and not necessarily to be the
fastest most performant solution. In some situations video-pipeline
is
even less performant than a custom solution. Depending on your requirements
it might be worth the effort to implement lower level packages or even
utilize C/C++
to reach higher performance benchmarks. To get a good
overview of the performance scenarios of video-pipeline
see the
Performance section.
A lot of the inner workings and infrastructure in video-pipeline
builds
off the hard work of these fantastic lower level packages and their
communities.
These components are separated so that each component could potentially run in parallel. “Potentially” because part of the scope of this project is to benchmark the various configurations that could be used. And also to help define clear boundaries between capturing, processing, and transporting video data.
The architecture itself follows a producer/consumer model where-as the
VideoStream
acts as the “Producer” and the rest of the components
in the pipeline (ie VideoProcessor
, VideoTransport
) act as
the “Consumer(s)”. Meaning that when an image frame is produced from
a VideoStream
source the frame is passed along to the next component
in the pipeline. This was done to follow a more intuitive streaming
approach to keep the design as explicit as possible, while not making
it too cumbersome to reason with.
When the VideoStream
“produces” an image frame it’s not really pushing
that new frame to the next part of the pipeline but rather the image frames
being captured are stored and then retrieved later by the pipeline itself.
This allows the VideoStream
to run semi-independently from the rest of
the system and allows the pipeline to govern the speed in which it samples
image frames. Letting the sample rate help stabilize the pipeline and
mitigate backpressure issues.
After the pipeline samples image frames from the VideoStream
the image
frames are then passed to the VideoProcessor
. The VideoProcessor
can
also accept a FrameFilter
allowing the client to process/manipulate image
frames in real-time. To learn more about the other types of FrameFilters
see the Filters section. The VideoProcessor
accepts each frame on the
same execution context as the the pipeline and does not process them right
away, instead the frames are queued up allowing for different modes of
processing. These processing modes determine which execution context
the image frame is processed in. The execution context is purposefully
vague to indicate that it doesn’t matter whether we’re using Threads
or Processes or the current execution context of the
pipeline - it’s up to the client!
Once the VideoProcessor
is done processing an image frame, the processed
frame is appended to an output queue called the “frame buffer”. It is then the
responsibility of the pipeline to pick up the processed image frames and pass
them along to a VideoTransport
. This transport component defines the basic
structure for handling a processed image frame and follows the same model.
This again allows the pipeline to drive the delivery of image frames based
on the needs of the other components in the pipeline. The primary types of
VideoTransport
s are TcpVideoTransport
and FileVideoTransport
.
The TcpVideoTransport
allows image frames to be transferred to another
machine over a TCP Socket making it fairly
easy to stream video data directly to another host machine. The
FileVideoTransport
on the other hand takes image frames and writes
them to disk. All VideoTransport
types can be configured with
various settings defined by their respective classes.
The general flow of video data can be described more succinctly with this diagram.