2. Architecture

video-pipeline consists of four high level components that describe how the flow of video data progresses through the system.

• VideoStream - Stream video frames from a video source and buffers the frames into memory.
• VideoProcessor - Processes image frames in parallel with a specified FrameFilter.
• FrameFilter - Transforms one image frame of video at a time. Used in the VideoProcessor.
• VideoTransport - Transport handler to stream content to a destination (i.e. disk/network).

These components and this document should give you a high level understanding of how video-pipeline works. Be aware that the core purpose of video-pipeline is to make it easier to understand how video processing works providing some deeper level of insight and not necessarily to be the fastest most performant solution. In some situations video-pipeline is even less performant than a custom solution. Depending on your requirements it might be worth the effort to implement lower level packages or even utilize C/C++ to reach higher performance benchmarks. To get a good overview of the performance scenarios of video-pipeline see the Performance section.

A lot of the inner workings and infrastructure in video-pipeline builds off the hard work of these fantastic lower level packages and their communities.

• imageio
• picamera
• scikit-image
• numpy

These components are separated so that each component could potentially run in parallel. “Potentially” because part of the scope of this project is to benchmark the various configurations that could be used. And also to help define clear boundaries between capturing, processing, and transporting video data.

The architecture itself follows a producer/consumer model where-as the VideoStream acts as the “Producer” and the rest of the components in the pipeline (ie VideoProcessor, VideoTransport) act as the “Consumer(s)”. Meaning that when an image frame is produced from a VideoStream source the frame is passed along to the next component in the pipeline. This was done to follow a more intuitive streaming approach to keep the design as explicit as possible, while not making it too cumbersome to reason with.

When the VideoStream “produces” an image frame it’s not really pushing that new frame to the next part of the pipeline but rather the image frames being captured are stored and then retrieved later by the pipeline itself. This allows the VideoStream to run semi-independently from the rest of the system and allows the pipeline to govern the speed in which it samples image frames. Letting the sample rate help stabilize the pipeline and mitigate backpressure issues.

After the pipeline samples image frames from the VideoStream the image frames are then passed to the VideoProcessor. The VideoProcessor can also accept a FrameFilter allowing the client to process/manipulate image frames in real-time. To learn more about the other types of FrameFilters see the Filters section. The VideoProcessor accepts each frame on the same execution context as the the pipeline and does not process them right away, instead the frames are queued up allowing for different modes of processing. These processing modes determine which execution context the image frame is processed in. The execution context is purposefully vague to indicate that it doesn’t matter whether we’re using Threads or Processes or the current execution context of the pipeline - it’s up to the client!

Once the VideoProcessor is done processing an image frame, the processed frame is appended to an output queue called the “frame buffer”. It is then the responsibility of the pipeline to pick up the processed image frames and pass them along to a VideoTransport. This transport component defines the basic structure for handling a processed image frame and follows the same model. This again allows the pipeline to drive the delivery of image frames based on the needs of the other components in the pipeline. The primary types of VideoTransports are TcpVideoTransport and FileVideoTransport. The TcpVideoTransport allows image frames to be transferred to another machine over a TCP Socket making it fairly easy to stream video data directly to another host machine. The FileVideoTransport on the other hand takes image frames and writes them to disk. All VideoTransport types can be configured with various settings defined by their respective classes.

The general flow of video data can be described more succinctly with this diagram.