Getting Started with Sprokit¶
In computer vision applications, the interaction between data structures (expressed in KWIVER as VITAL types) and algorithms (expressed in KWIVER as Arrows) can frequently be expressed as a pipeline of processing steps:
- Input processing to load images or video
- Manipulation and/or analysis of the imagery
- Output of resulting imagery and/or analytics in a useful format
A graphical representation of such a pipeline might look something like this:
The Manipulation/Analysis step, step 2, might be a collection of processing operations that build on one another to achieve some result, such as repsented in this graphical depiction of a more elaborate pipeline:
Because this type of processing architecture is so common, KWIVER includes a data flow architecture called Sprokit. Sprokit is a dynamic pipeline configuration and execution framework that combines all of KWIVER’s other components – VITAL types, Configuration blocks, and Arrows to create a powerful, dynamic system for expressing processing pipelines that address computer vision problems.
Sprokit pipelines consist of a series of Sprokit processes that are connected together through “ports” over which various VITAL types flow. A Sprokit pipeline can be a straightforward sequence of steps as shown in the first pipline Figure or can consist of many steps arranged with various branches into a more sophisticated processing system as shown in the second pipeline figure.
A key benefit of Sprokit is that it provides algorithm-independent support for system engineering issues. Much of the difficulty translating a system such as the first figure from a conceptually simple diagram into a functioning system lies in the mundane issues of data transport, buffering, synchronization, and error handling. By providing a common representation for data (via VITAL) and processing steps (via Sprokit), KWIVER allows the developer to focus on foundational algorithmic research subject to the constraint of a well-defined system interface.
Sprokit Pipeline Example¶
The easiest way to understand Sprokit is to work through an example of building and executing a pipeline using existing KWIVER Arrows. For this example, we will filter object detections using the confidence scores associated with the detection and then write them back to disk. The pipeline accepts a collection of bounding boxes as inputs. Every bounding box is characterized by the coordinates for the box, the confidence score, and a class type. The VITAL type detected_object_sets is used to represent these bounding boxes in the pipeline. The plugin_explorer application can be used to help construct the pipeline. After the pipeline is defined it can then be executed using pipeline_runner. Note that during this entire exercise no code is written or compiled.
Input¶
The first step is to define where the inputs come from and where they are
going. We’ll use KWIVER’s plugin_explorer application to identify the
processes that we want. The following command:
plugin_explorer --proc all --brief
Generates the following output (abbreviated for clarity in this document):
.
.
.
Process type: image_filter Apply selected image filter algorithm to incoming images.
Process type: image_writer Write image to disk.
Process type: image_file_reader Reads an image file given the file name.
Process type: detected_object_input Reads detected object sets from an input file.
Detections read from the input file are grouped into sets for each image
and individually returned.
Process type: detected_object_output Writes detected object sets to an output file.
All detections are written to the same file.
Process type: detected_object_filter Filters sets of detected objects using the detected_object_filter
algorithm.
Process type: video_input Reads video files and produces sequential images with metadata per frame.
.
.
.
We see detected_object_input and use the following command:
plugin_explorer --proc detected_object_input --detail
To get the following, more detailed information about detected_object_input:
Process type: detected_object_input
Description: Reads detected object sets from an input file.
Detections read from the input file are grouped into sets for each image
and individually returned.
Properties: _no_reentrant
-- Configuration --
Name : file_name
Default :
Description: Name of the detection set file to read.
Tunable : no
Name : reader
Default :
Description: Algorithm type to use as the reader.
Tunable : no
-- Input ports --
No input ports
-- Output ports --
Name : detected_object_set
Data type : kwiver:detected_object_set
Flags :
Description: Set of detected objects.
Name : image_file_name
Data type : kwiver:image_file_name
Flags :
Description: Name of an image file. The file name may contain leading path components.
What this tells us is that
- There is a
detected_object_inputprocess that takes afile_nameand areader(more on that in a moment) as a configuration parameter, - That it has no input ports
- That it produces a
detected_object_setand animage_file_nameon its output ports when it runs.
The ports in a process are the points at which one process can connect to another. Input ports of one type can be connected to output ports of the same type from a an earlier process in the pipeline. This particular process is referred to as an end cap, specifcally an input end cap for the pipeline. This is because it’s function is to load data external to the Sprokit pipeline (for example from a CSV file) and present it for processing on the Sprokit pipeline. Similarly, output end caps would have no output ports but would convert their input data to some form external to the Sprokit pipeline.
Of particular interest is the reader parameter, which lets us select the
particular arrow that we want to use to obtain our detected_object_set for
reading.
We can use the following plugin_explorer command to see what is available
for the configuration parameter:
plugin_explorer --algorithm detected_object_set_input --detail
Which results in the following output:
Plugins that implement type "detected_object_set_input"
---------------------
Info on algorithm type "detected_object_set_input" implementation "csv"
Plugin name: csv Version: 1.0
Detected object set reader using CSV format.
- 1: frame number
- 2: file name
- 3: TL-x
- 4: TL-y
- 5: BR-x
- 6: BR-y
- 7: confidence
- 8,9: class-name, score (this pair may be omitted or may repeat any
number of times)
-- Configuration --
---------------------
Info on algorithm type "detected_object_set_input" implementation "kw18"
Plugin name: kw18 Version: 1.0
Detected object set reader using kw18 format.
- Column(s) 1: Track-id
- Column(s) 2: Track-length (number of detections)
- Column(s) 3: Frame-number (-1 if not available)
- Column(s) 4-5: Tracking-plane-loc(x,y) (could be same as World-loc)
- Column(s) 6-7: Velocity(x,y)
- Column(s) 8-9: Image-loc(x,y)
- Column(s) 10-13: Img-bbox(TL_x,TL_y,BR_x,BR_y) (location of top-left &
bottom-right vertices)
- Column(s) 14: Area
- Column(s) 15-17: World-loc(x,y,z) (longitude, latitude, 0 - when
available)
- Column(s) 18: Timesetamp (-1 if not available)
- Column(s) 19: Track-confidence (-1 if not available)
-- Configuration --
---------------------
Info on algorithm type "detected_object_set_input" implementation "simulator"
Plugin name: simulator Version: 1.0
Detected object set reader using SIMULATOR format.
Detection are generated algorithmicly.
-- Configuration --
"center_x" = "100"
Description: Bounding box center x coordinate.
"center_y" = "100"
Description: Bounding box center y coordinate.
"detection_class" = "detection"
Description: Label for detection detected object type
"dx" = "0"
Description: Bounding box x translation per frame.
"dy" = "0"
Description: Bounding box y translation per frame.
"height" = "200"
Description: Bounding box height.
"max_sets" = "10"
Description: Number of detection sets to generate.
"set_size" = "4"
Description: Number of detection in a set.
"width" = "200"
Description: Bounding box width.
---------------------
Info on algorithm type "detected_object_set_input" implementation "kpf_input"
Plugin name: kpf_input Version: 1.0
Detected object set reader using kpf format.
-- Configuration --
As we can see, we have a number of choices including a CSV reader, a simulator, and some others. For this example we’ll use the CSV reader when we construct the pipeline.
Filter¶
Similarly, we can look at filters for detected_object_sets:
plugin_explorer --proc detected_object_input --detail
Which gives us:
Process type: detected_object_filter
Description: Filters sets of detected objects using the detected_object_filter
algorithm.
Properties: _no_reentrant
-- Configuration --
Name : filter
Default :
Description: Algorithm configuration subblock.
Tunable : no
-- Input ports --
Name : detected_object_set
Data type : kwiver:detected_object_set
Flags : _required
Description: Set of detected objects.
-- Output ports --
Name : detected_object_set
Data type : kwiver:detected_object_set
Flags :
Description: Set of detected objects.
And the associated Arrows:
Plugins that implement type "detected_object_filter"
---------------------
Info on algorithm type "detected_object_filter" implementation "class_probablity_filter"
Plugin name: class_probablity_filter Version: 1.0
Filters detections based on class probability.
This algorithm filters out items that are less than the threshold. The
following steps are applied to each input detected object set.
1) Select all class names with scores greater than threshold.
2) Create a new detected_object_type object with all selected class names
from step 1. The class name can be selected individually or with the
keep_all_classes option.
3) The input detection_set is cloned and the detected_object_type from
step 2 is attached.
-- Configuration --
"keep_all_classes" = "true"
Description: If this options is set to true, all classes are passed through this filter
if they are above the selected threshold.
"keep_classes" = ""
Description: A list of class names to pass through this filter. Multiple names are
separated by a ';' character. The keep_all_classes parameter overrides
this list of classes. So be sure to set that to false if you only want the
listed classes.
"threshold" = "0"
Description: Detections are passed through this filter if they have a selected
classification that is above this threshold.
We will use the class_probability_filter to only pass detections from all classes that are above a confidence value that we’ll set in our pipeline configuration file.
Output¶
Finally, we will select our output process, which has the following definition:
Process type: detected_object_output
Description: Writes detected object sets to an output file.
All detections are written to the same file.
Properties: _no_reentrant
-- Configuration --
Name : file_name
Default :
Description: Name of the detection set file to write.
Tunable : no
Name : writer
Default :
Description: Block name for algorithm parameters. e.g. writer:type would be used to
specify the algorithm type.
Tunable : no
-- Input ports --
Name : detected_object_set
Data type : kwiver:detected_object_set
Flags : _required
Description: Set of detected objects.
Name : image_file_name
Data type : kwiver:image_file_name
Flags :
Description: Name of an image file. The file name may contain leading path components.
-- Output ports --
This output process accepts a detected_object_set and image_file_name as input and writes out the result. We will look at our selection of arrows that we could use:
Plugins that implement type "detected_object_set_output"
---------------------
Info on algorithm type "detected_object_set_output" implementation "csv"
Plugin name: csv Version: 1.0
Detected object set writer using CSV format.
- 1: frame number
- 2: file name
- 3: TL-x
- 4: TL-y
- 5: BR-x
- 6: BR-y
- 7: confidence
- 8,9: class-name, score (this pair may be omitted or may repeat any
number of times)
-- Configuration --
---------------------
Info on algorithm type "detected_object_set_output" implementation "kw18"
Plugin name: kw18 Version: 1.0
Detected object set writer using kw18 format.
- Column(s) 1: Track-id
- Column(s) 2: Track-length (number of detections)
- Column(s) 3: Frame-number (-1 if not available)
- Column(s) 4-5: Tracking-plane-loc(x,y) (could be same as World-loc)
- Column(s) 6-7: Velocity(x,y)
- Column(s) 8-9: Image-loc(x,y)
- Column(s) 10-13: Img-bbox(TL_x,TL_y,BR_x,BR_y) (location of top-left &
bottom-right vertices)
- Column(s) 14: Area
- Column(s) 15-17: World-loc(x,y,z) (longitude, latitude, 0 - when
available)
- Column(s) 18: Timestamp (-1 if not available)
- Column(s) 19: Track-confidence (-1 if not available)
-- Configuration --
"tot_field1_ids" = ""
Description: Comma separated list of ids used for TOT field 1.
"tot_field2_ids" = ""
Description: Comma separated list of ids used for TOT field 2.
"write_tot" = "false"
Description: Write a file in the vpView TOT format alongside the computed tracks.
---------------------
Info on algorithm type "detected_object_set_output" implementation "kpf_output"
Plugin name: kpf_output Version: 1.0
Detected object set writer using kpf format.t
-- Configuration --
In this case, we’ll select the DIVA KPF writer when we assemble our pipeline.
Pipeline¶
A text file is used to construct the pipeline processes, their input and output port connections, and the configuration parameters.
We’ll construct a pipeline that has the following structure based on the information
we obtained from using plugin_explorer:
Here is the pipeline file that configures our selected input, filter, and output:
Which can be represented by the follwing pipeline file:
# --------------------------------------------------
process reader :: detected_object_input
file_name = sample_detected_objects.csv
reader:type = csv
# --------------------------------------------------
process filter :: detected_object_filter
filter:type = class_probablity_filter
filter:threshold = .5
connect from reader.detected_object_set to filter.detected_object_set
# --------------------------------------------------
process writer :: detected_object_output
file_name = sample_filtered_detected_objects.kpf
writer:type = kpf
connect from filter.detected_object_set to writer.detected_object_set
In this pipeline file we define three processes: reader, filter, and writer. We connect the detected_object_set output of reader to the detected_object_set input of filter. We configure filter to only pass detected_objects with a confidence above a threshold of 0.5 and then we pass its detected_object_set output port to our writer processes’ input port. We select a KPF writer for our writer process.
We can run the pipeline with the following command:
pipeline_runner --pipe sample_reader_filter_writer.pipe
When the pipeline runs it will read a set of detected_objects from the file sample_detected_objects.csv, filter out any that have a confidence less than 50%, and then write the remainder to a KPF file for further processing, etc.
PythonsProcesses¶
One of KWIVER’s great strengths (as provided by sprokit) is the ability to
create hybrid pipelines which combine C++ and Python processes in the same
pipeline. This greatly facilitates prototyping complex processing pipelines.
To test this out we’ll use a simple process called numbers which simply
generates numbers on a Sprokit port. We’ll also use a simple Python process
that prints the number called kw_print_number_process the code for which
can be seen in
[sprokit/processes/python/kw_print_number_process.py](sprokit/processes/python/kw_print_number_process.py).
As usual, we can lean about this process with the following command:
plugin_explorer --proc kw_print_number_process -d
Which produces the following output:
Process type: kw_print_number_process
Description: A Simple Kwiver Test Process
Properties: _no_reentrant, _python
Configuration:
Name : output
Default : .
Description: The path for the output file.
Tunable : no
Input ports:
Name : input
Type : integer
Flags : _required
Description: Where numbers are read from.
Output ports:
In order to get around limitations imposed by the Python Global
Interpreter Lock, we’ll use a different Sprokit scheduler for this
pipeline. The pythread_per_process scheduler which does essentially
what it says: it creates a Python thread for every process in the
pipeline:
pipeline_runner -S pythread_per_process -p </path/to/kwiver/source>/sprokit/pipelines/number_flow_python.pipe>
The previous pipeline, the numbers will be written to an output file,
this time numbers_from_python.txt