Image Data Types and Related Algorithms

Image

class image

The representation of an in-memory image.

This base image class represents an image with a dynamic data type. The underlying data type can be queried using pixel_traits(). To properly access individual pixels the data type must be known. The templated at<T>() member function provides direct access to pixels. Alternatively, cast the image itself into an image_of object. The typed image_of class is a bit easier to work with once the type is known, but this base class is useful in APIs that may operate on images of various types.

Memory Management

This image class supports two modes of memory management. Either the image owns its memory or it does not. If the image owns its memory the image::memory() function will return a shared pointer to that image_memory object. Otherwise, image::memory() will return nullptr. In both cases, image::first_pixel() returns a pointer to the first pixel of the memory that appears in the image. The address of the first pixel does not need to match the starting address of the image_memory. There can be multiple different views into the same memory (e.g. a cropped image view) and these views may use the same memory object with a different offsets to the first pixel, size, and step parameters.

Typically the image manages its own memory in a reference counted image_memory object. Creating a new image will allocate this memory, which can be accessed from image::memory(). Copying an image will make a shallow copy refering to the same memory object, and the memory will be deleted when all images are done with it.

There is a special constructor that allows construction of an image as a veiw into some existing memory. In this case the image does not own the memory and image::memory() will return nullptr. The user must ensure that the memory exists for the lifetime of the image.

Subclassed by kwiver::vital::image_of< uint8_t >, kwiver::vital::image_of< uint16_t >, kwiver::vital::image_of< T >

Public Functions

image(const image_pixel_traits &pt = image_pixel_traits())

Default Constructor.

Default Constructor

Parameters:: pt – Change the pixel traits of the image

image(size_t width, size_t height, size_t depth = 1, bool interleave = false, const image_pixel_traits &pt = image_pixel_traits())

Constructor that allocates image memory.

Constructor that allocates image memory

Create a new blank (empty) image of specified size.

Parameters:

width – Number of pixels in width
height – Number of pixel rows
depth – Number of image channels
pt – data type traits of the image pixels
interleave – Set if the pixels are interleaved

image(const void *first_pixel, size_t width, size_t height, size_t depth, ptrdiff_t w_step, ptrdiff_t h_step, ptrdiff_t d_step, const image_pixel_traits &pt = image_pixel_traits())

Constructor that points at existing memory.

Constructor that points at existing memory

Create a new image from supplied memory.

Parameters:

first_pixel – Address of the first pixel in the image. This does not have to be the lowest memory address of the image memory.
width – Number of pixels wide
height – Number of pixels high
depth – Number of image channels
w_step – pointer increment to get to next pixel column
h_step – pointer increment to get to next pixel row
d_step – pointer increment to get to next image channel
pt – data type traits of the image pixels

image(const image_memory_sptr &mem, const void *first_pixel, size_t width, size_t height, size_t depth, ptrdiff_t w_step, ptrdiff_t h_step, ptrdiff_t d_step, const image_pixel_traits &pt = image_pixel_traits())

Constructor that shares memory with another image.

Constructor that shares memory with another image

Create a new image from existing image.

Parameters:

mem – Shared memory block to be used
first_pixel – Address of the first pixel in the image. This does not have to be the lowest memory address of the image memory.
width – Number of pixels wide
height – Number of pixels high
depth – Number of image channels
w_step – pointer increment to get to next pixel column
h_step – pointer increment to get to next pixel row
d_step – pointer increment to get to next image channel
pt – data type traits of the image pixels

image(const image &other)

Copy Constructor.

Copy Constructor

The new image will share the same memory as the old image

Parameters:: other – The other image.

const image &operator=(const image &other): Assignment operator.

inline const image_memory_sptr &memory() const

Const access to the image memory

Access to the image memory

In most cases, when interacting with image data, you should use the first_pixel() function instead of memory(). The memory() function provides access to the underlying reference counted memory for advanced memory management applications. It returns a block of data that contains the image somewhere within. The block of data may contain only the pixels in this image, but it could also contain much more hidden data if the image is a crop or subsampling of an original image that was larger. The blocks of memory are typically shared between copies of this image, and each copy may have a different view into the memory.

This function may also return nullptr for a valid image that is a view into some external memory (first_pixel() is still valid in this case). Use caution when accessing the memory directly and always check that the memory is not nullptr. Never assume that the image data must be contained in the memory block returned by this function.

inline image_memory_sptr memory()

Access to the image memory

In most cases, when interacting with image data, you should use the first_pixel() function instead of memory(). The memory() function provides access to the underlying reference counted memory for advanced memory management applications. It returns a block of data that contains the image somewhere within. The block of data may contain only the pixels in this image, but it could also contain much more hidden data if the image is a crop or subsampling of an original image that was larger. The blocks of memory are typically shared between copies of this image, and each copy may have a different view into the memory.

This function may also return nullptr for a valid image that is a view into some external memory (first_pixel() is still valid in this case). Use caution when accessing the memory directly and always check that the memory is not nullptr. Never assume that the image data must be contained in the memory block returned by this function.

size_t size() const

The size of the image data in bytes.

The size of the image managed data in bytes

This size includes all allocated image memory, which could be larger than width*height*depth*bytes_per_pixel.

Note

This size only accounts for memory which is owned by the image. If this image was constructed as a view into third party memory then the size is reported as 0.

inline const void *first_pixel() const

Const access to the pointer to first image pixel

Access to the pointer to first image pixel

Returns a raw void pointer to the first pixel in the image. This is the starting point for iterating through the image using offsets of w_step(), h_step(), and d_step(). See also image_of::first_pixel() for a variant of this function that returns a pointer to the underlying pixel type.

See also

image_of::first_pixel()

Note

the address returned may differ from the starting address returned by image::memory() if the image is a window into a larger block of image memory.

Note

If the address returned is not nullptr but image::memory() returns nullptr, then this image is a view into external memory not owned by this image object.

inline void *first_pixel()

Access to the pointer to first image pixel

Returns a raw void pointer to the first pixel in the image. This is the starting point for iterating through the image using offsets of w_step(), h_step(), and d_step(). See also image_of::first_pixel() for a variant of this function that returns a pointer to the underlying pixel type.

See also

image_of::first_pixel()

Note

the address returned may differ from the starting address returned by image::memory() if the image is a window into a larger block of image memory.

Note

If the address returned is not nullptr but image::memory() returns nullptr, then this image is a view into external memory not owned by this image object.

inline size_t width() const: The width of the image in pixels.

inline size_t height() const: The height of the image in pixels.

inline size_t depth() const: The depth (or number of channels) of the image.

inline const image_pixel_traits &pixel_traits() const: The trait of the pixel data type.

inline ptrdiff_t w_step() const: The the step in memory to next pixel in the width direction.

inline ptrdiff_t h_step() const: The the step in memory to next pixel in the height direction.

inline ptrdiff_t d_step() const: The the step in memory to next pixel in the depth direction.

bool is_contiguous() const: Return true if the pixels accessible in this image form a contiguous memory block

bool operator==(image const &other) const

Equality operator.

Equality operator

Compares this image to another image to test equality.

See also

For deep equality comparison see equal_content

Note

This function computes only “shallow” equality. That is, the images are considered equal if they point to the same memory and have the dimensions and pixel step sizes. Deep equality testing requires stepping through and testing that the values of each pixel are the same even if the memory and possibly memory layout differ.

Parameters:: other – image to compare with

inline bool operator!=(image const &other) const

Inequality operator

Compares this image to another image to test inequality.

Note

This function computes only “shallow” inequality. Refer to the equality operator (==) for details.

Parameters:: other – image to compare with

template<typename T> inline T &at(size_t i, size_t j)

Access pixels in the first channel of the image

Parameters:

i – width position (x)
j – height position (y)

template<typename T> inline const T &at(size_t i, size_t j) const: Const access pixels in the first channel of the image.

template<typename T> inline T &at(size_t i, size_t j, size_t k): Access pixels in the image (width, height, channel)

template<typename T> inline const T &at(size_t i, size_t j, size_t k) const: Const access pixels in the image (width, height, channel)

image empty_copy(bool interleave = false) const

Allocate an uninitialized image with the same shape and type.

Parameters:: interleave – If true, channels are interleaved (RGBRGB…). If false, channels are stored in separate planes (RRR…GGG…BBB…).

image copy(bool interleave = false) const

Copy pixels into newly allocated memory.

Parameters:: interleave – If true, channels are interleaved (RGBRGB…). If false, channels are stored in separate planes (RRR…GGG…BBB…).

void copy_from(const image &other): Deep copy the image data from another image into this one.

void set_size(size_t width, size_t height, size_t depth)

Set the size of the image.

If the size has not changed, do nothing. Otherwise, allocate new memory matching the new size.

Parameters:

width – a new image width
height – a new image height
depth – a new image depth

image crop(size_t x_offset, size_t y_offset, size_t width, size_t height) const

Get a cropped view of the image.

Get a cropped view of the image. The cropped view shares memory with the original image so no deep copy is done.

Parameters:

x_offset – start of the crop region in x (width)
y_offset – start of the crop region in y (height)
width – width of the crop region
height – height of the crop region

Time Stamp

class timestamp

Frame time.

This class represents a timestamp for a single video frame. The time is stored in micro-seconds and frame numbers start at one.

A timestamp has the notion of valid time and valid frame. This is useful when dealing with interpolated timestamps. In this case, a timestamp may have a time, but no frame.

When comparing timestamps, they must be from the same domain. If not, then they are not comparable and all relative operators return false.

If both timestamps have a time, then they are ordered by that value. If both do not have time but both have frame numbers, they are ordered by frame number. If the timestamps do not have some way of being compared, all relational operators return false.

Public Functions

timestamp()

Default constructor.

Created an invalid timestamp.

explicit timestamp(time_usec_t t, frame_id_t f)

Constructor.

Creates a valid timestamp with specified time and frame number.

Parameters:

t – Time for timestamp in micro-seconds
f – Frame number for timestamp

inline bool is_valid() const

Is timestamp valid.

Both the time and frame must be set for a timestamp to be totally valid.

Returns:: true if both time and frame are valid

inline bool has_valid_time() const

Timestamp has valid time.

Indicates that the time has been set for this timestamp.

Returns:: true if time has been set

inline bool has_valid_frame() const

Timestamp has valid frame number.

Indicates that the frame number has been set for this timestamp.

Returns:: true if frame number has been set

inline time_usec_t get_time_usec() const

Get time from timestamp.

The time portion of the timestamp is returned in micro-seconds. The value will be undetermined if the timestamp does not have a valid time.

See also

has_valid_time()

Returns:: Frame time in micro-seconds

double get_time_seconds() const

Get time in seconds.

The time portion of the timestamp is returned in seconds and fractions.

Returns:: time in seconds.

inline frame_id_t get_frame() const

Get frame number from timestamp.

The frame number value from the timestamp is returned. The first frame in a sequence is usually one. The frame number will be undetermined if the timestamp does not have a valid frame number set.

See also

has_valid_frame()

Returns:: Frame number.

timestamp &set_time_usec(time_usec_t t)

Set time portion of timestamp.

Parameters:: t – Time for frame.

timestamp &set_time_seconds(double t)

Set time portion of timestamp.

Parameters:: t – Time for frame in seconds.

timestamp &set_frame(frame_id_t f)

Set frame portion of timestamp.

Parameters:: f – Frame number

timestamp &set_invalid()

Set timestamp totally invalid.

Both the frame and time are set to invalid

timestamp &set_time_domain_index(int dom)

Set time domain index for this timestamp.

Parameters:: dom – Time domain index
Returns:: Reference to this object.

std::string pretty_print() const

Format object in a readable manner.

This method formats a time stamp in a readable and recognizable manner suitable form debugging and logging.

Returns:: formatted timestamp

Image Container

class image_container

An abstract representation of an image container.

This class provides an interface for passing image data between algorithms. It is intended to be a wrapper for image classes in third-party libraries and facilitate conversion between various representations. It provides limited access to the underlying data and is not intended for direct use in image processing algorithms.

Subclassed by kwiver::arrows::gdal::image_container, kwiver::arrows::ocv::image_container, kwiver::arrows::qt::image_container, kwiver::arrows::vxl::image_container, kwiver::vital::simple_image_container, kwiver::vital::tiled_image_container

Public Functions

virtual ~image_container() = default: Destructor.

virtual size_t size() const = 0

The size of the image data in bytes.

The size of the image data in bytes

This size includes all allocated image memory, which could be larger than width*height*depth.

virtual size_t width() const = 0: The width of the image in pixels.

virtual size_t height() const = 0: The height of the image in pixels.

virtual size_t depth() const = 0: The depth (or number of channels) of the image.

inline virtual image_pixel_traits pixel_traits() const: Return the pixel traits of the image.

virtual image get_image() const = 0: Get an in-memory image class to access the data.

inline virtual image get_image(unsigned x_offset, unsigned y_offset, unsigned width, unsigned height) const: Get an in-memory image class to access a sub-image of the data.

inline virtual metadata_sptr get_metadata() const: Get metadata associated with this image.

inline virtual void set_metadata(metadata_sptr md): Set metadata associated with this image.

Image I/O Algorithm

Instantiate with:

kwiver::vital::algo::image_io_sptr img_io = kwiver::vital::algo::image_io::create("<impl_name>");

Arrow & Configuration	<impl_name> options	CMake Flag to Enable
GDAL	gdal	KWIVER_ENABLE_GDAL
FFmpeg	ffmpeg	KWIVER_ENABLE_FFMPEG
OpenCV	ocv	KWIVER_ENABLE_OPENCV
VXL	vxl	KWIVER_ENABLE_VXL

class image_io : public kwiver::vital::algorithm

An abstract base class for reading and writing images

This class represents an abstract interface for reading and writing images.

A note about the basic capabilities:

HAS_TIME - This capability is set to true if the image metadata supplies a timestamp. If a timestamp is supplied, it is made available in the metadata for the image. If the timestamp is not supplied, then the metadata will not have the timestamp set.

Subclassed by kwiver::arrows::core::image_io_tiled_multifile, kwiver::arrows::ffmpeg::ffmpeg_image_io, kwiver::arrows::gdal::image_io, kwiver::arrows::ocv::image_io, kwiver::arrows::qt::image_io, kwiver::arrows::vxl::image_io

Public Functions

kwiver::vital::image_container_sptr load(std::string const &filename) const

Load image from the file

Throws:

kwiver::vital::path_not_exists – Thrown when the given path does not exist.
kwiver::vital::path_not_a_file – Thrown when the given path does not point to a file (i.e. it points to a directory).

Parameters:

filename – the path to the file to load

Returns:

an image container refering to the loaded image

void save(std::string const &filename, kwiver::vital::image_container_sptr data) const

Save image to a file

Image file format is based on file extension.

Throws:

kwiver::vital::path_not_exists – Thrown when the expected containing directory of the given path does not exist.
kwiver::vital::path_not_a_directory – Thrown when the expected containing directory of the given path is not actually a directory.

Parameters:

filename – the path to the file to save
data – the image container refering to the image to write

kwiver::vital::metadata_sptr load_metadata(std::string const &filename) const

Get the image metadata

Throws:

kwiver::vital::path_not_exists – Thrown when the given path does not exist.
kwiver::vital::path_not_a_file – Thrown when the given path does not point to a file (i.e. it points to a directory).

Parameters:

filename – the path to the file to read

Returns:

pointer to the loaded metadata

algorithm_capabilities const &get_implementation_capabilities() const

Return capabilities of concrete implementation.

This method returns the capabilities for the current image reader/writer.

Returns:: Reference to supported image capabilities.

Convert Image Algorithm

Instantiate with:

kwiver::vital::algo::convert_image_sptr img_bypas = kwiver::vital::algo::convert_image::create("<impl_name>");

Arrow & Configuration	<impl_name> options	CMake Flag to Enable
Core	bypass	KWIVER_ENABLE_ARROWS

class convert_image : public kwiver::vital::algorithm

An abstract base class for converting base image type.

Arrows that implement this interface convert the input image type (e.g. BGR 16) to a different type (e.g. RGB 8). Concrete implementations usually work with a single image representation, such as VXL or OCV.

If you are lookling for an interface for an image transform that will change the value of a pixel, then use the image_filter interface.

Subclassed by kwiver::arrows::core::convert_image_bypass

Public Functions

virtual void set_configuration(kwiver::vital::config_block_sptr config): Set this algorithm’s properties via a config block.

virtual bool check_configuration(kwiver::vital::config_block_sptr config) const

Check that the algorithm’s currently configuration is valid.

Check that the algorithm’s current configuration is valid.

virtual kwiver::vital::image_container_sptr convert(kwiver::vital::image_container_sptr img) const = 0: Convert image base type.

Image Filter Algorithm

Instantiate with:

kwiver::vital::algo::image_filter_sptr img_filter = kwiver::vital::algo::image_filter::create("<impl_name>");

Arrow & Configuration	<impl_name> options	CMake Flag to Enable
VXL	vxl_aligned_edge_detection	KWIVER_ENABLE_VXL
VXL	vxl_average	KWIVER_ENABLE_VXL
VXL	vxl_color_commonality	KWIVER_ENABLE_VXL
VXL	vxl_convert_image	KWIVER_ENABLE_VXL
VXL	vxl_hashed_image_classifier_filter	KWIVER_ENABLE_VXL
VXL	vxl_high_pass_filter	KWIVER_ENABLE_VXL
VXL	vxl_morphology	KWIVER_ENABLE_VXL
VXL	vxl_pixel_feature_extractor	KWIVER_ENABLE_VXL
VXL	vxl_threshold	KWIVER_ENABLE_VXL

class image_filter : public kwiver::vital::algorithm

Abstract base class for image set filter algorithms.

This interface supports arrows/algorithms that do a pixel by pixel image modification, such as image enhancement. The resultant image must be the same size as the input image.

Subclassed by kwiver::arrows::vxl::aligned_edge_detection, kwiver::arrows::vxl::average_frames, kwiver::arrows::vxl::color_commonality_filter, kwiver::arrows::vxl::convert_image, kwiver::arrows::vxl::hashed_image_classifier_filter, kwiver::arrows::vxl::high_pass_filter, kwiver::arrows::vxl::morphology, kwiver::arrows::vxl::pixel_feature_extractor, kwiver::arrows::vxl::threshold

Public Functions

image_filter(): Return the name of this algorithm.

virtual kwiver::vital::image_container_sptr filter(kwiver::vital::image_container_sptr image_data) = 0

Filter a input image and return resulting image

This method implements the filtering operation. The method does not modify the image in place. The resulting image must be a newly allocated image which is the same size as the input image.

Parameters:: image_data – Image to filter.
Returns:: a filtered version of the input image

Split Image Algorithm

Instantiate with:

kwiver::vital::algo::split_image_sptr img_split = kwiver::vital::algo::split_image::create("<impl_name>");

Arrow & Configuration	<impl_name> options	CMake Flag to Enable
OpenCV	ocv	KWIVER_ENABLE_OPENCV
VXL	vxl	KWIVER_ENABLE_VXL

class split_image : public kwiver::vital::algorithm

An abstract base class for converting base image type.

Subclassed by kwiver::arrows::ocv::split_image, kwiver::arrows::vxl::split_image

Public Functions

virtual std::vector<kwiver::vital::image_container_sptr> split(kwiver::vital::image_container_sptr img) const = 0: Split image.

Video Input Algorithm

Instantiate with:

kwiver::vital::algo::video_input_sptr img_bypas = kwiver::vital::algo::video_input::create("<impl_name>");

class video_input : public kwiver::vital::algorithm

An abstract base class for reading videos

This class represents an abstract interface for reading videos. Once the video is opened, the frames are returned in order.

Use cases:

1) Reading video from a directory of images.

2) Reading video frames from a list of file names.

3) Reading video from mpeg/video file (one of many formats) (e.g. FMV)

4) Reading video from mpeg/video file (one of many formats) with cropping (e.g. WAMI). This includes Providing geostationary images by cropping to a specific region from an image. This may result in no data if the geo region and image do not intersect.

5) Reading video from network stream. (RTSP) This may result in unexpected end of video conditions and network related disruptions (e.g. missing frames, connection terminating, …)

A note about the basic capabilities:

HAS_EOV - This capability is set to true if the video source can determine end of video. This is usually the case if the video is being read from a file, but may not be known if the video is coming from a streaming source.

HAS_FRAME_NUMBERS - This capability is set to true if the video source supplies frame numbers. If the video source specifies a frame number, then that number is used when forming a time stamp. If the video does not supply a frame number, the time stamp will not have a frame number.

HAS_FRAME_TIME - This capability is set to true if the video source supplies a frame time. If a frame time is supplied, it is made available in the time stamp for that frame. If the frame time is not supplied, then the timestamp will not have the time set.

HAS_FRAME_DATA - This capability is set to true if the video source supplies frame images. It may seem strange for a video input algorithm to not supply image data, but happens with a reader that only supplies the metadata.

HAS_ABSOLUTE_FRAME_TIME - This capability is set to true if the video source supplies an absolute, rather than relative frame time. This capability is not set if an absolute frame time can not be found, or if the absolute frame time is configured as “none”.

HAS_METADATA - This capability is set if the video source supplies some type of metadata. The metadata could be in 0601 or 0104 data formats or a different source.

HAS_TIMEOUT - This capability is set if the implementation supports the timeout parameter on the next_frame() method.

IS_SEEKABLE - This capability is set if the video source can seek to a specific frame.

All implementations must support the basic traits, in that they are registered with a true or false value. Additional implementation specific (extended) traits may be added. The application should first check to see if a extended trait is registered by calling has_trait() since the actual implementation is set by a configuration entry and not directly known by the application.

Extended capabilities can be created to publish capabilities of non-standard video sources. These capabilities should be namespaced using the name (or abbreviation) of the concrete algorithm followed by the abbreviation of the capability.

Subclassed by kwiver::arrows::core::video_input_buffered_metadata_filter, kwiver::arrows::core::video_input_filter, kwiver::arrows::core::video_input_image_list, kwiver::arrows::core::video_input_metadata_filter, kwiver::arrows::core::video_input_pos, kwiver::arrows::core::video_input_splice, kwiver::arrows::core::video_input_split, kwiver::arrows::ffmpeg::ffmpeg_video_input, kwiver::arrows::ffmpeg::ffmpeg_video_input_clip, kwiver::arrows::ffmpeg::ffmpeg_video_input_rewire

Public Functions

virtual void open(std::string video_name) = 0

Open a video stream.

This method opens the specified video stream for reading. The format of the name depends on the concrete implementation. It could be a file name or it could be a URI.

Capabilities are set in this call, so they are available after.

Note

Once a video is opened, it starts in an invalid state (i.e. before the first frame of video). You must call next_frame() to step to the first frame of video before calling frame_image().

Parameters:: video_name – Identifier of the video stream.
Throws:: exception – if open failed

virtual void close() = 0

Close video stream.

Close the currently opened stream and release resources. Closing a stream that is already closed does not cause a problem.

virtual bool end_of_video() const = 0

Return end of video status.

This method returns the end-of-video status of the input video. true is returned if the last frame has been returned.

This method will always return false for video streams that have no ability to detect end of video, such as network streams.

Returns:: true if at end of video, false otherwise.

virtual bool good() const = 0

Check whether state of video stream is good.

This method checks the current state of the video stream to see if it is good. A stream is good if it refers to a valid frame such that calls to frame_image() and frame_metadata() are expected to return meaningful data. After calling open() the initial video state is not good until the first call to next_frame().

Returns:: true if video stream is good, false if not good.

virtual bool seekable() const = 0

Return whether video stream is seekable.

This method returns whether the video stream is seekable.

Returns:: true if video stream is seekable, false otherwise.

virtual size_t num_frames() const = 0

Get the number of frames in the video stream.

Get the number of frames available in the video stream.

Throws:: video_stream_exception – when there is an error in the video stream.
Returns:: the number of frames in the video stream, or 0 if the video stream is not seekable.

virtual bool next_frame(kwiver::vital::timestamp &ts, kwiver::vital::time_usec_t timeout = 0) = 0

Advance to next frame in video stream.

This method advances the video stream to the next frame, making the image and metadata available. The returned timestamp is for new current frame.

The timestamp returned may be missing either frame number or time or both, depending on the actual implementation.

Calling this method will make a new image and metadata packets available. They can be retrieved by calling frame_image() and frame_metadata().

Check the HAS_TIMEOUT capability from the concrete implementation to see if the timeout feature is supported.

If the video input is already an end, then calling this method will return false.

Parameters:

ts – [out] Time stamp of new frame.
timeout – [in] Number of microseconds to wait. 0 = no timeout.

Throws:

video_input_timeout_exception – when the timeout expires.
video_stream_exception – when there is an error in the video stream.

Returns:

true if frame returned, false if end of video.

virtual bool seek_frame(kwiver::vital::timestamp &ts, kwiver::vital::timestamp::frame_t frame_number, kwiver::vital::time_usec_t timeout = 0) = 0

Seek to the given frame number in video stream.

This method seeks the video stream to the requested frame, making the image and metadata available. The returned timestamp is for new current frame.

The timestamp returned may be missing the time.

Calling this method will make a new image and metadata packets available. They can be retrieved by calling frame_image() and frame_metadata().

Check the HAS_TIMEOUT capability from the concrete implementation to see if the timeout feature is supported.

If the frame requested does not exist, then calling this method will return false.

If the video input is not seekable then calling this method will return false.

Parameters:

ts – [out] Time stamp of new frame.
frame_number – [in] The frame to seek to.
timeout – [in] Number of microseconds to wait. 0 = no timeout.

Throws:

video_input_timeout_exception – when the timeout expires.
video_stream_exception – when there is an error in the video stream.

Returns:

true if frame returned, false if end of video.

virtual kwiver::vital::timestamp frame_timestamp() const = 0

Obtain the time stamp of the current frame.

This method returns the time stamp of the current frame, if any, or an invalid time stamp. The returned time stamp shall have the same value as was set by the most recent call to next_frame().

This method is idempotent. Calling it multiple times without calling next_frame() will return the same time stamp.

Returns:: The time stamp of the current frame.

virtual kwiver::vital::image_container_sptr frame_image() = 0

Get current frame from video stream.

This method returns the image from the current frame. If the video input is already an end, then calling this method will return a null pointer.

This method is idempotent. Calling it multiple times without calling next_frame() will return the same image.

Throws:: video_stream_exception – when there is an error in the video stream.
Returns:: Pointer to image container.

virtual video_raw_image_sptr raw_frame_image()

Return implementation-defined data for efficiently copying this frame’s image.

Using this method can help avoid the loss of efficiency and fidelity that comes with re-encoding an image, if no changes to the image are to be performed before writing it back out. May return nullptr, indicating the reader does not support this operation.

Returns:: Pointer to raw image data.

virtual kwiver::vital::metadata_vector frame_metadata() = 0

Get metadata collection for current frame.

This method returns the metadata collection for the current frame. It is best to call this after calling next_frame() to make sure the metadata and video are synchronized and that no metadata collections are lost.

Metadata typically occurs less frequently than video frames, so if you call next_frame() and frame_metadata() together while processing a video, there may be times where no metadata is returned. In this case an empty metadata vector will be returned.

Also note that the metadata collection contains a timestamp that can be used to determine where the metadata fits in the video stream.

In video streams without metadata (as determined by the stream capability), this method may return and empty vector, indicating no new metadata has been found.

Calling this method at end of video will return an empty metadata vector.

Metadata is returned as a vector, instead of a single object, to handle cases where there is multiple metadata packets between frames. This can happen in video streams with a fast metadata rate and slow frame rate. Multiple metadata objects can be also returned from video streams that contain metadata in multiple standards, such as MISB-601 and MISB-104.

In cases where there are multiple metadata packets between frames, it is inappropriate for the reader to try to select the best metadata packet. That is why they are all returned.

This method is idempotent. Calling it multiple times without calling next_frame() will return the same metadata.

Throws:: video_stream_exception – when there is an error in the video stream.
Returns:: Vector of metadata pointers.

virtual video_raw_metadata_sptr raw_frame_metadata()

Return implementation-defined data for efficiently copying this frame’s metadata.

Using this method can help avoid the loss of efficiency and fidelity that comes with re-encoding metadata, if no changes to the metadata are to be performed before writing it back out. May return nullptr, indicating the reader does not support this operation.

Returns:: Pointer to raw metadata.

virtual video_uninterpreted_data_sptr uninterpreted_frame_data()

Return an implementation-defined representation of uninterpreted data in this frame.

This method enables passage of miscellaneous data - such as audio, unrecognized metadata, or secondary image streams - to a video output when transcoding.

Returns:: Pointer to uninterpreted data.

virtual kwiver::vital::metadata_map_sptr metadata_map() = 0

Get metadata map for video.

This method returns a metadata map for the video assuming the video is seekable. If the video is not seekable it will return an empty map. Depending on the implementation if the metamap has not been previously requested then the video will have to loop over to create and store the metadata map.

In video streams without metadata (as determined by the stream capability), this method will return an empty map, indicating no metadata has been found.

Throws:: video_stream_exception – when there is an error in the video stream.
Returns:: Map of vectors of metadata pointers.

virtual double frame_rate()

Get frame rate from the video.

If frame rate is not supported, return -1.

Returns:: Frame rate.

virtual video_settings_uptr implementation_settings() const

Extract implementation-specific video encoding settings.

The returned structure is intended to be passed to a video encoder of similar implementation so that the output video can be encoded using the settings of the input video.

Returns:: Implementation video settings, or nullptr if none are needed.

algorithm_capabilities const &get_implementation_capabilities() const

Return capabilities of concrete implementation.

This method returns the capabilities for the currently opened video.

Returns:: Reference to supported video capabilities.

Video Output Algorithm

Instantiate with:

kwiver::vital::algo::video_output_sptr video_writer = kwiver::vital::algo::video_output::create("<impl_name>");

Arrow & Configuration	<impl_name> options	CMake Flag to Enable
FFmpeg	ffmpeg	KWIVER_ENABLE_FFMPEG

class video_output : public kwiver::vital::algorithm

An abstract base class for writing videos.

This class represents an abstract interface for writing videos. Once the video is opened, frames may be added in order.

Subclassed by kwiver::arrows::ffmpeg::ffmpeg_video_output

Public Functions

virtual void open(std::string video_name, video_settings const *settings) = 0

Open a video stream.

This method opens the specified video stream for writing. The format of the name depends on the concrete implementation. It could be a file name, a directory, or a URI.

Parameters:

video_name – Identifier of the video stream.
settings – Additional information used to configure the video output.

Throws:

exception – Thrown if opening the video stream failed.

virtual void close() = 0

Close video stream.

Close the currently opened stream and release resources. Closing a stream that is already closed does not cause a problem.

virtual bool good() const = 0

Check whether state of video stream is good.

This method checks the current state of the video stream to see if it is good. A stream is good if it is ready to receive images and/or metadata.

Returns:: true if video stream is good, false if not good.

virtual void add_image(kwiver::vital::image_container_sptr const &image, kwiver::vital::timestamp const &ts) = 0

Add a frame image to the video stream.

This method writes the next frame image to the video stream. The timestamp should be greater than that of the previously written frame, as many implementations are unable to write frames out of order.

Throws:: video_stream_exception – Thrown if is an error in the video stream.

virtual void add_image(video_raw_image const &image)

Add a raw frame image to the video stream.

This method writes the raw image to the video stream. There is no guarantee that this functions correctly when intermixed with non-raw images.

virtual void add_metadata(kwiver::vital::metadata const &md) = 0

Add metadata collection to the video stream.

This method adds metadata to the video stream. Depending on the implementation, the metadata may be written immediately, or may be deferred until the next frame is written. For this reason, the metadata’s timestamp should be greater than that of the previously written frame.

For implementations that do not support metadata, this method does nothing.

Throws:: video_stream_exception – Thrown if is an error in the video stream.

virtual void add_metadata(video_raw_metadata const &md)

Add a frame of raw metadata to the video stream.

This method writes the raw metadata to the video stream. There is no guarantee that this functions correctly when intermixed with non-raw metadata.

virtual void add_uninterpreted_data(video_uninterpreted_data const &misc_data)

Add a frame of uninterpreted data to the video stream.

This method writes the uninterpreted data to the video stream.

virtual vital::video_settings_uptr implementation_settings() const

Extract implementation-specific video encoding settings.

The returned structure is intended to be passed to a video encoder of similar implementation to produce similarly formatted output. The returned value may not be identical to the one passed to this object via open().

Returns:: Implementation video settings, or nullptr if none are needed.

algorithm_capabilities const &get_implementation_capabilities() const

Return capabilities of concrete implementation.

This method returns the capabilities of the algorithm implementation.

Returns:: Reference to supported algorithm capabilities.

Public Static Attributes

static const algorithm_capabilities::capability_name_t SUPPORTS_FRAME_RATE

Writer supports a global frame rate.

This capability indicates if the implementation is able to write a video which specifies a global, constant frame rate.

static const algorithm_capabilities::capability_name_t SUPPORTS_FRAME_TIME

Writer can write per-frame time codes.

This capability indicates if the implementation is able to write distinct time codes for each frame. Note that writers without this capability may still use the frame time for error checking, insertion of “blank” frames, or synchronization with metadata.

static const algorithm_capabilities::capability_name_t SUPPORTS_METADATA

Writer can write metadata.

This capability indicates if the implementation is able to write metadata.

static const algorithm_capabilities::capability_name_t SUPPORTS_UNINTERPRETED_DATA

Writer can write uninterpreted data.

This capability indicates if the implementation can take data which a video input did not interpret and put it back in the video stream.

Code Example  // All algorithms are implemented/encapsulated in an arrow, and operate on
  // vital classes
  // There are various algorithms (arrows) that kwiver provides that you can use
  // to analyze imagry
  // In this example, while we will look at a few algorithms, this example
  // highlights the vital data types used by algorithms
  // These vital data types can then be used as inputs or outputs for
  // algorithms.
  // The vital data types are a sort of common 'glue' between dispart algorithms
  // allowing them to work together.

  // Image I/O algorithms are derived from the kwiver::vital::image_io algorithm
  // interface

  // While we could instantiate a particular algorithm object directly with this
  // code
  // kwiver::arrows::ocv::image_io ocv_io;
  // kwiver::arrows::vxl::image_io vxl_io;
  // This would require our application to include specific headers be include
  // in our code
  // and require our application to directly link to OpenCV and cause a
  // dependency

  // A key feature of the KWIVER architecture is the ability to dynamically load
  // available algorithms at runtime.
  // This ability allow you to write your application with a set of basic data
  // types and algorithm interfaces and
  // then dynamically replace or reconfigure algorithms at run time without
  // needing to recompile
  // New algorithms can be dropped on disk at and KWIVER can run them
  // The first thing to do is to tell kwiver to load up all it's plugins (which
  // includes all the algorithms)
  kwiver::vital::plugin_manager::instance().load_all_plugins();

  // Refer to this page :
  // http://kwiver.readthedocs.io/en/latest/vital/images.html
  // Documenting the types and algorithms associated with images:
  //               Various implementations of the algorithm,
  //               The string to use to specify creation of a specific
  // implementation,
  //               The KWIVER CMake option that builds the specific
  // implementation

  ///////////////
  // Image I/O //
  ///////////////

  // The main image libraries used in KWIVER are the OpenCV and VXL libraries
  kwiver::vital::algo::image_io_sptr ocv_io =
    kwiver::vital::create_algorithm< kwiver::vital::algo::image_io >( "ocv" );
  kwiver::vital::algo::image_io_sptr vxl_io =
    kwiver::vital::create_algorithm< kwiver::vital::algo::image_io >( "vxl" );

  // The image_io interface is simple, and has a load and save method
  // These methods will operate on the vital object image_container
  // The image_container is intended to be a wrapper for image to facilitate
  // conversion between
  // various representations. It provides limited access to the underlying
  // data and is not intended for direct use in image processing algorithms.
  kwiver::vital::image_container_sptr ocv_img = ocv_io->load( "./cat.jpg" );
  kwiver::vital::image_container_sptr vxl_img = vxl_io->load( "./cat.jpg" );

  // Let's use OpenCV to display the images
  // NOTE, this requires that our application CMakeLists properly
  // find_package(OpenCV)
  // And that we tell our application CMake targets about OpenCV (See the
  // CMakeLists.txt for this file)
  cv::Mat mat;
  // First, convert the image to an OpenCV image object
  mat =
    kwiver::arrows::ocv::image_container::vital_to_ocv(
      ocv_img->get_image(),
      kwiver::arrows::ocv::image_container::BGR_COLOR );
  cv::namedWindow( "Image loaded by OpenCV", cv::WINDOW_AUTOSIZE ); // Create a
                                                                    // window
                                                                    // for
                                                                    // display.
  cv::imshow( "Image loaded by OpenCV", mat );                     // Show our
                                                                   // image
                                                                   // inside it.
  cv::waitKey( 2000 );                                                  // Wait
                                                                        // for
                                                                        // 2s
  cv::destroyWindow( "Image loaded by OpenCV" );

  // We can do the same, even if the image was originally loaded with VXL
  mat =
    kwiver::arrows::ocv::image_container::vital_to_ocv(
      vxl_img->get_image(),
      kwiver::arrows::ocv::image_container::BGR_COLOR );
  cv::namedWindow( "Image loaded by VXL", cv::WINDOW_AUTOSIZE ); // Create a
                                                                 // window for
                                                                 // display.
  cv::imshow( "Image loaded by VXL", mat );                     // Show our
                                                                // image inside
                                                                // it.
  cv::waitKey( 2000 );                                              // Wait for
                                                                    // 2s
  cv::destroyWindow( "Image loaded by VXL" );

  //////////////////
  // Image Filter //
  //////////////////

  // Currently, there is no arrow implementing image filtering
  // kwiver::vital::algo::image_filter_sptr _filter =
  // kwiver::vital::algo::image_filter::create("<impl_name>");

  /////////////////
  // Split Image //
  /////////////////

  // These algorithms split an image in half (left and right)
  kwiver::vital::algo::split_image_sptr ocv_split =
    kwiver::vital::create_algorithm< kwiver::vital::algo::split_image >(
      "ocv" );
  kwiver::vital::algo::split_image_sptr vxl_split =
    kwiver::vital::create_algorithm< kwiver::vital::algo::split_image >(
      "vxl" );

  std::vector< kwiver::vital::image_container_sptr > ocv_imgs =
    ocv_split->split( vxl_img );
  for( kwiver::vital::image_container_sptr i : ocv_imgs )
  {
    mat = kwiver::arrows::ocv::image_container::vital_to_ocv(
      i->get_image(),
      kwiver::arrows::ocv::image_container::BGR_COLOR );
    cv::namedWindow( "OpenCV Split Image", cv::WINDOW_AUTOSIZE ); // Create a
                                                                  // window for
                                                                  // display.
    cv::imshow( "OpenCV Split Image", mat );                     // Show our
                                                                 // image inside
                                                                 // it.
    cv::waitKey( 2000 );                                               // Wait
                                                                       // for 2s
    cv::destroyWindow( "OpenCV Split Image" );
  }

  std::vector< kwiver::vital::image_container_sptr > vxl_imgs =
    ocv_split->split( ocv_img );
  for( kwiver::vital::image_container_sptr i : vxl_imgs )
  {
    mat = kwiver::arrows::ocv::image_container::vital_to_ocv(
      i->get_image(),
      kwiver::arrows::ocv::image_container::BGR_COLOR );
    cv::namedWindow( "VXL Split Image", cv::WINDOW_AUTOSIZE ); // Create a
                                                               // window for
                                                               // display.
    cv::imshow( "VXL Split Image", mat );                     // Show our image
                                                              // inside it.
    cv::waitKey( 2000 );                                            // Wait for
                                                                    // 2s
    cv::destroyWindow( "VXL Split Image" );
  }
}