Configuration Usage

Introduction

The vital config_block supports general configuration tasks where a general purpose key/value pair is needed. The configuration block is used to specify, communicate and set configurable items in many different situations. The two major users of the configuration support are algorithms and processes. In addition, there are a few other places that they are used also.

Configurations are usually established in an external file which are read and converted to an internal config_block object. This is the typical way to control the behaviour of the software. Configuration blocks can also be created pragmatically such as when specifying an expected set of configurable items.

When algorithms are used within processes, the configuration entries are specified as a block in the pipe file. The process takes the appropriate config subblock and passes it to the set_nested_algo_configuration() method to instantiate and configure the algorithm.

From File to config_block

Using config_block_io to directly convert config file into block. This can be used by a main program that manages configs and algorithms directly. The read_config_file() uses a complex set of rules to locate config files based on host system and application name.

Establishing Expected Config

The expected configuration is established using macros. The general form for the PLUGGALBE_IMPL() macro which sets configuration parameters is shown below.

PLUGGABLE_IMPL(
  <class name>,
  <description>,
  <list of PARAM/PARAM_DEFAULT macros>)

An example configuration for the filter_tracks algorithm is shown below.

class KWIVER_ALGO_CORE_EXPORT filter_tracks
  : public vital::algo::filter_tracks
{
public:
  PLUGGABLE_IMPL(
    filter_tracks,
    "Filter tracks by track length or matrix matrix importance.",
    PARAM_DEFAULT(
      min_track_length, unsigned int,
      "Filter the tracks keeping those covering "
      "at least this many frames. Set to 0 to disable.",
      3 ),
    PARAM_DEFAULT(
      min_mm_importance, double,
      "Filter the tracks with match matrix importance score "
      "below this threshold. Set to 0 to disable.",
      1.0 )
  )

Once an algorithm has been created, the configuration parameters can be modified or set using the ->set_value() method. The following example shows how to modify the configuration values in the filter_features_scale algorithm.

plugin_manager::instance().load_all_plugins();

  // Generate instance of the filter_tracks algorithm
  algo::filter_tracks_sptr filter_algo =
    create_algorithm< algo::filter_tracks >( "core" );

  // Get the configuration of the filter_tracks algorithm
  config_block_sptr config = filter_algo->get_configuration();

  // Adjust configuration parameter values to filter 3 features
  // and set ``top_fraction`` to 0.3
  config->set_value( "min_track_length", 5 );
  config->set_value( "min_mm_importance", 0.8 );

  // Set the updated configuration to the filter algorithm
  filter_algo->set_configuration( config );

This expected configuration serves as documentation for the algorithm or process configuration items when it is displayed by other tools. It is also used to validate the configuration supplied at run time to make sure all expected items are present.

Usage by Algorithms

Algorithms specify their expected set of configurable items using the PLUGGALBE_IMPL() macro as described above. This macro then defines the get_configuration() and set_configuration() methods.

The run time configuration is passed to an algorithm through the set_configuration() method. This method typically extracts the expected configuration values and saves them locally for the algorithm to use. When a configuration is read from the file, there is no guarantee that all expected configuration items are present and attempting to get a value that is not present generates an exception.

The recommended way to avoid this problem is to use the expected configuration, as created by the macro, then supply any missing entries using the set_configuration_internal() method. The following code snippet shows how this is done.

// Set this algorithm's properties via a config block
void
<algorithm>
::set_configuration_internal( vital::config_block_sptr in_config )
{

  // Starting with our generated vital::config_block to ensure that assumed values are present
  // An alternative is to check for key presence before performing a get_value() call.
  vital::config_block_sptr const& config = this->get_configuration();

  // Merge in supplied config to cause these values to overwrite the defaults.
  config->merge_config( in_config );
}

Instantiating Algorithms

Algorithms can be used directly in application code. The actual implementation of the abstract algorithm interface is specified through a config block.

Lets first look at the code that will instantiate the configured algorithm and then look at the contents of the configuration file.

The following code snippet instantiates a draw_detected_object_set algorithm.

// this pointer will be used to reference the algorithm after it is created.
vital::algo::draw_detected_object_set_sptr m_algo;

// Get algorithm configuration
auto algo_config = get_config(); // or an equivalent call

// Check config so it will give run-time diagnostic of config problems
if ( ! vital::algo::draw_detected_object_set::check_nested_algo_configuration( "draw_algo", algo_config ) )
{
  LOG_ERROR( logger, "Configuration check failed." );
}

vital::algo::draw_detected_object_set::set_nested_algo_configuration( "draw_algo", algo_config, m_algo );
if ( ! d->m_algo )
{
  LOG_ERROR( logger, "Unable to create algorithm." );
}

After the configuration is extracted, it is passed to the check_nested_algo_configuration() method to determine if the configuration has the basic type entry and the requested type is available. If the type entry is missing or the specified implementation is not available, a detailed log message is generated with the available implementations.

If the configuration is acceptable, the set_nested_algo_configuration() call will actually instantiate and configure the selected algorithm implementation.

The name that is supplied to these calls, “draw_algo” in this case, is used access the configuration block for this algorithm.

The following configuration file snippet can be used to configure the above algorithm.:

block draw_algo
  type = ocv    # select the ocv instance of this algorithm

  block ocv     # configure the 'ocv' instance
    alpha_blend_prob   = true
    default_line_thickness   = 1.25
    draw_text   = false
  endblock # for ocv
endblock  # for draw_algo

The outer block labeled “draw_algo” specifies the configuration to be used for the above code snippet. The config entry “type” specifies which implementation of the algorithm to instantiate. The following block labeled “ocv” is used to configure the algorithm after it is instantiated. The block labeled “ocv” is used for algorithm type “ocv”. If the algorithm type was “foo”, then the block “foo” would be used to configure the algorithm.

Verifying a Configuration

When a configuration file (or configuration section of a pipe file) is read in, there is no checking of the configuration key names. There is no way of knowing which configuration items are valid or expected and which ones are not. If a name is misspelled, which sometimes happens, it will be misspelled in the configuration block. This can lead to hours of frustration diagnosing a problem.

A configuration can be checked against a baseline using the config_difference class. This class provides methods to determine the differences between a reference configuration and one created from an input file. The difference between these two configurations is presented in two different ways. It provides a list of keys that are baseline config and not in the supplied config. These are the config items that were expected but not supplied. It also provides a list of keys that are in the supplied config but not in the expected config. These items are supplied but not expected.

The following code snippet shows how to report the difference between two config blocks.

//                                    ref-config                received-config
kwiver::vital::config_difference cd( this->get_configuration(), config );
const auto key_list = cd.extra_keys();
if ( ! key_list.empty() )
{
  // This may be considered an error in some cases
  LOG_WARN( logger(), "Additional parameters found in config block that are not required or desired: "
            << kwiver::vital::join( key_list, ", " ) );
}

key_list = cd.unspecified_keys();
if ( ! key_list.empty() )
{
  LOG_WARN( logger(), "Parameters that were not supplied in the config, using default values: "
            << kwiver::vital::join( key_list, ", " ) );
}

Not all applications need to check both cases. There may be good reasons for not specifying all expected configuration items when the default values are as expected. In some cases, unexpected items that are supplied by the configuration may be indications of misspelled entries.

Config Management Techniques

The configuration file reader provides several alternatives for managing the complexity of a large configuration. The block / endblock construct can be used to shorten config lines and modularize the configuration. The include directove can be used to share or reuse portions of a config.

Starting with the example config section that selects an algorithm and configures it:

algorithm_instance_name:type = type_name
algorithm_instance_name:type_name:algo_param = value
algorithm_instance_name:type_name:threshold = 234

The block construct can be used to simplify the configuration and make it easier to navigate.:

block algorithm_instance_name
  type = type_name
  block  type_name
    algo_param = value
    threshold = 234
  endblock
endblock

In cases where the configuration block is extensive or used in multiple applications, that part of the configuration can exist as a stand-alone file and be included where it is needed.:

block  algorithm_instance_name
  include type_name.conf
endblock

where type_name.conf contains:

type = type_name
block   type_name
  algo_param = value
  threshold = 234
endblock

Environment variables and config macros can be combined to provide a level of adaptability to config files. Using the environment macro in an include directive can provide run time agility without requiring the file to be edited. The following is an example of selecting a different include file based on mode.:

include $ENV{MODE}/config.file.conf

Using enums in config entries

Quite often a configuration parameter can only take a fixed number of values such as when the user is trying to configure an enum. The enum support in vital directly supports converting strings to enum values with the use of the enum_converter and enum support in the config block. The enum converter will verify that the supplied string represents an enum value, and throw an error if it does not. The list of valid enum strings is provided to assist in documenting config entries.

The following code snippets show examples on how to use the ENUM_CONVERTER macro.:

 #include <vital/util/enum_converter.h>

ENUM_CONVERTER(
  method_converter, inpainting_method, { "mask", METHOD_mask },
  { "navier_stokes", METHOD_navier_stokes } )

ENUM_CONVERTER(
  morphology_converter, morphology_mode,
  { "erode", MORPHOLOGY_erode }, { "dilate", MORPHOLOGY_dilate },
  { "open", MORPHOLOGY_open }, { "close", MORPHOLOGY_close },
  { "none", MORPHOLOGY_none } );