API Reference

We provide a public, stable API consisting of tested modules. However, in members of the API you may encounter experimental features (e.g. arguments or functions). These have the prefix experimental_ and are exempted from stability guarantees.

The functions of the API are exported (apart from their original modules) also in the panoptic_parts namespace, so they can be imported and used as:

import panoptic_parts as pp
pp.decode_uids(uids)

Label format handling

panoptic_parts.utils.format.decode_uids(uids, *, return_sids_iids=False, return_sids_pids=False, experimental_noinfo_id=- 1, experimental_dataset_spec=None, experimental_correct_range=False)

Decode the compact panoptic-parts uids into consituent ids.

Given the universal ids uids encoded according to the panoptic-parts format described in https://panoptic-parts.readthedocs.io/en/stable/label_format.html, this function returns element-wise the semantic ids (sids), instance ids (iids), and part ids (pids). Optionally it returns the sids_iids and sids_pids as well. sids_iids represent the semantic-instance-level (two-level) labeling, e.g., sids_iids from Cityscapes-Panoptic-Parts ≡ ids from Cityscapes-Original. sids_pids represent the semantic-part-level (semantics) labeling.

Examples

• decode_uids(23, return_sid_pid=True) → (23, -1, -1, 23)

• decode_uids(23003, return_sid_pid=True) → (23, 3, -1, 23)

• decode_uids(2300304, return_sid_pid=True) → (23, 3, 4, 2304)

• decode_uids(tf.constant([1, 12, 1234, 12345, 123456, 1234567])) → ([ 1, 12, 1, 12, 1, 12],

  [-1, -1, 234, 345, 234, 345], [-1, -1, -1, -1, 56, 67])

• decode_uids(np.array([[1, 12], [1234, 12345]])) → ([[ 1, 12], [ 1, 12]],

  [[ -1, -1], [234, 345]], [[-1, -1], [-1, -1]])

Each output has the same type and shape as uids (not shown for clarity).

Parameters

• uids – The panoptic-parts uids. Can be a tf.Tensor of dtype tf.int32 and arbitrary shape, or a np.ndarray of dtype np.int32 and arbitrary shape, or a torch.tensor of dtype torch.int32 and arbitrary shape, or a Python int, or a np.int32 integer, with elements encoded according to the panoptic-parts format.

• return_sids_iids (bool) – Optionally return sids_iids.

• return_sids_pids (bool) – Optionally return sids_pids.

• experimental_noinfo_id (int) – The integer representing the “no info”/void value.

• experimental_dataset_spec (typing.Optional[panoptic_parts.specs.dataset_spec.DatasetSpec]) – a DatasetSpec is used a) for removing the part-level instance information layer from the pids, this layer is not useful for Part-aware Panoptic Segmentation but is present in the encoded uids of some datasets (e.g. PPP), b) for ids range validity checking and correction according to that DatasetSpec (provide experimental_correct_range=True) for this functionality.

• experimental_correct_range (bool) – If a DatasetSpec is provided, the invalid ids according to that DatasetSpec, will be replaced with the experimental_noinfo_id value.

Returns

There are 4 return signatures according to the given return_* keyword arguments. All return values have the same type and shape as uids, where non-relevant/void pixels have value -1.

if return_sids_iids and return_sids_pids are False (default behavior):

sids, iids, pids = decode_uids(uids)

if return_sids_iids is True:

sids, iids, pids, sids_iids = decode_uids(uids, return_sids_iids=True)

if return_sids_pids is True:

sids, iids, pids, sids_pids = decode_uids(uids, return_sids_pids=True)

if return_sids_iids and return_sids_pids are both True:

sids, iids, pids, sids_iids, sids_pids = decode_uids(uids, return_sids_iids=True, return_sids_pids=True)

sids have no -1. iids have -1 for pixels labeled with semantic-level labels only. pids have -1 for pixels labeled with semantic-level or semantic-instance-level labels only. sids_iids: have no -1. sids_pids: have no -1.

panoptic_parts.utils.format.encode_ids(sids, iids, pids)

Given semantic ids (sids), instance ids (iids), and part ids (pids) this function encodes them element-wise to uids according to the hierarchical format described in README.

This function is the opposite of decode_uids, i.e., uids = encode_ids(decode_uids(uids)).

Parameters

• sids – all of the same type with -1 for non-relevant pixels with elements according to hierarchical format (see README). Can be: tf.Tensor of dtype tf.int32 and arbitrary shape, or np.ndarray of dtype np.int32 and arbitrary shape, or torch.tensor of dtype torch.int32 and arbitrary shape, or Python int, or np.int32 integer.

• iids – all of the same type with -1 for non-relevant pixels with elements according to hierarchical format (see README). Can be: tf.Tensor of dtype tf.int32 and arbitrary shape, or np.ndarray of dtype np.int32 and arbitrary shape, or torch.tensor of dtype torch.int32 and arbitrary shape, or Python int, or np.int32 integer.

• pids – all of the same type with -1 for non-relevant pixels with elements according to hierarchical format (see README). Can be: tf.Tensor of dtype tf.int32 and arbitrary shape, or np.ndarray of dtype np.int32 and arbitrary shape, or torch.tensor of dtype torch.int32 and arbitrary shape, or Python int, or np.int32 integer.

Returns

same type and shape as the args according to hierarchical format (see README).

Return type

uids

Visualization

panoptic_parts.utils.visualization.random_colors(num)

Returns a list of num random Python int RGB color tuples in range [0, 255]. Colors can be repeated. This is desired behavior so we don’t run out of colors.

Parameters

num – Python int, the number of colors to produce

Returns

a list of tuples representing RGB colors in range [0, 255]

Return type

colors

panoptic_parts.utils.visualization.uid2color(uids, sid2color=None, experimental_deltas=(60, 60, 60), experimental_alpha=0.5)

Generate an RGB palette for all unique uids in uids. The palette is a dictionary mapping each uid from uids to an RGB color tuple, with values in range [0, 255]. A uid is an up to 7-digit integer that is interpreted according to our panoptic parts format (see README), i.e., decode_uids(uid) = (sid, iid, pid).

The colors are generated in the following way:

• if uid represents a semantic-level label, i.e. uid = (sid, N/A, N/A), then `sid2color`[sid] is used.

• if uid represents a semantic-instance-level label, i.e. uid = (sid, iid, N/A), then a random shade of sid2color`[sid] is generated, controlled by `experimental_deltas. The shades are generated so they are as diverse as possible and the variability depends on the number of iids per sid. The more the instances per sid in the uids, the less discriminable the shades are.

• if uid represents a semantic-instance-parts-level label, i.e. uid = (sid, iid, pid), then a random shade is generated as in the semantic-instance-level case above and then it is mixed with a single color from the parula colormap, controlled by experimental_alpha. A different parula colormap is generated for each sid to achieve best discriminability of parts colors per sid.

If sid2color is not provided (is None) then random colors are used. If sid2color is provided but does not contain all the sids of uids an error is raised.

Example usage in {cityscapes, pascal}_panoptic_parts/visualize_from_paths.py.

Parameters

• uids – a list of Python int, or a np.int32 np.ndarray, with elements following the panoptic parts format (see README)

• sid2color – a dict mapping each sid of uids to an RGB color tuple of Python ints with values in range [0, 255], sids that are not present in uids will be ignored

• experimental_deltas – the range per color (Red, Green, Blue) in which to create shades, a small range provides shades that are close to the sid color but makes instance colors to have less contrast, a higher range provides better contrast but may create similar colors between different sid instances

• experimental_alpha – the mixing coeffient of the shade and the parula color, a higher value will make the semantic-instance-level shade more dominant over the parula color

Returns

a dict mapping each uid to a color tuple of Python int in range [0, 255]

Return type

uid2color

Misc

panoptic_parts.utils.utils.safe_write(path, image, **params)

Saves image to path by creating all intermediate directories. If the path already exists it does not override it and returns False. Extra params passed to the PIL.Image.save writer can provided by keyword arguments (e.g. optimize=True or compress_level=9).

Parameters

• path – a path passed to os.path.exists, os.makedirs and PIL.Image.save()

• image – a numpy image passed to PIL.Image.fromarray()

Returns

False is path exists. True if the image is successfully written.

Code Reference

Documented/Undocumented functionality of the rest of the code his repo lies here. This functionality will be added to the API in the future. Until then, the following functions may be moved or be unstable.

Dataset & Evaluation specifications

class panoptic_parts.specs.dataset_spec.DatasetSpec(spec_path)

This class creates a dataset specification from a YAML specification file, so properties in the specification are easily accessed. Moreover, it provides defaults and specification checking.

Specification attribute fields:

• l: list of str, the names of the scene-level semantic classes

• l_things: list of str, the names of the scene-level things classes

• l_stuff: list of str, the names of the scene-level stuff classes

• l_parts: list of str, the names of the scene-level classes with parts

• l_noparts: list of str, the names of the scene-level classes without parts

• scene_class2part_classes: dict, mapping for scene-level class name to part-level class names,

  the ordering of elements in scene_class2part_classes.keys() and scene_class2part_classes.values() implicitly defines the sid and pid respectively, which can be retrieved with the functions below

• sid2scene_class: dict, mapping from sid to scene-level semantic class name

• sid2scene_color: dict, mapping from sid to scene-level semantic class color

• sid_pid2scene_class_part_class: dict, mapping from sid_pid to a tuple of

  (scene-level class name, part-level class name)

Specification attribute functions:

• scene_class_from_sid(sid)

• sid_from_scene_class(name)

• part_classes_from_sid(sid)

• part_classes_from_scene_class(name)

• scene_color_from_scene_class(name)

• scene_color_from_sid(sid)

• scene_class_part_class_from_sid_pid(sid_pid)

• sid_pid_from_scene_class_part_class(scene_name, part_name)

Examples (from Cityscapes Panoptic Parts):

• for the ‘bus’ scene-level class and the ‘wheel’ part-level class it holds: - ‘bus’ in l_things → True - ‘bus’ in l_parts → True - sid_from_scene_class(‘bus’) → 28 - scene_color_from_scene_class(‘bus’) → [0, 60, 100] - part_classes_from_scene_class(‘bus’) → [‘UNLABELED’, ‘window’, ‘wheel’, ‘light’, ‘license plate’, ‘chassis’] - sid_pid_from_scene_class_part_class(‘bus’, ‘wheel’) → 2802

Experimental (format/API may change):

• l_allparts: list of str, a list of all parts in str with format f”{scene_class}-{part_class}”, contains at position 0 the special ‘UNLABELED’ class

Notes

• A special ‘UNLABELED’ semantic class is defined for the scene-level and part-level abstractions.

  This class must have sid/pid = 0 and is added by befault to the attributes of this class if it does not exist in yaml specification.

• It holds that: - the special ‘UNLABELED’ class ∈ l, l_stuff, l_noparts - l = l_things ∪ l_stuff - l = l_parts ∪ l_noparts

• sids are continuous and zero-based

• iids do not need to be continuous

• pids are continuous and zero-based per sid

part_classes_from_scene_class(name)

part_classes_from_sid(sid)

scene_class_from_sid(sid)

scene_class_part_class_from_sid_pid(sid_pid)

scene_color_from_scene_class(name)

scene_color_from_sid(sid)

sid_from_scene_class(name)

sid_pid_from_scene_class_part_class(scene_name, part_name)

class panoptic_parts.specs.eval_spec.PartPQEvalSpec(spec_path)

This class creates an evaluation specification from a YAML specification file and provides convenient attributes from the specification and useful functions. Moreover, it provides defaults and specification checking.

class panoptic_parts.specs.eval_spec.SegmentationPartsEvalSpec(spec_path)

This class creates an evaluation specification from a YAML specification file and provides convenient attributes from the specification and useful functions. Moreover, it provides defaults and specification checking.

Accessible specification attributes:

• dataset_spec: the associated dataset specification

• Nclasses: the number of evaluated classes (including ignored and background)

• scene_part_classes: list of str, the names of the scene-part classes for evaluation,

  ordered by the eval id

• eid_ignore: the eval_id to be ignored in evaluation

• sid_pid2eval_id: dict, maps all sid_pid (0-99_99) to an eval_id,

  according to the template in specification yaml

• sp2e_np: np.ndarray, shape: (10000,), sid_pid2eval_id as an array for dense gathering,

  position i has the sid_pid2eval_id[i] value

Member functions:

Visualization

panoptic_parts.visualization.visualize_label_with_legend.visualize_from_paths(datasetspec_path, label_path)

Visualizes in a pyplot window a label from the provided path.

For visualization pixels are colored on:

• semantic-level: according to colors defined in dataspec.sid2scene_color

• semantic-instance-level: with random shades of colors defined in dataspec.sid2scene_color

• semantic-instance-parts-level: with a mixture of parula colormap and the shades above

See panoptic_parts.utils.visualization.uid2color for more information on color generation.

Parameters

• datasetspec_path – a YAML file path, including keys: sid2scene_color, scene_class_part_class_from_sid_pid

• label_path – a label path, will be passed to Pillow.Image.open

panoptic_parts.utils.visualization.experimental_colorize_label(label, *, sid2color=None, return_sem=False, return_sem_inst=False, emphasize_instance_boundaries=True, return_uid2color=False, experimental_deltas=(60, 60, 60), experimental_alpha=0.5)

Colorizes a label with semantic-instance-parts-level colors based on sid2color. Optionally, semantic-level and semantic-instance-level colorings can be returned. The option emphasize_instance_boundaries will draw a 4-pixel white line around instance boundaries for the semantic-instance-level and semantic-instance-parts-level outputs. If a sid2color dict is provided colors from that will be used otherwise random colors will be generated. See panoptic_parts.utils.visualization.uid2color for how colors are generated.

Parameters

• label – 2-D, np.int32, np.ndarray with up to 7-digit uids, according to format in README

• sid2color – a dictionary mapping sids to RGB color tuples in [0, 255], all sids in labels must be in sid2color, otherwise provide None to use random colors

• return_sem – if True returns sem_colored

• return_sem_inst – if True returns sem_inst_colored

Returns

3-D, np.ndarray with RGB colors in [0, 255],

colorized label with colors that distinguish scene-level semantics, part-level semantics, and instance-level ids

sem_colored: 3-D, np.ndarray with RGB colors in [0, 255], returned if return_sem=True,

colorized label with colors that distinguish scene-level semantics

sem_inst_colored: 3-D, np.ndarray with RGB colors in [0, 255], returned if return_sem_inst=True,

colorized label with colors that distinguish scene-level semantics and part-level semantics

Return type

sem_inst_parts_colored

panoptic_parts.utils.visualization._generate_shades(center_color, deltas, num_of_shades)

panoptic_parts.utils.visualization._num_instances_per_sid(uids)

panoptic_parts.utils.visualization._num_parts_per_sid(uids)

panoptic_parts.utils.visualization._sid2iids(uids)

panoptic_parts.utils.visualization._sid2pids(uids)

Evaluation

class panoptic_parts.utils.experimental_evaluation_IOU.ConfusionMatrixEvaluator_v2(eval_spec, filepaths_pairs, pred_reader_fn, experimental_validate_args=False)

Bases: object

Computes the confusion matrix for the provided ground truth and prediction pairs filepaths using a tf.data pipeline for fast execution.

A standard use of this class is:

evaluator = ConfusionMatrixEvaluator_v2(eval_spec, list_of_filepaths_pairs, pred_reader_fn) confusion_matrix = evaluator.compute_cm() metrics = compute_metrics_with_any_external_function(confusion_matrix)

compute_cm()

print_metrics(*args, **kwargs)

Misc

panoptic_parts.utils.utils.compare_pixelwise(l1, l2)

Compare numpy arrays l1, l2 with same shape and dtype in a pixel-wise manner and return the unique tuples of elements that do not match for the same spatial position.

Parameters

• l1 (np.ndarray) – array 1

• l2 (np.ndarray) – array 2

Examples (supposing the following lists are np.ndarrays):

• compare_pixelwise([1,2,3], [1,2,4]) → [[3], [4]]

• compare_pixelwise([1,2,4,3], [1,2,3,5]) → [[3, 4], [5, 3]]

Returns

unique_diffs: 2D, with columns having the differences for the same position sorted in ascending order using the l1 elements

Return type

np.ndarray

panoptic_parts.utils.utils._sparse_ids_mapping_to_dense_ids_mapping(ids_dict, void, length=None, dtype=np.int32)

Create a dense np.array from an ids dictionary. The array can be used for indexing, e.g. numpy advanced indexing or tensorflow gather. This method is useful to transform a dictionary of uids to class mappings (e.g. {2600305: 3}), to a dense np.array that has in position 2600305 the value 3. This in turn can be used in gathering operations. The reason that the mapping is given in a dictionary is due to its sparseness, e.g. we may not want to hard-code an array with 2600305 elements in order to have the mapping for the 2600305th element.

ids.values() and void must have the same shape and dtype.

The length of the dense_mapping is infered from the maximum value of ids_dict.keys(). If you need a longer dense_mapping provide the length in length.

Parameters

• ids_dict – dictionary mapping ids to numbers (usually classes),

• void – int, list of int, tuple of int, the positions of the dense array that don’t appear in ids_dict.keys() will be filled with the void value,

• length – the length of the dense mapping can be explicitly provided

• dtype – the dtype of the returned dense mapping