evaluators package

Subpackages

Submodules

evaluators.pyodEvaluator module

class evaluators.pyodEvaluator.PyodEvaluator(config, models, data_test)

Bases: object

Evaluator for Python Outlier Detection models

evaluate()

Evaluate PyOD models.

  • OutlierDetectionVisualizer: Visualization of test results.

evaluators.pytorchEvaluator module

class evaluators.pytorchEvaluator.PytorchEvaluator(config, model, dataloader)

Bases: object

Evaluator for PyTorch models.

adapt_input(output, labels, time, event_name)

Adapt input size for evaluation purposes

Parameters:

  • output (Tensor) – Predicted outputs for the event to be evaluated.

  • labels (Tensor) – Labels for the event to be evaluated.

  • time (List) – Timestep of the event to be evaluated.

  • event_name (List) – Name of the event to be evaluated.

Returns:

Predicted outputs for the event to be evaluated.

Return type:

Tensor

Returns:

Labels for the event to be evaluated.

Return type:

Tensor

:return Timestep of the event to be evaluated. :rtype int :return Name of the event to be evaluated. :rtype int

connect_events(events, task)

Assign the same label to events closer than min_dist parameter in space (postprocessing)

Parameters:

  • events (np.array) – Event categorical masks after performing detection.

  • task (str) – Evaluation task to be performed (‘characterization’ or ‘xai’).

Returns:

Event categorical masks after connecting events (postprocessing).

Return type:

np.array

define_events(inference_outputs, task)

Identify events and get their ids (unique values)

Parameters:

  • inference_outputs (List[Tensor]) – Predicted outputs during the inference stage.

  • task (str) – Evaluation task (‘characterization’ or ‘xai’)

Returns:

Events obtained after applying binary extreme event detection on top of the probabilistic outputs.

Return type:

List

evaluate(inference_outputs, subset)
Evaluate PyTorch models.

  • ClassificationVisualizer: Visualization of results (only available for detection task)

  • ClassificationCharacterizer: Characterization of detected events (only available for detection task)

  • XAI: Explainable AI for the interpretation of results (available both for detection and impact assessment tasks).

Parameters:

  • inference_outputs (List[Tensor]) – Predicted outputs during the inference stage.

  • subset (str) – Data subset to be evaluated (‘train’, ‘val’ or ‘test’)

Returns:

Predicted outputs during the inference stage.

Return type:

List[Tensor]

inference(subset)

Inference stage

Parameters:

subset (str) – Data subset to be evaluated (‘train’, ‘val’ or ‘test’)

Returns:

Outputs obtained during the inference stage.

Return type:

dict

remove_scant_labels(tmp_test_outputs, connectivity)

Remove scant labels after applying binary/categorical detection (postprocessing).

1) Remove small holes -> replace with ones (non drought areas surrounded by drought areas) Maximum area to be filled defined by the threshold.

2) Remove small objects -> replace with zeros (drought areas surrounded by non-drought areas) Minimum area to retain the drought object defined by the threshold

Parameters:

  • tmp_test_outputs (List[Tensor]) – Predicted outputs (binary detection) for the test subset.

  • connectivity (int) – Connectivity parameter (4 or 8)

Returns:

Predicted outputs (binary/categorical detection) after removing scant labels/postprocessing.

Return type:

List[Tensor]

set_threshold(task)

Determine the optimum threshold for binary extreme event detection based on the validation data subset.

Parameters:

task (str) – Evaluation task to be performed (‘characterization’ or ‘xai’).

Returns:

Optimum threshold for binary extreme event detection.

Return type:

int

Module contents