Evaluation Metrics
- deepod.metrics.auc_pr(y_true, y_score)[source]
Calculates the area under the Precision-Recall (PR) curve.
- Parameters:
y_true (np.array, required) – True binary labels. 0 indicates a normal timestamp, and 1 indicates an anomaly.
y_score (np.array, required) – Predicted anomaly scores. A higher score indicates a higher likelihood of being an anomaly.
- Returns:
The score of the area under the PR curve.
- Return type:
- deepod.metrics.auc_roc(y_true, y_score)[source]
Calculates the area under the Receiver Operating Characteristic (ROC) curve.
- Parameters:
y_true (np.array, required) – True binary labels. 0 indicates a normal timestamp, and 1 indicates an anomaly.
y_score (np.array, required) – Predicted anomaly scores. A higher score indicates a higher likelihood of being an anomaly.
- Returns:
The score of the area under the ROC curve.
- Return type:
- deepod.metrics.point_adjustment(y_true, y_score)[source]
adjust the score for segment detection. i.e., for each ground-truth anomaly segment, use the maximum score as the score of all points in that segment. This corresponds to point-adjust f1-score. This function is copied/modified from the source code in [Zhihan Li et al. KDD21]
- Parameters:
y_true (np.array, required) – Data label, 0 indicates normal timestamp, and 1 is anomaly.
y_score (np.array, required) – Predicted anomaly scores, higher score indicates higher likelihoods to be anomaly.
- Returns:
Adjusted anomaly scores.
- Return type:
np.array
- deepod.metrics.tabular_metrics(y_true, y_score)[source]
Calculates evaluation metrics for tabular anomaly detection.
- Parameters:
y_true (np.array, required) – Data label, 0 indicates normal timestamp, and 1 is anomaly.
y_score (np.array, required) – Predicted anomaly scores, higher score indicates higher likelihoods to be anomaly.
- Returns:
A tuple containing:
- auc_roc (float):
The score of area under the ROC curve.
- auc_pr (float):
The score of area under the precision-recall curve.
- f1 (float):
The score of F1-score.
- Return type:
- deepod.metrics.ts_metrics(y_true, y_score)[source]
Calculates evaluation metrics for time series anomaly detection.
- Parameters:
y_true (np.array, required) – Data label, 0 indicates normal timestamp, and 1 is anomaly.
y_score (np.array, required) – Predicted anomaly scores, higher score indicates higher likelihoods to be anomaly.
- Returns:
A tuple containing:
- roc_auc_score (float):
The score of area under the ROC curve.
- average_precision_score (float):
The score of area under the precision-recall curve.
- best_f1 (float):
The best score of F1-score.
- best_p (float):
The best score of precision.
- best_r (float):
The best score of recall.
- Return type:
- deepod.metrics.ts_metrics_enhanced(y_true, y_score, y_test)[source]
This function calculates additional evaluation metrics for time series anomaly detection. It returns a variety of metrics, including those sourced from the code in [A Huet et al. KDD22] and [J Paparrizos et al. VLDB22]. The function requires three inputs: y_true (data label), y_score (predicted anomaly scores), and y_test (predictions of events).
- Parameters:
y_true (np.array) – Data label, where 0 indicates a normal timestamp and 1 indicates an anomaly.
y_score (np.array) – Predicted anomaly scores, where a higher score indicates a higher likelihood of being an anomaly.
y_test (np.array) – Predictions of events, where 0 indicates a normal timestamp and 1 indicates an anomaly.
- Returns:
A tuple containing:
- auroc (float):
The score of the area under the ROC curve after point adjustment.
- aupr (float):
The score of the area under the precision-recall curve after point adjustment.
- best_f1 (float):
The best score of F1-score after point adjustment.
- best_p (float):
The best score of precision after point adjustment.
- best_r (float):
The best score of recall after point adjustment.
- affiliation_precision (float):
The score of affiliation precision.
- affiliation_recall (float):
The score of affiliation recall.
- vus_r_auroc (float):
The score of range VUS ROC.
- vus_r_aupr (float):
The score of range VUS PR.
- vus_roc (float):
The score of VUS ROC.
- vus_pr (float):
The score of VUS PR.
- Return type:
References
Guansong Pang, Chunhua Shen, and Anton van den Hengel. Deep anomaly detection with deviation networks. In Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining, 353–362. 2019.
Lukas Ruff, Robert Vandermeulen, Nico Görnitz, Lucas Deecke, Shoaib Siddiqui, Alexander Binder, Emmanuel Müller, and Marius Kloft. Deep one-class classification. International conference on machine learning, 2018.