ROC-AUC 與 PR-AUC：稀有事件為何需要多種評估指標

Imagine you are building a system to monitor social media during a wildfire.

Thousands of posts appear every hour. Most are ordinary discussions, but a small number may contain important information about an evacuation, a blocked road, or an emerging hazard.

Your classifier is designed to flag potentially relevant posts for further review.

You evaluate it and obtain a ROC-AUC of 0.95.

That sounds excellent.

But when you examine the alerts, you discover that most of the flagged posts are irrelevant.

How can a model have a high ROC-AUC and still generate so many false alarms?

The answer becomes clearer when we compare ROC-AUC with precision-recall metrics, especially when the positive class is rare.

In this tutorial, we will construct a small, reproducible Python example, calculate both metrics, and explore what they tell us about the same classifier.

• The problem with rare events

Consider a dataset containing 10,000 social media posts.

Only 100 contain the event we want to identify. The remaining 9,900 do not.

This is an imbalanced classification problem.

The positive class is the event we want to detect. It represents only 1% of the dataset.

This situation appears in a plethora of applications:

• Fraud detection, where fraudulent transactions are uncommon.
• Security monitoring, where genuine attacks are rare.
• Predictive maintenance, where equipment failures occur infrequently.
• Crisis monitoring, where important event reports are a small fraction of all posts.

A model can achieve 99% accuracy simply by predicting that every post is negative.

It would identify none of the 100 relevant events.

This illustrates an important lesson: a metric is only useful when it reflects the problem we are trying to solve.

• What do ROC-AUC and PR-AUC measure?

Before writing code, let us distinguish the metrics.

ROC-AUC: How well does the model rank positive examples?

A Receiver Operating Characteristic (ROC) curve plots:

• True Positive Rate (TPR): the proportion of positive examples correctly identified.
• False Positive Rate (FPR): the proportion of negative examples incorrectly flagged.

The true positive rate is also called recall or sensitivity.

The true positive rate is also called recall or sensitivity:

TPR = TP / (TP + FN)

FPR = FP / (FP + TN)

ROC-AUC summarizes the ROC curve across classification thresholds. It measures how well the model ranks positive examples above negative examples.

A score of 1.0 represents perfect ranking; a score near 0.5 indicates performance similar to a random ranking under the usual binary-classification interpretation.

However, the false positive rate uses all negative examples as its denominator. When there are thousands of negatives, a seemingly small rate can still produce a large number of false alarms.

Precision-recall: How useful are the positive predictions?

A precision-recall (PR) curve plots precision against recall at different thresholds.

Precision answers the question:

Of all the posts flagged by the model, how many are actually relevant?

Precision = TP / (TP + FP)

Recall answers a different question:

Of all the relevant posts, how many did the model find?

Recall = TP / (TP + FN)

The area under the precision-recall curve summarizes the trade-off between these two quantities.

In scikit-learn, we will use Average Precision (AP) as our summary of the PR curve. AP is a common PR-curve summary, but it is not the same as calculating the trapezoidal area under the curve.

For imbalanced datasets, precision-recall analysis often reveals the practical consequences of false positives more clearly than ROC analysis alone (Saito & Rehmsmeier, 2015).

• Build a deliberately simple example

Instead of training a complicated model, let us construct a synthetic dataset with known outcomes.

This is intentional: the goal is to isolate the behaviour of the metrics, not to claim that we have built a production-ready detector.

Our classifier assigns the following scores:

Higher scores indicate that the model considers a post more likely to be positive.

Notice something interesting: the 495 negative posts with the highest scores rank ahead of the positive posts.

This is a useful teaching example because it lets us see how a low false positive rate can coexist with a substantial number of false alerts.

Let us implement it.

import numpy as np

from sklearn.metrics import (
    average_precision_score,
    confusion_matrix,
    precision_score,
    recall_score,
    roc_auc_score,
)

# 10,000 posts: 100 positive and 9,900 negative.
y_true = np.concatenate([
    np.zeros(9_900, dtype=int),
    np.ones(100, dtype=int),
])

# Scores aligned with the labels above.
# Negative posts: 495 high scores, then 9,405 low scores.
# Positive posts: 80 medium-high scores, then 20 lower scores.
y_score = np.concatenate([
    np.full(495, 0.90),
    np.full(9_405, 0.10),
    np.full(80, 0.80),
    np.full(20, 0.60),
])

# Threshold for converting scores into binary decisions.
threshold = 0.50
y_pred = (y_score >= threshold).astype(int)

# Ranking metrics.
roc_auc = roc_auc_score(y_true, y_score)
average_precision = average_precision_score(y_true, y_score)

# Metrics at the selected threshold.
precision = precision_score(y_true, y_pred, zero_division=0)
recall = recall_score(y_true, y_pred, zero_division=0)

tn, fp, fn, tp = confusion_matrix(
    y_true, y_pred, labels=[0, 1]
).ravel()

accuracy = (tp + tn) / len(y_true)

print(f"ROC-AUC: {roc_auc:.4f}")
print(f"Average Precision: {average_precision:.4f}")
print(f"Precision at threshold {threshold}: {precision:.4f}")
print(f"Recall at threshold {threshold}: {recall:.4f}")
print(f"Accuracy at threshold {threshold}: {accuracy:.4f}")

print(f"True positives: {tp}")
print(f"False positives: {fp}")
print(f"True negatives: {tn}")
print(f"False negatives: {fn}")

The expected output is:

ROC-AUC: 0.9500
Average Precision: 0.1449
Precision at threshold 0.5: 0.1681
Recall at threshold 0.5: 1.0000
Accuracy at threshold 0.5: 0.9505

True positives: 100
False positives: 495
True negatives: 9405
False negatives: 0

The contrast is striking.

The ROC-AUC is 0.95, which indicates strong overall ranking performance on this synthetic dataset. Yet the average precision is only about 0.145, and at the selected threshold, fewer than 17% of flagged posts are genuinely positive.

The example demonstrates why these metrics answer different questions.

We found every positive case.

That gives us 100% recall.

But we also flagged 495 negative posts.

Our precision is:

Precision: = 100 (100 + 495), approx 0.1681

In other words, out of 595 flagged posts, only 100 are positive.

The model has achieved perfect recall at this threshold, but the alert workload is substantial.

Imagine a crisis-response team that must manually investigate every alert. Reviewing 595 posts to find 100 relevant ones could be expensive, especially if the system operates continuously.

That is the operational side of the metric.

ROC-AUC tells us about ranking across thresholds. Precision tells us how much confidence we should place in the positive predictions produced at a particular threshold.

Neither should be interpreted in isolation.

• Visualize both curves

import matplotlib.pyplot as plt

from sklearn.metrics import (
    average_precision_score,
    precision_recall_curve,
    roc_auc_score,
    roc_curve,
)

# ROC curve.
fpr, tpr, _ = roc_curve(y_true, y_score)

# Precision-recall curve.
precision_values, recall_values, _ = precision_recall_curve(
    y_true, y_score
)

roc_auc = roc_auc_score(y_true, y_score)
average_precision = average_precision_score(y_true, y_score)
prevalence = y_true.mean()

fig, (ax_roc, ax_pr) = plt.subplots(1, 2, figsize=(12, 5))

# Left: ROC.
ax_roc.plot(fpr, tpr, label=f"ROC-AUC = {roc_auc:.3f}")
ax_roc.plot([0, 1], [0, 1], "--", label="Random ranking")
ax_roc.set_xlabel("False Positive Rate")
ax_roc.set_ylabel("True Positive Rate (Recall)")
ax_roc.set_title("ROC Curve")
ax_roc.legend()
ax_roc.grid(True, alpha=0.3)

# Right: Precision-recall.
ax_pr.plot(
    recall_values,
    precision_values,
    label=f"Average Precision = {average_precision:.3f}",
)
ax_pr.axhline(
    prevalence,
    linestyle="--",
    label=f"Positive prevalence = {prevalence:.1%}",
)
ax_pr.set_xlabel("Recall")
ax_pr.set_ylabel("Precision")
ax_pr.set_title("Precision-Recall Curve")
ax_pr.legend()
ax_pr.grid(True, alpha=0.3)

plt.tight_layout()
plt.show()

The ROC curve shows strong overall ranking performance, with a ROC-AUC of 0.95.

However, this metric can obscure the impact of false positives when the dataset is highly imbalanced.

The precision-recall curve highlights this issue: 495 negative posts receive higher scores than all the positive posts, reducing precision despite the high ROC-AUC.

The dashed line in the PR plot represents positive-class prevalence. With random rankings, expected Average Precision is approximately equal to this prevalence, or 0.01 (1%) in this example.

One caution: this dataset uses intentionally constructed score groups. Real models typically produce different score distributions, so their curves may have different shapes.

• Does Pr-AUC always matter more?

Not necessarily.

ROC-AUC remains useful when comparing ranking behaviour across thresholds, and it may be particularly valuable when both classes matter to the task.

Precision-recall metrics are especially informative when the positive class is rare, and the quality of positive predictions matters operationally. This is why they are widely considered alongside ROC analysis in imbalanced classification research (Davis & Goadrich, 2006; Saito & Rehmsmeier, 2015).

The choice ultimately depends on the cost of the errors.

Consider two scenarios.

Scenario A: Missing an event is extremely costly.

In early crisis detection, you may prefer a lower threshold to capture as many potentially important posts as possible. This increases recall but can also increase the number of false alarms.

Scenario B: Human review capacity is limited.

If a small team can inspect only a limited number of posts, precision and precision-recall behaviour become particularly important. A model that flags too many irrelevant posts may overwhelm the people who are supposed to investigate them.

The threshold should therefore reflect the task, available resources, and relative cost of false positives and false negatives.

AUC values alone do not choose the threshold for you.

• How should we apply this to anomaly detection and fact-checking?

This distinction is especially important in social-media analysis during emergencies.

An anomaly detector might identify an unusual cluster of posts, a sudden change in message volume, or content that differs from typical patterns.

That output is a signal for further investigation. It is not evidence that a post is false.

The system still needs to examine the claim, assess available evidence, and account for uncertainty.

Evaluation should reflect those separate responsibilities. Anomaly detection should be tested for its ability to prioritize relevant cases, while verification should be evaluated against suitable reference evidence and labels.

This is one of the design principles I explored in my MSc thesis on anomaly detection and fact-checking of social-media data in crisis management: unusual content should be prioritized for analysis without automatically treating it as misinformation.

The distinction between detection and verification is not merely philosophical. It changes what the system measures, how its outputs are interpreted, and which errors matter.

• Conclusion

When positive cases are rare, a model can look impressive under one metric while exposing weaknesses under another.

ROC-AUC summarizes ranking behaviour across thresholds. Precision-recall analysis makes the relationship between retrieved positives and false alarms more visible. Precision, recall, and the confusion matrix then help us understand what happens at the operating threshold we actually intend to use.

My practical checklist is:

• Inspect the class distribution. Understand how rare the positive class is before interpreting scores.

• Report more than accuracy. Consider ROC-AUC, average precision, precision, recall, and the confusion matrix.

• Choose thresholds based on consequences. Think about the workload created by false positives and the harm caused by missed events.

• Validate on representative data. A model's performance on a controlled dataset does not automatically establish its usefulness in a live environment.

Most importantly, evaluate the model against the task it is intended to support.

In AI-assisted decision-making, the goal is not simply to produce a high score. It is to understand what that score establishes—and what remains uncertain.

References and further reading

• Saito, T., & Rehmsmeier, M. (2015). The Precision-Recall Plot Is More Informative than the ROC Plot When Evaluating Binary Classifiers on Imbalanced Datasets. PLOS ONE, 10(3), e0118432.
• Davis, J., & Goadrich, M. (2006). The Relationship Between Precision-Recall and ROC Curves. Proceedings of the 23rd International Conference on Machine Learning, 233–240.
• scikit-learn. ROC-AUC documentation: https://scikit-learn.org/stable/modules/model_evaluation.html#receiver-operating-characteristic-roc
• scikit-learn. Average Precision documentation: https://scikit-learn.org/stable/modules/model_evaluation.html#precision-recall-and-f-measures

Enjoyed this breakdown? Feel free to check out my website at ioannidis.dev for more machine learning guides and articles.