ROC AUC Calculator

ROC AUC is the area under the receiver operating characteristic curve, and it equals the probability that a binary classifier gives a randomly chosen positive example a higher score than a randomly chosen negative one. This ROC AUC calculator takes true labels and model scores and returns the AUC, the curve, the Gini coefficient, average precision and the Youden-optimal threshold.

What does a ROC curve plot?

A classifier that outputs a score, such as the probability from a sigmoid output unit, does not make a yes-or-no decision until you choose a threshold. Every row scoring at or above the threshold is called positive and the rest negative. The ROC curve records what happens to two rates as that threshold slides from above the highest score down to the lowest.

The vertical axis is the true positive rate (TPR, also called recall or sensitivity): the share of real positives that were caught. The horizontal axis is the false positive rate (FPR): the share of real negatives that were wrongly flagged. A very high threshold flags nothing and sits at (0, 0). A very low one flags everything and sits at (1, 1). A ROC curve calculator traces the path between those corners, one step per distinct score.

How is ROC AUC calculated?

\(TP\), \(FP\), \(FN\) and \(TN\) are the true positive, false positive, false negative and true negative counts at one threshold. With the curve points ordered from \((0, 0)\) to \((1, 1)\), the area is the sum of the trapezoids between neighbouring points:

Here \(k\) is the number of distinct scores and point 0 is the origin. The same number comes from comparing every positive with every negative:

\(P\) and \(N\) are the numbers of positive and negative rows, \(s_p\) is the score of a positive and \(s_n\) the score of a negative. Each pair where the positive scores higher counts as 1, a tie counts as one half, and the total is divided by the number of pairs. Two related outputs use simple formulas: \(\text{Gini} = 2 \cdot \text{AUC} - 1\), and average precision \(\text{AP} = \sum_n (R_n - R_{n-1}) P_n\), where \(R_n\) and \(P_n\) are recall and precision at the n-th threshold.

Worked example: counting the pairs

The default input has six rows. The positives scored 0.9, 0.7 and 0.6, and the negatives scored 0.8, 0.55 and 0.4. That makes 3 × 3 = 9 positive-negative pairs, and the table shows which of them the positive wins.

There are no ties, so AUC = 7 / 9 = 0.7778 and Gini = 2 × 0.7778 − 1 = 0.5556. The threshold route gives the same area. Lowering the threshold through the six scores produces these curve points:

The curve moves right three times, each by 1/3. The average height of those three steps is 1/3, 1 and 1, so the area is 1/3 × 1/3 + 1/3 × 1 + 1/3 × 1 = 7/9. Youden’s J peaks at the threshold 0.6, where all three positives are caught for one false positive. Recall rises at the thresholds 0.9, 0.7 and 0.6, where precision is 1, 2/3 and 3/4, so average precision is 1/3 × (1 + 2/3 + 3/4) = 29/36 = 0.8056.

How do you interpret a ROC AUC value?

Labels such as “good” or “excellent” attached to fixed AUC bands are conventions from particular fields, not properties of the metric. Whether 0.8 is useful depends on the task, on what the current baseline achieves and on what each kind of error costs. Compare models on the same data, and report a second metric that reflects the decision you will actually make.

Why can ROC AUC look optimistic on imbalanced data?

FPR divides false positives by the number of negatives. When negatives vastly outnumber positives, a large count of false alarms is still a small FPR, so the curve hugs the left edge and the area stays high. Precision divides by the number of rows flagged instead, so it notices those false alarms. Precision is the conditional probability of a true positive given a positive prediction, and it falls as the negative class grows.

The Imbalanced preset shows the gap with 2 positives among 20 rows. ROC AUC is 34 / 36 = 0.9444, yet average precision is only 0.75, and at the threshold 0.61 the FPR is 2 / 18 = 0.1111 while half of the four flagged rows are wrong. If the positive class is rare and it is the class you care about, look at the precision-recall curve and average precision alongside ROC AUC. A random ranking scores 0.5 on ROC AUC whatever the class balance, whereas its expected average precision is roughly the share of positives in the data.

How do you choose a classification threshold?

AUC summarises every threshold at once, so it cannot tell you which one to deploy. Youden’s J = TPR − FPR is one rule: it picks the curve point farthest above the chance diagonal and treats a missed positive and a false alarm as equally bad. That assumption rarely holds exactly. If a false negative costs far more than a false positive, a lower threshold than the J-optimal one is the better choice, and the reverse is true when false alarms are expensive.

Use the slider to move along the curve and read the counts at each cut-off. Those four counts are a full confusion matrix, and precision and recall at the chosen point combine into an F1 score. Pick the threshold on validation data, not on the test set you report.

Where is ROC AUC used in machine learning?

• Model selection. It is a standard scoring option for cross-validation and hyperparameter search on binary classifiers (scoring="roc_auc" in scikit-learn), because it needs no threshold.
• Ranking problems. Credit scoring, fraud queues and medical triage act on the order of the scores. Credit risk teams usually quote the same quantity as the Gini coefficient.
• Alongside a calibration metric. AUC uses only the order of the scores, so it says nothing about whether a predicted 0.8 means 80%. Pair it with log loss from the cross-entropy loss calculator, and see the other tools in the model evaluation calculators hub.

Common mistakes

• Passing hard predictions. Feeding 0/1 class predictions instead of scores leaves a single threshold, so the curve collapses to one point and the AUC is usually understated.
• Scoring the wrong class. Using the probability of class 0 as the score returns 1 − AUC. An AUC well below 0.5 usually means this.
• Reading AUC as accuracy. An AUC of 0.9 does not mean 90% of predictions are correct. It is about pairs of rows, not single decisions.
• Assuming calibration. Any transformation that keeps the order of the scores leaves AUC unchanged, so a high AUC does not make the scores trustworthy probabilities.
• Evaluating a fold with one class. With no positives or no negatives the metric is undefined. Stratify the splits.
• Ignoring ties. Tied scores must move as one diagonal step. Stepping through tied rows one at a time makes the answer depend on row order.

How do you compute ROC AUC in Python?

scikit-learn provides sklearn.metrics.roc_auc_score for the area, roc_curve for the points and average_precision_score for the precision-recall summary. This reproduces the worked example, then repeats the AUC by counting pairs in NumPy.

import numpy as np
from sklearn.metrics import roc_auc_score, roc_curve, average_precision_score

y_true = np.array([1, 0, 1, 1, 0, 0])
y_score = np.array([0.9, 0.8, 0.7, 0.6, 0.55, 0.4])

print(round(roc_auc_score(y_true, y_score), 6))            # 0.777778
print(round(average_precision_score(y_true, y_score), 6))  # 0.805556

fpr, tpr, thresholds = roc_curve(y_true, y_score, drop_intermediate=False)
best = np.argmax(tpr - fpr)
print(thresholds[best], tpr[best], round(fpr[best], 6))    # 0.6 1.0 0.333333

# The same AUC by counting positive-negative pairs
pos = y_score[y_true == 1]
neg = y_score[y_true == 0]
diff = pos[:, None] - neg[None, :]
auc = ((diff > 0).sum() + 0.5 * (diff == 0).sum()) / diff.size
print(round(auc, 6))                                       # 0.777778

By default roc_curve drops points that lie on a straight line between their neighbours, which changes how many thresholds it returns but not the area. This page keeps every distinct score, matching drop_intermediate=False. In Keras, tf.keras.metrics.AUC approximates the area from a fixed grid of thresholds (200 by default), so it can differ slightly from the exact value.

Frequently asked questions

What is a good ROC AUC score?

There is no universal cut-off. 0.5 means the scores rank no better than chance and 1 means a perfect ranking, but whether a value in between is good depends on the problem, the baseline you are comparing against and the cost of each kind of error.

Can ROC AUC be less than 0.5?

Yes. A value below 0.5 means the scores rank negatives above positives more often than not. Reversing the scores gives 1 minus that AUC, so a very low value often points to a flipped label or the probability of the wrong class.

What is the difference between ROC AUC and average precision?

ROC AUC summarises true positive rate against false positive rate, and its chance level is 0.5 whatever the class balance. Average precision summarises precision against recall, and its chance level is about the share of positives, so it is more sensitive to false positives when positives are rare.

How is the Gini coefficient related to AUC?

Gini equals 2 times AUC minus 1. It rescales the area so that a random ranking scores 0 and a perfect ranking scores 1. An AUC of 0.7778 corresponds to a Gini of 0.5556.

Related calculators

Once a threshold is fixed, the confusion matrix calculator and the F1 score calculator describe the classifier at that single operating point.

More tools that pair with this one:

• All model evaluation calculators
• Log loss for predicted probabilities
• Turn a logit into a probability
• Find a score cut-off by percentile