How can we quantify the quality of the prediction of Machine Learning models?¶

There are a number of scores defined and computed for various machine learning problems. Here we categorize the scoring methods based on the problem category: Classification, Regression and Clustering and include the most frequently used measures.

Classification

accuracy score: fraction of the correct prediction to total predictions

precision-recall curve: shows the precision ( $P=\frac{T_p}{T_p+F_p}$ ) against recall ( $R=\frac{T_p}{T_p+F_n}$ ) by varying a decision parameter where $T_p$ , $T_n$ , $F_p$ , and $F_n$ refer to the true positive count, the true negative count, the false positive count, and the false negative count, respectively.

f1-score: f1 score is computed as $2\frac{P*R}{P+R}$ where $P$ and $R$ refer to precision and recall respectively.

roc curve: shows the true positive rate (TPR = $\frac{T_p}{T_p+F_n}$ ) against the false positive rate (FPR = $\frac{F_p}{F_p+T_n}$ ) by varying a decision parameter.

confusion matrix: entry i,j in this matrix shows the number of observations in group i but predicted to be in group j.

Clustering

adjusted rand index: is the corrected for chance version of rand index which is a measure of similarity between two data clusterings and represents the chance of occurrence of agreements for any pair of elements.

adjusted mutual information: is the corrected for chance version of mutual information score which is bounded by the entropies of each cluster.

contingency index: entry i,j in this matrix shows the number of true members of cluster i predicted to be in cluster j.

Regression

mean squared error: defined as the average of square difference between actual and predicted output

mean absolute error: defined as the average of absolute difference between actual and predicted output

explained variance score: defined as 1 minus variance of error divided by variance of actual output, that is:

$1 - \frac{Var\{ y - \hat{y}\}}{Var\{y\}}$

R² score: defined as 1 minus sum of squared error divided by sum of squared difference from output mean, that is:

$1 - \frac{\sum_{i=1}^{n} (y_i - \hat{y}_i)^2}{\sum_{i=1}^{n} (y_i - \bar{y})^2}\;\mathrm{where}\;\bar{y} = \frac{1}{n} \sum_{i=1}^{n} y_i$