• No results found

3.10 Performance Loss/Gain on Classifiers

4.1.6 Result of analysis of Analysis of Variance (ANOVA)

4.1.6.5 ANOVA test on all classifiers

Analysis of the performance of classifiers used in the study on all the datasets was carried out using the ROC_AUC metric. The results are presented in Table 4.33.

It can be observed from Table 4.33 that SVM was the least performing classifier on the DM dataset and was alone in its subset out of the four subsets created. DECISION FOREST gave the best performance of all the classifiers and it shared similar classification characteristics with BAGGING, RANDOM COMMITTEE, RANDOM FOREST, DECISION TREE, MULTICLASS CLASSIFIER, STACKING, BOOSTING, REP TREE, MLP and RIPPER in the same subset ‘A’ respectively.

The p–value = Sig. (Significance) ≈ 0.675 > 0.05, so both the null and alternative hypothesis were retained for SSS Result dataset. The mean values do not differ amongst the 14 classifiers so there was no need to perform a post hoc test. However, the post hoc Tukey-Kramer (Tukey’s W) multiple comparison analysis was still carried out. One homogeneous subset table was created as shown in the result displayed in Table 4.33. All the 14 classifiers were grouped together in one subset ‘A’. DECISION FOREST classifier gave the best performance while RIPPER classifier gave the least performance.

Four homogeneous subsets were created for the CM dataset. RANDOM FOREST classifier gave the best performance out of all classifiers and is in the same subset ‘A’

with SVM, RANDOM COMMITTEE, 1B3, BOOSTING, STACKING, DECION FOREST, DECISION TREE, RANDOM TREE, MULTICLASSCLASSIFIER and BAGGING classifiers. REPTREE gave the least performance of all the classifiers and is in the same subset ‘D’ as RIPPER, MLP, BAGGING, MULTICLASS CLASSIFIER, RANDOM TREE, DECISION TREE and STACKING.

Therefore, the ANOVA test result which was also validated by Friedman’s test showed that all the classifiers behaved similarly on all the 14 different data sampling schemes with SSS Result dataset. Any of the classifiers can be used for classification of this dataset.

UNIVERSITY OF IBADAN LIBRARY

175

Considering base classifiers, Decision Tree classifier gave the best performance on DM and SSS Result dataset. The SVM classifier surpassed other classifiers on CM dataset.

For ensembles, BAGGING, a homogeneous ensemble with Decision Tree classifier as the base classifier gave the best performance on DM and SSS Result dataset while RANDOM FOREST, also a homogeneous ensemble gave the best performance CM dataset. BOOSTING, a homogeneous ensemble with Decision Tree classifier as the base classifier gave the least performance on DM dataset. STACKING, a heterogeneous ensemble had the least performance on SSS Result dataset while BAGGING had the least performance on CM dataset. SVM, RIPPER and REPTREE classifiers had the poorest performance on all three dataset respectively.

UNIVERSITY OF IBADAN LIBRARY

176

Table 4.33: ANOVA on all classifiers with all data sampling schemes using ROC_AUC metric

DM SSS Result CM

S/N Classifiers Mean Subset Mean Subset Mean Subset

1 DECISION FOREST 0.8764 A 0.8550 A 0.6007 A,B,C,D

2 BAGGING 0.8743 A 0.8543 A 0.5736 A,B,C,D

3 RANDOM COMMITTEE 0.8729 A 0.8436 A 0.6243 A,B,C

4 RANDOM FOREST 0.8657 A,B 0.8471 A 0.6350 A

5 DECISION TREE 0.8629 A,B 0.8507 A 0.5950 A,B,C,D

6 MULTICLASSCLASSIFIER 0.8607 A,B,C 0.8493 A 0.5800 A,B,C,D

7 STACKING 0.8521 A,B,C 0.8386 A 0.6007 A,B,C,D

8 BOOSTING 0.8479 A,B,C 0.8393 A 0.6129 A,B,C

9 REP TREE 0.8379 A,B,C 0.8507 A 0.5379 D

10 MLP 0.8029 A,B,C 0.8057 A 0.5650 B,C,D

11 RIPPER 0.7700 A,B,C 0.7579 A 0.5636 C,D

12 1B3 0.7521 B,C 0.8471 A 0.6143 A,B,C

13 RANDOM TREE 0.7421 C 0.8443 A 0.5914 A,B,C,D

14 SVM 0.6143 D 0.7779 A 0.6336 A,B

UNIVERSITY OF IBADAN LIBRARY

177

4.1.6.6 ANOVA on all datasets with all data sampling schemes using performance loss/gain metric

The result of ANOVA test on performance loss/gain on both enhanced and existing data sampling schemes is presented in Table 4.34.

Five homogenous subsets were created for the DM dataset. One of the enhanced data sampling schemes, SMOTE300ENN performed best. This established an improvement (gain) on performance with respect to the RAW DATA. SMOTE300ENN, SMOTEENN and SMOTE300NCL are all members of the same subset ‘E’. The CNN data sampling performed least and is alone in its subset ‘A’.

Similarly, six homogeneous subsets were created for SSS Result dataset.

SMOTE300ENN gave the best improvement and in the same subset ‘E’ as ENN, SMOTEENN, 5ENN, SMOTE300NCL and SMOTENCL data sampling schemes.

CNN data sampling scheme performed least and alone in its subset ‘E’.

Seven homogeneous subsets were created for CM dataset. SMOTE300RUS and SMOTE300ENN are in the same subset ‘G’ and gave good performance. The data sampling schemes with the least performance were are CNN and RUS respectively and are both in the same subset ‘A’.

Therefore, ANOVA test result as confirmed by Friedman test on performance loss/gain metric on all dataset revealed that two (SMOTE300ENN and SMOTE300NCL) of the enhanced data sampling schemes were ranked out of the best seven of the fourteen data sampling schemes across all datasets. CNN data sampling scheme gave the worst performance across all dataset.

The average performance loss/gain metric on all the data sampling schemes across the three datasets is presented in Table 4.35. The results in the table also corroborates the result of analysis.

UNIVERSITY OF IBADAN LIBRARY

178

Table 4.34: ANOVA on all datasets with all data sampling schemes using performance loss/gain metric

DM SSS Result CM

S/N Data sampling schemes

Mean Subset Mean Subset Mean Subset

1 SMOTERUS -0.0948 B,C,D 0.0279 B,C -0.1618 E,F

2 SMOTENCL -0.1150 C,D -0.0785 C,D,E -0.0542 B,C

3 SMOTEENN -0.1934 D,E -0.2762 E -0.0851 B,C,D

4 SMOTE300RUS -0.1155 C,D -0.0197 C,D -0.2245 G

5 SMOTE300NCL -0.1552 C,D,E -0.0948 C,D,E -0.1435 D,E,F

6 SMOTE300ENN -0.2711 E -0.2804 E -0.1776 F,G

7 SMOTE300 -0.1128 C,D -0.0387 C,D,E -0.1039 C,D,E

8 SMOTE -0.0573 B,C -0.0142 C,D -0.0248 B

9 RUS 0.0156 B 0.0851 B 0.0470 A

10 NCL -0.0990 B,C,D -0.0761 C,D,E -0.0328 B

11 ENN -0.0507 B,C -0.2784 E -0.0580 B,C

12 CNN 0.1473 A 0.2807 A 0.0836 A

13 5ENN -0.0387 B,C -0.2580 E -0.0786 B,C

UNIVERSITY OF IBADAN LIBRARY

179

Table 4.35: Summary of performance gain/loss on performance of the scheme compared to the RAW DATA in percentage

SCHEMES SMOTE 5ENN CNN ENN NCL RUS SMOTE SMOTE SMOTE SMOTE SMOTE SMOTE SMOTE

DATASETS 300ENN 300 ENN NCL RUS 300NCL 300RUS

DM 27 4 -15 5 10 -2 6 11 19 12 9 16 12

SSS Result 28 26 -28 28 8 -9 1 4 28 8 -3 9 2

CM 18 8 -8 6 3 5 2 10 9 5 16 14 22

AVERAGE 24 13 -17 13 7 -2 3 8 19 8 7 13 12

UNIVERSITY OF IBADAN LIBRARY

180

Figure 4.28: Chart showing the summary of performance gain/loss on performance of the scheme compared to the RAW DATA in percentage

-40 -30 -20 -10 0 10 20 30 40

DM SSS Result CM AVERAGE

UNIVERSITY OF IBADAN LIBRARY

181