RQ1: How often does TestI predict file changes associated with a task?

TestI helps to predict changes in controllers and MVC slices, but not for all tasks

TestI recall for CF configuration (content filtered by controllers) and for NF configuration (no content filtering).

TestI precision for CF configuration (content filtered by controllers) and for NF configuration (no content filtering).

Our routing mechanism does not strongly compromise the predictive ability of TestI

Precision and recall measures of NF configuration (no content filtering) when using Rails routes and inferred routes.

Precision and recall measures of CF configuration (content filtered by controllers) when using Rails routes and inferred routes.

Wilcoxon Signed-Rank test applied to precision and recall measures of TestI when using Rails routes and inferred routes

Hypothesis pvalue r Inferred routes precision = Rails routes precision 0.3426 0.11 Inferred routes recall = Rails routes recall 0.0039 0.34 Inferred routes controller precision = Rails routes controller precision 0.6044 0.06 Inferred routes controller recall = Rails routes controller recall 0.0142 0.28

TestI has higher predictive power for tasks with higher test coverage

Correlation between test coverage per task and TestI predictive ability

Correlation rho pvalue Precision & Coverage 0.82 2.8e-19 Recall & Coverage 0.47 2.3e-05 CF-Precision & CF-Coverage 0.78 2.7e-16 CF-Recall & CF-Coverage 0.50 5.1e-06

Discarding test precondition and postcondition compromises TestI recall with slight improvement in precision

Precision and recall measures of WF configuration (content filtered by when steps) and for NF configuration (no content filtering).

Discarding changed tests compromises TestI recall with slight improvement in precision

Precision and recall measures of TestI computed by considering only created tests (CT) and for NF configuration (no content filtering).

Wilcoxon Signed-Rank test applied to precision and recall measures of the main configurations of TestI

Hypothesis pvalue r TestI-NF Precision = TestI-CT Precision 2.0e-14 0.36 TestI-NF Recall = TestI-CT Recall 1.9e-31 0.54 TestI-NF Precision = TestI-CF Precision 1.8e-35 0.58 TestI-NF Recall = TestI-CF Recall 3.8e-44 0.65 TestI-NF Precision = TestI-WF Precision 1.5e-05 0.20 TestI-NF Recall = TestI-WF Recall 5.7e-58 0.75

TestI performs better as a predictor of changes in controllers and MVC slices

Precision values for the 8 possible TestI configurations: NF (no filter), CT (created-tests), CF (controller-filter), CTCF (created-tests-controller-filter), WF (when-filter), WCF (when-controller-filter), CTWF (created-tests-when-filter), CTWCF (created-tests-when-controller-filter)

Recall values for the 8 possible TestI configurations: NF (no filter), CT (created-tests), CF (controller-filter), CTCF (created-tests-controller-filter), WF (when-filter), WCF (when-controller-filter), CTWF (created-tests-when-filter), CTWCF (created-tests-when-controller-filter)

Wilcoxon Signed-Rank test combined with the Bonferroni correction method applied to precision and recall measures for all configurations of TestI

Hypothesis pvalue r TestI-NF Precision = TestI-CT Precision 5.5e-13 0.356 TestI-NF Precision = TestI-CF Precision 5.0e-34 0.578 TestI-NF Precision = TestI-CTCF Precision 1.0e-33 0.575 TestI-NF Precision = TestI-WF Precision 4.1e-04 0.201 TestI-NF Precision = TestI-CTWF Precision 9.3e-10 0.308 TestI-NF Precision = TestI-WCF Precision 2.4e-36 0.597 TestI-NF Precision = TestI-CTWCF Precision 3.2e-32 0.562 TestI-CT Precision = TestI-CF Precision 8.0e-20 0.440 TestI-CT Precision = TestI-CTCF Precision 2.6e-28 0.527 TestI-CT Precision = TestI-WF Precision 3.4e-02 0.150 TestI-CT Precision = TestI-CTWF Precision 3.3e-01 0.117 TestI-CT Precision = TestI-WCF Precision 2.2e-27 0.518 TestI-CT Precision = TestI-CTWCF Precision 2.6e-27 0.517 TestI-CF Precision = TestI-CTCF Precision 9.9e-05 0.216 TestI-CF Precision = TestI-WF Precision 1.1e-22 0.471 TestI-CF Precision = TestI-CTWF Precision 3.8e-15 0.384 TestI-CF Precision = TestI-WCF Precision 4.9e-06 0.243 TestI-CF Precision = TestI-CTWCF Precision 8.4e-06 0.238 TestI-CTCF Precision = TestI-WF Precision 2.6e-28 0.527 TestI-CTCF Precision = TestI-CTWF Precision 1.5e-22 0.470 TestI-CTCF Precision = TestI-WCF Precision 1.0e+00 0.031 TestI-CTCF Precision = TestI-CTWCF Precision 4.8e-01 0.111 TestI-WF Precision = TestI-CTWF Precision 9.4e-08 0.275 TestI-WF Precision = TestI-WCF Precision 2.1e-35 0.589 TestI-WF Precision = TestI-CTWCF Precision 1.2e-31 0.557 TestI-CTWF Precision = TestI-WCF Precision 1.3e-25 0.501 TestI-CTWF Precision = TestI-CTWCF Precision 4.1e-27 0.515 TestI-WCF Precision = TestI-CTWCF Precision 5.7e-01 0.108 TestI-NF Recall = TestI-CT Recall 5.3e-30 0.542 TestI-NF Recall = TestI-CF Recall 1.1e-42 0.648 TestI-NF Recall = TestI-CTCF Recall 4.5e-23 0.475 TestI-NF Recall = TestI-WF Recall 1.6e-56 0.746 TestI-NF Recall = TestI-CTWF Recall 5.6e-59 0.762 TestI-NF Recall = TestI-WCF Recall 2.4e-15 0.387 TestI-NF Recall = TestI-CTWCF Recall 2.2e-04 0.208 TestI-CT Recall = TestI-CF Recall 2.0e-51 0.711 TestI-CT Recall = TestI-CTCF Recall 5.7e-42 0.642 TestI-CT Recall = TestI-WF Recall 3.3e-15 0.385 TestI-CT Recall = TestI-CTWF Recall 1.8e-50 0.705 TestI-CT Recall = TestI-WCF Recall 1.1e-26 0.511 TestI-CT Recall = TestI-CTWCF Recall 6.0e-15 0.382 TestI-CF Recall = TestI-CTCF Recall 3.4e-16 0.397 TestI-CF Recall = TestI-WF Recall 1.4e-59 0.766 TestI-CF Recall = TestI-CTWF Recall 5.1e-63 0.788 TestI-CF Recall = TestI-WCF Recall 1.3e-22 0.470 TestI-CF Recall = TestI-CTWCF Recall 1.1e-30 0.548 TestI-CTCF Recall = TestI-WF Recall 2.2e-46 0.675 TestI-CTCF Recall = TestI-CTWF Recall 9.7e-58 0.754 TestI-CTCF Recall = TestI-WCF Recall 1.7e-02 0.160 TestI-CTCF Recall = TestI-CTWCF Recall 1.8e-21 0.458 TestI-WF Recall = TestI-CTWF Recall 3.7e-24 0.486 TestI-WF Recall = TestI-WCF Recall 3.9e-49 0.695 TestI-WF Recall = TestI-CTWCF Recall 7.1e-29 0.532 TestI-CTWF Recall = TestI-WCF Recall 3.9e-55 0.737 TestI-CTWF Recall = TestI-CTWCF Recall 2.1e-46 0.675 TestI-WCF Recall = TestI-CTWCF Recall 1.0e-13 0.365