combine-is-the-third-outcome-of-an-adoption-test

# Register three outcomes, not two: replace, combine, discard A decision rule of the form *adopt the challenger only if it beats the incumbent on held-out data* is a good rule with a missing branch. It has two destinations, replace and discard, and it sends everything that is not better to the same place as everything that is worthless. Those are not the same thing. A method can be worse than the incumbent on every row you measured and still be right on rows the incumbent gets wrong. That is not an edge case. It is the normal situation for two methods that read the same input in structurally different ways, and it is the entire reason ensembling works. A comparison of two accuracies is a comparison of two margins over the same population; it says nothing about *which* rows each one owns, so a replace-only gate cannot see it. ## What the missing branch cost, once One run split its population in two and handled the halves with different machinery. On the harder half — 79 labelled rows, where a structural trick that worked elsewhere did not apply — it ran a challenger against its incumbent with a pre-registered rule behind each: | on those 79 rows | accuracy | |---|---| | incumbent: an absolute yes/no probability at a fixed threshold | 0.798 | | challenger: a forced choice over rivals the run manufactured | 0.646 | Discarded. The bar said *strictly greater, and more wins than losses*; the challenger lost 4 to 16 with an exact p of 0.012. That is a correct application of th...