The true "test" dataset for a generalised task

The standard ML pipeline is well known: you want to teach a task to an algorithm. Take your data which encodes that task, split it into training, validation, and test sets. Train the ML on the training set, using the validation set to tune hyper-parameters.

Then, finally, check that the algorithm has learnt the task by evaluating it on the test set. This will ensure that it hasn’t overfitted to the training data.

Particularly important: the test set must come from the same distribution as the training set, or else its test is useless.

The case for very distinct test sets

That three-way split works very well – as long as the task is “process any data drawn from a similar distribution in a similar way”.

For a more general task, such as “describe how a program would run, from its code”, one should draw the test dataset from a very different part of the distribution.

For example, train and validation could contain Python code, and the test might be entirely in Java. Then, if the test is a success, we would be confident that the task really is “describe how a program would run, from its code” and not “describe how a very specific subset of Python programs would run, from their code”.

To reach that objective, the train and validation sets could be diversified themselves – maybe train is Python and C++, validation is Python, C++, and Lisp. But in any case, the test set should be as out of distribution as possible from the train and validation set, while still in-distribution for the task. That way a successful outcome is much stronger evidence of task proficiency.

And what is “in-distribution to the task”? Well, that’s whatever the programmer wants it to be. If the task is truly “describe how a program would run, from its code” then Java is certainly in-distribution. If the desired task is merely “describe how a Python program would run, from its code”, then the test set must be in Python – but it would be ideal if it was a completely different style of Python code,…