I've seen it suggested that the barn wasn't on Gallifrey, because it's on a world with a blue sky (though The Five Doctors provides an obvious counterexample). Presumably, if it was somewhere else, the Doctor would have been there on the equivalent of a field trip.