we just posted a preprint in which we report and closely analyse differences in DNN representations that result from a minimal intervention: changing the random seed for the weights prior training. While the resulting network performances are more or less identical, the networks exhibit drastic differences in their internal representations (especially in intermediate and higher DNN layers).
The tool we use for the analyses can also be used to investigate how representations change across layers, different networks, and learning trajectories. Our background is in computational neuroscience, but we hope that the technique and results will be of use to ML researchers, too.
I have written a short summary on twitter: https://twitter.com/TimKietzmann/status/1215620270679044096