Sorry if this has already been discussed, but I've been reading some deep learning papers and it seems like a lot of the choice of architecture is wishy-washy stuff that we just have to “accept” for some reason. I know that explaining a deep network is almost impossible, but that makes it really difficult to decide whether a given network outperforms another, especially when considering artificial data sets, because:
In short, everything seems like it's a question of throwing more layers at the problem until you reach some sort of solution that you can claim is “better” than others', except since nothing is standardized, even for a particular type of problem, it can result in widely varying results and reproducibility.
How do you manage this? How do you not take it with a tablespoon of salt? I'm rather new to this, which is probably why I'm unable to see past this.