I know it’s not an interesting topic compared to…well…almost everything else in machine learning, but there’s a 2015 NeurIPS paper (https://papers.nips.cc/paper/5656-hidden-technical-debt-in-machine-learning-systems.pdf ) on the subject that’s been making the rounds again in terms of papers from this year that cite it. There was this reaction piece that was posted on HN the other day that does a lot better in terms of recommending specific tools (https://news.ycombinator.com/item?id=23266575) but bar is pretty low in this sub-field (compared to GAN or RL research, for example). I know this has been a huge problem at EVERY company I’ve been at (both small and large, especially the large).
Has anyone seen anything good in terms of automated tools for dealing with technical debt in machine learning? Are there code smells or anti-patterns that aren’t mentioned in either of these articles (that are unique to ML and not usually associated with typical Software Engineering)? This seems like one of those topics that even experienced ML engineers and researchers struggle with in terms of practical implementation (theoretical fixes are easy by comparison).