Interpretability has become a key objective in the research, development and implementation of machine learning algorithms. However, existing notions of interpretability may not be conducive to how meaning emerges in algorithmic systems that employ ML algorithms. In this provocation, we suggest that hermeneutic analysis can be used to probe assumptions in interpretability. First, we propose three levels of interpretability that may be analyzed: formality, achievability, and linearity. Second, we discuss how the three levels have surfaced in prior work, in which we conducted an explicitation interview with a developer to understand decision-making in an algorithmic system implementation. Third, we suggest that design practice may be needed to move beyond analytic deconstruction, and showcase two design projects that exemplify possible strategies. In concluding, we suggest how the proposed approach may be taken up in future work and point to research avenues.