Department of Mathematics and Computer Science

Context

Machine Learning has become indispensable in many application areas, such as in medicine, traffic control but also in research. One area that often receives little attention is that of the humanities. While many text-based humanities disciplines have long adapted computational approaches and are considered the foundation of digital humanities (Jockers 2013), ''digital art history'' is amongst the newer developments in the field. With the ever growing numbers of digitized visual sources in the form of images from museum collections, image archives, or libraries, the potential of digital technologies and interactive systems that support the retrieval, processing, and analysis of relevant resources for art historical inquiry is eminent. Computer vision (CV) techniques can especially support art historical research and are, thus, gaining more and more attention in this field (e.g., Seguin 2018, Bell and Ommer 2018).

Problem

One promising example for an application that integrates CV into a ready-to-use tool is imgs.ai - a deep visual search engine. However, the user interface does not support potential users (subject-matter experts from art history) in interpreting the results given by the system and how the parameters or the choice of embeddings actually affect the search result. However, such systems that should be comprehensible without a deeper technical understanding. In this context, Explainable AI (XAI) algorithms provide a promising approach to tackle this challenge by aiming to support human understanding and sense-making of the results by providing explanations. These XAI algorithms can be categorized into explanation methods (e.g. local, global, example-based or counterfactual) (Liao et al. 2020); therefore, the scope and type of the explanations provided differ as do their content and representation. The overall goal of XAI algorithms is to provide interpretability and transparency (e.g., Carvalho et al. 2019), since research has shown that the degree to which the ML results can be interpreted by explanations can enhance user understanding, which, in turn, leads to more trust (e.g., Stumpf et al. 2007). Therefore, explanations need to be carefully designed in so-called Explanation User Interfaces for their context of use. Ideally, researchers should be enabled to reflect and critically assess the results and outputs of the system in such a way that it enriches and informs their research process.

Objectives

The objective of this thesis is to delineate and map the design space for possible XAI algorithms that can be used for designing an Explanation User Interfaces for the Computer Vision-based search interface of imgs.ai. By adapting the proposed methodology by Liao et al. (2021) suitable Explanation User Interfaces should be proposed and empirically evaluated. At the end of the thesis, a critical reflection on the design process should be done.

Procedure

• State of the Art analysis of XAI algorithm in the field of CV
• Evaluation of existing algorithms regarding suitability for the use case
• Collect and analyze requirements from use case
• Map requirements to existing XAI algorithms
• Prototypically implement at least three user interfaces that provide different approaches for providing explanations
• Evaluate these prototypes by Mechanical Turk
• Reflect and discuss the implications of each implemented method with regard to their suitability for non-technical expert users

References

Bell, Peter and Björn Ommer. “Computer Vision und Kunstgeschichte — Dialog zweier Bildwissenschaften”. In: Computing Art Reader. Einführung in die digitale Kunstgeschichte. Heidelberg: arthistoricum.net, (Computing in Art and Architecture, Vol. 1) (2018). https://hci.iwr.uni-heidelberg.de/sites/default/files/publications/files/1523349512/413-17-83318-2-10-20181210.pdf

Carvalho, Diogo V, Eduardo M Pereira, und Jaime S Cardoso. „Machine Learning Interpretability: A Survey on Methods and Metrics“. Electronics 8, Nr. 8 (2019): 832–34. https://doi.org/10.3390/electronics8080832.

Jockers, Matthew Lee. "Macroanalysis: digital methods and literary history". University of Illinois Press (2013).

Liao, Q Vera, Daniel Gruen, und Sarah Miller. „Questioning the AI: Informing Design Practices for Explainable AI User Experiences“ (2020). https://doi.org/10.1145/3313831.3376590.

Liao, Q. Vera, Milena Pribić, Jaesik Han, Sarah Miller, und Daby Sow. „Question-Driven Design Process for Explainable AI User Experiences“. arXiv:2104.03483 [cs] (2021). http://arxiv.org/abs/2104.03483.

Benoît Laurent Auguste Seguin, “Making large art historical photo archives searchable” (2018). https://infoscience.epfl.ch/record/261212

Stumpf, Simone, Vidya Rajaram, Lida Li, Margaret Burnett, Thomas Dietterich, Erin Sullivan, Russell Drummond, und Jonathan Herlocker. „Toward harnessing user feedback for machine learning“. In Proceedings of the 12th international conference on Intelligent user interfaces. IUI ’07. New York, NY, USA: Association for Computing Machinery (2007). https://doi.org/10.1145/1216295.1216316.

Web Sources

• http://imgs.ai/