Springe direkt zu Inhalt

Explainable Recommender Systems in Wikidata

Requirements

BSc.:

  • Required: Experience in web development (html/javascript, python)
  • Preferred: Completion of lecture on "User-Centered Design" course and seminar on “Interactive Intelligent Systems”

MSc.:

  • Required: Experience in web development (html/javascript, python)
  • Desirable: Experience in statistics / data analysis
  • Preferred: Completion of the lecture on "User-Centered Design", the seminar on “Interactive Intelligent Systems” and the lecture on "Wissenschaftliches Arbeiten in der Informatik"
Discipline
Recommender Systems, Explanatory User Interfaces
Degree
Bachelor of Science (B.Sc.) or Master of Science (M.Sc.)

Contents

Context

Wikidata [1] serves as the central database for structured data for its sister projects, such as Wikipedia or Wiktionary, and beyond. Similar to Wikipedia, Wikidata is organised in pages which correspond to items. These items have a unique identifier (e.g. Q30); they are described by terms and statements. Statements describe items by their characteristics and in their simplest form consisting of a property and at least one value (e.g. a string, time, co-ordinates or another item). This property-value pair is called a claim; a statement can consist of various claims. One goal of Wikidata’s community is to improve the data quality and knowledge diversity of these statements [2]. Thus, members of the Wikidata community have developed recommender systems to support them in this process. Examples are the Property Suggester [3], Recoin [4], or Snoopy [5].

Problem

These approaches have been compared by Zangerle et al. [6]. The results suggest that two aspects are especially important for the quality of the recommendations: first, the incorporation of classifying properties and the use of contextual information.

However, these insights focus on the technical realization of a recommender system. User studies are rare. This is surprising, since a recommender system should represent the community’s understanding of the needed properties (aka statements). At the same time, understanding the reasoning behind the provided recommendations might help Wikidata’s editors to make more valuable contributions, since the editor can evaluate its suitability.

Objectives

The goal of this thesis is to solve this situation by providing explanations. Explanations are used to clarify certain algorithmic decisions and make these decisions understandable to the user [7, 8, 9]. Thus, explanations can be provided by the recommendation system, to make the inner working of the system transparent to the user. At the same time, explanations can be requested by the user to resolve misunderstanding. Explanations serve different information needs which need to be determined. These needs depend on the experience of the Wikidata editor. Thus, by building a simple user model (based on past edits of the user), the provision of the explanations should be dependent on the experience of the user on Wikidata.

Possible Procedure

Focus of B.Sc. Thesis:

  • Contextual research into recommender systems in Wikidata [6]
  • Overview on user-centered explanations based on provided articles [7,8,9]
  • Select one perspective and derive design rationales
  • Iterative, user-centered development of a prototype for providing explanations in the context of the Property Suggester
  • User testing (e.g. think-aloud testing)

Focus of M.Sc. Thesis:

  • Systematic literature review on requirements for user-centered explanations drawing from the recommender systems and the machine learning literature [7,8,9]
  • Contextual research into Wikidata’s recommender
      • Quantitative data analysis of usage statistics for the Property Suggester
      • Qualitative analysis of community discussions (talk-pages, mailing lists) for the Property Suggester and interviews with the community
  • Concept development of requirements for an explainable recommender system
  • Iterative prototype development
  • Development and conduction of an experimental framework for prototype evaluation
  • Analysis, evaluation and discussion of results

References

[1] https://www.wikidata.org/wiki/Wikidata:Main_Page

[2] Müller-Birn, C.; Karran, B.; Janette Lehmann, J. and Luczak-Rösch, M. (2015): Peer-production System or Collaborative Ontology Engineering Effort: What is Wikidata? In: Proceedings of the 11th International Symposium on Open Collaboration (OpenSym ’15), 20:1–20:10. https://doi.org/10.1145/2788993.2789836

[3] https://github.com/wikimedia/wikibase-property-suggester-scripts

[4] https://www.wikidata.org/wiki/Wikidata:Recoin

[5] Zangerle, E.; Gassler, W. and Specht, G. (2010): Recommending Structure in Collaborative Semistructured Information Systems. In: RecSys ’10: Proceedings of the third ACM conference on Recommender systems. Barcelona, Spain: ACM, pp. 141–145. https://doi.org/10.1145/1864708.1864762

[6] Zangerle, E.; Gassler, W.; Pichl, M.; Steinhauser, S. and Specht, G. (2016): An Empirical Evaluation of Property Recommender Systems for Wikidata and Collaborative Knowledge Bases. In: Proc. of the 12th International Symposium on Open Collaboration (OpenSym ’16). New York, USA: ACM, Article 18, pp. 1–8. https://doi.org/10.1145/2957792.2957804

[7] Herlocker, J. L.; Konstan, J. A. and Riedl, J. (2000): Explaining collaborative filtering recommendations. Presented at the the 2000 ACM conference, New York, USA: ACM, pp. 241–250. https://doi.org/10.1145/358916.358995

[8] Ribera, M. and Lapedriza, À. (2019): Can we do better explanations? A proposal of user-centered explainable AI. IUI Workshops. http://ceur-ws.org/Vol-2327/IUI19WS-ExSS2019-12.pdf

[9] Liao, Q. V.; Gruen, D. and Miller, S. (2020): Questioning the AI: Informing Design Practices for Explainable AI User Experiences. https://arxiv.org/abs/2001.02478

Further literature is being provided for proposal writing process.