V+Ü Empirical Evaluation in Informatics (Empirische Bewertung in der Informatik) SS 2015

This is the homepage of the lecture Empirical Evaluation in Informatics (Vorlesung "Empirische Bewertung in der Informatik") and its corresponding tutorial (Übung).


As an engineering discipline, Informatics is constantly developing new artifacts such as methods, languages/notations or concrete software systems. In most cases, the functional efficiency and effectiveness of these solutions for the intended purpose is not obvious -- especially not in comparison to other already existing solutions for the same or similar purpose.

For this reason, methods for evaluating the efficacy of these solutions must be a routine part of Informatics -- a fact which unfortunately only slowly has become recognized. Evaluation is needed by those who create new solutions (that is in research and development), but also by the users, as these need to evaluate the expected efficacy specifically for their situation. These evaluations need to be empirical (that is based on observation), because the problems are nearly always too complicated for an analytical (that is a purely thought-based) approach.

This lecture presents the most important empirical evaluation methods and explains where these have been used (using examples) and should be used, how to use them and what to consider when doing so.



Requirements/target group, classification, credit points etc.

see entry in the KVV course syllabus


  • KVV (course syllabus): All participants need to have registered in the KVV.

  • For the tutorials every participant needs to have registered in the KVV.
    • Subscribe to »Empirische Bewertung in der Informatik S15«



Examination modalities

Necessary criteria for obtaining the credit points:


Most of the linked documents and videos can only be accessed from the FU Berlin network (externally you receive a 403/Forbidden: "You don't have permission to access ...", use a VPN connection in this case).

Lecture topics

The lecture divides into three sections:

The individual lectures:
  1. (13.4.2015) Introduction - The role of empiricism:
    • Term "empirical evaluation"; theory, construction, empiricism; status of empiricism in Informatics
    • Hypothetical examples of use
    • quality criteria: reliability, relevance
    • Note: scale types
  2. (20.4.2015) The scientific method: (Video 2014-04)
    • Science and methods for gaining insights; classification of Informatics
    • The scientific method; variables, hypotheses, control; internal and external validity; validity, reliability, relevance
  3. (27.4.2015) How to lie with statistics: (Video 2014-05)
    • When looking at somebody else's conclusions from data: What is actually meant? What specifically? How can they know it? What is not said?
    • Does the measurement distort the meaning? Is the sample biased?, etc.
    • Material: book on the topic; Study on alternative ink; article with arguments against hypothesis testing: "The earth is round (p < 0.05)".

  4. (04.5.2015) Empirical approach: (Video 2015-05)
    • steps: formulate aim and question; select method and design study; create study situation; collect data; evaluate findings; interpret results.
    • example: N-version programming (article, reply to the criticisms against it)
  5. (11.5.2015) Survey: (Video 2014-05)
    • example: relevance of different topics in Informatics education (article)
    • method: selection of aims; selection of group to be interviewed; design and validation of the questionnaire; execution of the survey; evaluation; interpretation
  6. (18.5.2015) Controlled experiment: (Video 2014-05)
    • example 1: flow charts versus pseudo-code (article, criticized prior work)
    • method: control and constancy; problems with reaching constancy; techniques for reaching constancy
    • example 2: use of design pattern documentation (article)
  7. (1.6.2015) Quasi experiment: (Video 2015-06)
    • example 1: comparison of 7 programming languages (article, detailed technical report)
    • method: like controlled experiment, but with incomplete control (mostly: no randomization)
    • example 2: influence of work place conditions on productivity (article)
  8. (8.6.2015) Benchmarking: (Video 2014-06)
    • example 1: SPEC CPU2000 (article)
    • Benchmark = measurement + task + comparison; problems (costs, task selection, overfitting); quality characteristics (accessibility, effort, clarity, portability, scalability, relevance) (article)
    • example 2: TREC (article)

  9. (15.6.2015) Data analysis - basic terminology: (Video 2014-06)
  10. (22.6.2015) Data analysis - techniques: (Slide video 2015-06, until slide 20 only, then the recorder went on strike)
    • Samples and populations; average value; variability; comparison of samples: significance test, confidence interval; bootstrap; relations between variables: plots, linear models, correlations, local models (loess)
    • Article: "A tour through the visualization zoo"

  11. (29.6.2015) Case study: (Video 2014-07)
    • example 1: Familiarization with a software team (article)
    • method: characteristics of case studies; what is the 'case'?; use of many data types; triangulation; validity dimensions
    • example 2: An unconventional methods for für requirements inspections (article)
  12. (06.7.2015) Other methods: (Video 2014-07)
    • The method landscape; simulation; software archeology (studies on the basis of existing data); literature study;
    • example simulation: scaling of P2P file sharing (article)
    • example software archeology: code decline (article)
    • example literature study: a model of the effectiveness of reviews (article)

  13. (13.7.2015) Summary and advice:
    • Role of empiricism; quality criteria; generic method; advantages and disadvantages of the methods; practical advice (for data analysis; for conclusion-drawing; for final presentation); outlook

Aims of the tutorials

Practice sheets

(These links will be added continuously as the course proceeds.)

Changes over the years



Should you have comments or suggestions concerning this page, you may add them here (possibly with date and name):