The research approach of the SE group

09 Aug 2023 - 17:03 | Version 11 | UnknownUser

Describes which research methods we use under which circumstances and why.

Research doctrine

Software engineering methodology as a research field, like all kinds of engineering, is fundamentally constructive: It intends to "show the way" how to solve some class of problems reliably, effectively, and efficiently.

Most software engineering research groups interpret this fact in such a way, that they work in 'construction mode' most of the time: They tirelessly invent new methods, new notations, and new tools for supporting them.

Looking at the results of this research, however, one finds that most of it has more or less failed: The vast majority of these methods, notations, and tools is either no improvement at all (compared to the previous state of the art) or its scope of application is so embarrassingly narrow that hardly anybody has even any opportunity of picking them up.

In our view, helpful contributions to software engineering are difficult. This is why we approach any topic using empirical research at first: Before one can usefully construct, one needs to understand the situation. The 'construction-only' mode of SE research implicitly claims that this understanding can be gained purely analytically or intuitively.

We do not believe that this is true. Since software engineering always involves complex action and interaction of human beings, psychological and sociological insights are required for understanding the situation and these insights can only be gained empirically.

Therefore, our research approach always starts from empirical investigations. Depending on the topic under investigation, we use quantitative methods or qualitative methods or both. We typically pick under-researched topics and therefore start with qualitative methods to gain a thorough and valid understanding. This can then be followed by constructive research and/or quantitative research, but we rarely get to the latter.

We view each individual study as a contribution to answering some more general question and judge the contribution by its credibility and its relevance. Only once we believe to have gained sufficient understanding of how a situation really is will we start attacking a software engineering methodology problem by means of construction.

And as an overarching principle, we strongly believe in common sense as a useful guide for research as well as for publishing about research.

Quantitative empirical research

Quantitative research focuses on observations that can be expressed in numbers. The underlying concept (usually called a 'variable' by quantitative researchers) may be subjective (i.e. requiring interpretation or judgement to convert it to a number) or objective (measurable in some automated way not requiring human judgement).

This section shortly describes the main methods of quantitative research and when we use them. There are other methods (many of which can be considered variants of those given here), but they are of less importance to our work and were left out for brevity's sake.

For an introduction to quantitative research methods for software engineering, see the materials for the course "Empirische Methoden in der Informatik".

Controlled experiment

Basic idea: Vary only one relevant parameter and keep everything else constant. When people are involved, this can only be done by means of averaging out the differences between people by considering a group of them at once (rather than a single individual).
Strengths: If done successfully, a controlled experiment establishes a cause-effect relationship: The changes observed when varying the parameter must be caused by the parameter change, as nothing else has changed.
Weaknesses: (1) A single experiment can only prove that the cause-effect relationship exists in the given setting. It cannot make sure the finding generalizes to other settings. (2) The need for not-too-small groups of highly skilled participants. Taken together, these two problems make controlled experiments a very expensive research method.
When to use: (1) To prove some hotly debated cause-effect relationship does indeed (at least sometimes) exist. (2) To measure the size of an effect.

Survey

Basic idea: Ask many people for the same pieces of information in order to obtain an overview of an objective situation or of subjective evaluations, preferences, etc.
Strengths: (1) The only method for measuring attitudes. (2) Allows for gathering a good amount of data of significant breadth in a short time at low cost.
Weaknesses: (1) It is usually very difficult to obtain answers from a representative sample of whatever population would be of interest. This restricts generalization. (2) It is difficult (often impossible) to assess the correctness of the answers obtained if the questions concern facts rather than attitudes.
When to use: (1) When breadth of information is more important than precision. (2) When subjective information is sought (perceptions, attitudes).

Qualitative empirical research

Qualitative research uses the full spectrum of possible observations, not confining itself to observations that can be expressed quantiatively (although these may be used as well).

This section shortly describes the main methods of qualitative research that are relevant for our work and characterizes when we use them. Many other qualitative methods exist, but they are of less importance to our work and were left out here for brevity's sake.

Grounded Theory Methodology (GTM)

Basic idea: Inductively structure your observations into a theory that explains what is going on. While doing so, incrementally seek those observations that contribute the most to the process.
Strengths: Can yield very rich and still solid results.
Weaknesses: The researcher plays a very prominent role, hence it is difficult to make GTM results reproducible.
When to use: When only shallow understanding of some phenomenon is available and deeper understanding is required.

See Wikipedia:Grounded_Theory.

Case study

Basic idea: Strongly focus on a small number of units of observation (the cases, perhaps just one) and characterize them by making use of the broadest possible set of observation sources, typically including longitudinal observation. Unify these sources so as to create a coherent description. One typical result is hypotheses to be investigated in subsequent research.
Strengths: Can be applied under almost all circumstances (few prerequisites).
Weaknesses: The results often have limited value unless some kind of generalizability can be shown.
When to use: When description (rather than explanation) is sought. The research question typically asks "Why" or "How" something is done.

Constructive research

Constructive research means providing new things rather than talking about things that already exist.

The typical manner of constructive research in software engineering is (1) devising methods and (2) building software tools for supporting some method. If and when we do constructive research, it will typically be in a context in which we performed qualitative research before and have now found an interesting approach for possible improvement. We then usually take an Wikipedia:Action_Research approach.

Interestingly, the border between analytic research and constructive research is not as clear as it seems at first, because theories, the most desired output of analytic research, are constructions, too. Conversely, any output of constructive research implicitly contains some amount of theory (embedded in the rationale why the construction was created as it was) and hence has some component of analytic research as well.

Publishing doctrine

We also do our best to satisfy the following principles when we publish our research.

Substance

We do not believe in the leas publishable unit style of scientific publication. Rather, we attempt to form the most coherent whole we can when we design a publication. The goal is to maximize the insight gained per reading effort.

We believe that this is "the right thing" to do from the point of view of the scientific process and hope that the scientific community will align its reward systems appropriately for each of our members when this member explains what we did and why.

Clarity

We try to impress our readers by clarity, rather than by complexity. In particular, this means that when it comes to presenting quantitative data or results of statistical analyses, we tend to strongly prefer graphical plots over tables, significance test p-values, and other unillustrative formats.

We attempt to write prose that is simple and straightforward, avoids excessive jargon, and intelligently bends conventions whenever those threaten to get in the way of readability.

Again, our goal is to maximize the insight gained per reading effort.

Transparency and detail

We want our results to be highly credible. We think that all too many publications in software engineering lack credibility. Often this is because they present a proposal but provide no empirical evaluation of its properties. In other cases, an evaluation is present, but too many skeptical questions remain unanswered when reading its description, which raises a lot of suspicion.

For this reason (and a number of other reasons as well, see below) we believe that a high degree of transparency is important for good research. One main method for achieving such transparency is reporting on the research on a fine level of detail.

This is why we often provide the following two kinds of additional material, although preparing such material requires a lot of additional effort:

Technical reports: A technical report can describe a study, its evaluation, and the results of this evaluation without space constraints. This medium thus eliminates almost all excuses for being fuzzy or vague about what was done, how it was done, and what was the result. In particular, it allows for describing the setup and context of the study in full detail, which can be very important for meta analyses as well as for many kinds of surveys. It also allows to present all those elements of a study's results that were deemed too uninteresting to be included in journal or conference articles. This helps avoid publication bias (the "file-drawer effect").
Data packages: A data package contains raw materials and data used for an empirical study (such as test programs, input data etc.) or produced during the study (such as raw or preprocessed observation data, evaluation scripts etc.). Publishing these data both maximizes transparency and allows for reuse in further studies.

See our publications page for a list of technical reports and a list of data packages.

Comments

SWTIDSR