Replication of a controlled experiment on design patterns (Bachelorarbeit)

This work has two goals:

• performing the 1997 experiment described in Patmain once again
• preparing an up-to-date experiment package that makes it easy for others to perform additional replications

Such additional replications will be called for in the context of the RESER workshop (Replication in Software Engineering Research) for 2011 and beyond.

This thesis is performed by Martin Liesenberg and supervised by Lutz Prechelt.

---

Task

1. Review the starting point: The Patmain article and the old experiment package.
2. Translate the original experiment programs from the package from C++ to Java. The new programs should be adequate Java programs (with no C++ feel) and must maintain the proper design differences between the Pattern and Alternative versions.
3. Build a web portal that guides subjects through the experiment and that performs basic data gathering (time-taking, result program upload).
4. Pilot-test the experiment and correct its setup as needed
5. Perform the experiment with (as many as possible) volunteer participants from the Softwaretechnik-Projekt
6. Publish the replication package for others to use (e.g. set up the portal to be publicly visible, create some documentation)
7. Evaluate the experiment replication and compare with the previous results, including those of the previous replication published in 2002.
8. Extend the published package by the R scripts created for the evaluation and some documentation.
9. Write bachelor thesis

Requirements for the portal

There are three roles:

1. Experimenter: A person who manages, runs, and probably evaluates one replication (or potentially several)
2. Participant: A person who takes part in one replication as a subject
3. Portal Administrator: The person who manages the portal and has access to the union of all data sets created by the replications.

Below, there may be some pairs of sometimes-equivalent terms: experiment/replication, participant/subject. Consistency would be better.

Use case "Status and open/close replication"

• 1. Experimenter logs in.
• 2. Portal shows status page: A table listing each of this experimenter's replications with name (a hyperlink to the participant start page), date opened, #subjects finished, #subjects active, last subject action timestamp, and experimenter action options:
  • "create subject numbers": creates 25 new subject numbers and displays a page with 25 entries to be printed out and cut into pieces ("participant chits") with scissors. Each chit contains the URL for this replication and one subject number. One chit is handed to each participants. Superfluous chits can be thrown away.
  • "show status": Show a table that shows the times (minutes) for each subtask for each participant (plus experiment name and subject number).
    Include (at the bottom) the still-active participants to show their progress.
  • "close experiment": mark the experiment as closed. No subject can submit any further data. This is to make sure the data retrieved by "retrieve data" is well-defined.
    If someone has closed the experiment prematurely, a new one can be opened instead. Remaining participants then need to be given new chits. Partial results of still-active participants will be mostly lost, though, so make sure close experiment is never called accidentally.
  • "retrieve data": Download a ZIP file containing the results. * 3. Experimenter can optionally input a not-previously used experiment name * 4. Portal opens a replication of this name and displays the updated status page

Better GUI: The "name" hyperlink leads to a details page that covers the same information and options as the table entry but in a verbose manner with plenty of explanation.

Subject number: is a random five-digit number (10000 to 99999). This guards against typos and acts as a kind of pseudo-password at the same time. Watch out not to create duplicates (not even across replications, this simplifies the evaluation).

Experiment name: should contain these characters only: a-z, A-Z, -, 0-9.

Results ZIP file:

• Contains a TAB-separated-value file with one entry per finished participant. The entry contains the subject number, group number, answers to the questionnaire, time (in seconds) for each step.
  • A header should declare what is what
  • Data refering to the pattern versus alternative versions of the same program goes into the same variable, but data for a different program never does.
  • The set of variables is fixed throughout all groups. This means there will be empty fields if not everybody works on all four programs.
  • Missing values (in particular inapplicable ones, due to group number) are noted as NA (R constant for "not available")
  • Absolute times for start and end of each participants should also be present, preferably in POSIXct format (an integer as per the Unix clock definition)
  • A web-client-side time zone would also be nice, to compute time-of-day. The name of this file is replicationname-finished.tsv.
• Contains another such file of the same structure for the participants that have not (yet) finished The name of this file is replicationname-unfinished.tsv.
• Contains one ZIP file per participant and program. The name of this file is subjectnumber-programname-PATorALT.zip. The contents are just as they were uploaded by the participant.

Use case "Participate"

• 1. Participant visits the link printed on her participant chit.
• 2. Portal validates that experiment is open and asks for subject number.
• 3. Participant inputs subject number as printed on her participant chit.
• 4. Portal validates that participant number is valid and fresh, opens a session, and starts guiding participant through the experiment.
• 5. Participant works as guided.
• 6. After last step, portal bids farewell and marks session and subject number as closed (counts participant as finished)

Alternatives:

• 1b. If session is still open, guidance continues at last previously unsubmitted step and marks step as interrupted. Time measurement continues throughout.
• 2b. Portal declares experiment closed

Guidance and groups:

• The subject number modulo 4 is the group number and determines through which programs the participant will be guided and in what order.

Guidance and time measurement method:

• At the beginning of working on each program, a ZIP file is offered for download that contains the sources in an Eclipse-ready format, along with instructions for how to set them up in Eclipse.
• Participants should be asked to not look at the code then but rather to call their first task. This will not always work. During evaluation, one should probably attempt to estimate a typical time that participants need for setting up the files and then use the measured setup time minus estimated setup time plus time-for-first-task for the first task. We could also insert a trivial additional first task to obtain a better measurement for the actual intended first task.

Guidance and data validation:

• At the end of working on this program, the participant will be asked to upload her work results, again as a ZIP file.
• The portal should make sure that all of the intended files are there somewhere in this ZIP and make sure that at least one of them was modified. This will guard against various kinds of mistakes.

Account management requirements

• Registration (username, clear name, email address) creates a user account
• Password reminder sends the password, given the email address (may apply to more than one account) or the username
• Log in requires username+password and creates a session (via a cookie) with timeout 4 days.

Admin requirements

Functions available to the portal administrator:

• List experimenters
• Take on the identity of any experimenter and act as if she was that other user.
• Status page for all replications of all experimenters at once
• Retrieve data for all replications of all experimenters at once. Just stuff all ZIP files into yet another ZIP file.
• Retrieve structured data for all replications of all experimenters at once. A ZIP file containing just the two .tsv files (as described above) in their global form.

Progress

• 2010-09-01: first contact
• 2010-09-02: Bachelorarbeit officially registered
• 2010-09-02: Boolean program translated
• 2010-09-03: Graphics and Communication programs translated
• 2010-09-04: Stockticker translated and first steps in Tapestry
• 2010-09-06: more steps in Tapestry, first steps in Hibernate, portal requirements analysis and definition
• 2010-09-08: first simple pages for the portal
• 2010-09-09: login pages for 'users' and experimenters
• 2010-09-10: decision on which programs to use for first experiment, based on that portal guidance implemented
• 2010-09-13: Hibernate integration into the portal
• 2010-09-14: experimenter - experiment interaction logic partly implemented
• 2010-09-15: redesign of the user login to simplify guidance through the experiment work flow, added possibility for experimenter to create users for experiments
• 2010-09-16: design of questionnaire - pre-experimemnt and post-task
• 2010-09-17: a lot of small changes to the underlying design most of which proved useless
• 2010-09-19: adjustments to the guidance to include questionnaires
• 2010-09-20: finally hibernate is working!
• 2010-09-21: fixed storing of experiment subject related information
• 2010-09-22: fixed time stamps setup
• 2010-09-23: layout added
• 2010-09-24: (perform experiment with participants from Softwaretechnik-Projekt) postponed due to last minute bug
• 2010-09-24: last minute bug fixed
• 2010-09-26: added real user management for experimenters(login logic, own experiments,etc.), page to edit experiments
• 2010-09-28: started to implement feedback - design issues and internal organization
• …
• …
• 2010-11-25: Submit bachelor thesis

Information for external users of the portal

see PatmainReplicationInfo

Comments