You are here: SE » ThesesHome » ThesisDPPXIII

Verbesserte Präsenz durch Screensharing für ein Werkzeug zur verteilten Paarprogrammierung (ThesisDPPXIII)

Improved presence through screen sharing for a distributed pair programming tool (Bachelorarbeit)

von Stephan Lau, betreut von Karl Beecher und Stephan Salinger

Antrittsvortrag / Preparation presentation

is here

Ziele

Saubere und schnelle Implementierung von Screensharing in Saros wink

Planung / Approximate timeline

  • registration: 17.11.2009
  • delivery: 9.2.2010
  • ill: 21.12.09 - 8.1.10 (19 days)

  1. Week 1-2 9. - 20.11
    • base architecture and requirements for screensharing
    • start to implement
  2. Week 3-4 23.11 - 4.12
    • start integration in saros
      • UI
      • bytestream via XMPP
    • only passive features first
  3. Week 5-6 7.12 - 18.12
    • additional features
      • active screensharing
  4. Week 7-8 21.12 - 1.1.2010
    • prepare feature for release
    • alternative encoders
    • test/optimize
  5. Week 9-10 4. - 15.1
    • deal with scientific questions, theoretical preparation for thesis
      • evaluate screensharing
        • what has been done?
        • how is screensharing accepted (by saros' users)?
        • how (or does) screensharing improves saros?
        • are there remaining problems?
      • screensharing <→ collaboration awareness
        • how are they related?
        • can they be combined?
  6. Week 11-12 18.1 - 5.2.
      • final work on thesis

Anforderungen / Requirements

  • functional
    • client (consumer)
      1. see screen (or part) of remote desktop
      2. zoom into desktop
        • useful when desktop has higher resolution than video
      3. click and/or point on the remote desktop
        • send this activities to server
      4. resize video to size of player
    • server/host (provider)
      1. provide local desktop
        • capture desktop, put it in videostream
      2. execute actions send by client
      3. pretend overload of connection between server and client
        • control bitrate of stream and adjust when necessary
  • non-functional
    1. remote desktop is viewed in realtime or as fast as possible
    2. source of transmitted images and used encoder is replacable (by architecture)
    3. effective encoding
      • use xuggler
    4. fallback when native encoder not installed

Optional

Will be implemented when time is left.

  • focus sharing on one or more windows
  • support multi-monitor environments
  • switch between screen and webcam
  • record session, maybe with activities
  • share some parts of eclipse only
    • e.g. a console or something
  • one host, many clients
  • don't use slow codecs (e.g. libx264) on slow machines
    • ability to detect this

Anwendungsfälle / Use Cases

see ThesisDPPXIIIUseCases

Implementierung

see ThesisDPPXIIIProgramming

Literatur

see ThesisDPPXIIILiteratur

Status

StreamServices

This layer adds the ability to define services/features which need pure streams (Java's InputStream / OutputStream).

Current state

running, but sending is not reliable enough

Working features

  • session negotiation
  • smooth termination of sessions
  • configure characteristics of streams to meet non-functional requirements
    • size of buffer
    • minimal size of chunks transferred
    • maximal delay between sending sending data

Problems

  • =BinaryChannel= deadlocks sometimes → data lost, stream broken
  • when an OutputStream is closed which still contains data they will be lost
    • reason: MetaPacket send immediately, but data is queued

Concrete TODO's

  • solve stream-close problem (s.a.)
  • review errorhandling
  • =StreamSessionOutputStream= : public methods to get buffer's state
  • =SteramSessionInputSession= : count read bytes, public available

Nice-to-have features

  • integrate SubMonitor into streams methods
    • not really necessary, but DataTransferManager uses it

ScreenSharing (or VideoSharing)

This is the main feature of this thesis. It's purpose is to see the desktop of a participant in a saros-session. The implementation is a general framework to supply any sequence of images from one to another saros-user.

Current state

basically runs, a session can be started

Working features

  • start a session, see remote desktop
  • capture mouse- and keyboardevents in player
    • switch between fullscreen and mouse following
  • set up basic preferences via eclipses preferences

Problems

  • xuggler
    • resampling of images takes a lot of computation (more than compressing the picture) because it's not accelarated by hardware
    • on mac only supported for 64-bit java 1.6
      • good argument to implement different en-/decoder wink
  • player
    • draw and keep initial background
    • scrollable or resize input
  • desktop/screen
    • zoom-in/-out when following mouse
  • preferences
    • validate new chosen values (e.g. invalid container for xuggler)

Concrete TODO's

Nice-to-have features

Not sure I can make it for this thesis.

  • execute click/type action on host (active screensharing)
    • user-rights management in preferences
  • record session (transmitted stream and activities)

Berichte

Vorbereitungsphase / Preparatory stage

Week ( 1.10 - 2.10.09)

Activities
  • took part in testing of release 9.10.2
  • search existing libraries/programms in java for screensharing or screencapture (on javaforge, berlios, freshmeat, sourgeforge)

Results
  • invitationprocess seems to be buggy → not released yet

Problems
  • not found many existing tools.. am I the first one using this in java!?!?

Comments

Goals
  • get an overview to screensharing
  • find some more javastuff concerning this topic

Week ( 5.10 - 9.10.09)

Activities

Results
  • seems that I have build screenshare on my own
    • use a simple videostream for transmitting the screen
  • did not found other existing screensharingtools or apis for dynamic streaming of content
  • chosen an api for videoencoding Xuggler
    • based on ffmpeg
      • wide range of supported codes
      • active development
      • fast encoding due to native code
    • easy to install on windows, mac, linux with integrated and compiled ffmpeg

Problems
  • videoencoding is very complex
    • choose a preferred container and videoformat
    • understand lots of possible settings an encoder uses

Comments

Goals

  • running prototype

Week ( 12.10 - 16.10.09)

Activities

  • explored xuggler

Results

  • found 'hidden' settings for encoders in xuggler

Problems

  • most settings applied not to codec h.264

Comments

  • was ill from 13.10 to 16.10

Goals

  • see last week

Week ( 19.10 - 23.10.09)

Activities

  • worked on prototype
  • set up libx264 encoder to use cbr (constant bitrate encoding)
  • meet the given bitrate and don't exceed it
  • first checkin of prototype into svn

Results

  • prototype
    • client-server program, streaming via network
    • dynamic resizing of inputframes to encoder
    • console in server to change and view current enviroment
    • change fps while streaming
    • use vbv-buffer (in ffmpeg) to guarantee the bitratelimit (in some cases, esp. low bitrates, without vbv some peaks appeared)

Problems

  • due to update of xuggler, 'libx264' needed other settings than default ones
    • presets by ffmpeg did not work
    • examples by xuggler used not cbr-mode
    • bruteforce trying to find appropriate settings
  • in cbr mode x264 has a noticable delay
    • seems that encoder collects all frames between two keyframes, encodes them at once to keep given bitrate and finally pushes them into videostream

Comments

  • xuggler updated to 3.3 on 21.10, so problems on setting up 'libx264' disappeared (see last week)

Goals

  • release saros
  • make weekly reports
  • prototype
    • deal with temporary low bandwidth

Week ( 26.10 - 30.10.09)

Activities

  • improved prototype
  • releasemanager in this cycle
  • tested new release

Results

  • prototype
    • extended console
    • threaded server
    • change resolution while streaming → problem
    • introduced and second stream for controlling the transmission
      • client sends statistics about delay etc. to server
  • releasemanager
    • collected all changes since last release
    • prepared branch for testing

Problems

  • resizing the livestream is not possible
    • used container (flv) does not support more than one videostream
    • other containerformats don't seem to work frown, sad smile
      • either not streamable (mpeg4, …)
      • or don't work w. xuggler? (mkv, ogg, …)
  • setting a new framerate on the fly causes a big delay on client
    • reason can't be figured out, maybe is the old timebase (based on old framerate) the reason
    • try to drop frames instead of changing framerate

Comments

Goals

  • prepare the presentation for thursday next week
    • plan advance for the next weeks
    • give an overview about screensharing
    • details of planned implementation
      • features
      • requirements
      • integration into saros
  • find reason why some containerformats don't work in xuggler
  • prototype
    • deal with temporary low bandwidth

Working stage

Week ( 2.11 - 6.11.09)

Activities

Results

Problems

Comments

  • not done
    • find reason why some containerformats don't work in xuggler

Goals

  • complete base architecture for screenshareapp
  • send some bugreports to xuggler

Week( 9.11 - 13.11.09)

Activities

  • worked on base architecture
  • meeting with Stephan and Karl on wednesday

Results

  • wrote usecases (ThesisDPPXIIIUseCases) and requirements
  • developed modules (functionality, responsibilities)
  • meeting
    • shorten iterations to 2 weeks
    • immediately integrate screensharing into saros
  • some bugreports for xuggler

Problems

Comments

  • this week was not really productive, i increase it!

Goals

  • adapt plan due to meeting
  • finish architecture
  • start integration in saros

Week ( 16.11 - 20.11.09)

Activities

  • preparation for implementation (requirements engineering)
  • worked on implementation
  • adapted plan

Results
  • architecture and modules, see ThesisDPPXIIIProgramming
  • started to code (see my repository)
    • ported prototype to new architecture
    • local testing via piped streams possible

Problems
  • still some open questions for architecture
    • who controls when a frame will be rendered, encoder or connectionmanager?
    • what is needed at clients side to initiate and handle connections?
      • need some details concerning saros, should be done this week
    • how can i handle configurations?

Comments

Goals
  • integrate into saros
    • view for clients player
    • bytestreams for connection between two users

Week( 23.11 - 27.11.09)

Activities
  • programmed feature
    • startup with saros
    • get a network stream

Results
  • programming
    • hooked into picocotainer
    • tried to get network stream
  • entered tests 3.2 - 3.7 in testlink

Problems
  • is not easy to get network stream
    • find a way to get one via jingle, but this seems not to be accepted → meeting tuesday

Comments

Goals
  • find solution for network architecture (meeting tuesday)
    • let saros be packetbased, wrap a stream with them
    • get a real stream via jingle/socks
    • not completely my part, sandor and henning have to help me
  • start ui
    • player
    • session initiation, negotiation

Week( 30.11 - 4.12.09)

Activities
  • worked on streaming with saros
  • assisted releasemanger
  • tested saros for release

Results
  • developed framework for streambased services
    • simple filetransfer-test worked

Problems

Comments

Goals
  • test streaming with video, finish it completely
  • start ui stuff

Week( 7.12 - 11.12.09)

Activities
  • worked at our new streaming interface
  • played a bit with swt and eclipse views
  • implemented file sending based on streaming

Results
  • streaming: improved it
    • performance (chunkSize and maximalDelay)
    • reliability, send packets same order they arrive
    • introduced soft stop
  • UI
    • created first view (canvas with simple statusbar)
    • discovered swt: widgets, layouts, …

Problems
  • not sure whether streaming is now reliable enough… spend already a lot of time there, but sometimes strange errors occur
Comments

  • don't get so far to test videosharing with stream interface

Goals
  • integrate view into videosharing
  • create a StreamService for video, test it in saros

Week( 14.12 - 18.12.09)

Activities
  • worked on view for playing video
  • implemented example service for sending single files in a session
  • improved streaming
  • integration of videosharing-prototype into saros

Results
  • videosharing
    • created branch, moved classes
    • created simple view for showing video with canvas and statusbar
  • streaming
    • can add serialized objects to metapackets
    • developed improving concepts
      • streams need to be closed
      • output should block when input is not reading
  • file sending
    • show speed in task
    • cancellable

Problems
  • videoplayerview is swt, but images are awt: need to embed an awt canvas
  • sometimes streaming hangs without obvious reason, need to figure out

Comments

Goals

  • improve streaming (s.a.)
    • esp. robustness (errorhandling, disposing)
  • make videosharing work in saros
    • embed awt-canvas in player
    • create preferences (pages)
    • catch input in player's canvas, send them to host

Week( 21.12 - 25.12.09)

Ill

Week( 28.12.09 - 1.1.10)

Ill

Week( 4.1 - 8.1.10)

Ill

Week( 11.1 - 15.1.10)

Activities
  • finish screensharing and streaming for release
  • documented state of work in wiki
  • write paper about technical realization

Results
  • screensharing
    • disposing
    • fast awt-canvas for displaying video
    • catch input in videoplayer and send to host
    • pure java encoder taken from smack-demo
  • streaming
    • errorhandling/disposing
    • many fixes
  • paper - 50% done

Problems
  • alternative encoder is too slow, works only w. 1 FPS
    • try to optimize

Comments
  • screensharing not ready for release
  • streaming could be ready
    • christopher fixes binarychannel
    • must be reviewed (and afterwards fixed)
    • any feature uses it

Goals
  • make features ready for use for release next week
    • optimize other encoder
    • UI
      • players toolbar
      • resizing
      • preference dialogues
  • paper finished until wednesday

Week( 18.1 - 22.1.10)

Activities
  • writing mentioned paper
  • make screensharing ready for release
  • make streaming ready for release
  • reviewed some patches for release

Results
  • paper done
    • no feedback yet
  • screensharing works
    • UI minor bugs

Problems
  • seems that there's not enough to review both features
    • maybe not released or only experimental?
  • alternative encoder
    • really bad performance, only 1 fps at low resolution, even at fast machines

Comments
  • only core features in screensharing work

Goals
  • release a new version for Saros
  • finish streaming and sending file during session

Week( 25.1 - 29.1.10)

Activities
  • helped releasing and testing saros 10.1.29
  • worked on streaming and screensharing
Results
  • streaming
    • created patch for reviewing
  • screensharing
    • worked towards creating patch, not done yet

Problems

Comments

Goals
  • next weeks
    • get screensharing stable
    • focus on thesis
    • improve streaming according to reviews
  • next week
    • relax @ holiday in austria

Week( 1.2 - 5.2.10)

holiday in austria

Week( 8.2 - 12.2.10)

Activities
  • worked on thesis
  • improved streaming

Results
  • created outline
  • read and chose some papers, see ThesisDPPXIIILiteratur
  • streaming
    • almost done, thread-pools are missing

Problems

Comments

Goals
  • write parts about saros, pp, dpp and ca
  • create new patch for streaming to be reviewed

Week (15.2 - 19.2.10)

Activities
  • read some more papers
  • prepared streaming for release

Results
  • finished streaming patch, which will be reviewed for release

Problems

Comments
  • ill

Goals
  • get better

Week (22.2 - 26.2.10)

Activities
  • adapted patch for streaming

Results
  • streaming-patch
    • adapted due to hennings and flo's review

Problems
  • patch done too close to release, not released

Comments
  • still ill

Goals
  • get finally better
  • find some papers concerning video (esp. bandwidth-adaption)
  • write parts about saros, pp, dpp, ca and related work

Kommentare

Wer will, wer will, wer hat noch nicht? Ich würde mich über jede Anregung, Anmerkung, Kritik freuen. smile