Team:ETHZ Basel/InformationProcessing/Visualization

From 2010.igem.org

(Difference between revisions)
(→Visualization)
(→User Experience)
 
(25 intermediate revisions not shown)
Line 2: Line 2:
{{ETHZ_Basel10_InformationProcessing}}
{{ETHZ_Basel10_InformationProcessing}}
-
= Visualization =
+
= Visualization and Man-Machine Interface =
-
== User Experience ==
+
== Background ==
-
The images of the microscope are visualized live at a personal computer which is connected to the microscope over a local network or the internet, thus it is possible to control the E. lemming from another office or city. To enhance the user experience, all detected cells in the microscope image are highlighted by letting them glow blue (see example movie below). The user can select a cell with the joystick, which is then highlighted by a yellow glowing to simplify the tracking. Furthermore, the current direction of the cell is visualized by a semi-transparent yellow light cone and the path the cell has already travelled is marked by yellow dots.
+
<html><div class="thumb tright"><div class="thumbinner" style="width:402px;">
 +
<iframe title="YouTube video player" class="youtube-player" type="text/html" width="400" height="325" src="http://www.youtube.com/embed/hwtOBgQCAAA?rel=0&hd=1" frameborder="0"></iframe>
 +
<div class="thumbcaption"><div class="magnify"><a href="http://www.youtube.com/watch?v=hwtOBgQCAAA&hd=1" class="external" title="Enlarge"><img src="/wiki/skins/common/images/magnify-clip.png" width="15" height="11" alt="" /></a></div><b>The real-time representation of the current microscope image</b>. <br><i>Blue dots</i>: the detected <i>E. coli</i> cells. <i>Yellow dot</i>: the currently selected E. lemming. <i>Yellow cone</i>: the current swimming direction of E. lemming. <i>Red thin line</i>: the reference direction. <i>Yellow dotted line</i>: the current path of the E. lemming. <br>Note that the movement process is not under the influence of the controller.</br> </div></div></div></html>
 +
To control E. lemming, the scientist has to be aware of a huge amount of information, which has to be visualized in real-time. Only with this information, the user is able to choose an E. lemming to control, decide on a reference direction, and interfere in the experiments. This information includes
 +
* the current microscopy image;
 +
* all detected, not detected and falsely detected cells;
 +
* the selected cell itself;
 +
* its swimming direction;
 +
* the reference direction;
 +
* its path;
 +
* and movements of the stage.
 +
Since this information has to be absorbed by the experimentalist at a rate higher than three frames per second, a textual representation is not adequate. We thus decided to represent the information in an intuitive way by incorporating it into the microscope images.
-
The user can select a direction with the joystick he wants the E. lemming to go, which is visualized by a red line. The controller (see [Team:ETHZ_Basel/Modeling/Controller Controller]) then automatically tries, by sending red and far-red light signals, to force the E. lemming in this direction. Alternatively the user can send the light signals on his own by using the buttons of the joystick.
+
To enable the scientist to interfere with the experiment without having to interrupt, and thus possibly losing the currently controlled cell, we established two input devices: the user can either control the experiment and the E. lemming using the keyboard or the joystick. With both devices he/she is able to:
 +
* select a cell to control;
 +
* activate or deactivate the controller;
 +
* set the reference direction;
 +
* increase or decrease the threshold for cell detection;
 +
* or, alternatively, manually induce red or far-red light pulses of various length (instead of using the controllers provided in the toolbox).  
 +
If using a force-feedback joystick, the current swimming direction is furthermore given as a small force on the direction of the joystick, thus intuitively increasing the amount of information available for the user.
-
If the cell moves out of the current image of the microscope, this is detected by the controller and the microscope is automatically moved so that the cell is always in the central 50% of the image. The user furthermore can change during the simulation the threshold of the cell detection algorithm (see [https://2010.igem.org/Team:ETHZ_Basel/InformationProcessing/CellDetection Cell Detection]) to detect either more cells (but also possibly more false positive ones) or less cells (which may reduce the number of true positives).
+
Both, the visual representation of the current state of the experiment as well as the user input had to be designed to require low computational efforts not to unnecessarily slow down the imaging.
-
== Example Movie ==
+
== User Experience ==
-
<html>
+
[[Image:userExperience.jpg|thumb|400px|'''Example Image of the Visualization''']]The images of the microscope are visualized live at a personal computer which is connected to the microscope over a local network or internet, thus it is possible to control the E. lemming from another office or city.  
-
<object classid="clsid:02BF25D5-8C17-4B23-BC80-D3488ABDDC6B" width="704" height="480" codebase="http://www.apple.com/qtactivex/qtplugin.cab">
+
-
<param name="autoplay" value="false" />
+
-
<param name="controller" value="true" />
+
-
<param name="pluginspage" value="http://www.apple.com/quicktime/download/indext.html" />
+
-
<param name="target" value="myself" />
+
-
<param name="type" value="video/quicktime" />
+
-
<param name="src" value="https://static.igem.org/mediawiki/2010/8/89/Visualization.mov" />
+
-
<embed src="https://static.igem.org/mediawiki/2010/8/89/Visualization.mov" width="704" height="480" autoplay="false" controller="true" border="0" pluginspage="http://www.apple.com/quicktime/download/indext.html" target="myself"></embed>
+
-
</object>
+
-
</html>
+
-
The cells were placed in a 50 &mu;m (?) high flow chamber ([https://2010.igem.org/Team:ETHZ_Basel/InformationProcessing/MicroscopeSetup Details]).
+
=== Visual Enhancements ===
-
The excitation for the bright field images (40x) was 60ms, the period something around 0.3s, aproximately corresponding to the fps rate of the movie. To increase speed, the binning in the microscope was set to 2, so that the images were 672 x 512 grayscale (16 bit) instead of the maximal resolution of the microscope of 1344 x 1024 pixels, and every pixel  The images were made out-of-focus to simplify cell detection.
+
All detected cells in the microscope image are highlighted by letting them glow blue (see example picture). The user can select a cell to control with the joystick or the keyboard, which is then glowing yellow instead of blue. The current swimming direction is indicated by a transparent yellow light cone, imitating the floodlight of cars or a flashlight, and the reference direction is depicted as a thin red line. The path the cell has passed during the experiment is marked by small yellow dots, similar to footprints in snow. Finally, the automatic movement of the stage of the microscope, when following a cell which swims out of the current field of view, is indicated by arrows in the direction of movement, which are popping up for a moment.

Latest revision as of 01:13, 28 October 2010

Visualization and Man-Machine Interface

Background

The real-time representation of the current microscope image.
Blue dots: the detected E. coli cells. Yellow dot: the currently selected E. lemming. Yellow cone: the current swimming direction of E. lemming. Red thin line: the reference direction. Yellow dotted line: the current path of the E. lemming.
Note that the movement process is not under the influence of the controller.
To control E. lemming, the scientist has to be aware of a huge amount of information, which has to be visualized in real-time. Only with this information, the user is able to choose an E. lemming to control, decide on a reference direction, and interfere in the experiments. This information includes

  • the current microscopy image;
  • all detected, not detected and falsely detected cells;
  • the selected cell itself;
  • its swimming direction;
  • the reference direction;
  • its path;
  • and movements of the stage.

Since this information has to be absorbed by the experimentalist at a rate higher than three frames per second, a textual representation is not adequate. We thus decided to represent the information in an intuitive way by incorporating it into the microscope images.

To enable the scientist to interfere with the experiment without having to interrupt, and thus possibly losing the currently controlled cell, we established two input devices: the user can either control the experiment and the E. lemming using the keyboard or the joystick. With both devices he/she is able to:

  • select a cell to control;
  • activate or deactivate the controller;
  • set the reference direction;
  • increase or decrease the threshold for cell detection;
  • or, alternatively, manually induce red or far-red light pulses of various length (instead of using the controllers provided in the toolbox).

If using a force-feedback joystick, the current swimming direction is furthermore given as a small force on the direction of the joystick, thus intuitively increasing the amount of information available for the user.

Both, the visual representation of the current state of the experiment as well as the user input had to be designed to require low computational efforts not to unnecessarily slow down the imaging.

User Experience

Example Image of the Visualization
The images of the microscope are visualized live at a personal computer which is connected to the microscope over a local network or internet, thus it is possible to control the E. lemming from another office or city.

Visual Enhancements

All detected cells in the microscope image are highlighted by letting them glow blue (see example picture). The user can select a cell to control with the joystick or the keyboard, which is then glowing yellow instead of blue. The current swimming direction is indicated by a transparent yellow light cone, imitating the floodlight of cars or a flashlight, and the reference direction is depicted as a thin red line. The path the cell has passed during the experiment is marked by small yellow dots, similar to footprints in snow. Finally, the automatic movement of the stage of the microscope, when following a cell which swims out of the current field of view, is indicated by arrows in the direction of movement, which are popping up for a moment.

Retrieved from "http://2010.igem.org/Team:ETHZ_Basel/InformationProcessing/Visualization"