Team:ETHZ Basel/InformationProcessing/InformationFlow

From 2010.igem.org

(Difference between revisions)
(References)
 
(20 intermediate revisions not shown)
Line 3: Line 3:
= Information Flow =
= Information Flow =
-
[[Image:MicroscopePipeline.jpg|thumb|400px|The imaging pipeline from the microscope to Matlab/Simulink]]
+
== Information Acquisition ==
-
The microscope is connected to a workstation using the core drivers and interfaces of μManager (see Stuurman et al. (2007) or [http://www.micro-manager.org]). To provide a mechanism to change the cell's input signal depending on its fluorescence signal, we developed the microscope software μPlateImager, which enables for parallel acquisition of images and the modification of light input signals. μPlateImager uses the Java interface of the μManager core to control the microscope and can be configured by a separate platform-independent visual user interface. Since the communication with the microscope already requires a significant amount of system resources, it was necessary to swap the image processing, cell detection, and controller part to a second workstation to increase the possible frame rate. μPlateImager can therefore be controlled over the network or the internet by a GUI (graphical user interface). This GUI uses the yet undocumented Java MATLAB Interface (JMI) to start up a Matlab (The MathWorks, Natick, MA) process based on the open source project matlabcontrol (see [http://code.google.com/p/matlabcontrol/]). It furthermore starts up a Simulink model and transfers the microscope control to several of the blocks of the Lemming Toolbox, a Simulink toolbox allowing for the block based interconnection of the main controller and image processing parts, like cell detection, tracking the visualization or the like.
+
-
<br clear="all">
+
-
== Toolkit (Simulink) ==
+
===&mu;PlateImager===
-
[[Image:Simulink.jpg|thumb|400px|Graphical user interface with which the image analysis pipeline and the control algorithm can be defined]]A Matlab script is executed by &mu;PlateImager, which transfers the control over the microscope and automatically starts the GUI based on Simulink. The GUI consists of several blocks representing the single steps of the image analysis and the control of the microscope. First, the microscope block triggers the microscope to make an image and sends this image to the cell detection block, which detects all cells in the current image and tracks them between the several frames. This information is send to the next block which selects the cell to be controlled. Furthermore, by comparing the change in the position of the E. lemming over several consecutive images, the direction of the E. lemming is estimated. This data together with the raw image is send to the display.<br clear="all">
+
-
[[Image:Screen.jpg|thumb|400px|Output on the screen. The cone is visualizing the current direction of the selected E. coli.]]The direction of the cell is automatically compared to the direction it should go. This direction can be intuitively defined by the user using a joystick. The force feedback functionality of the joystick is used to give the user an intuitive feedback of the current direction of the E. lemming. If the difference between the actual direction of the E. lemming and the direction the user defined is too high, tumbling is automatically induced by a red light (660nm) pulse. Otherwise tumbling is supressed by a far red light (748nm). Alternatively the user can induce the pulses directly using the buttons of the joystick.
+
[[Image:MicroscopePipeline.jpg|thumb|400px|'''The imaging pipeline from the microscope to Matlab/Simulink.''' Images obtained from the microscope are sent through the network to the client, where they are processed. The connection between the server and the client is mediated by the &mu;PlateImager microscope software. ]]
-
If the cell is moving out of the image the microscope moves automatically such that the cell is always approximately in the center of the image.
+
The microscope is connected to a workstation using the core drivers and interfaces of &mu;Manager (see [[Team:ETHZ Basel/InformationProcessing/InformationFlow#References| [1]]] or [http://www.micro-manager.org [2]]). To provide a mechanism to change the cell's input signal depending on the direction towards which it is swimming, we developed the microscope software &mu;PlateImager, which enables the parallel acquisition of images and the modification of light input signals.
-
<br clear="all">
+
 
 +
&mu;PlateImager uses the Java interface of the &mu;Manager core to control the microscope and can be configured by a separate platform-independent visual user interface. Since the communication with the microscope already requires a significant amount of system resources, it was necessary to swap the image processing, cell detection, and controller part to a second workstation to increase the possible frame rate.
 +
 
 +
&mu;PlateImager can be controlled over the network or the Internet by a GUI (graphical user interface). This GUI uses the yet undocumented Java MATLAB Interface (JMI) to start up a Matlab process, based on the open source project matlabcontrol (see [http://code.google.com/p/matlabcontrol/ [3]]). It furthermore starts up a Simulink model and transfers the microscope control to several of the blocks of the [[Team:ETHZ_Basel/Achievements/Matlab_Toolbox| Lemming Toolbox]].
 +
 
 +
== Information Processing ==
 +
We constructed a Simulink model based on the [[Team:ETHZ_Basel/Achievements/Matlab_Toolbox| Lemming Toolbox]]. The model consists of several blocks representing the single steps of image analysis and controller and blocks enabling the microscope control. Since it contains several feedback loops, we here only give an overview over the main information flow, skipping some of the however important connections.
 +
 
 +
=== Simulink blocks ===
 +
[[Image:realTimeModel.jpg|thumb|400px|'''Overview of the Simulink Blocks of the Information Processing path.''' The Simulink Blocks we created allow us to combine the image analysis pipeline, the user's input, the control algorithm and the visualization''']]
 +
First, the microscope block triggers &mu;PlateImager and therefore the microscope is indirectly triggered. An out-of focus image is taken by the microscope and sent by &mu;PlateImager to the cell detection block. Based on a user definable threshold, the [https://2010.igem.org/Team:ETHZ_Basel/InformationProcessing/CellDetection '''Cell Detection Block'''] returns the current positions of the cells having the highest score. These positions are send to the [https://2010.igem.org/Team:ETHZ_Basel/InformationProcessing/CellDetection '''Cell Tracking Block'''], which compares the positions of the currently detected cells with the positions of the cells detected in the previous frames. If the distance of a currently detected cell from its position in one of the previous frames is smaller than a threshold, the block assigns the same unique ID (UID) to the cell, otherwise a new UID is generated. Furthermore the cell tracking algorithm allows the cell to persist for several successive frames if it gets lost.
 +
 
 +
The cell positions, together with their UIDs are sent to the '''Cell Selection Block'''. This block determines the direction of the currently selected cell by calculating the change in position by comparing its current position to the position it had in the previous frames. Furthermore, it allows the user to switch between the cells with the help of the buttons of a connected joystick.
 +
 
 +
With the joystick the user defines a reference direction, which serves, together with the actual direction of the E. lemming, as input for the '''Controller Block'''. The controller calculates, depending on the actual and previous direction of the E. lemming and the reference direction, if red light or far-red light should be activated. This information is passed to the '''Red Light Block''' and '''Far-Red Light Block''', which trigger &mu;Manager, therewith the microscope, to activate or deactivate the light sources.
 +
 
 +
The '''Stage Control Block''' gets as input the current position of the E. lemming and decides if the E. lemming is on the verge of moving out of the field of view. If so, it induces the x/y-stage of the microscope to change for a defined amount of &mu;m. The current stage position is passed to the '''Cell Tracking Block''' as an additional input to allow for the tracking of cells over a long distance.
 +
 
 +
Finally, several of the previous described blocks' outputs are passed to the [https://2010.igem.org/Team:ETHZ_Basel/InformationProcessing/Visualization '''Visualization Block'''], which displays the received information on the computer screen or the beamer.<br clear="all">
 +
 
 +
=References=
 +
[1] [http://jcs.biologists.org/cgi/content/full/120/21/3715 Wollman and Stuurman: High throughput microscopy: from raw images to discoveries. 2007. Journal of Cell Science. 120 (21). p3715-22].
 +
 
 +
[2] http://www.micro-manager.org/
 +
 
 +
[3] http://code.google.com/p/matlabcontrol/

Latest revision as of 22:50, 27 October 2010

Information Flow

Information Acquisition

μPlateImager

The imaging pipeline from the microscope to Matlab/Simulink. Images obtained from the microscope are sent through the network to the client, where they are processed. The connection between the server and the client is mediated by the μPlateImager microscope software.

The microscope is connected to a workstation using the core drivers and interfaces of μManager (see [1] or [2]). To provide a mechanism to change the cell's input signal depending on the direction towards which it is swimming, we developed the microscope software μPlateImager, which enables the parallel acquisition of images and the modification of light input signals.

μPlateImager uses the Java interface of the μManager core to control the microscope and can be configured by a separate platform-independent visual user interface. Since the communication with the microscope already requires a significant amount of system resources, it was necessary to swap the image processing, cell detection, and controller part to a second workstation to increase the possible frame rate.

μPlateImager can be controlled over the network or the Internet by a GUI (graphical user interface). This GUI uses the yet undocumented Java MATLAB Interface (JMI) to start up a Matlab process, based on the open source project matlabcontrol (see [3]). It furthermore starts up a Simulink model and transfers the microscope control to several of the blocks of the Lemming Toolbox.

Information Processing

We constructed a Simulink model based on the Lemming Toolbox. The model consists of several blocks representing the single steps of image analysis and controller and blocks enabling the microscope control. Since it contains several feedback loops, we here only give an overview over the main information flow, skipping some of the however important connections.

Simulink blocks

Overview of the Simulink Blocks of the Information Processing path. The Simulink Blocks we created allow us to combine the image analysis pipeline, the user's input, the control algorithm and the visualization

First, the microscope block triggers μPlateImager and therefore the microscope is indirectly triggered. An out-of focus image is taken by the microscope and sent by μPlateImager to the cell detection block. Based on a user definable threshold, the Cell Detection Block returns the current positions of the cells having the highest score. These positions are send to the Cell Tracking Block, which compares the positions of the currently detected cells with the positions of the cells detected in the previous frames. If the distance of a currently detected cell from its position in one of the previous frames is smaller than a threshold, the block assigns the same unique ID (UID) to the cell, otherwise a new UID is generated. Furthermore the cell tracking algorithm allows the cell to persist for several successive frames if it gets lost.

The cell positions, together with their UIDs are sent to the Cell Selection Block. This block determines the direction of the currently selected cell by calculating the change in position by comparing its current position to the position it had in the previous frames. Furthermore, it allows the user to switch between the cells with the help of the buttons of a connected joystick.

With the joystick the user defines a reference direction, which serves, together with the actual direction of the E. lemming, as input for the Controller Block. The controller calculates, depending on the actual and previous direction of the E. lemming and the reference direction, if red light or far-red light should be activated. This information is passed to the Red Light Block and Far-Red Light Block, which trigger μManager, therewith the microscope, to activate or deactivate the light sources.

The Stage Control Block gets as input the current position of the E. lemming and decides if the E. lemming is on the verge of moving out of the field of view. If so, it induces the x/y-stage of the microscope to change for a defined amount of μm. The current stage position is passed to the Cell Tracking Block as an additional input to allow for the tracking of cells over a long distance.

Finally, several of the previous described blocks' outputs are passed to the Visualization Block, which displays the received information on the computer screen or the beamer.

References

[1] Wollman and Stuurman: High throughput microscopy: from raw images to discoveries. 2007. Journal of Cell Science. 120 (21). p3715-22.

[2] http://www.micro-manager.org/

[3] http://code.google.com/p/matlabcontrol/