NucleusEditor

From FarsightWiki
(Difference between revisions)
Jump to: navigation, search
(Image Viewer)
 
(93 intermediate revisions by 9 users not shown)
Line 1: Line 1:
The main purpose of the NucleusEditor program is to view and segment images of nuclei, and then efficiently edit the segmentation using pattern analysis tools (PACE).   
+
The main purpose of the NucleusEditor program is to view and segment images of nuclei, and then efficiently edit the segmentation using pattern analysis tools (PACE).  An auxiliary function of the system includes segmentation of 2D Histopathology images.
  
 
== Image Viewer ==
 
== Image Viewer ==
  
NucleusEditor allows users to view image data in a variety of file formats (tif, pic, lsm, etc). An image may be opened by selecting File|Load Image...
+
NucleusEditor allows users to view image data in a variety of file formats (tif, pic, lsm, etc). The image must be in 8-bit grayscale and must have "bright" foreground objects under a "dark" background to segment properly. An image may be opened by selecting File/Load Image...
  
[[Image:Loadimage.JPG|frame|center|'''Load Image Example''']]
+
[[Image:Loadimage2.JPG|thumb|800px|center|'''Load Image Example''']]
  
== Segment Cell Nuclei ==
+
=== XML Images ===
  
To begin segmentation, an image must already be loaded and visible in NucleusEditor.  The [[Nuclear_Segmentation |nuclear segmentation algorithm]] is triggered by selecting File|Start Segmentation...  The program will use the visible image for the segmentation algorithm.  If the image has multiple channels, the program will search for the channel that contains the string "Nuclei" in its nameBefore the segmentation algorithm can begin it must be initialized with a few parametersThese parameters can be chosen automatically using an innovative algorithm developed by Yousef Al-Kofahi, or they can be supplied in the form of a parameters file.  The user must choose an option from the dialog box that will appear.  When the segmentation begins, a new toolbar will be added to the NucleusEditor that shows the progress of the segmentation algorithm and provides a STOP button to cancel the segmentation.
+
Often [[Glossary#Multi-Channel Images|multi-channel images]] are divided into multiple files; one file for each channel.  We have defined an XML file format for specifying this type of imageEach file can be given a channel name and a channel color.
  
Nuclear Segmentation is achieved in 3 major steps:
+
[[Image:Xmlimage.JPG|frame|center|'''XML Image Definition''']]
* 1. Initial Segmentation
+
* 2. Seed Editing
+
* 3. Final Segmentation
+
  
[[Image:Segment.JPG]]
+
== Segment Cell Nuclei ==
  
Upon completion of the initial segmentation, the program will pause and allow the user to inspect the result and edit the seeds [not available]. At this point the user may either continue segmentation (by pressing the "Continue" button), abort segmentation (by pressing the "Stop" button), or modify the parameters and start over (by changing the parameter file, pressing the "Stop" button, and then re-starting the segmentation using the saved parameter file).   
+
To begin [[Glossary#Segmentation|segmentation]], an image must already be loaded and visible in NucleusEditor.  The [[Nuclear_Segmentation |nuclear segmentation algorithm]] is triggered by selecting ''Tools / Segment Nuclei...''  The program will use the visible image for the segmentation algorithm, and ask the user to select the desired [[Glossary#Channel|channel]] from within that image. Before the segmentation algorithm can begin it must be initialized with a few [[Glossary#Parameters|parameters]].  These parameters will be chosen automatically using an innovative algorithm developed by Yousef Al-Kofahi. When the segmentation begins, an activity toolbar will be added to the NucleusEditor that shows the progress of the segmentation algorithm and provides a STOP button to cancel the segmentation.   
  
After the final segmentation, the program will compute features and show this metadata (mainly [[Intrinsic_Features_of_Blobs|object features]]) using multiple data visualization tools that are all actively linked, allowing the data to be viewed in multiple spaces simultaneously. A cluster of objects to be selected based on any combination of operations in any of the multiple spaces. We term this method [[EVS/ALISA |actively linked multiple spaces architecture (ALISA)]]. NucleusEditor provides this linked-view architecture where objects can be visualized in image or geometric views, table views, plot views, and histogram views. Operations performed in one view are immediately visible in all of the views.
+
Nuclear Segmentation is achieved in 2 major steps:
 +
* 1. Segmentation
 +
* 2. Editing
  
To save s segmentation result, select File|Save Result
+
[[Image:Segment3.JPG|thumb|800px|center]]
 
+
[[Image:Result.JPG]]
+
  
 +
Upon completion of the segmentation, the program will compute features and show this metadata (mainly [[Intrinsic_Features_of_Blobs|object features]]) using multiple data visualization tools that are all actively linked, allowing the data to be viewed in multiple spaces simultaneously. There is a method to select a cluster of objects based on any combination of operations in any of the multiple spaces. We term this method [[EVS/ALISA |actively linked multiple spaces architecture (ALISA)]]. NucleusEditor provides this linked-view architecture where objects can be visualized in image or geometric views, table views, plot views, and histogram views. Operations performed in one view are immediately visible in all of the views.
  
 
In the ALISA mode a number of operations become active in each view:
 
In the ALISA mode a number of operations become active in each view:
 
* Segmentation Result
 
* Segmentation Result
** Show/Hide Object Boundaries {View|Show Boundaries or Ctrl+B)
+
** Show/Hide Object Boundaries (View|Show Boundaries or Ctrl+B)
** Select/Deselect Objects {by clicking on them)
+
** Show/Hide Object ID Numbers (View|Show Object IDs or Ctrl+I)
 +
** Show/Hide Object Centroids (View|Show Object Centroids)
 +
** Add Table (View|New Table)
 
** Add Scatter Plot (View|New Scatter)
 
** Add Scatter Plot (View|New Scatter)
 +
** Add Histogram (View|New Histogram)
 +
** Modify gray-scale image intensity (View|Adjust Image Intensity or Ctrl+G)
 
** Clear Selections (Editing|Clear Selections or Ctrl+C)
 
** Clear Selections (Editing|Clear Selections or Ctrl+C)
 
** Edit Functions (Edit Menu - see below)
 
** Edit Functions (Edit Menu - see below)
 +
** Apply an Exclusion Margin to the image (Editing|Apply Exclusion Margin...)
 +
** Zoom using + and - keys
 +
** Select/Deselect Objects (Hold Ctrl key to select more than one object)
 
* Scatter Plots
 
* Scatter Plots
 
** Select/Deselect Objects  
 
** Select/Deselect Objects  
 
*** Select/deselect by clicking on single object
 
*** Select/deselect by clicking on single object
 
*** Clear all selections (Options|Clear Selections or Ctrl+C)
 
*** Clear all selections (Options|Clear Selections or Ctrl+C)
*** Select group by holding CTRL key and creating a lasso via a series of clicks
+
*** Select a group of objects by holding down the SHIFT key, and clicking on a series of points to sketch out a polygon that acts as a "lasso".
 
** Options|Normalize - Ctrl+N (show features in normalized space)
 
** Options|Normalize - Ctrl+N (show features in normalized space)
 
** Tools|Find Outliers (ALISA - see below)
 
** Tools|Find Outliers (ALISA - see below)
 
** Options (can change columns for X,Y and coloring)
 
** Options (can change columns for X,Y and coloring)
 
* Table
 
* Table
** View|Sort table by... (To select column to sort by - in ascending order)
+
** View|Filter table by... (To select items meeting specific criteria)
 
** Show/Hide table columns (View|Visible Columns)
 
** Show/Hide table columns (View|Visible Columns)
 +
 +
[[Image:Result.JPG|thumb|800px|center|'''Final Result Example''']]
 +
 +
== Image Processing Pipeline ==
 +
 +
In many cases the automatic nuclear segmentation parameters will require modifications for satisfactory results. This is possible by using the project processor capability within the Nucleus Editor. This function uses an XML definition file as input. This definition file can specify required process steps and parameters for each step. For example, it can be used to limit the number of features that are calculated and specify that certain associative features are desired. See an example of one of these files below. If you have specific needs please contact the Roysam lab for assistance in creating a process file. Once the file has been created, execute the pipeline by first loading the image, then choosing "File|Process image..."
 +
 +
[[Image:ProjectProcessorDefinition.JPG|thumb|800px|center|'''Project Processor Definition File''']]
  
 
== Using the PACE Concept ==
 
== Using the PACE Concept ==
  
== Editing Segmentation Results ==
+
Since automated segmentation algorithms are not perfect, there is a compelling need to develop efficient methods to identify and correct the automated segmentation errors. Due to the large size of the datasets, the identification and correction of the cells to be edited have to be done very fast. In order to achieve this, NucleusEditor implements kernel based pattern analysis algorithms where we can do group or cluster editing of multiple errors simultaneously. This dramatically reduces the amount of manual effort required compared to unassisted edit-based methods. We term this methodology PACE (Pattern Assisted Cluster Editing). By observing the nature of the errors, it will be possible to asses the performance of the segmentation algorithm used in NucleusEditor and modify it to reduce similar errors in the future.
 +
 
 +
ALISA is designed to allow group editing of objects. The idea is for the user to select a group of objects based on their features. This can be done manually, or using automated pattern analysis algorithms. Manual selection of objects is aided by scatter plots of features. The user can choose the features displayed along the x and y axes of scatter plots. To visualize more than 2 features at a time, ALISA allows the user to display an unlimited number of scatter plots. Each point on the scatter plot corresponds to a specific object in the image (with a unique identifier (ID)). Clicking the mouse on a point selects and highlights the corresponding object in all other views. The user can also draw a polygon on a scatter plot to select a group of objects visually.
 +
 
 +
Computer-assisted selection of objects can also be performed. Features that highlight specific types of errors in segmentation (we call them diagnostic features) can also be defined and these objects selected and edited in their feature-space. Outlier detection tools are provided to assist in finding unusual objects to be inspected and/or edited. Outliers may correspond to segmentation errors (typical) or unusual cells (rare but possible). For outlier detection, the one-class [http://en.wikipedia.org/wiki/Support_vector_machine Support Vector Machine] (SVM) is employed [libSVM]. The one-class SVM proves to be a very useful tool for outlier detection, often highlighting a variety of quirky objects that should be inspected more closely, and edited when necessary. The outliers are shown in the segmentation, and scatter views, by a simple color coding scheme. The user can quickly locate, inspect and edit the outliers due to the linked nature of the views.
 +
 
 +
=== Clues ===
 +
 
 +
Two simple features have been shown to be particularly useful in detecting segmentation errors: volume and percent shared boundary.  The following edit operations tend to correlate to these features in this way:
 +
 
 +
* Delete: small volume & zero (or small) percent shared boundary
 +
* Merge: small volume & large percent shared boundary
 +
* Split: high volume &| high surface area
 +
 
 +
=== Methods ===
 +
 
 +
Two main techniques can be used to locate objects that need to be edited: filters and outlier detection.
 +
 
 +
* Filters:  The table view provides two very useful functions: View|Visible Columns and View|Filters.  The NucleusEditor provides a large wealth of feature data, most of which is noise to the editing process.  Change the Visible Columns to only include those features that are useful while editing (ex. id, volume, shared boundary, surface area). Next we may want to select all of the objects with very small volume.  On the scatter plot it is very hard to select just one of these objects and view it because they are too tightly clustered together.  By filtering the table by volume, it becomes very easy to find all of the objects with small volumes. [[Image:Filter.JPG|thumb|800px|none|'''Filtering''']]
 +
 
 +
 
 +
* Finding Outliers: The Nucleus Editor provides a tool for finding outliers in a set of data.  This tool can be activated by selecting Tools|Detect Outliers.  The outlier tool uses the one-class SVM to find outliers.  The outlier tool seems to be most effective when only 2 or 3 features are used.  If the outliers are not immediately color-coded in the scatter plot it could be because the incorrect column for coloring is selected, please select the "outlier?" column by choosing Options|Set Color Column. From the scatter plot it is now easy to find sub-populations of objects that meet specific criteria (ex. small volume & high shared boundary - possible merge candidates). [[Image:Outliers3.JPG|thumb|800px|none|'''Finding Outliers''']]
 +
 
 +
=== Edits ===
 +
 
 +
At this stage in the process sub-populations of objects requiring specific edits should be identified, either by sorting the table or by finding regions (outliers) in the scatter plot.  NucleusEditor provides the following three operations to edit cells, which will become active when a valid segmentation result is loaded:
 +
* Delete
 +
* Merge
 +
* Split
 +
* Add
 +
* Fill
 +
The goal is to implement these operations using smart algorithms that allow them to be carried out in groups.  These "smart algorithms" are still in development.  The idea is to identify a large group of objects that need to be merged (for example), and simply tell the program to merge them.  This could mean merging some of the selected objects with other selected objects, or it could mean merging a selected object with another, non-selected, object.  Similarly, the split function should attempt to split an object into the most optimal number of objects with reasonable boundaries. As soon as objects are edited all views will be updated (however you will need to re-run the outlier detector to update outliers).  Be sure to select File|Save Result to save results and any edits that have been performed.
 +
 
 +
==== Add ====
 +
 
 +
Adding new object requires detecting new foreground pixels from the existing background.  Start the add operation by selected "Editing|Add Objects" from the menu or by "Ctrl+A".  Draw a box around the new object in the middle z-slice of that object by clicking the left mouse button. [[Image:Add.JPG|thumb|800px|none|'''Add new object''']]
 +
 
 +
==== Delete ====
 +
 
 +
In order to delete objects we simply select them in any one of the available views and choose the "Editing|Delete Objects" menu item within the Nuclear Segmentation Window or by "Ctrl+D". In the following example all cells with small volume and zero shared boundary will be deleted, they have first been selected in the table view.
 +
 
 +
==== Merge ====
 +
 
 +
Merging objects requires the selection of two or more objects.  Only neighboring objects will be merged to form a new object. Merging may be initiated from the "Editing|Merge Cells" menu item or by "Ctrl+M".  Note that if the selected cells are not neighbors, they will not be merged.  In this example we have merged object 394 and 396 to become a new object: 504.
 +
 
 +
==== Split ====
 +
 
 +
Splitting objects is achieved by specifying two or more [[Glossary#Seeds|seed points]]. Start the splitting operation by selecting "Editing|Split Objects X-Y" from the menus.  Left-click within the objects to place 2 seed points.  Objects will be split as soon as two seed points have been specified.  Choose "Editing|Split Objects X-Y" again from the menu to exit splitting mode.
 +
 
 +
Objects may also also be split at a specific z-plane. This is accomplished by Choosing "Editing|Split Objects At Z. All selected objects will be split at the current z slice.
 +
 
 +
==== Fill ====
  
Existing nuclear segmentation results can be loaded by selecting File|Load Result...  
+
Sometimes segmented objects can be made of up multiple (non-touching) connected components, or they may have holes in them. These errors can be fixed using the "Edit|Fill Objects" menu item.
  
NucleusEditor provides the following three operations to edit cells, which will become active when a valid segmentation result is loaded:
+
=== Edit History Log ===
* Delete Cells
+
* Merge Cells
+
* Splitting (Cells)
+
  
=== Delete Cells ===
+
When edits are made to a nuclear segmentation, an entry is made to a log file.  This log file is stored in the same location as the input image.
  
In order to "Delete a cell", we first pick the cell and click on "Delete Cells" menu item.
+
== Save/Load Results ==
Alternatively, we can press "D" after clicking on the cell. In the following example, we want to delete the cell with id-13. As soon as this cell is deleted all the views will be updated accordingly. In this example, the cell 13 and all the corresponding data presented in that row will be deleted.
+
  
[[Image:editing1-delete.JPG]]
+
A segmentation has multiple components, each can be saved separately or as a complete project.
 +
Image: This is the input image. It may be important to save this if preprocessing.
 +
Result: This is the result image.  It is stored as a label image
 +
Table: This is the information that is in the table (features).
 +
Project: Creates a project file (XML) that contains the filenames of all of the project files, including the ones named above.
  
=== Merge Cells ===
+
A convenience function is also available for saving the currently displayed image (image + result + boundaries + ids + centroids + etc).
 +
The image saved is 2D.
  
Merging Cells requires the user to select two neighboring cells first. After clicking on the "Merge Cells" menu item, the cells will be merged and the new feature values will be computed and added into the table with a new id. The feature values associated with the old cells will be removed. In the following example, we merge the cells with id 18 and 37. The last id that we have in the table is 54. After the merge operation, the new cell will have id 55 in the table.
+
== KPLS Classifier ==
  
Similar to the "Delete Cell" operation, the user can use the keyboard and press "M" to merge the cells. Note that if the selected cells are not neighbors, they will not be merged.
+
After the image has been segmented and the edits have been made, the KPLS classifier can be used to classify the cells in the image. For classifying the cells, the user needs to provide a reliable set of training samples. This can be done through the following actions
  
[[Image:editing2-merge.JPG]]
+
=== Train ===
 +
In order to train the classifier we select the "Train" Option under Tools->Classifer menu. this opens the '''Training Dialog'''.
 +
We can either save the set of training samples ( called the '''model''') or simply specify the sample IDs for the number of classes present and test the classifier's performance for this training set. 
 +
 +
[[Image:Train.JPG|frame|center|'''Train Menu ''']]
  
=== Splitting (Cells) ===
+
'''Training Set''' : We can enter the IDs that best represent each class. Clicking the '''Add Class''' button adds another row to the dialog where the ids for the next class can be entered. Clicking the '''Remove Class''' button removes the last class added and any ids entered for the class. After all the IDs have been added for the classes, we click the '''Done''' button to complete the training.
  
Cells can be split after we are in the splitting mode. The splitting menu has a submenu that has two items:
+
'''Note''': The IDs can be overlayed onto the nuclei by selecting the '''Show Object Centroids''' item in the '''View''' menu.
* Start Splitting
+
* End Splitting
+
  
We start splitting operation with "Start Splitting" submenu item and select two seed points inside the cell. When we click on points in a cell these points will be colored with red. Once the seed points are ready we can split the selected cell by clicking on "End Splitting".
+
[[Image:Training_Set.jpg|thumb|800px|center|'''Train''']]
  
In the following example, We first choose "Start Splitting" to split the cell with id-1.
+
We can also save the set of training samples as a model to reuse it for similar images. Clicking the Save Model opens a pop up and we can save the model as an XML file in the location of our preference. The Training Dialog box will close automatically after the model is saved.  
  
[[Image:editing3-split0.JPG]]
+
'''Training by Loading a Model''': We can also train the classifier by loading a model. This can be done in the Training Dialog by clicking the '''Load Model'''. This opens a popup which can be used to select the file containing the model which was previously saved. We need to make sure that this model is applicable to the image currently being processed. The Training Dialog box will close automatically after the model is loaded.  
  
We first zoom in using the mouse and click on two seed points.  
+
After the training process, an additional column called '''train''' is added to the table. All the IDs which have not been used for training will have the value -1 for this column and the IDs used for training will have the respective class value.
  
[[Image:editing3-split1.JPG]]
 
  
After splitting the cell with id-1, the split cell will be removed and two new cells (id-56 and 57) with new features will be added into the table and other views.
+
=== Classify ===
  
[[Image:editing3-split2.JPG]]
+
In order to classify the objects we select the "Classify" Option under Tools->Classifier menu. This opens the '''Pattern Analysis Wizard'''. We can select the set of features that we want to be used to classify the objects from the menu.  
  
== Logging Editing History ==
+
[[Image:Pattern Analysis Wizard.jpg|frame|center|'''Pattern Analysis Wizard''']]
  
A typical XML result file consists of a set of objects whose structure is given in the following figure. Each object is defined with a set of features. In the following result file, an object corresponds to a cell nuclei which is identified by a unique id. A cell can be either valid or invalid depending on the editing operation. Initially all cells are valid and will be loaded. If a "delete" operation is applied to the cell , it will no longer be valid. But we still keep this cell data in the result file for logging purposes.
+
Clicking Finish will classify the objects in the image . After the training process, an additional column called "prediction" is added to the table. This column will contain the prediction made by the KPLS classifier based on the features selected for classification.
  
[[Image:farsight-object.JPG]]
+
To visualize the classification results from the image displayed, we can select the '''View Object Centroids''' from the view menu after the classification. This displays the centroids of different classes in different colors.
  
Every editing operation (delete, merge, or split) is recorded by creating an editing history log for each cell in the result file. We define an <EditHistory> element for each cell and list the editing operations with a proper timestamp and cell ids. There are three cases:
+
== Keyboard Shortcuts ==
  
'''- Delete:''' We just note that this cell has been deleted (valid=0)
+
Ctrl+L Load Project
 +
Ctrl+S Save Project
 +
Ctrl+Q Quit
  
[[Image:farsight-object-delete.JPG]]
+
Ctrl+B: Hide/show Boundaries
 +
Ctrl+I: Hide/Show Object IDs
  
'''- Merge :''' In this case, two cells (id1, and id2) will be merged and a new cell will be created. Therefore, we first delete the first two cells and update their edit histories with this information, and then we update the edit history of the new cell noting that it is created by merging the cells with id1 and id2.
+
Ctrl+C: Clear Selections
 +
Ctrl+A: Add Object (trigger command, then draw a box)
 +
Ctrl+M: Merge Objects (all selected objects will be merged selected neighbors)
 +
Ctrl+D: Delete Objects (removes all selected objects)
 +
Ctrl+F: Fill Objects
 +
Ctrl+T: Split Objects in Z direction at current Z
 +
Ctrl+P: Split X-Y (Select this, then click two points as seeds).
  
[[Image:farsight-object-merge.JPG]]
+
-/_  zoom out
 +
=/+ zoom in
  
'''- Split :''' We note that two new cells will be created (with their ids) after splitting the cell of interest. In addition, the edit histories of each new cell will be updated with the id of the cell they are splitted from.
+
Use the number keys to show/hide channels (starting with 0)
  
[[Image:farsight-object-split.JPG]]
+
== Summary ==
  
== Edit-Based Validation - Group Editing ==
+
An "edited" segmentation is also a [[Validation_Methods|"validated"]] segmentation, and the results can be used for hypothesis-driven research utilizing object features, associative object measurements, and the other tools that are part of the [[FARSIGHT Toolkit]].
  
Since automated segmentation algorithms are not perfect, there is a compelling need to develop efficient methods to identify and correct the automated segmentation errors. Due to the large size of the datasets, the identification and correction of the cells to be edited have to be done very fast. In order to achieve this NucleusEditor implements kernel based pattern analysis algorithms where we can do group or cluster editing of multiple errors simultaneously. This dramatically reduces the amount of manual effort required compared to unassisted edit-based methods. We term this methodology PACE (Pattern Assisted Cluster Editing). By observing the nature of the errors and analyzing them, it can be possible to asses the performance of the algorithms used in NucleusEditor for this purpose and modify them to prevent such errors in the future. We explain ALISA and Group Editing in more detail at [[EVS|EVS]] and [[Validation_Methods|Validation Methods]].
 
  
 +
See the [[getting_there|"getting there"]] page for a suggested editing/validation procedure.
  
Once the corrected segmentation results are obtained with this way, hypothesis-driven and discovery driven research can be conducted by utilizing object features, associative object measurements, and the other tools that are part of the [[FARSIGHT Toolkit|FARSIGHT_Toolkit]].
 
  
 +
Contributors to the Nucleus Editor:
 +
*Isaac Abbott
 +
*Yousef Al-Kofahi
 +
*Aytekin Vargun
 +
*Raghav Padmanabhan
  
This page was prepared by Aytekin Vargun (with Isaac Abbott's contributions)
+
This page was prepared by Isaac Abbott & Aytekin Vargun

Latest revision as of 16:11, 21 February 2012

The main purpose of the NucleusEditor program is to view and segment images of nuclei, and then efficiently edit the segmentation using pattern analysis tools (PACE). An auxiliary function of the system includes segmentation of 2D Histopathology images.

Contents

Image Viewer

NucleusEditor allows users to view image data in a variety of file formats (tif, pic, lsm, etc). The image must be in 8-bit grayscale and must have "bright" foreground objects under a "dark" background to segment properly. An image may be opened by selecting File/Load Image...

Load Image Example

XML Images

Often multi-channel images are divided into multiple files; one file for each channel. We have defined an XML file format for specifying this type of image. Each file can be given a channel name and a channel color.

XML Image Definition

Segment Cell Nuclei

To begin segmentation, an image must already be loaded and visible in NucleusEditor. The nuclear segmentation algorithm is triggered by selecting Tools / Segment Nuclei... The program will use the visible image for the segmentation algorithm, and ask the user to select the desired channel from within that image. Before the segmentation algorithm can begin it must be initialized with a few parameters. These parameters will be chosen automatically using an innovative algorithm developed by Yousef Al-Kofahi. When the segmentation begins, an activity toolbar will be added to the NucleusEditor that shows the progress of the segmentation algorithm and provides a STOP button to cancel the segmentation.

Nuclear Segmentation is achieved in 2 major steps:

  • 1. Segmentation
  • 2. Editing
Segment3.JPG

Upon completion of the segmentation, the program will compute features and show this metadata (mainly object features) using multiple data visualization tools that are all actively linked, allowing the data to be viewed in multiple spaces simultaneously. There is a method to select a cluster of objects based on any combination of operations in any of the multiple spaces. We term this method actively linked multiple spaces architecture (ALISA). NucleusEditor provides this linked-view architecture where objects can be visualized in image or geometric views, table views, plot views, and histogram views. Operations performed in one view are immediately visible in all of the views.

In the ALISA mode a number of operations become active in each view:

  • Segmentation Result
    • Show/Hide Object Boundaries (View|Show Boundaries or Ctrl+B)
    • Show/Hide Object ID Numbers (View|Show Object IDs or Ctrl+I)
    • Show/Hide Object Centroids (View|Show Object Centroids)
    • Add Table (View|New Table)
    • Add Scatter Plot (View|New Scatter)
    • Add Histogram (View|New Histogram)
    • Modify gray-scale image intensity (View|Adjust Image Intensity or Ctrl+G)
    • Clear Selections (Editing|Clear Selections or Ctrl+C)
    • Edit Functions (Edit Menu - see below)
    • Apply an Exclusion Margin to the image (Editing|Apply Exclusion Margin...)
    • Zoom using + and - keys
    • Select/Deselect Objects (Hold Ctrl key to select more than one object)
  • Scatter Plots
    • Select/Deselect Objects
      • Select/deselect by clicking on single object
      • Clear all selections (Options|Clear Selections or Ctrl+C)
      • Select a group of objects by holding down the SHIFT key, and clicking on a series of points to sketch out a polygon that acts as a "lasso".
    • Options|Normalize - Ctrl+N (show features in normalized space)
    • Tools|Find Outliers (ALISA - see below)
    • Options (can change columns for X,Y and coloring)
  • Table
    • View|Filter table by... (To select items meeting specific criteria)
    • Show/Hide table columns (View|Visible Columns)
Final Result Example

Image Processing Pipeline

In many cases the automatic nuclear segmentation parameters will require modifications for satisfactory results. This is possible by using the project processor capability within the Nucleus Editor. This function uses an XML definition file as input. This definition file can specify required process steps and parameters for each step. For example, it can be used to limit the number of features that are calculated and specify that certain associative features are desired. See an example of one of these files below. If you have specific needs please contact the Roysam lab for assistance in creating a process file. Once the file has been created, execute the pipeline by first loading the image, then choosing "File|Process image..."

Project Processor Definition File

Using the PACE Concept

Since automated segmentation algorithms are not perfect, there is a compelling need to develop efficient methods to identify and correct the automated segmentation errors. Due to the large size of the datasets, the identification and correction of the cells to be edited have to be done very fast. In order to achieve this, NucleusEditor implements kernel based pattern analysis algorithms where we can do group or cluster editing of multiple errors simultaneously. This dramatically reduces the amount of manual effort required compared to unassisted edit-based methods. We term this methodology PACE (Pattern Assisted Cluster Editing). By observing the nature of the errors, it will be possible to asses the performance of the segmentation algorithm used in NucleusEditor and modify it to reduce similar errors in the future.

ALISA is designed to allow group editing of objects. The idea is for the user to select a group of objects based on their features. This can be done manually, or using automated pattern analysis algorithms. Manual selection of objects is aided by scatter plots of features. The user can choose the features displayed along the x and y axes of scatter plots. To visualize more than 2 features at a time, ALISA allows the user to display an unlimited number of scatter plots. Each point on the scatter plot corresponds to a specific object in the image (with a unique identifier (ID)). Clicking the mouse on a point selects and highlights the corresponding object in all other views. The user can also draw a polygon on a scatter plot to select a group of objects visually.

Computer-assisted selection of objects can also be performed. Features that highlight specific types of errors in segmentation (we call them diagnostic features) can also be defined and these objects selected and edited in their feature-space. Outlier detection tools are provided to assist in finding unusual objects to be inspected and/or edited. Outliers may correspond to segmentation errors (typical) or unusual cells (rare but possible). For outlier detection, the one-class Support Vector Machine (SVM) is employed [libSVM]. The one-class SVM proves to be a very useful tool for outlier detection, often highlighting a variety of quirky objects that should be inspected more closely, and edited when necessary. The outliers are shown in the segmentation, and scatter views, by a simple color coding scheme. The user can quickly locate, inspect and edit the outliers due to the linked nature of the views.

Clues

Two simple features have been shown to be particularly useful in detecting segmentation errors: volume and percent shared boundary. The following edit operations tend to correlate to these features in this way:

  • Delete: small volume & zero (or small) percent shared boundary
  • Merge: small volume & large percent shared boundary
  • Split: high volume &| high surface area

Methods

Two main techniques can be used to locate objects that need to be edited: filters and outlier detection.

  • Filters: The table view provides two very useful functions: View|Visible Columns and View|Filters. The NucleusEditor provides a large wealth of feature data, most of which is noise to the editing process. Change the Visible Columns to only include those features that are useful while editing (ex. id, volume, shared boundary, surface area). Next we may want to select all of the objects with very small volume. On the scatter plot it is very hard to select just one of these objects and view it because they are too tightly clustered together. By filtering the table by volume, it becomes very easy to find all of the objects with small volumes.
    Filtering


  • Finding Outliers: The Nucleus Editor provides a tool for finding outliers in a set of data. This tool can be activated by selecting Tools|Detect Outliers. The outlier tool uses the one-class SVM to find outliers. The outlier tool seems to be most effective when only 2 or 3 features are used. If the outliers are not immediately color-coded in the scatter plot it could be because the incorrect column for coloring is selected, please select the "outlier?" column by choosing Options|Set Color Column. From the scatter plot it is now easy to find sub-populations of objects that meet specific criteria (ex. small volume & high shared boundary - possible merge candidates).
    Finding Outliers

Edits

At this stage in the process sub-populations of objects requiring specific edits should be identified, either by sorting the table or by finding regions (outliers) in the scatter plot. NucleusEditor provides the following three operations to edit cells, which will become active when a valid segmentation result is loaded:

  • Delete
  • Merge
  • Split
  • Add
  • Fill

The goal is to implement these operations using smart algorithms that allow them to be carried out in groups. These "smart algorithms" are still in development. The idea is to identify a large group of objects that need to be merged (for example), and simply tell the program to merge them. This could mean merging some of the selected objects with other selected objects, or it could mean merging a selected object with another, non-selected, object. Similarly, the split function should attempt to split an object into the most optimal number of objects with reasonable boundaries. As soon as objects are edited all views will be updated (however you will need to re-run the outlier detector to update outliers). Be sure to select File|Save Result to save results and any edits that have been performed.

Add

Adding new object requires detecting new foreground pixels from the existing background. Start the add operation by selected "Editing|Add Objects" from the menu or by "Ctrl+A". Draw a box around the new object in the middle z-slice of that object by clicking the left mouse button.
Add new object

Delete

In order to delete objects we simply select them in any one of the available views and choose the "Editing|Delete Objects" menu item within the Nuclear Segmentation Window or by "Ctrl+D". In the following example all cells with small volume and zero shared boundary will be deleted, they have first been selected in the table view.

Merge

Merging objects requires the selection of two or more objects. Only neighboring objects will be merged to form a new object. Merging may be initiated from the "Editing|Merge Cells" menu item or by "Ctrl+M". Note that if the selected cells are not neighbors, they will not be merged. In this example we have merged object 394 and 396 to become a new object: 504.

Split

Splitting objects is achieved by specifying two or more seed points. Start the splitting operation by selecting "Editing|Split Objects X-Y" from the menus. Left-click within the objects to place 2 seed points. Objects will be split as soon as two seed points have been specified. Choose "Editing|Split Objects X-Y" again from the menu to exit splitting mode.

Objects may also also be split at a specific z-plane. This is accomplished by Choosing "Editing|Split Objects At Z. All selected objects will be split at the current z slice.

Fill

Sometimes segmented objects can be made of up multiple (non-touching) connected components, or they may have holes in them. These errors can be fixed using the "Edit|Fill Objects" menu item.

Edit History Log

When edits are made to a nuclear segmentation, an entry is made to a log file. This log file is stored in the same location as the input image.

Save/Load Results

A segmentation has multiple components, each can be saved separately or as a complete project.

Image: This is the input image. It may be important to save this if preprocessing.
Result: This is the result image.  It is stored as a label image
Table: This is the information that is in the table (features).
Project: Creates a project file (XML) that contains the filenames of all of the project files, including the ones named above.

A convenience function is also available for saving the currently displayed image (image + result + boundaries + ids + centroids + etc). The image saved is 2D.

KPLS Classifier

After the image has been segmented and the edits have been made, the KPLS classifier can be used to classify the cells in the image. For classifying the cells, the user needs to provide a reliable set of training samples. This can be done through the following actions

Train

In order to train the classifier we select the "Train" Option under Tools->Classifer menu. this opens the Training Dialog. We can either save the set of training samples ( called the model) or simply specify the sample IDs for the number of classes present and test the classifier's performance for this training set.

Train Menu

Training Set : We can enter the IDs that best represent each class. Clicking the Add Class button adds another row to the dialog where the ids for the next class can be entered. Clicking the Remove Class button removes the last class added and any ids entered for the class. After all the IDs have been added for the classes, we click the Done button to complete the training.

Note: The IDs can be overlayed onto the nuclei by selecting the Show Object Centroids item in the View menu.

Train

We can also save the set of training samples as a model to reuse it for similar images. Clicking the Save Model opens a pop up and we can save the model as an XML file in the location of our preference. The Training Dialog box will close automatically after the model is saved.

Training by Loading a Model: We can also train the classifier by loading a model. This can be done in the Training Dialog by clicking the Load Model. This opens a popup which can be used to select the file containing the model which was previously saved. We need to make sure that this model is applicable to the image currently being processed. The Training Dialog box will close automatically after the model is loaded.

After the training process, an additional column called train is added to the table. All the IDs which have not been used for training will have the value -1 for this column and the IDs used for training will have the respective class value.


Classify

In order to classify the objects we select the "Classify" Option under Tools->Classifier menu. This opens the Pattern Analysis Wizard. We can select the set of features that we want to be used to classify the objects from the menu.

Pattern Analysis Wizard

Clicking Finish will classify the objects in the image . After the training process, an additional column called "prediction" is added to the table. This column will contain the prediction made by the KPLS classifier based on the features selected for classification.

To visualize the classification results from the image displayed, we can select the View Object Centroids from the view menu after the classification. This displays the centroids of different classes in different colors.

Keyboard Shortcuts

Ctrl+L Load Project
Ctrl+S Save Project
Ctrl+Q Quit
Ctrl+B: Hide/show Boundaries
Ctrl+I: Hide/Show Object IDs
Ctrl+C: Clear Selections
Ctrl+A: Add Object (trigger command, then draw a box)
Ctrl+M: Merge Objects (all selected objects will be merged selected neighbors)
Ctrl+D: Delete Objects (removes all selected objects)
Ctrl+F: Fill Objects
Ctrl+T: Split Objects in Z direction at current Z
Ctrl+P: Split X-Y (Select this, then click two points as seeds).
-/_  zoom out
=/+ zoom in

Use the number keys to show/hide channels (starting with 0)

Summary

An "edited" segmentation is also a "validated" segmentation, and the results can be used for hypothesis-driven research utilizing object features, associative object measurements, and the other tools that are part of the FARSIGHT Toolkit.


See the "getting there" page for a suggested editing/validation procedure.


Contributors to the Nucleus Editor:

  • Isaac Abbott
  • Yousef Al-Kofahi
  • Aytekin Vargun
  • Raghav Padmanabhan

This page was prepared by Isaac Abbott & Aytekin Vargun

Personal tools