An experimental study of pathologist's navigation patterns in virtual microscopy
© Roa-Peña et al; licensee BioMed Central Ltd. 2010
Received: 1 September 2010
Accepted: 18 November 2010
Published: 18 November 2010
In virtual microscopy, a sequential process of captures of microscopical fields, allows to construct a virtual slide which is visualized using a specialized software, called the virtual microscopy viewer. This tool allows useful exploration of images, composed of thousands of microscopical fields of view at different levels of magnification, emulating an actual microscopical examination. The aim of this study was to establish the main pathologist's navigation patterns when exploring virtual microscopy slides, using a graphical user interface, adapted to the pathologist's workflow. Four pathologists with a similar level of experience, graduated from the same pathology program, navigated six virtual slides. Different issues were evaluated, namely, the percentage of common visited image regions, the time spent at each and its coincidence level, that is to say, the region of interest location. In addition, navigation patterns were also assessed, i.e., mouse movement velocities and linearity of the diagnostic paths. Results suggest that regions of interest are determined by a complex combination of the visited area, the time spent at each visit and the coincidence level among pathologists. Additionally, linear trajectories and particular velocity patterns were found for the registered diagnostic paths.
A very recent field, known as virtual microscopy, has made possible digital exploration of histological slides, archiving of these slides for later analysis and easy access to this information. A slide digitization is called a virtual slide (VS) and is constructed from sequential captures of microscopical fields . These virtual slides are high resolution images whose visualization requires a specialized software, called the virtual microscopy viewer, a specific tool devised for running over images composed of thousands of microscopical fields of view.
Complex mechanisms are involved in the genesis of the navigation patterns, guided by the training time and triggered by the contents of the slide . Coarsely, diagnosis in pathology can be considered as a process composed of four sequential steps: look, see, recognize and understand . A definitive diagnosis is achieved by following a standard methodology with two coarse phases: first examination is carried out at the lower magnification (panoramic) in order to locate relevant information in terms of a spatial organization of the histological sample (scanning), while the second and further examination is conducted for analysis of the slide contents which implies changing the magnification (zoom) [3–5]. This analysis is performed through navigation of the zoomed areas , on which gentle movements are generally required. This learned strategy has been observed in multiple studies in which it has been possible to determine the existence of these two phases. Tsuchihashi et al. studied one pathologist exploring twenty different slides in telepathology. This investigation identified two patterns: exploration at low magnification and analysis at higher magnifications . Crowley et al. recorded on videotape diagnoses performed on four histological slides by fifteen pathologists, distributed into three different categories: novices, intermediates and experts . Results showed that intermediates and experts exhibited very similar patterns to the two described before, i.e., a general search strategy and selection of areas to revisit them at higher power. Tiersma et al. investigated visual exploration patterns in pathology using an eye tracker mechanism.
Results showed two patterns: scanning (saccadic eye movements) and selection (eye fixation over specific points for further exploration) . Finally, Krupinski et al., using an eye tracker system in a group of pathologists, demonstrated that visual exploration is characterized by a rapid determination of ROIs, which likely contain diagnostic information . It is worthy to strengthen out that the experimental setup of that study allowed only scanning patterns, i.e., magnification changes were not available.
Overall, the experience with images in virtual microscopy should be very similar to an actual optical examination. Therefore, design of friendly and useful graphical user interfaces (GUIs) is a fundamental issue. Indeed, navigation patterns arise from two intermixed processes: the motor control associated to some movement automations and a refined search information process, which reflects the level of expertise. Yet patterns may be different, the more expert is the group of pathologist the more similar are the locations they visit when exploring a histological slide . It is important to keep in mind that the virtual microscopy tool does not imitate exactly a conventional light microscope, but rather it has the purpose of allowing a pathologist to navigate at any resolution, while the VS is always available at the lowest resolution, that is to say, the expert always conserves a thorough panorama of the VS.
This study aims to determine the main factors involved in the genesis of the navigation patterns from a particular diagnostic path. These patterns are the result of the interaction between the image contents and the expert experience. Other studies have focused before on studying either the attentional mechanisms that guide trajectories in this type of images, or the general mechanisms at the very base of the interaction of an expert with an image. On the contrary, our study integrates these two visions and introduces new elements as the magnification changes. Finally, as far as we know, this is the first study which actually evaluates how a pathologist moves the stage, in other words, we dedicated our endeavors to figure out the influence of both the image contents and the pathologist methodology in an actual interaction context.
A total of four randomly selected histological specimens were digitized and six VS were assembled, using an acquisition system composed of a Carl Zeiss Axiostar Plus microscope, a Sony high resolution digital video camera Handycam DCR-HC85 (640 × 480 pixels) and two Carl Zeiss adapters: 426126 and 456006 (Carl Zeiss, Light Microscopy, Gottingen, Germany). Hematoxylin-eosin tissue samples from endomyometrium, gallbladder, prostate and a uterus leiomyoma were used for this study.
Histopathological slides were selected from a set of routine cases from the Pathology Department at the National University of Colombia, a laboratory of medium complexity. These samples were selected by an expert pathologist. The endomyometrium and the leyomioma were obtained from a total hysterectomy of a patient with abnormal uterine hemorrhage due to multiple leyomiomas; the endomyometrium had only focal epithelial componet and the leyomyoma was basically composed of smooth muscular tissue with quite homogeneous distribution. The gallbladder specimen was obtained from a colecistectomy and the prostate sample was obtained by transrectal biopsy, case in which the tissue is characterized by the presence of glands supported by stroma. Finally, the uterus leiomyoma, obtained from a myomectomy, is basically composed of a muscular tissue whose distribution is quite homogeneous. They were digitized and six different images were assembled. Sizes in pixels were 53280 × 39360, 42480 × 15840, 33840 × 21600, 53280 × 39360, 42480 × 15840 and 49680 × 28320 which stand for an effective area of 11.97 × 8.84, 9.54 × 3.56, 11.16 × 6.36, 11.97 × 8.84, 9.54 × 3.56, 11.16 × 6.36 mm2 (pixel size of 1.98 μm2), respectively. Mega-images were stitched using automatic registration with cross correlation as the similarity measure and were stored in JPEG2000 format for latter access and navigation .
Pathologist navigation patterns were recorded using a virtual microscope prototype whose GUI was adapted to the pathologist requirements . This design exploits the importance of low magnifications for exploration and analysis at high resolutions for diagnosis. The GUI is composed of a thumbnail and an auxiliary window. The former displays the lowest resolution thumbnail image, in which a rectangular re-sizable window allows a required selection. The thumbnail window is set to a desired size at the beginning, while the auxiliary window is constantly varying, according to the magnification level of the selected ROI in the thumbnail window. Displacements of a particular ROI were only allowed in the thumbnail window through drag and drop operations. Finally, for each requested ROI, its position, size, resolution and time were recorded for later analysis.
The aim of the present study was to compare navigation patterns when expert pathologists are exploring virtual slides. Four expert pathologists participated in this study, all of them had similar years of experience (about five years), and were graduated from the same pathology school program. Each pathologist was previously trained on virtual microscopy using two test virtual slides. Average, the training time was 20 minutes so that at the end of this time they were free to navigate the slides at will, until they could reach a probable diagnosis and organ identification. Six virtual slides were chosen, as parts of full histological slides, with a relative size which varied between 10% to a 30% of the whole histological sample. An experienced pathologist, with at least five years of experience, selected the digitized area. For the sake of the experiment, images shown to the pathologist belonged to areas in which it was difficult to determine both the organ and the pathological entity so they were forced to spend more time, exploiting the navigation tool .
The six VS were randomly displayed for each of the pathologists so that the examination order was always different. Each pathologist was asked to run over the virtual slide, up to a diagnosis was set, using the same screen monitor they use in their routine computer work (CCFL (220 nits) WXGA (1280 × 800 of 13, 3")). During examinations, every pathologist action was recorded for later analysis, namely, we recorded the ROI location, the ROI size related to the thumbnail image, the time any action (drop or drag) was carried out and the magnification level.
The percentage of visited area was calculated for the group of pathologists and the whole set of VS.
The percentage of coincidence among the different visited areas, i.e., at least two pathologists examined the same image region within the navigation.
For each pathologist, the time spent per region was also computed.
Coincidence among ROIs, defined as the common visited areas and longer visits was calculated.
Mouse movement velocities were registered and analyzed.
Linearity of trajectories: Euclidian distance was computed and compared against the actual trajectory distance, i.e., a ratio between the two distances was calculated.
Regions of Interests
As previously discussed, we suppose that ROIs are defined by a combination of the three criteria presented hereafter:
Percentage of visited area
Percentage of visited area for the group of pathologists in the whole set of VSs.
Percentage of visited area per image
Pathologists were forced to further explore the image, attempting a maximum level of information, but in general this was hardly established since information was not enough as to consolidate a diagnosis. Results support this statement since the percentage of explored image was larger than a 50%.
Percentage of Coincidence
Influence of the image contents in the navigation pattern observed from our experiments was assessed by measuring the coincidence level in the visited regions, namely, the area percentage which was visited by more of one pathologist.
Percentage of coincidence in visited areas
As expected, samples where information is located show higher levels of coincidence. The first virtual slide corresponds to prostatic tissue, in which there is a large number of glands, so pathologists dedicated most of the navigation exploring them: the coincidence level is high (larger than 50%). The second virtual slide corresponds to an endomyometrial sample, its epithelial component is quite located and constitutes a very small area of the sample so the coincidence level is also high. The third virtual slide corresponds to a fragment of a leiomyoma with a predominant stromal component. The areas visited by the pathologists corresponded to structures with a luminal space, seen at the low magnification. Interestingly, as it was not clear if they corresponded to glands or vessels, the resultant coincidence level was again high. The fourth virtual slide was clearly a leiomyoma and yet the visited area was large (86%), the coincidence level was only 69%. The fifth virtual slide was a gallbladder, the epithelial component is minimum but scattered and then the area to explore large, the coincidence level was smaller than the other images. Finally, the sixth virtual slide corresponded to a prostate, in which the epithelial component was sparse and hence also the coincidence level and the area to explore.
A potential ROI could arise either when every pathologist stops at particular image location and therefore information therein is relevant, or when the pathologist spans a longer period in a precise area so that even if the interest in the region is not shared among the group of experts, there exists a potential source of knowledge. In consequence, we also evaluated the time every pathologist required for examining regions as the total time of visit per pixel in the thumbnail window, which was estimated by accumulating the set of visit times and computing their average. A ROI (in the time sense) was then defined as the region composed of those pixels for which this quantity was larger than the mean.
Coincidence in ROIs
One important question we addressed consisted in determining, whether or not the ROIs defined by spatial preferences or time, would coincide. A standard measure of the degree of intersection between regions was used: the Jaccard coefficient, a measurement of the similarity between sample sets, defined as the intersection divided by the union of the sample sets. This coefficient has a maximum value of 1 when there is total agreement and zero when there is none.
Figure 6 shows different navigations over the same virtual slide. In all of them the navigation is composed of a scanning phase and zooms, at different ROIs. Interestingly, three pathologists used the default window (a, b, c) and the fourth one enlarged the window to cover a maximum area.
Path linearity among ROIs
The average coefficient of linearity is herein defined as the distance between two ROIs divided by the actual run distance. This coefficient has a maximum value of 1 when there is a linear trajectory and lower when is not linear.
The present investigation was addressed to determine main pathologist's navigation patterns when using virtual microscopy slides, using a GUI adapted to the pathologist's workflow. Four pathologists with a similar level of experience, graduated from the same pathology program, navigated six virtual slides. Provided that a microscopical navigation is an interactive process, this study was devised to establish the relative importance of both image contents and navigation patterns of pathologists with high degree of expertise. A contribution of this work is that our GUI allowed to study not only scanning patterns, as described in the literature , but also magnification changes, a scenario really close to what pathologists are doing in their daily routine. Likewise, we explored the concept of ROI from different perspectives, either by analyzing the number of visits to a particular VS location as described in other previous investigations , and more importantly, by taking into account two new issues, i.e., the time a pathologist dedicates to explore this particular location and the coincidence level among pathologists, determinant factors which have not been evaluated so far. In addition, we also assessed the different paths described during these navigations, namely, mouse movement velocities and linearity of the diagnostic paths.
In previous studies, ROIs have been defined as areas to which the examiner is rapidly attracted to look and may contain diagnostic information , where visual stimulus is analyzed in detail  or by regions that are prioritized by the image content (first criterion) . However, such definitions are still incomplete becasue a particular examiner can be rapidly drawn to regions which require a further exploration for classifying the type of information. Such pattern can be easily observed in difficult cases in which relevant information is hidden or information associated to the case is not enough as to establish an objective judgment. This leads us to acknowledge that the first path approximation is insufficient and that other criteria should be included. Classical psychophysical theory claims that the time spent in any particular task is directly related to the degree an examiner is familiar with a particular pattern . We decided then to include the time spent at examining, as an evaluation criterion (second criterion) of what a ROI means in pathology, because the common pathology workflow consists in developing technical skills as to search abnormalities, a very harsh picture in many pathologies. In the present investigation, this factor resulted crucial since our GUI permitted to zoom in at any of the available preparations and so we could evaluate the importance of a particular image locus, not only because the examiner stopped there, but basically because information was valuable and required actual analysis. This fact could be established because we could compare the time spent at any of these image loci. Finally, another analysis direction (third criterion) could be the coincidence area visited during navigations, a factor which can be much more objectively included as a criterion, even though its inclusion in clinical routine is very difficult because the number of pathologists examining the same slide is very rarely larger than two. Overall, our results have supported the importance of simultaneously taking into account the three issues, mentioned before, as the base of an actual ROI definition because: (a) Pathologists are effectively attracted by some regions, as inferred from Table 1, with percentages of visited areas from 44% to 91%, a variable figure which directly depends on the diagnostic difficulty. Recall that the present study was devised for studying navigation patterns and therefore information related to the case was not available, this factor did increase the navigation time but even in these hard conditions, pathologists did not need to visit the entire virtual slide. Images associated to larger visited areas are consistent with the ones in which it was more difficult to determine the organ and/or the pathologic entity. (b) The interest for particular image loci is shared by most of the pathologists, a claim inferred from a coincidence level nearby to 70.5%, as observed in Table 2. Finally, we found that the time spent at examining each of these regions was at least a 50% of their navigation time, on specific regions previously defined, whereby some of the original regions were ruled out when considering the time factor. This evidence suggests that none of these factors could be considered as the base for defining the ROI and probably complex combinations of them are required for specific applications.
Some recent works have approached the problem of automatic ROI determination in histopathological images [11–15]. Most of these works have been devoted to segment cancer, for instance, breast tumors and colon biopsy images [13, 15], since in these images the relevant regions are basically those containing abnormal architectural patterns. Nevertheless, the automatic ROI determination is still a challenging problem in many other cases in which relevant information is part of the normal patterns . The problem of determining ROIs is a complex combinations of different issues, among others, the high variability at the level of the low image features (architecture, texture and color) , artifacts at any level of the pathology workflow chain (sampling, sections, dyes, microscopes) , and finally, the semantic gap between the low level image representation and the concepts formulated by a pathologists during the diagnosis process . The present work proposed an alternative way to setting ROIs at taking advantage of the previously recorded navigations, i.e., information about visited area and times. There are two main applications for which determining ROIs can be fundamental: telepathology and training. In the former case, determining ROIs may accelerate the process of sending information to the specialist while in the latter, educational environments may use VS prototypes for training specialists . In the case of educational applications, slides are previously navigated by a set of expert pathologists and the strategies herein proposed could be used to automatically select ROIs, without additional image processing overload. Moreover, the mouse patterns can also be used to speeding up navigations, using strategies such as cache and prefetching . Finally, patterns generated during actual navigation could be used as training samples to devise automatic navigation strategies, based on machine learning .
Classically, exploration patterns in medical imaging have been studied by both tracking the visual system or analyzing the diagnostic path complexity . However, due to the restrictions of visual systems for acquiring and interpreting high resolution images and the limited display capabilities of the computational resources , new alternatives for exploring virtual slides have been recently proposed . In actual virtual microscopy scenarios, any diagnostic task requires the use of a device (a kind of joystick), mainly mouse devices, to point out relevant information over a low resolution version of the VS. The kind of patterns generated by the interaction of an expert and a virtual slide is obviously related to the type of interface. Overall, navigation patterns in virtual microscopy have been recently studied , and the few reported studies use very complicated interfaces such as eye trackers, which may bias the observed patterns. As far as we know, there exists only another study that has recorded diagnostic paths during actual navigations  but its analysis is totally different since therein, authors do not analyze velocities, and time is not included as a criterion at defining what a ROI is. Such study is focused on exploring prefetching and caching as possibilities to reduce navigation and transfer times. This study reported partial image covering, as observed in the present investigation, and an average linearity coefficient of 0.41 for the complete diagnostic path. Interestingly, they also found a 0.85 linearity coefficient when the analysis was carried out on three consecutive steps of the diagnostic path. On the contrary, we assessed linearity for every single step that is part of the diagnostic path, when using virtual slides in a conventional laptop and mostly under the scanning phase. Our results showed a 0.8 linearity coefficient, a quite coherent figure when comparing with the study mentioned before. In addition, we identified velocity patterns in scanning tasks, consisting in a rapid increase of velocity when pathologists leave the ROI and a decreasing profile velocity when the expert is nearby to a new ROI.
So far an optimal GUI design in virtual microscopy is still an open problem. Many virtual microscopes try to emulate the experience of navigating a real microscope, that is to say, to move a microscopical stage while zooming in and out. The simpler exploration strategy consists in using a unique window, which stands for the microscopical objective and provides interactions with a virtual stage by means of mouse panning, while the zoom operations are simulated by clicking. This strategy is of course closely related to a real microscope exploration, however if an expert might leap between two high magnification regions, that expert must zoom out from the first region, displaces the field of view to the second region and then zoom in. This pattern constitutes a natural movement with any actual microscope, but for a virtual device, it ignores the main display capacities of a virtual interface. This complex set of operations can be drastically reduced by taking advantage of both digital storage and display potentialities. The GUI herein presented approached this problem by "focus & context" , a set of techniques that combines a "focus view" (the auxiliary window), i.e., the GUI is partially charged of displaying the high degree of detail, and a "context view" (thumbnail window) is charged of presenting the VS at low resolution. A fundamental hypothesis in the present work for the presented design is that an expert filters information out using the "context view" and then switches to the "focus view", on which the process of information refinement and diagnostic, is achieved. Therefore, this focus view occupies most of the available area, while the smaller part for the context provides orientation during interaction. For the sake of interaction, this focus view is placed within the context view, allowing a scanning-like display as well as an additional view of the whole contents. This design compensates many disadvantages of ordinary scanning because every interaction can then be executed using only the content view so that leaps between high magnification regions become simple displacements of this focus view within the context view.
The authors would like to thank The pathology Department of the National University, Doctors Yobany Sánchez, Jinneth Acosta and Viviana Arias who helped us with their experience in the navigations assessments. This work was partially supported by the Colombian Institute for the Advancement of Science and Technology (COLCIENCIAS), Grant no. 1101-487-26000.
- Afework A, Beynon M, Bustamante F, Demarzo A, Ferreira R, Miller R, Silberman M, Saltz J, Sussman A, Tsang H: Digital Dynamic Telepathology - the Virtual. AMIA Symp, Volume CS-TR-3892. 1998, 912-6.
- Krupinski E, Tillack A, Richter L, Henderson J, Bhattacharyya A, Scott K, Graham A, Descour M, Davis J, Weinstein R: Eye-movement study and human performance using telepathology virtual slides implications for medical education and differences with experience. Human Pathology. 2006, 37 (12): 1543-1556. 10.1016/j.humpath.2006.08.024.View ArticlePubMed
- Bussolati G: Dissecting the pathologists brain: mental processes that lead to pathological diagnoses. Virchows Arch. 2006, 448 (6): 739-743. 10.1007/s00428-006-0164-y.View ArticlePubMed
- Tsuchihashi Y, Mazaki T, Nakasato K, Morishima M, Nagata H, Tofukuji I, Shirakata H, Naito K, Akasaka Y: The basic diagnostic approaches used in robotic still-image telepathology. Journal of Telemedicine and Telecare. 1999, 5 (S1): 115-117. 10.1258/1357633991932801.View Article
- Crowley R, Naus G, Stewart J, Friedman C: Development of visual diagnostic expertise in pathology an information processing study. Journal of the American Medical Informatics Association. 2003, 10: 30-51. 10.1197/jamia.M1123.View Article
- Tiersma E, Peters A, Mooij A, Fleuren G: Visualising scanning patterns of pathologists in the grading of cervical intraepithelial neoplasia. Journal of clinical pathology. 2003, 59 (9):
- Iregui M, Gómez F, Romero E: Strategies for efficient virtual microscopy in pathological samples using JPEG2000. Micron. 2007, 38: 700-713. 10.1016/j.micron.2007.04.008.View ArticlePubMed
- Privitera C, Stark L: Evaluating image processing algorithms that predict regions of interest. Pattern Recognition Letters. 1998, 19: 1037-1043. 10.1016/S0167-8655(98)00077-4.View Article
- Nguyen A, Chandran V, Sridharan S: Gaze tracking for region of interest coding in JPEG 2000. Image Communication. 2006, 21: 359-377.
- Fei-Fei L, Rullen RV, Koch C, Perona P: Why does natural scene categorization require little attention? Exploring attentional requirements for natural and synthetic stimuli. Visual Cognition. 2005, 12 (5): 893-924. 10.1080/13506280444000571.View Article
- Demir C, Yener B: Automated cancer diagnosis based on histopathological images: a systematic survey. Tech. rep., Rensselaer Polytechnic Institute, Department of Computer Science, TR-05-09. 2005
- Kayser K, Molnar B, Weinstein R: Virtual Microscopy: Fundamentals, Applications, Perspectives of Electronic Tissue-based Diagnosis. 2006, Berlin: VSV Interdisciplinary Medical Publishing
- Oger M, Belhomme P, Klossa J, Michels J, Elmoataz A: Automated region of interest retrieval and classification using spectral analysis. Diagnostic Pathology. 2008, 3 (Suppl 1): S17-10.1186/1746-1596-3-S1-S17.PubMed CentralView ArticlePubMed
- Kayser K, Schultz H, Goldmann T, Gortler J, Kayser G, Vollmer E: Theory of sampling and its application in tissue based diagnosis. Diagnostic Pathology. 2009, 4: 6-10.1186/1746-1596-4-6.PubMed CentralView ArticlePubMed
- Tosun A, Kandemir M, Cenk S, Gunduz-Demir C: Object-oriented texture analysis for the unsupervised segmentation of biopsy images for cancer detection. Pattern Recogn. 2009, 42 (6): 1104-1112. 10.1016/j.patcog.2008.07.007.View Article
- Gilbertson J, Yagi Y: Histology, imaging and new diagnostic work-flows in pathology. Diagnostic Pathology. 2008, 3 (Suppl 1): S14-10.1186/1746-1596-3-S1-S14.PubMed CentralView ArticlePubMed
- Pena G, Andrade-Filho J: How does a pathologist make a diagnosis?. Arch Pathol Lab Med. 2009, 133: 124-132.PubMed
- Fontelo P, DiNino E, Johansen K, Khan A, Ackerman M: Virtual Microscopy: Potential Applications in Medical Education and Telemedicine in Countries with Developing Economies. Proceedings of the 38th Hawaii International Conference on System Sciences. 2005
- Schrader T, Niepage S, Leuthold T, Saeger K, Schluns K, Hufnagl P, Kayser K, Dietel M: The diagnostic path, a useful visualisation tool in virtual microscopy. Diagnostic Pathology. 2006, 1: 40-10.1186/1746-1596-1-40.PubMed CentralView ArticlePubMed
- Williams D, Porter J, Yoon G, Guirao A, Hofer H, Chen L, Cox I, Macrae S: How Far Can We Extend the Limits of Human Vision?. 2004, Slack Incorporated, chap 3: 19-38.
- Wienert S, Beil M, Saeger K, Hufnagl P, Schrader T: Integration and acceleration of virtual microscopy as the key to successful implementation into the routine diagnostic process. Diagnostic Pathology. 2009, 4: 3-10.1186/1746-1596-4-3.PubMed CentralView ArticlePubMed
- Keahey T: The Generalized Detail-In-Context Problem. Proceedings Vis'98, IEEE Visualization. 1998, Research Triangle Park NC
This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.