Image model embeddings for digital pathology and drug development via self-supervised learning

ABSTRACTWhole slide images (WSIs) contain rich pathology information which can be used to diagnose cancer, characterize the tumour microenvironment (TME), assess patient prognosis, and provide insights into the likelihood of whether a patient may respond to a given treatment. However, since WSI availability is generally scarce during early stage clinical trials, the applicability of deep learning models to new and ongoing drug development in early stages is typically limited. WSIs available in public repositories, such as The Cancer Genome Atlas (TCGA), enable an unsupervised pretraining approach to help alleviate data scarcity. Pretrained models can also be utilised for a range of downstream applications such as automated annotation, quality control (QC), and similar image search.In this work we present DIME (Drug-development Image Model Embeddings), a pipeline for training image patch embeddings for WSIs via self-supervised learning. We compare inpainting and contrastive learning approaches for embedding training in the DIME pipeline, and demonstrate state-of-the-art performance at image patch clustering. In addition, we show that the resultant embeddings allow for training effective downstream patch classifiers with relatively few WSIs, and apply this to an AstraZeneca-sponsored phase III clinical trial. We also highlight the importance of effective colour normalisation for implementing histopathology analysis pipelines, regardless of the core learning algorithm. Finally, we show via subjective exploration of embedding spaces that the DIME pipeline clusters interesting histopathological artefacts, suggesting a possible role for the method in QC pipelines. By clustering image patches according to underlying morphopathologic features, DIME supports subsequent qualitative exploration by pathologists and has the potential to inform and expediate biomarker discovery and drug development.

Fully Convolutional Network and Visual Saliency-Based Automatic Optic Disc Detection in Retinal Fundus Images

We present in this paper a novel optic disc detection method based on a fully convolutional network and visual saliency in retinal fundus images. Firstly, we employ the morphological reconstruction-based object detection method to locate the optic disc region roughly. According to the location result, a 400 × 400 image patch that covers the whole optic disc is obtained by cropping the original retinal fundus image. Secondly, the Simple Linear Iterative Cluster approach is utilized to segment such an image patch into many smaller superpixels. Thirdly, each superpixel is assigned a uniform initial saliency value according to the background prior information based on the assumption that the superpixels located on the boundary of the image belong to the background. Meanwhile, we use a pretrained fully convolutional network to extract the deep features from different layers of the network and design the strategy to represent each superpixel by the deep features. Finally, both the background prior information and the deep features are integrated into the single-layer cellular automata framework to gain the accurate optic disc detection result. We utilize the DRISHTI-GS dataset and RIM-ONE r3 dataset to evaluate the performance of our method. The experimental results demonstrate that the proposed method can overcome the influence of intensity inhomogeneity, weak contrast, and the complex surroundings of the optic disc effectively and has superior performance in terms of accuracy and robustness.

Zebra Crossing Detection and Time Scheduling Accuracy, Enhancement Optimization Using Artificial Intelligence

Zebra crossing detection is a fundamental function of the electronic travel aid. It can locate the zebra crossing and estimate its direction to help the visually impaired to cross the road safely. In contrast to the conventional methods, a regression approach is adopted to detect zebra crossing based on convolutional neural networks. Specifically, a fixed‐size window slides across the image captured at the intersection. The image patches are sequentially fed to the logistic regression model to identify the zebra crossing. Then the image patch of zebra crossing is fed to the regression model to predict the direction. The parameters of models are optimized by the ANN back propagation algorithm before predictions. Compared with existing methods, the proposed method can improve the precision‐recall performance of the zebra crossing identification and reduce the root mean square error of predicted directions.

Single Image Defogging Method Based on Image Patch Decomposition and Multi-Exposure Image Fusion

Bad weather conditions (such as fog, haze) seriously affect the visual quality of images. According to the scene depth information, physical model-based methods are used to improve image visibility for further image restoration. However, the unstable acquisition of the scene depth information seriously affects the defogging performance of physical model-based methods. Additionally, most of image enhancement-based methods focus on the global adjustment of image contrast and saturation, and lack the local details for image restoration. So, this paper proposes a single image defogging method based on image patch decomposition and multi-exposure fusion. First, a single foggy image is processed by gamma correction to obtain a set of underexposed images. Then the saturation of the obtained underexposed and original images is enhanced. Next, each image in the multi-exposure image set (including the set of underexposed images and the original image) is decomposed into the base and detail layers by a guided filter. The base layers are first decomposed into image patches, and then the fusion weight maps of the image patches are constructed. For detail layers, the exposure features are first extracted from the luminance components of images, and then the extracted exposure features are evaluated by constructing gaussian functions. Finally, both base and detail layers are combined to obtain the defogged image. The proposed method is compared with the state-of-the-art methods. The comparative experimental results confirm the effectiveness of the proposed method and its superiority over the state-of-the-art methods.

Pansharpening of WorldView-2 Data via Graph Regularized Sparse Coding and Adaptive Coupled Dictionary

The spectral mismatch between a multispectral (MS) image and its corresponding panchromatic (PAN) image affects the pansharpening quality, especially for WorldView-2 data. To handle this problem, a pansharpening method based on graph regularized sparse coding (GRSC) and adaptive coupled dictionary is proposed in this paper. Firstly, the pansharpening process is divided into three tasks according to the degree of correlation among the MS and PAN channels and the relative spectral response of WorldView-2 sensor. Then, for each task, the image patch set from the MS channels is clustered into several subsets, and the sparse representation of each subset is estimated through the GRSC algorithm. Besides, an adaptive coupled dictionary pair for each task is constructed to effectively represent the subsets. Finally, the high-resolution image subsets for each task are obtained by multiplying the estimated sparse coefficient matrix by the corresponding dictionary. A variety of experiments are conducted on the WorldView-2 data, and the experimental results demonstrate that the proposed method achieves better performance than the existing pansharpening algorithms in both subjective analysis and objective evaluation.