Image Pre-Processing Method of Machine Learning for Edge Detection with Image Signal Processor Enhancement

Even though computer vision has been developing, edge detection is still one of the challenges in that field. It comes from the limitations of the complementary metal oxide semiconductor (CMOS) Image sensor used to collect the image data, and then image signal processor (ISP) is additionally required to understand the information received from each pixel and performs certain processing operations for edge detection. Even with/without ISP, as an output of hardware (camera, ISP), the original image is too raw to proceed edge detection image, because it can include extreme brightness and contrast, which is the key factor of image for edge detection. To reduce the onerousness, we propose a pre-processing method to obtain optimized brightness and contrast for improved edge detection. In the pre-processing, we extract meaningful features from image information and perform machine learning such as k-nearest neighbor (KNN), multilayer perceptron (MLP) and support vector machine (SVM) to obtain enhanced model by adjusting brightness and contrast. The comparison results of F1 score on edgy detection image of non-treated, pre-processed and pre-processed with machine learned are shown. The pre-processed with machine learned F1 result shows an average of 0.822, which is 2.7 times better results than the non-treated one. Eventually, the proposed pre-processing and machine learning method is proved as the essential method of pre-processing image from ISP in order to gain better edge detection image. In addition, if we go through the pre-processing method that we proposed, it is possible to more clearly and easily determine the object required when performing auto white balance (AWB) or auto exposure (AE) in the ISP. It helps to perform faster and more efficiently through the proactive ISP.

Camera ISP Modification to Enable Image De-rendering

A camera's image signal processor (ISP) is dedicated hardware that performs a series of processing steps to render a captured raw sensor image to its final display-referred output suitable for viewing and sharing. It is often desirable to be able to revert – or de-render – the ISP-processed image back to the original raw sensor image. Undoing the ISP rendering, however, is not an easy task. This is because ISPs perform many nonlinear routines in the rendering pipeline that are difficult to invert. Moreover, modern cameras often apply scene-specific image processing, resulting in a wide range of possible ISP parameters. In this paper, we propose a modification to the ISP that allows the ISP-rendered image to be reverted back to a raw image. Our approach works by appending a fixed-sampling of the raw sensor values to all captured images. The appended raw samples comprise no more than 8 rows of pixels in the full-sized image and represent a negligible overhead given that 12–16 MP sensors typically have 3000 rows of pixels or more. The appended pixels are rendered along with the captured image to the final output. From these rendered raw samples, a reverse mapping function can be computed to undo the ISP processing. We demonstrate that this method performs almost on par with competing state-ofthe-art approaches for ISP de-rendering while offering a practical solution that is integrable to current camera ISP hardware.

Handheld endomicroscope using a fiber-optic harmonograph enables real-time and in vivo confocal imaging of living cell morphology and capillary perfusion

Confocal laser endomicroscopy provides high potential for noninvasive and in vivo optical biopsy at the cellular level. Here, we report a fully packaged handheld confocal endomicroscopic system for real-time, high-resolution, and in vivo cellular imaging using a Lissajous scanning fiber-optic harmonograph. The endomicroscopic system features an endomicroscopic probe with a fiber-optic harmonograph, a confocal microscope unit, and an image signal processor. The fiber-optic harmonograph contains a single mode fiber coupled with a quadrupole piezoelectric tube, which resonantly scans both axes at ~ 1 kHz to obtain a Lissajous pattern. The fiber-optic harmonograph was fully packaged into an endomicroscopic probe with an objective lens. The endomicroscopic probe was hygienically packaged for waterproofing and disinfection of medical instruments within a 2.6-mm outer diameter stainless tube capable of being inserted through the working channel of a clinical endoscope. The probe was further combined with the confocal microscope unit for indocyanine green imaging and the image signal processor for high frame rate and high density Lissajous scanning. The signal processing unit delivers driving signals for probe actuation and reconstructs confocal images using the auto phase matching process of Lissajous fiber scanners. The confocal endomicroscopic system was used to successfully obtain human in vitro fluorescent images and real-time ex vivo and in vivo fluorescent images of the living cell morphology and capillary perfusion inside a single mouse.

Effective ISP Tuning Framework Based on User Preference Feedback

This paper presents an effective tuning framework between CMOS Image Sensor (CIS) and Image Signal Processor (ISP) based on user preference feedback. One of key issue in ISP tuning is how to apply individual's subjectivity of Image Quality (IQ) in systematic way. In order to mitigate this issue, we propose a framework that efficiently surveys user preference of IQ and select ISP parameter based on those preferences. The overall processes are done on large-scale image database generated by an ISP simulator. In preference survey part, we make clusters that consist of perceptually similar images and gather user’s feedback on representative images of each cluster. Next, for training user preference, we train a DNN model according to general preference, and fine-tune model to optimize individuals preference based on user feedback. The model provides ISP candidate most similar to the preferences. In order to assess performance, the proposed framework was evaluated with a state-of-art CIS and ISP system. The experimental results indicate that the proposed framework converges the IQ score according to user feedback and find the ISP parameters that have higher quality IQ as compared with hand-tuned results.

DNN-based ISP Parameter Inference Algorithm for Automatic Image Quality Optimization

In camera development, because the image quality is subjective and the tuning complexity is increasing, building a correlated model with image signal processor (ISP) pipeline is very demanding task. In order to overcome those problems, this paper proposes an automatic image quality tuning framework based on Deep Neural Network (DNN). The image quality metric (IQM) have been defined to quantifies subjective image quality, which effectively represents the actual user perception. In this way, fast reproduction of the desired image has been possible through the minimized computing resource. Proposed Optimization methodology consists of Phase 1, a ISP modeling, and Phase 2, parameter optimization. Phase 1 construct a model between the parameters of ISP and the image quality metric. At phase 2, we add partially connected layer at input layer in order to optimize the parameters of ISP. Using backpropagation approach, the network selectively updates only the weights of partial connections, which allow to automatically derive the optimal parameters for high quality image. This idea has been implemented and experimented through commercial 16 Mega pixel resolution CMOS image sensor (CIS) and the state-of-the art ISP.