High-tech and Artificial Intelligence Make the Invisible Visible

Research and Development in Biosystems Engineering

HawkSpex® Mobile

Modern smartphones are veritably ubiquitous. Many users count on using their smartphones for more than just communication. Integrated sensors (e.g. camera, GPS, gyroscope) play a crucial role. Typically, specially developed software, an app, provides more extensive functions to users.
A whole group of such added features are transitioning toward low-priced, easy-touse and constantly available meters and monitors.

Many apps base their assessment of the properties of a scanned object, in the broadest sense, on its chemical composition and thus scan far more than just size, shape, color, texture, etc.. In other words, they require a hyperspectral camera. Some apps scan foods for freshness, ripeness or treatment, personalize recommendations for cosmetic care products, detect forged documents, counterfeit medicines or fake luxury apparel as well as concealed touchups of car paint damaged by an accident.

“Professional” apps frequently employ hyperspectral scanners with special and usually expensive sensors. The cameras installed in smartphones are equipped with three-channel color sensors (red-green-blue), making them unsuitable for such tasks.

HawkSpex® Mobile takes an entirely different but surprisingly simple route. A combination of adjustable illumination and a color camera, a kind of inverse spectroscopy, disperses light in narrowband spectral channels. This is fundamental to the implementation of hyperspectral scanning.

To do this, the smartphone’s display switches to a sequence of different colors in rapid succession on its entire display and synchronously takes pictures with the front camera, thus measuring the light reflected by the defined illuminated target object. The sequence of lighting colors produces a spectral image in fractions of a second. Specially developed methods of artificial intelligence convert the recorded raw spectral data into information in the context of the concrete application. An app enables user interaction, synchronizes illumination and picture- taking, and establishes a connection to a spectral application database.

Users can correct disparities in the features of different models’ displays and front cameras immediately after they install the app or later. Paper money serves as a readily available calibration standard with remarkably constant spectral properties. The user scans a bill in good condition once. The app compares it with a stored reference from the Fraunhofer IFF’s spectral laboratory and computes a correction factor, which is automatically employed in the background every time the app is used afterward.

Schematic of a spectral scan with HawkSpex® Mobile

The display is successively illuminated with the three primary colors, separately and together, while the front camera synchronously takes color pictures. This combination produces a twelve channel spectral effect. When the display is dark, an image is used to compensate ambient light.

HawkSpex® Gadget

HawkSpex® Mobile can be used to create numerous apps based on smartphones. Numerous applications can be implemented with this system. Ambitious users can disconnect the basic solution described from their smartphones and use it as HawkSpex® Gadget with enhanced spectral capabilities. The LED illumination and photosensors employed make it possible to configure the spectral bands for applications beyond the smartphone while basically retaining the sequential scanning principle. HawkSpex® Gadget can be controlled by the same app or operated as a separate device.

HawkSpex® Gadget System Model

 

HawkSpex® Gadget produces virtually any spectral bands and thus not only a larger potential number of bands but also the ultraviolet and infrared spectrum. Its retention of the principle of HawkSpex® scanning along with its universal usability, energy consumption and price make it superior to conventional spectrometers.

 

 

Experience HawkSpex® Mobile first hand

Scan the 3DQR® code on the right and use the 3DQR® app (available from Apple App Store and Google Play Store) to see an animation of HawkSpex® Mobile in action.

Digital Engineering for Soft Sensing by Means of Artificial Intelligence

The conventional approach to measuring a wide variety of parameters is to employ sensors that measure a proxy variable that correlates simply with the actual variable. Such sensors are also called “hardware sensors”. Usually curves are plotted. Direct measurements cannot be taken in many cases, e.g. when

  1. parameters are not directly measurable because sensor systems are inadequate,
  2. sensors are too expensive, especially when they would have to be used in many points in a process or,
  3. sensors are used in an environment in which they would wear quickly.

So-called soft sensors (virtual sensors) are used here. Such sensor systems consist of two basic components. One component consists of one or more sensors, which deliver indirect variables that have a correlation with the target variables that is systematic but also mathematically complex or indescribable analytically. The second component is a physical or empirical model that ascertains the target variables from the sensor data. While a physical model delivers a formal description, the empirical model contains an approximation of the function that maps the sensor data on the target variable.

© Fraunhofer IFF

Soft sensor system development.

The Biosystems Engineering Expert Group develops artificial intelligence and machine learning systems that create empirical models quickly and efficiently on the basis of systematic sample data. The models’ distinct features are:

  1. Scalability: Processing complexity can be gradually modified for existing hardware.

  2. Robustness: The model is robust for variations in data that are irrelevant to the computation of the target variable.

  3. Automation: Modeling and method selection are largely automatable.

  4. Real-time capability: Model adaption and knowledge retrieval satisfy real-time standards.

Visual Chemometrics and Automatic Analysis Spectral Data

Collected spectral data generally supports conclusions about the material composition of analyzed samples. Spatially resolved spectral data (e.g. through spectral cameras or MALDI imaging) yields extremely complex data sets that require special methods of analysis in order to extract the relevant information they contain. The Biosystems Engineering Expert Group is pursuing extensive research on this to develop modified algorithms that combine classic statistics and the latest methods des machine learning and der artificial intelligence. This research is in turn the point of departure for the development of custom solutions in different fields of application.

© Fraunhofer IFF

Hyperspectral imaging can analyze scenery in a multitude of, frequently narrow-band wavelength ranges. This produces an image stack containing local information resolved two-dimensionally. Every image pixel thus becomes the vector that contains the spectral fingerprint at this point in the scanned wavelength range. Typically, several hundred spectra are scanned.

The Biosystems Engineering Expert Group develops artificial intelligence and machine learning systems that create empirical models quickly and efficiently on the basis of systematic sample data. The models’ distinct features are:

  1. Scalability: Processing complexity can be gradually modified for existing hardware.

  2. Robustness: The model is robust for variations in data that are irrelevant to the computation of the target variable.

  3. Automation: Modeling and method selection are largely automatable.

  4. Real-time capability: Model adaption and knowledge retrieval satisfy real-time standards.

Publications

Bollenbeck, F. & Seiffert, U. Application-adaptive Dissimilarity Measures for Hyperspectral Images, Machine Learning Reports, 2010, 4, 23-27.

Seiffert, U. & Bollenbeck, F., Clustering of Hyperspectral Image Signatures Using Neural Gas, Machine Learning Reports, 2010, 4, 49-59.

Seiffert, U.; Bollenbeck, F.; Mock, H.-P. & Matros, A. Clustering of Crop Phenotypes by Means of Hyperspectral Signatures Using Artificial Neural Networks, Proceedings of the 2nd IEEE Workshop on Hyperspectral Imaging and Signal Processing: Evolution in Remote Sensing WHISPERS 2010, IEEE Press, 2010, 31-34.

Backhaus, A.; Bollenbeck, F. & Seiffert, U. Robust classification of the nutrition state in crop plants by hyperspectral imaging and artificial neural networks, In Proc. 3rd Workshop on Hyperspectral Image and Signal Processing: Evolution in Remote Sensing, WHISPERS 2011, IEEE Press, 2011.

Smart Image Processing

The Biosystems Engineering Expert Group develops underlying algorithms and custom tools and system solutions for smart image and signal processing. Applications are geared toward high-dimensional biological data and automatic and high throughput analysis of them, especially incorporating existing inexplicit expertise. The methodology employed includes not only classic mathematical/statistical approaches but increasingly also machine learning and artificial intelligence.

Publications

Seiffert, U.: Content Adaptive Compression of Images Using Neural Maps. Proc. of the International Workshop on Self-Organizing Maps WSOM 2005, Paris, France, pp. 227-234, 2005.

Herzog, A.; Herrmann, C.S.; Seiffert U.; Michaelis, B. & Braun, K.: Quantitative 3-D analysis of dendritic spines using geometric models. Society of Neuroscience 2010, 516.11.

Automatic Creation of Geometric Models of Biological Objects and Registration of Functional Data

Apart from representing pure morphology, three-dimensional (spatial) or four-dimensional (spatiotemporal) geometric models of biological objects predominantly serve to visualize experimental data in their spatiotemporal context. This function is frequently called an atlas. Depending on the demands on the model (e.g. spatial resolution, visibility of certain structures), different imaging systems are used:

  1. Nuclear magnetic resonance spectroscopy
  2. Confocal laser scanning microscopy
  3. Computed tomography

In some cases, the three-dimensional correlation must first be established by two-dimensional images (e.g. histological sections).

© Fraunhofer IFF

Segmented three-dimensional model of a barley seed automatically generated from approximately 2,500 single histological sections. One of the real cross-sections employed to generate the model is visualized at its correct spatial position as is a virtual longitudinal section in which any views can be generated. This work was performed in cooperation with the Seed Biology Group at the Leibniz Institute of Plant Genetics and Crop Plant Research Gatersleben (IPK).

© Fraunhofer IFF

Three-dimensional histological atlas of a barley seed with an automatically registered functional data set. Pictured here is the spatial distribution of a certain peptide as a function of a longitudinal section determined by means of mass spectrometric imaging (MSI). This work was performed in cooperation with the Applied Biochemistry Group at the Leibniz Institute of Plant Genetics and Crop Plant Research Gatersleben (IPK).

© Fraunhofer IFF

Interactive visualization of an automatically generated three-dimensional model of a leaf stem in the CAVE at the Fraunhofer IFF Magdeburg. The model was generated in cooperation with the Cell Biology and Imaging Group of James Hutton Institute Dundee, Scotland.

The custom models generated are being used in a number of projects to register other structural information of other modalities or functional data. This makes it possible to map certain biological functions in spatial or spatiotemporal structural patterns. To do so, the Biosystems Engineering Expert develops mathematical algorithms and tools adjusted for groups, which facilitate automatic registration of the aforementioned data.

Certain biological problems, e.g. extensive mutants screening, can only be addressed when such tools are available for high throughput.

Different three-dimensional projection systems can impressively visualize such virtual computerized models for individual verification as well as for training and presentation. The Fraunhofer IFF Magdeburg has a number of systems suitable for this.

Publications

Bollenbeck, F. & Seiffert, U. Joint Registration and Segmentation of Histological Volume Data by Diffusion-Based Label Adaption, 20th International Conference on Pattern Recognition (ICPR), 2010 , 2010, 2440 -2443.

Backhaus, A.; Kuwabara, A.; Bauch, M.; Monk, N.; Sanguinetti, G. & Fleming, A., LEAFPROCESSOR: a new leaf phenotyping tool using contour bending energy and shape cluster analysis, New Phytologist, 2010, 187, 251-261.

Bollenbeck, F.; Pielot, R.; Weier, D.; Weschke, W. & Seiffert, U., Inter-modality Registration of NMRi and Histological Section Images using Neural Networks Regression in Gabor Feature Space, Proceedings of the IEEE Symposium on Computational Intelligence for Image Processing CIIP 2009, IEEE Press, 2009, 27-34.

Bollenbeck, F.; Kaspar, S.; Mock, H.-P.; Weier, D. & Seiffert, U.; Rajasekaran, S. (Ed.) Three-dimensional Multimodality Modelling by Integration of High-Resolution Interindividual Atlases and Functional MALDI-IMS Data, Bioinformatics and Computational Biology, Springer, 2009, 5462, 126-139.