GPU-GIST – a case of generalized database indexing on modern hardware

2017 ◽  
Vol 59 (3) ◽  
Author(s):  
Felix Beier ◽  
Kai-Uwe Sattler

AbstractA lot of different indexes have been developed for accelerating search operations on large data sets. Search trees, representing the most prominent class, are ubiquitous in database management systems but are also widely used in non-DBMS applications. An approach for lowering the implementation complexity of these structures are index frameworks like generalized search trees (GiST). Common data management operations are implemented within the framework which can be specialized by data organization and evaluation strategies in order to model the actual index type. These frameworks are particularly useful in scientific and engineering applications where characteristics of the underlying data set are not known a priori and a lot of prototyping is required in order to find suitable index structures for the workload.However, existing frameworks only abstract data organization and data maintenance aspects to model different index families, while traversal operations for executing searches are implemented serially. This paper presents an approach for enabling parallel processing in GiST in order to leverage the full power of parallel processor architectures for different index implementations at once. Further, results of a prototypical implementation are evaluated on a hybrid CPU/GPU system architecture to verify the applicability of this generic framework idea on different hardware platforms.

Author(s):  
Lior Shamir

Abstract Several recent observations using large data sets of galaxies showed non-random distribution of the spin directions of spiral galaxies, even when the galaxies are too far from each other to have gravitational interaction. Here, a data set of $\sim8.7\cdot10^3$ spiral galaxies imaged by Hubble Space Telescope (HST) is used to test and profile a possible asymmetry between galaxy spin directions. The asymmetry between galaxies with opposite spin directions is compared to the asymmetry of galaxies from the Sloan Digital Sky Survey. The two data sets contain different galaxies at different redshift ranges, and each data set was annotated using a different annotation method. The results show that both data sets show a similar asymmetry in the COSMOS field, which is covered by both telescopes. Fitting the asymmetry of the galaxies to cosine dependence shows a dipole axis with probabilities of $\sim2.8\sigma$ and $\sim7.38\sigma$ in HST and SDSS, respectively. The most likely dipole axis identified in the HST galaxies is at $(\alpha=78^{\rm o},\delta=47^{\rm o})$ and is well within the $1\sigma$ error range compared to the location of the most likely dipole axis in the SDSS galaxies with $z>0.15$ , identified at $(\alpha=71^{\rm o},\delta=61^{\rm o})$ .


2020 ◽  
Vol 6 ◽  
Author(s):  
Jaime de Miguel Rodríguez ◽  
Maria Eugenia Villafañe ◽  
Luka Piškorec ◽  
Fernando Sancho Caparrini

Abstract This work presents a methodology for the generation of novel 3D objects resembling wireframes of building types. These result from the reconstruction of interpolated locations within the learnt distribution of variational autoencoders (VAEs), a deep generative machine learning model based on neural networks. The data set used features a scheme for geometry representation based on a ‘connectivity map’ that is especially suited to express the wireframe objects that compose it. Additionally, the input samples are generated through ‘parametric augmentation’, a strategy proposed in this study that creates coherent variations among data by enabling a set of parameters to alter representative features on a given building type. In the experiments that are described in this paper, more than 150 k input samples belonging to two building types have been processed during the training of a VAE model. The main contribution of this paper has been to explore parametric augmentation for the generation of large data sets of 3D geometries, showcasing its problems and limitations in the context of neural networks and VAEs. Results show that the generation of interpolated hybrid geometries is a challenging task. Despite the difficulty of the endeavour, promising advances are presented.


2006 ◽  
Vol 39 (2) ◽  
pp. 262-266 ◽  
Author(s):  
R. J. Davies

Synchrotron sources offer high-brilliance X-ray beams which are ideal for spatially and time-resolved studies. Large amounts of wide- and small-angle X-ray scattering data can now be generated rapidly, for example, during routine scanning experiments. Consequently, the analysis of the large data sets produced has become a complex and pressing issue. Even relatively simple analyses become difficult when a single data set can contain many thousands of individual diffraction patterns. This article reports on a new software application for the automated analysis of scattering intensity profiles. It is capable of batch-processing thousands of individual data files without user intervention. Diffraction data can be fitted using a combination of background functions and non-linear peak functions. To compliment the batch-wise operation mode, the software includes several specialist algorithms to ensure that the results obtained are reliable. These include peak-tracking, artefact removal, function elimination and spread-estimate fitting. Furthermore, as well as non-linear fitting, the software can calculate integrated intensities and selected orientation parameters.


1997 ◽  
Vol 1997 ◽  
pp. 143-143
Author(s):  
B.L. Nielsen ◽  
R.F. Veerkamp ◽  
J.E. Pryce ◽  
G. Simm ◽  
J.D. Oldham

High producing dairy cows have been found to be more susceptible to disease (Jones et al., 1994; Göhn et al., 1995) raising concerns about the welfare of the modern dairy cow. Genotype and number of lactations may affect various health problems differently, and their relative importance may vary. The categorical nature and low incidence of health events necessitates large data-sets, but the use of data collected across herds may introduce unwanted variation. Analysis of a comprehensive data-set from a single herd was carried out to investigate the effects of genetic line and lactation number on the incidence of various health and reproductive problems.


2019 ◽  
Vol 8 (2S11) ◽  
pp. 3523-3526

This paper describes an efficient algorithm for classification in large data set. While many algorithms exist for classification, they are not suitable for larger contents and different data sets. For working with large data sets various ELM algorithms are available in literature. However the existing algorithms using fixed activation function and it may lead deficiency in working with large data. In this paper, we proposed novel ELM comply with sigmoid activation function. The experimental evaluations demonstrate the our ELM-S algorithm is performing better than ELM,SVM and other state of art algorithms on large data sets.


2009 ◽  
Vol 2 (1) ◽  
pp. 421-475 ◽  
Author(s):  
A. Velo ◽  
F. F. Pérez ◽  
X. Lin ◽  
R. M. Key ◽  
T. Tanhua ◽  
...  

Abstract. Data on carbon and carbon-relevant hydrographic and hydrochemical parameters from previously non-publicly available cruise data sets in the Artic Mediterranean Seas (AMS), Atlantic and Southern Ocean have been retrieved and merged to a new database: CARINA (CARbon IN the Atlantic). These data have gone through rigorous quality control (QC) procedures to assure the highest possible quality and consistency. The data for most of the measured parameters in the CARINA database were objectively examined in order to quantify systematic differences in the reported values, i.e. secondary quality control. Systematic biases found in the data have been corrected in the data products, i.e. three merged data files with measured, calculated and interpolated data for each of the three CARINA regions; AMS, Atlantic and Southern Ocean. Out of a total of 188 cruise entries in the CARINA database, 59 reported pH measured values. Here we present details of the secondary QC on pH for the CARINA database. Procedures of quality control, including crossover analysis between cruises and inversion analysis of all crossover data are briefly described. Adjustments were applied to the pH values for 21 of the cruises in the CARINA dataset. With these adjustments the CARINA database is consistent both internally as well as with GLODAP data, an oceanographic data set based on the World Hydrographic Program in the 1990s. Based on our analysis we estimate the internal accuracy of the CARINA pH data to be 0.005 pH units. The CARINA data are now suitable for accurate assessments of, for example, oceanic carbon inventories and uptake rates and for model validation.


2021 ◽  
Author(s):  
Kezia Lange ◽  
Andreas C. Meier ◽  
Michel Van Roozendael ◽  
Thomas Wagner ◽  
Thomas Ruhtz ◽  
...  

<p>Airborne imaging DOAS and ground-based stationary and mobile DOAS measurements were conducted during the ESA funded S5P-VAL-DE-Ruhr campaign in September 2020 in the Ruhr area. The Ruhr area is located in Western Germany and is a pollution hotspot in Europe with urban character as well as large industrial emitters. The measurements are used to validate data from the Sentinel-5P TROPOspheric Monitoring Instrument (TROPOMI) with focus on the NO<sub>2</sub> tropospheric vertical column product.</p><p>Seven flights were performed with the airborne imaging DOAS instrument, AirMAP, providing continuous maps of NO<sub>2</sub> in the layers below the aircraft. These flights cover many S5P ground pixels within an area of about 40 km side length and were accompanied by ground-based stationary measurements and three mobile car DOAS instruments. Stationary measurements were conducted by two Pandora, two zenith-sky and two MAX-DOAS instruments distributed over three target areas, partly as long-term measurements over a one-year period.</p><p>Airborne and ground-based measurements were compared to evaluate the representativeness of the measurements in time and space. With a resolution of about 100 x 30 m<sup>2</sup>, the AirMAP data creates a link between the ground-based and the TROPOMI measurements with a resolution of 3.5 x 5.5 km<sup>2</sup> and is therefore well suited to validate TROPOMI's tropospheric NO<sub>2</sub> vertical column.</p><p>The measurements on the seven flight days show strong variability depending on the different target areas, the weekday and meteorological conditions. We found an overall low bias of the TROPOMI operational NO<sub>2</sub> data for all three target areas but with varying magnitude for different days. The campaign data set is compared to custom TROPOMI NO<sub>2</sub> products, using different auxiliary data, such as albedo or a priori vertical profiles to evaluate the influence on the TROPOMI data product. Analyzing and comparing the different data sets provides more insight into the high spatial and temporal heterogeneity in NO<sub>2</sub> and its impact on satellite observations and their validation.</p>


2021 ◽  
Vol 14 (11) ◽  
pp. 2369-2382
Author(s):  
Monica Chiosa ◽  
Thomas B. Preußer ◽  
Gustavo Alonso

Data analysts often need to characterize a data stream as a first step to its further processing. Some of the initial insights to be gained include, e.g., the cardinality of the data set and its frequency distribution. Such information is typically extracted by using sketch algorithms, now widely employed to process very large data sets in manageable space and in a single pass over the data. Often, analysts need more than one parameter to characterize the stream. However, computing multiple sketches becomes expensive even when using high-end CPUs. Exploiting the increasing adoption of hardware accelerators, this paper proposes SKT , an FPGA-based accelerator that can compute several sketches along with basic statistics (average, max, min, etc.) in a single pass over the data. SKT has been designed to characterize a data set by calculating its cardinality, its second frequency moment, and its frequency distribution. The design processes data streams coming either from PCIe or TCP/IP, and it is built to fit emerging cloud service architectures, such as Microsoft's Catapult or Amazon's AQUA. The paper explores the trade-offs of designing sketch algorithms on a spatial architecture and how to combine several sketch algorithms into a single design. The empirical evaluation shows how SKT on an FPGA offers a significant performance gain over high-end, server-class CPUs.


Author(s):  
V. Suresh Babu ◽  
P. Viswanath ◽  
Narasimha M. Murty

Non-parametric methods like the nearest neighbor classifier (NNC) and the Parzen-Window based density estimation (Duda, Hart & Stork, 2000) are more general than parametric methods because they do not make any assumptions regarding the probability distribution form. Further, they show good performance in practice with large data sets. These methods, either explicitly or implicitly estimates the probability density at a given point in a feature space by counting the number of points that fall in a small region around the given point. Popular classifiers which use this approach are the NNC and its variants like the k-nearest neighbor classifier (k-NNC) (Duda, Hart & Stock, 2000). Whereas the DBSCAN is a popular density based clustering method (Han & Kamber, 2001) which uses this approach. These methods show good performance, especially with larger data sets. Asymptotic error rate of NNC is less than twice the Bayes error (Cover & Hart, 1967) and DBSCAN can find arbitrary shaped clusters along with noisy outlier detection (Ester, Kriegel & Xu, 1996). The most prominent difficulty in applying the non-parametric methods for large data sets is its computational burden. The space and classification time complexities of NNC and k-NNC are O(n) where n is the training set size. The time complexity of DBSCAN is O(n2). So, these methods are not scalable for large data sets. Some of the remedies to reduce this burden are as follows. (1) Reduce the training set size by some editing techniques in order to eliminate some of the training patterns which are redundant in some sense (Dasarathy, 1991). For example, the condensed NNC (Hart, 1968) is of this type. (2) Use only a few selected prototypes from the data set. For example, Leaders-subleaders method and l-DBSCAN method are of this type (Vijaya, Murthy & Subramanian, 2004 and Viswanath & Rajwala, 2006). These two remedies can reduce the computational burden, but this can also result in a poor performance of the method. Using enriched prototypes can improve the performance as done in (Asharaf & Murthy, 2003) where the prototypes are derived using adaptive rough fuzzy set theory and as in (Suresh Babu & Viswanath, 2007) where the prototypes are used along with their relative weights. Using a few selected prototypes can reduce the computational burden. Prototypes can be derived by employing a clustering method like the leaders method (Spath, 1980), the k-means method (Jain, Dubes, & Chen, 1987), etc., which can find a partition of the data set where each block (cluster) of the partition is represented by a prototype called leader, centroid, etc. But these prototypes can not be used to estimate the probability density, since the density information present in the data set is lost while deriving the prototypes. The chapter proposes to use a modified leader clustering method called the counted-leader method which along with deriving the leaders preserves the crucial density information in the form of a count which can be used in estimating the densities. The chapter presents a fast and efficient nearest prototype based classifier called the counted k-nearest leader classifier (ck-NLC) which is on-par with the conventional k-NNC, but is considerably faster than the k-NNC. The chapter also presents a density based clustering method called l-DBSCAN which is shown to be a faster and scalable version of DBSCAN (Viswanath & Rajwala, 2006). Formally, under some assumptions, it is shown that the number of leaders is upper-bounded by a constant which is independent of the data set size and the distribution from which the data set is drawn.


2019 ◽  
Vol 15 (S350) ◽  
pp. 406-407
Author(s):  
Sacha Foschino ◽  
Olivier Berné ◽  
Christine Joblin

AbstractObservations of the mid-infrared (mid-IR, 3-15 μm) spectra of photo-dissociation regions reveal ubiquitous, broad and intense emission bands, the aromatic infrared bands (AIBs), attributed to polycyclic aromatic hydrocarbons (PAHs). Studies of the AIBs showed spectral variations (e.g. in the band positions) between different astrophysical objects, or even within single object, thanks to hyperspectral images. The James Webb Space Telescope (JWST) will allow to get further spectral and spatial details compared to former space observatories. This will come with large data sets, which will require specific tools in order to perform efficient scientific analysis.We propose in this study a method based on blind signal separation to reduce the analysis of such large data set to that of a small number of elementary spectra, spectrally representative of the data set and physically interpretable as the spectra of populations of mid-IR emitters. The robustness and fastness of the method are improved compared to former algorithms. It is tested on a ISO-SWS data set, which approaches the best the characteristics of JWST data, from which four elementary spectra are extracted, attributed to cationic, neutral PAHs, evaporating very small grains and large and ionized PAHs.


Sign in / Sign up

Export Citation Format

Share Document