Scalable Attentive Sentence Pair Modeling via Distilled Sentence Embedding

Recent state-of-the-art natural language understanding models, such as BERT and XLNet, score a pair of sentences (A and B) using multiple cross-attention operations – a process in which each word in sentence A attends to all words in sentence B and vice versa. As a result, computing the similarity between a query sentence and a set of candidate sentences, requires the propagation of all query-candidate sentence-pairs throughout a stack of cross-attention layers. This exhaustive process becomes computationally prohibitive when the number of candidate sentences is large. In contrast, sentence embedding techniques learn a sentence-to-vector mapping and compute the similarity between the sentence vectors via simple elementary operations. In this paper, we introduce Distilled Sentence Embedding (DSE) – a model that is based on knowledge distillation from cross-attentive models, focusing on sentence-pair tasks. The outline of DSE is as follows: Given a cross-attentive teacher model (e.g. a fine-tuned BERT), we train a sentence embedding based student model to reconstruct the sentence-pair scores obtained by the teacher model. We empirically demonstrate the effectiveness of DSE on five GLUE sentence-pair tasks. DSE significantly outperforms several ELMO variants and other sentence embedding methods, while accelerating computation of the query-candidate sentence-pairs similarities by several orders of magnitude, with an average relative degradation of 4.6% compared to BERT. Furthermore, we show that DSE produces sentence embeddings that reach state-of-the-art performance on universal sentence representation benchmarks. Our code is made publicly available at https://github.com/microsoft/Distilled-Sentence-Embedding.

P6594Granger Causality-based analysis to accurately identify specific electrophenotypes of myocardial fibrillation

Background Clinical identification of fibrillation drivers remains challenging in both atrial and ventricular fibrillation (VF). In this study, we developed novel tools using granger causality (GC) analysis for quantifying the causal relationship between neighbouring fibrillatory signals. We tested whether it was adaptable to low resolution, limited coverage and sequentially acquired data for quantifying global organisation of VF and mapping regions with stable rotational drivers (RDs). Methods Eighteen Sprague-Dawley rat hearts were perfused ex vivo for optical mapping studies. VF with differing degrees of organisation was induced with carbenoxolone (10–50μM, n=8), or prior maturation of patchy ventricular fibrosis (n=10) generated by ischaemia-reperfusion. After phase mapping, the data was downsampled to 25% of full resolution to develop validated GC-based tools. The causality pairing index (CPI), a global measure of organisation, quantified propagational effects between all neighboring signals. Low-resolution GC-vector maps localized areas harboring RDs and quantified the prevalence of RDs over time using a novel index called circular interdependence value (CIV). These GC-based tools were then adapted to analyze low-resolution multi-electrode electrograms of sixteen persistent-AF (psAF) patients presenting for a first ablation procedure. Results A spectrum of fibrillatory organisation and mechanisms in VF was observed. In rat VF there was a positive correlation between CPI and the number of stable RDs (R2=0.41, p=0.004), and CIV showed a significant difference in driver vs non-driver regions (0.91±0.05 vs 0.35±0.06, p=0.0002). Similarly, in psAF patients, there was a positive correlation between CPI and the number of stable RDs (R2=0.56, p≤0.001). GC vector mapping showed that 8/16 of patients had at least one RD area, and 8/16 had chaotic activity with no RDs. Conclusion Mechanisms of myocardial fibrillation occurs along a spectrum between organized activity with discrete areas harboring RDs and disorganised myocardial activation with no RDs. GC maps can be utilised for identifying regions localising RDs with sequential mapping in limited spatial resolution and coverage. In psAF GC-based analysis accurately identified specific fibrillatory mechanisms from low-resolution mapping. GC vector mapping holds potential for development with human fibrillation data as a mapping tool for driver guided ablation. Acknowledgement/Funding BHF Programme Grant PG/16/17/32069