Speech enhancement (SE) aims to reduce noise in speech signals. Most SE techniques focus only on addressing audio information. In this work, inspired by multimodal learning, which utilizes data from different modalities, and the recent success of convolutional neural networks (CNNs) in SE, we propose an audio-visual de…
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,742 papers · 148 categories
Trend · papers per month
48 results for “audio-visual integration”
Speech enhancement (SE) aims to reduce noise in speech signals. Most SE techniques focus only on addressing audio information. In this work, inspired by multimodal learning, which utilizes data from different modalities, and the recent success of convolutional neural networks (CNNs) in SE, we propose an audio-visual de…
AV-CPL uses continuous pseudo-labels for AVSR combining labeled and unlabeled data.
problem Improving AVSR performance with labeled and unlabeled data.
method Semi-supervised method using continuous pseudo-labels generated by the same AVSR model.
result Significant improvements in VSR performance on LRS3 dataset.
Curiosity enhanced by audio-visual associations improves learning efficiency.
problem Challenges in reinforcement learning, especially predicting the future.
method Exploits multiple modalities (audio and vision) to predict novel associations.
result Improves exploration and learning efficiency in various environments.
In this paper, we analyzed how audio-visual speech enhancement can help to perform the ASR task in a cocktail party scenario. Therefore we considered two simple end-to-end LSTM-based models that perform single-channel audio-visual speech enhancement and phone recognition respectively. Then, we studied how the two model…
Automatic speech recognition can potentially benefit from the lip motion patterns, complementing acoustic speech to improve the overall recognition performance, particularly in noise. In this paper we propose an audio-visual fusion strategy that goes beyond simple feature concatenation and learns to automatically align…
Paper proposes new principles and framework for AVC learning from user-generated videos.
problem Challenges in learning audio-visual correspondence from short-term user-generated videos.
method Introduced new principles and a framework to facilitate AVC learning from videos' themes.
result Proposed approach outperformed baseline by 23.15% on KWAI-AD-AudVis corpus.
In this paper we address the problem of tracking multiple speakers via the fusion of visual and auditory information. We propose to exploit the complementary nature of these two modalities in order to accurately estimate smooth trajectories of the tracked persons, to deal with the partial or total absence of one of the…
Data clustering has received a lot of attention and numerous methods, algorithms and software packages are available. Among these techniques, parametric finite-mixture models play a central role due to their interesting mathematical properties and to the existence of maximum-likelihood estimators based on expectation-m…
Inspired by brain's modality fusion, this paper detects active speakers from audio and video.
problem Detecting active speakers in noisy environments.
method Inspired by brain's superior colliculus, combines audio and visual data through specialized neural networks and a novel fusion layer.
result Achieved results greatly surpassing initial expectations, confirming the effectiveness of the proposed method.
Traditional multi-view learning approaches suffer in the presence of view disagreement,i.e., when samples in each view do not belong to the same class due to view corruption, occlusion or other noise processes. In this paper we present a multi-view learning approach that uses a conditional entropy criterion to detect v…
We address the problems of multi-domain and single-domain regression based on distinct and unpaired labeled training sets for each of the domains and a large unlabeled training set from all domains. We formulate these problems as a Bayesian estimation with partial knowledge of statistical relations. We propose a worst-…
Investigating Audio, Visual, and Text Fusion Methods for End-to-End Automatic Personality Predictioncs.AI
We propose a tri-modal architecture to predict Big Five personality trait scores from video clips with different channels for audio, text, and video data. For each channel, stacked Convolutional Neural Networks are employed. The channels are fused both on decision-level and by concatenating their respective fully conne…
Two-stream model recognizes affect from audio and video.
problem Human affect recognition in real-world settings.
method Two-stream aural-visual analysis model with separate audio and visual processing.
result Model achieves promising results on Aff-Wild2 database.
Paper proposes self-supervised method for accurate speaker diarization.
problem Speaker diarization without massive labeling effort.
method Introduces dynamic triplet loss and multinomial loss for audio-video synchronization.
result Best model yields +8% F1-score improvement and diarization error rate reduction.
It has been suggested in developmental psychology literature that the communication of affect between mothers and their infants correlates with the socioemotional and cognitive development of infants. In this study, we obtained day-long audio recordings of 10 mother-infant pairs in order to study their affect communica…
The Audio/Visual Emotion Challenge and Workshop (AVEC 2019) "State-of-Mind, Detecting Depression with AI, and Cross-cultural Affect Recognition" is the ninth competition event aimed at the comparison of multimedia processing and machine learning methods for automatic audiovisual health and emotion analysis, with all pa…
AV-ASR system improves speech recognition with visual context.
problem Improving speech recognition accuracy with visual information.
method Transformer-based architecture with multiresolution and multimodal training.
result Multiresolution training speeds up convergence and improves WER by 18%.
Large-scale datasets have played a significant role in progress of neural network and deep learning areas. YouTube-8M is such a benchmark dataset for general multi-label video classification. It was created from over 7 million YouTube videos (450,000 hours of video) and includes video labels from a vocabulary of 4716 c…
Automatic prediction of emotion promises to revolutionise human-computer interaction. Recent trends involve fusion of multiple data modalities - audio, visual, and physiological - to classify emotional state. However, in practice, collection of physiological data `in the wild' is currently limited to heartbeat time ser…
Speaker-Independent Speech-Driven Visual Speech Synthesis using Domain-Adapted Acoustic Modelseess.AS
Speech-driven visual speech synthesis involves mapping features extracted from acoustic speech to the corresponding lip animation controls for a face model. This mapping can take many forms, but a powerful approach is to use deep neural networks (DNNs). However, a limitation is the lack of synchronized audio, video, an…
Co-Localization of Audio Sources in Images Using Binaural Features and Locally-Linear Regressioncs.SD
This paper addresses the problem of localizing audio sources using binaural measurements. We propose a supervised formulation that simultaneously localizes multiple sources at different locations. The approach is intrinsically efficient because, contrary to prior work, it relies neither on source separation, nor on mon…
SEMI uses multisensory incongruity to self-supervise exploration in reinforcement learning.
problem Efficient exploration in reinforcement learning with sparse or missing rewards.
method SEMI incentivizes exploration by maximizing multisensory incongruity, measured in perception and action incongruity.
result SEMI improves sample efficiency and learns skills without external rewards.
NeoMLP improves neural fields by adding self-attention for better downstream tasks.
problem Complexity and lack of conditioning mechanisms in neural fields.
method Transformed MLP into NeoMLP with self-attention and weight-sharing.
result NeoMLP outperforms state-of-the-art methods in downstream tasks.
The study examines stability of Hamiltonian Poisson integrators on both integrable and non-integrable systems.
problem Investigating stability properties of Hamiltonian Poisson integrators.
method Examples of Lotka-Volterra dynamics and numerical investigations of a non-integrable system are used.
result The existence of a modified Hamiltonian is crucial for the stability of Hamiltonian Poisson integrators.
Geometrically interprets integrability of geodesic flow using web theory.
problem Integrability of geodesic flow by quadratic integrals.
method Geometric interpretation through web theory and integrable billiards construction.
result Constructs integrable billiards on surfaces with quadratic geodesic integrals.
Proof shows volume equals integral points for certain manifolds.
problem Counting integral points on affine manifolds.
method Rational Ehrhart theory and Fourier analysis.
result Volume equals number of integral points for integral-integral affine manifolds.
Rough geometric integrationmath.DG
Integrates rough geometric forms on manifolds.
problem Integrating rough forms on complex manifolds.
method Combines Whitney's geometric integration and sewing approaches.
result Introduced distributional k-forms for integration.
The paper defines and analyzes set-valued stochastic integrals for Lévy processes.
problem Defining and analyzing set-valued stochastic integrals for Lévy processes.
method Extending classical definitions to convoluted integrals with square-integrable kernels, and proving properties of set-valued convoluted stochastic integrals.
result Set-valued convoluted stochastic integrals can be explosive and take extended vector values.
New integration theory on topological spaces, including fractals.
problem Developing a universal integration theory for arbitrary topological spaces.
method Introducing a new integration framework using unital magma valued functions and measures.
result Integration, differentiation, and orientation defined for arbitrary topological spaces.
We discuss a recurrent geometrical method, due to Élie Cartan and von Weber ([1],[11]) enabling us to determine, step by step, the maximal integral manifolds of a not necessarily integrable nor regular Pfaffian system. The dimensions of such integral manifolds can, of course, vary from point to point but more so can va…
We define a non-absolutely convergent integration on integral currents of dimension 1 in Euclidean space. This integral is closely related to the Henstock-Kurzweil and Pfeffer Integrals. Using it, we prove a generalized Fundamental Theorem of Calculus on these currents. A detailed presentation of Henstock-Kurzweil Inte…
The paper defines and proves the existence of decompositions of integral varifolds.
problem Existence of integral varifold decompositions.
method Introducing and proving the existence of decompositions of integral varifolds into countably many integral varifolds.
result Existence of decompositions of integral varifolds whose first variation is representable by integration.
Counterexample shows Ito integrand needn't be locally square integrable.
problem Ito integrand's square integrability condition is not always met.
method Provided a counterexample to Ito's Lemma's integrability condition.
result Ito integrand needn't be locally square integrable.
Local high-degree polynomial integrals of geodesic flows and the generalized hodograph methodmath.DS
Study integrable geodesic flows on 2-surfaces with high-degree polynomial first integrals.
problem Integrable geodesic flows on 2-surfaces with high-degree polynomial first integrals.
method Semi-Hamiltonian systems of PDEs and generalized hodograph method.
result Construction of many local explicit and implicit integrable examples with polynomial first integrals of degrees 3, 4, 5.
The article constructs stochastic integration in Riemannian manifolds.
problem No specific problem stated; focuses on the construction of stochastic integration.
method Functional-analytic approach to stochastic integration in Riemannian manifolds.
result There are infinitely many stochastic integrals, and they are related by a simple formula.
We introduce renormalized integrals which generalize conventional measure theoretic integrals. One approximates the integration domain by measure spaces and defines the integral as the limit of integrals over the approximating spaces. This concept is implicitly present in many mathematical contexts such as Cauchy's pri…
TQ separates sampling and integration for high-dimensional integrals.
problem High-dimensional integration challenges in science.
method Tree Quadrature (TQ) constructs a surrogate model using regression trees.
result TQ outperforms existing methods in up to 15 dimensions.
Method finds differential equations for integrable billiard tables.
problem Finding differential equations for integrable billiard tables.
method Introducing a method to find differential equations for functions defining tables.
result Illustrated method in three billiard systems.
Constructs Lie groupoid integrating elliptic tangent bundles and Poisson structures.
problem Existence and integration of elliptic tangent bundles and Poisson structures.
method Explicit construction of Lie groupoids and local models for symplectic integration.
result Necessary and sufficient topological condition for integration of elliptic tangent bundles.
Investigates integrable systems with linear periodic integral for e(3) Lie algebra.
problem Analyzes singularities and topological properties of integrable systems.
method Examines singularities of Liouville foliation, bifurcation diagram, transformations of Liouville tori, and isoenergy surfaces.
result Discovers topological properties of integrable systems with linear periodic integral.
We use neural networks as control variates with geometric integration techniques.
problem Analytic integration of neural network approximations for variance reduction.
method Integration domain subdivision using computational geometry for MLPs with continuous piecewise linear activation functions.
result Neural networks can be used as control variates with geometric integration methods.
Integral foliated simplicial volume is a version of simplicial volume combining the rigidity of integral coefficients with the flexibility of measure spaces. In this article, using the language of measure equivalence of groups we prove a proportionality principle for integral foliated simplicial volume for aspherical m…
This paper is an exposition of heuristics related to Witten's functional integral, relating it to Vassiliev invariants and to the Kontsevich integrals that can be used to produce Vassiliev invariants of knots and links.In particular, we give a simplified version of the appearance of the Kontsevich integrals in the pert…
Surveying integrability of Lie algebroids and structures.
problem Integrability of Lie algebroids and structures.
method Survey and recent results on integrability.
result Recent findings on local and global integrability.
This paper provides an existence-and-uniqueness theorem characterizing the stochastic integral with respect to a Wiener process. The integral is represented as a mapping from the space of measurable and adapted pathwise locally integrable processes to the space of continuous adapted processes. It is characterized in te…
We construct an infinite-dimensional symplectic 2-groupoid as the integration of an exact Courant algebroid. We show that every integrable Dirac structure integrates to a "Lagrangian" sub-2-groupoid of this symplectic 2-groupoid. As a corollary, we recover a result of Bursztyn-Crainic-Weinstein-Zhu that every integrabl…
Integrable LCK manifoldsmath.DG
Integrable LCK manifolds characterized as Kähler Lie algebras.
problem Characterizing LCK manifolds with integrable anti-Lee forms.
method Examining LCK manifolds with integrable anti-Lee forms and applying to Lie algebras.
result Unimodular integrable LCK Lie algebras are Kähler Lie algebras with specific derivations.