New method enforces orthogonality in convolutional layers for improved robustness.
problem Improving adversarial robustness in deep learning models.
method Applying the Cayley transform to skew-symmetric convolutions in the Fourier domain.
result The proposed method preserves orthogonality and enhances adversarial robustness compared to existing techniques.
New method prevents gradient attenuation in Lipschitz constrained convolutional networks.
problem Gradient norm attenuation in Lipschitz constrained convolutional networks.
method Block Convolution Orthogonal Parameterization (BCOP) to train scalable, expressive, provably Lipschitz convolutional networks.
result Empirically, BCOP parameterization is competitive with existing approaches to provable adversarial robustness and Wasserstein distance estimation.
Random convolutional networks can be fooled with adversarial examples.
problem Existence of adversarial examples for random convolutional networks.
method Utilizing isoperimetric inequalities on the special orthogonal group so(d). result Adversarial examples exist for various random convolutional networks.
Initialization of parameters in deep neural networks has been shown to have a big impact on the performance of the networks (Mishkin & Matas, 2015). The initialization scheme devised by He et al, allowed convolution activations to carry a constrained mean which allowed deep networks to be trained effectively (He et al.…
New optimization algorithms on orthogonal group for machine learning.
problem Efficient optimization on the orthogonal group for machine learning tasks.
method Stochastic geometric algorithms on Lie groups.
result Strong performance on diverse machine learning tasks.
This paper tackles spatio-temporal information preservation in machine learning.
problem Conventional machine learning assumes orthogonal data attributes, disrupting spatio-temporal information.
method Shift-invariant k-means, convolutional dictionary learning, and spatio-temporal hypercomplex encoding schemes are proposed.
result Gabor feature extraction outperforms convolutional dictionary learning in spatio-temporal information preservation.
Anti-transfer learning prevents misleading representations for speech tasks.
problem Misleading representations learned from orthogonal tasks in speech processing.
method Penalizes similarity between activations of a network and another trained on an orthogonal task.
result Improves classification accuracy and invariance to the orthogonal task.
Convolutional nets require fewer samples than fully-connected nets for image classification.
problem Understanding why convolutional nets are more sample-efficient than fully-connected nets.
method Construction of a natural distribution and target function to demonstrate a sample complexity gap.
result Convolutional nets require O(1) samples for a single target function, while fully-connected nets require Ω(d2) samples. LOT improves adversarial robustness by training 1-Lipschitz convolution layers.
problem Improving adversarial robustness of deep neural networks.
method LOT: Layer-wise Orthogonal Training for 1-Lipschitz convolution layers.
result LOT significantly enhances certified robustness of Lipschitz-bounded models.
In recent years, state-of-the-art methods in computer vision have utilized increasingly deep convolutional neural network architectures (CNNs), with some of the most successful models employing hundreds or even thousands of layers. A variety of pathologies such as vanishing/exploding gradients make training such deep n…
Motivated by recently published methods using frequency decompositions of convolutions (e.g. Octave Convolutions), we propose a novel convolution scheme to stabilize the training and reduce the likelihood of a mode collapse. The basic idea of our approach is to split convolutional filters into additive high and low fre…
Orthogonal initialization does not speed up training in ultra-wide neural networks.
problem Exploring the effect of orthogonal initialization on training speed in deep neural networks.
method Study of neural tangent kernel dynamics in FCNs and CNNs with orthogonal initialization.
result The NTK of orthogonally-initialized networks remains constant during training, suggesting no speedup in the NTK regime.
Orthogonium offers unified, efficient layers for robust deep learning.
problem Fragmented and computationally demanding implementations of orthogonal and 1-Lipschitz layers.
method Unified, efficient PyTorch library providing orthogonal and 1-Lipschitz layers.
result Reduced overhead and standardized tools for robust experimentation.
This paper seeks to answer the question: as the (near-) orthogonality of weights is found to be a favorable property for training deep convolutional neural networks, how can we enforce it in more effective and easy-to-use ways? We develop novel orthogonality regularizations on training deep CNNs, utilizing various adva…
Regularization techniques are widely used to improve the generality, robustness, and efficiency of deep convolutional neural networks (DCNNs). In this paper, we propose a novel approach of regulating DCNN convolutional kernels by a structured filter bank. Comparing with the existing regularization methods, such as $\el…
Different neural networks trained on the same dataset often learn similar input-output mappings with very different weights. Is there some correspondence between these neural network solutions? For linear networks, it has been shown that different instances of the same network architecture encode the same representatio…
Convolutional networks predict turbulence from wall quantities.
problem Predicting turbulence fields from wall-shear-stress components and wall pressure.
method Two CNN models: FCN and FCN-POD, trained on DNS data.
result FCN and FCN-POD models outperform EPOD in predicting turbulence fields.
Improved robustness of 1D CNNs for heart arrhythmia classification.
problem Improving the robustness of 1D CNNs for classification tasks.
method Parameterization using Cayley transform and controllability Gramian for Lipschitz-bounded CNNs.
result Improved robustness of trained Lipschitz-bounded 1D CNNs for heart arrhythmia classification.
Convolutional operator learning is gaining attention in many signal processing and computer vision applications. Learning kernels has mostly relied on so-called patch-domain approaches that extract and store many overlapping patches across training signals. Due to memory demands, patch-domain methods have limitations w…
LipKernel adds robustness to CNNs by enforcing Lipschitz bounds.
problem Improving robustness of CNNs in real-time applications.
method Dissipative layers parameterized by LMIs and 2-D Roesser model.
result Orders of magnitude faster run-time compared to state-of-the-art methods.
We present QuickNet, a fast and accurate network architecture that is both faster and significantly more accurate than other fast deep architectures like SqueezeNet. Furthermore, it uses less parameters than previous networks, making it more memory efficient. We do this by making two major modifications to the referenc…
Designs CNNs for better image reconstruction.
problem Image reconstruction from limited data.
method Parseval convolution operators and chaining of elementary modules.
result CNN-based algorithm yields better results than sparsity-based methods.
Deep learning algorithms have achieved excellent performance lately in a wide range of fields (e.g., computer version). However, a severe challenge faced by deep learning is the high dependency on hyper-parameters. The algorithm results may fluctuate dramatically under the different configuration of hyper-parameters. A…
We introduce Parseval networks, a form of deep neural networks in which the Lipschitz constant of linear, convolutional and aggregation layers is constrained to be smaller than 1. Parseval networks are empirically and theoretically motivated by an analysis of the robustness of the predictions made by deep neural networ…
LOTOS improves ensemble robustness by promoting orthogonal transformations.
problem Transferability of adversarial examples threatens robustness of classification models.
method LOTOS promotes orthogonality among sub-spaces of transformations in ensemble models.
result LOTOS increases robust accuracy of ensembles by 6 percentage points against black-box attacks.
Enhanced autoencoders improve ROMs for PDEs by capturing essential properties.
problem Autoencoders struggle to capture essential properties for accurate ROMs.
method Introduced symmetric Convolutional AutoEncoders (CAEs) that preserve manifold properties.
result Symmetric CAEs yield more accurate latent trajectories and robust models.
Study reveals an equivalence principle for the spectrum of random inner-product kernel matrices in polynomial scaling.
problem Understanding the spectrum of random kernel matrices in polynomial scaling regimes.
method Investigates random matrices with nonlinear kernel functions applied to inner products of uniformly distributed vectors.
result The spectrum of the random kernel matrix is asymptotically equivalent to a simpler matrix model through free additive convolution.
The increasingly photorealistic sample quality of generative image models suggests their feasibility in applications beyond image generation. We present the Neural Photo Editor, an interface that leverages the power of generative neural networks to make large, semantically coherent changes to existing images. To tackle…
The study extends Jacobi-orthogonality to indefinite scalar product spaces.
problem Generalizing Jacobi-orthogonality to indefinite scalar product spaces.
method Comparing principles, investigating tensor relations, proving properties.
result Every quasi-Clifford tensor is Jacobi-orthogonal; certain tensors are Jacobi-dual or Osserman.
New characterization of Osserman tensors using Jacobi-orthogonality.
problem Characterizing Osserman tensors.
method Introducing Jacobi-orthogonality as a new potential characterization.
result Jacobi-orthogonal tensors are Osserman, and all known Osserman tensors are Jacobi-orthogonal.
Study isotropy groups for complex orthogonal and skew-symmetric matrices.
problem Understanding isotropy subgroups of orthogonal similarity transformations.
method Analysis of group structure of nonsingular block matrices.
result Group structure of isotropy subgroups related to block Toeplitz matrices.
New findings on Kähler manifolds restrict orthogonal coordinates existence.
problem Existence of orthogonal coordinates on Kähler manifolds.
method Algebraic and geometric techniques applied to Kähler manifolds.
result No nontrivial self-dual Kähler 4-manifolds or Ricci-flat Kähler 4-manifolds support orthogonal coordinates.
Constructs orthogonal coordinates in curved spaces.
problem Separating variables in curved spaces.
method Explicit construction of orthogonal coordinates and transformations.
result Explicit formulas for Killing tensors and Stäckel matrices.
OPT framework improves neural network generalization by learning an orthogonal transformation.
problem Improving neural network generalization.
method Orthogonal over-parameterized training (OPT) framework that minimizes hyperspherical energy.
result OPT framework provably minimizes hyperspherical energy and improves empirical generalization.
The paper studies surfaces in a bounded domain with orthogonal boundaries and proves curvature estimates.
problem Estimating the area of surfaces with orthogonal boundaries in a bounded domain.
method Weak formulation of orthogonality for curvature varifolds, classification of vanishing curvature varifolds.
result Existence of an orthogonal 2-varifold that minimizes L2 curvature in the integer rectifiable class. Orthogonal random features approximate a Bessel kernel, offering sharper bounds than random Fourier features.
problem Approximating Gaussian kernel efficiently for large datasets.
method Use of Haar orthogonal matrices to construct orthogonal random features and analyze their bias and variance.
result Orthogonal random features approximate a Bessel kernel, not the Gaussian kernel, with sharper bounds.
Proposes FOAGP for efficient orthogonal effect decomposition of black-box computer experiments.
problem Challenges in sensitivity analysis of black-box computer experiments with complex, nonlinear functional outputs.
method Functional-output orthogonal additive Gaussian process (FOAGP) with conditional orthogonality constraint.
result Demonstrates effectiveness in orthogonal effect decomposition and variance decomposition through simulations and real-world application.
Novel prior for orthogonal functions improves functional component estimation.
problem Improving orthogonality in functional principal component analysis.
method Sequential adaptive priors for orthogonal functions using hierarchical conditionally normal distributions.
result Proposed prior leads to nearly orthogonal posterior estimates.
Method constructs orthogonal curvilinear coordinates in constant curvature spaces.
problem Creating orthogonal coordinates in spaces of constant curvature.
method Modification of Krichever's method for Euclidean space, applied to constant curvature spaces.
result Examples of orthogonal coordinate systems on the sphere and hyperbolic plane constructed.
ManifoldFlow relaxes fixed-spectrum Stiefel layers to learn a positive spectrum.
problem Fixed-spectrum Stiefel layers impose rigid spectral constraints.
method Introduces ManifoldFlow, a relaxation that learns a positive spectrum while keeping the basis on the Stiefel manifold.
result Learnable SPD spectrum improves performance in various settings.
We study the Chern-Simons partition function of orthogonal quantum group invariants, and propose a new orthogonal Labastida-Mariño-Ooguri-Vafa conjecture as well as degree conjecture for free energy associated to the orthogonal Chern-Simons partition function. We prove the degree conjecture and some interesting cases o…
It is well known that Convolutional Neural Networks (CNNs) have significant redundancy in their filter weights. Various methods have been proposed in the literature to compress trained CNNs. These include techniques like pruning weights, filter quantization and representing filters in terms of a basis functions. Our ap…
A Clifford algebra model for M"obius geometry is presented. The notion of Ribaucour pairs of orthogonal systems in arbitrary dimensions is introduced, and the structure equations for adapted frames are derived. These equations are discretized and the geometry of the occuring discrete nets and sphere congruences is disc…
DONUT improves treatment effect estimation by enforcing orthogonality constraints.
problem Estimating treatment effects from observational data is challenging due to unobserved outcomes.
method DONUT uses a regularization framework that formalizes unconfoundedness as orthogonality, leading to deep orthogonal networks.
result DONUT outperforms state-of-the-art methods in estimating average treatment effects.
New convergence guarantees for learning with unknown nuisance parameters.
problem Learning problems with unknown nuisance parameters.
method Stochastic gradient optimization with Neyman orthogonality and approximately orthogonalized updates.
result Stochastic gradient algorithms can converge under conditions of nuisance parameters.
MuonEq improves training of matrix-valued parameters by rebalancing momentum before orthogonalization.
problem Training matrix-valued parameters with orthogonalized-update optimizers like Muon.
method MuonEq introduces three lightweight pre-orthogonalization equilibration schemes: two-sided row/column normalization (RC), row normalization (R), and column normalization (C).
result Row/column normalization acts as a zeroth-order surrogate for whitening and improves the geometry seen by orthogonalization.
A new algorithm POGO optimizes thousands of orthogonal matrices efficiently.
problem Optimizing thousands of orthogonal constraints at scale is computationally expensive.
method Revisits Landing algorithm, uses modern adaptive optimizers, reduces hyperparameters.
result POGO optimizes thousands of orthogonal matrices in minutes, outperforming alternatives.
Batch normalization makes deep neural networks' representations increasingly orthogonal.
problem Orthogonality of deep neural network representations.
method Random linear transformations in successive batch-normalizations.
result Orthogonality of representations improves SGD performance.