A common assumption about neural networks is that they can learn an appropriate internal representations on their own, see e.g. end-to-end learning. In this work we challenge this assumption. We consider two simple tasks and show that the state-of-the-art training algorithm fails, although the model itself is able to r…
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,695 papers · 148 categories
Trend · papers per month
48 results for “internal representations”
DORA analyzes deep neural networks' internal representations to detect spurious correlations.
problem Detecting spurious correlations in deep neural networks' internal representations.
method DORA uses Extreme-Activation (EA) distance measure to assess representation similarities.
result Identifies internal representations capable of detecting spurious correlations.
Current machine learning techniques proposed to automatically discover a robot kinematics usually rely on a priori information about the robot's structure, sensors properties or end-effector position. This paper proposes a method to estimate a certain aspect of the forward kinematics model with no such information. An …
Neural networks learn distance-based representations, not just intensity.
problem Understanding how neural networks interpret and learn from internal activations.
method Manipulated ReLU and Absolute Value activations to observe sensitivity to distance and intensity perturbations.
result Neural networks are highly sensitive to small distance-based perturbations, challenging the intensity-based interpretation.
Method determines latent dimensionality in international trade flows.
problem Finding meaningful low-dimensional latent features in high-dimensional international trade data.
method Proposes a latent dimension determination method based on clustering of nonnegative RESCAL decompositions.
result Validates the latent features against empirical economic facts.
Enhances RL agents with predictive internal representations.
problem Improving model-free reinforcement learning agents.
method Introduces Deep InfoMax (DIM) objective to train predictive internal representations.
result Successfully learned predictive representations in synthetic settings.
Recurrent neural networks (RNNs) are a vital modeling technique that rely on internal states learned indirectly by optimization of a supervised, unsupervised, or reinforcement training loss. RNNs are used to model dynamic processes that are characterized by underlying latent states whose form is often unknown, precludi…
Conformal Prediction in Hierarchical Classification with Constrained Representation Complexitystat.ML
New algorithms for hierarchical classification using conformal prediction.
problem Valid prediction sets in hierarchical classification tasks.
method Extended split conformal prediction framework with two inference algorithms.
result Empirical evaluations show effectiveness in achieving nominal coverage.
MuZero visualizes its internal representations to stabilize planning.
problem Stability issues in MuZero's planning process.
method Visualized MuZero's latent representations and proposed regularization techniques.
result Action trajectories diverge between observation embeddings and internal state transitions, leading to instability.
Graphs are general and powerful data representations which can model complex real-world phenomena, ranging from chemical compounds to social networks; however, effective feature extraction from graphs is not a trivial task, and much work has been done in the field of machine learning and data mining. The recent advance…
This paper evaluates and validates cluster results using external and internal evaluation methods.
problem Evaluating and validating the quality of clustering results.
method External evaluation using Homogeneity, Correctness, and V-measure scores; internal evaluation using Silhouette Index and Sum of Square Errors.
result Validation of the number of clusters using dendrogram and statistical frequency distribution.
Homomorphism Autoencoder -- Learning Group Structured Representations from Observed Transitionscs.LG
Autoencoder learns group representations from actions, improving future prediction accuracy.
problem Learning internal models of interactions with the real world.
method Homomorphism autoencoder with group representation trained on equivariance-derived loss.
result Agents can predict future actions with improved accuracy.
Probably the most important problem in machine learning is the preliminary biasing of a learner's hypothesis space so that it is small enough to ensure good generalisation from reasonable training sets, yet large enough that it contains a good solution to the problem being learnt. In this paper a mechanism for {\em aut…
New approach extracts AI model representations for steering and monitoring.
problem Understanding and steering AI models' internal knowledge representation.
method Feature learning advances for scalable linear concept representations.
result Concept representations enable steering and monitoring of AI models.
A new framework detects anomalous inputs to DNNs.
problem Detecting adversarial and out-of-distribution inputs in DNNs.
method Unsupervised meta-algorithm with configurable components based on statistical testing.
result The proposed method effectively detects adversarial and OOD inputs.
This study investigates abrupt learning dynamics in Transformers, revealing plateau formation and internal representation collapse.
problem Abrupt learning in Transformers, particularly during the loss plateau.
method Investigates mechanisms of abrupt learning in shallow Transformers, focusing on attention maps and hidden states.
result Reveals plateau formation, internal representation collapse, and strong repetition bias in outputs.
LLMs prefer Bitcoin under crisis frames, affecting financial decisions.
problem Testing whether LLMs have built-in biases towards specific financial assets.
method Developed a three-level audit protocol to examine Bitcoin's representation and influence in LLMs.
result An identifiable internal feature in LLMs can be perturbed to move financial choices, but only within measurable limits.
Study finds neural dialog models struggle with conversational tasks.
problem Insufficient understanding of dialog by neural models.
method Analysis of internal representations and evaluation of model performance.
result Neural dialog models lack key conversational skills like answering questions and inferring contradiction.
Goal-directed manipulation of representations is a key element of human flexible behaviour, while consciousness is often related to several aspects of higher-order cognition and human flexibility. Currently these two phenomena are only partially integrated (e.g., see Neurorepresentationalism) and this (a) limits our un…
Bidirectional Quaternion Long-Short Term Memory Recurrent Neural Networks for Speech Recognitioneess.AS
Recurrent neural networks (RNN) are at the core of modern automatic speech recognition (ASR) systems. In particular, long-short term memory (LSTM) recurrent neural networks have achieved state-of-the-art results in many speech recognition tasks, due to their efficient representation of long and short term dependencies …
We introduce the BriarPatch, a pixel-space intervention that obscures sensitive attributes from representations encoded in pre-trained classifiers. The patches encourage internal model representations not to encode sensitive information, which has the effect of pushing downstream predictors towards exhibiting demograph…
New algorithm constrains SOMs to create supervised low-dimensional mappings.
problem Creating supervised mappings in neural networks with known internal topology.
method Developed Supervised Topological Maps (STMs) by modifying SOMs to incorporate target distances.
result STMs allow for supervised generation of new data with known internal structure.
The worldwide trade network has been widely studied through different data sets and network representations with a view to better understanding interactions among countries and products. Here we investigate international trade through the lenses of the single-layer, multiplex, and multi-layer networks. We discuss diffe…
Vision Transformers show different internal representations compared to CNNs.
problem Understanding how Vision Transformers solve image classification tasks.
method Comparative analysis of ViT and CNN architectures on image classification benchmarks.
result ViT has more uniform representations across all layers, while CNNs have more varied representations.
Moment Pooling reduces latent space dimensions in machine learning models.
problem High-dimensional latent spaces in machine learning models are hard to interpret.
method Moment Pooling extends Deep Sets networks to arbitrary multivariate moments.
result Latent dimensions as small as 1 can achieve similar performance to higher dimensions.
The utility of learning a dynamics/world model of the environment in reinforcement learning has been shown in a many ways. When using neural networks, however, these models suffer catastrophic forgetting when learned in a lifelong or continual fashion. Current solutions to the continual learning problem require experie…
New model explains how concepts grow based on experience.
problem Existing models assume fixed representation; new model allows for growth.
method Geometric framework with MDL criterion for basis extension.
result Conceptual growth is selective and conservative, exposing or amplifying residual error.
Reinforcement learning (RL) algorithms allow artificial agents to improve their selection of actions to increase rewarding experiences in their environments. Temporal Difference (TD) Learning -- a model-free RL method -- is a leading account of the midbrain dopamine system and the basal ganglia in reinforcement learnin…
Two-Stage Stochastic International Portfolio Optimisation under Regular-Vine-Copula-Based Scenarioscs.CE
In this paper, we present a two-stage stochastic international portfolio optimisation model to find an optimal allocation for the combination of both assets and currency hedging positions. Our optimisation model allows a "currency overlay", or a deviation of currency exposure from asset exposure, to provide flexibility…
Study benchmarks 26 clustering validity measures.
problem Determining the best clustering solution from candidates.
method Enhanced revision of previous methodology with three sub-methodologies.
result Comprehensive evaluation of 26 internal validity indexes.
TabPFN's internal geometry topology correlates with dataset reliability.
problem Understanding TabPFN's behavior on structurally difficult tabular geometries.
method Using zigzag persistent homology, studying TabPFN's internal representations on synthetic tabular tasks with known topology.
result Topology of TabPFN's internal representation geometry is strongly associated with dataset-level reliability.
Using the work of Bonahon-Dreyer and Fock-Goncharov, one can construct a real-analytic parameterization for the PSL(n,R) Hitchin component of a surface S, that is explicitly analogous to the Fenchel-Nielsen coordinates on the Teichmuller space of S. Given a Hitchin representation, we give a lower bound on the "length" …
We propose a novel learning method for multilayered neural networks which uses feedforward supervisory signal and associates classification of a new input with that of pre-trained input. The proposed method effectively uses rich input information in the earlier layer for robust leaning and revising internal representat…
New approach uses prior knowledge to improve neural network representations.
problem Difficulty in learning robust internal representations without a large training set.
method Incorporates prior information into a pre-trained reasoning module.
result Improves representation learning in diverse self-supervised learning settings.
Transformers infer tasks from context via two modes, geometrically shaped task vectors explain their behavior.
problem Understanding how transformers infer tasks from context and the geometric properties of task vectors.
method Synthetic setting to train small transformers, mathematical characterization of task-vector geometry and inference modes.
result Task-vector geometry shapes in-distribution and out-of-distribution behavior of transformers.
The discovery of novel materials and functional molecules can help to solve some of society's most urgent challenges, ranging from efficient energy harvesting and storage to uncovering novel pharmaceutical drug candidates. Traditionally matter engineering -- generally denoted as inverse design -- was based massively on…
Sparse Transformers degrade semantic information first, with early layers encoding more.
problem Understanding how sparse Transformers affect learned representations and semantic information.
method Probed Transformers with progressively pruned weights to observe changes in semantic information and model behavior.
result Complex semantic information is first to degrade in sparse Transformers, with early layers encoding more.
Study examines flaws in probing LLMs' knowledge and introduces a new method.
problem Flaws in existing methods for probing the veracity of LLMs' internal knowledge.
method sAwMIL (Sparse-Aware Multiple-Instance Learning) combining multiple-instance learning with conformal prediction.
result LLMs encode a third type of signal distinct from true and false.
Recurrent neural networks (RNNs) are powerful architectures to model sequential data, due to their capability to learn short and long-term dependencies between the basic elements of a sequence. Nonetheless, popular tasks such as speech or images recognition, involve multi-dimensional input features that are characteriz…
An internal model of the own body can be assumed a fundamental and evolutionary-early representation as it is present throughout the animal kingdom. Such functional models are, on the one hand, required in motor control, for example solving the inverse kinematic or dynamic task in goal-directed movements or a forward t…
This work characterizes how data augmentation shapes neural representations.
problem Understanding the impact of data augmentation on neural network representations.
method Embedding neural network hidden representations into a metric space invariant to transformations, analyzing shape-space trajectories.
result Increasing data augmentation strength leads to well-behaved trajectories in the embedded space, and different augmentation types steer representations in distinct directions.
Develops non-standard analysis for coherent risk estimation.
problem Estimating coherent risk measures in financial contexts.
method Non-standard analysis, hyperfinite representations, discrete Kusuoka formulae, plug-in asymptotics.
result Uniform almost sure consistency and asymptotic normality of spectral plug-in estimators.
Internal Lagrangians derived from variational principles.
problem Reproducing the principle of stationary action in variational geometry.
method Introducing stationary points of internal Lagrangians, establishing connections with symmetries and conservation laws, and investigating relations between non-degenerate and internal Lagrangians.
result Noether's theorem reformulated in terms of internal Lagrangians.
Model predicts internal fraud in retail banking is cyclical and influenced by corruption.
problem Predicting and mitigating internal fraud losses in retail banking.
method Developed a dynamic model considering internal factors and macroeconomic indicators.
result Internal fraud losses are pro-cyclical and positively affected by corruption perceptions.
Deep networks are well-known to be fragile to adversarial attacks. We conduct an empirical analysis of deep representations under the state-of-the-art attack method called PGD, and find that the attack causes the internal representation to shift closer to the "false" class. Motivated by this observation, we propose to …
A 12D spinor encodes fermions in a 4D Kaluza-Klein model.
problem Encoding fermions in a 4D spacetime from a higher-dimensional perspective.
method Using a spacetime with , encoding fermions in 64 spinor components.
result The 64 spinor components couple to Standard Model gauge fields in chiral representations.
Transformer architectures show significant promise for natural language processing. Given that a single pretrained model can be fine-tuned to perform well on many different tasks, these networks appear to extract generally useful linguistic features. A natural question is how such networks represent this information in…
New measure shows how LSTM models compose hierarchical representations.
problem Understanding how LSTM models capture compositional structure in language.
method Novel measure of interdependence between word meanings in LSTM internal gates.
result High interdependence can hurt generalization and reveals hierarchical structure learning.