arXiv research

In theoretical analysis of deep learning, discovering which features of deep learning lead to good performance is an important task. In this paper, using the framework for analyzing the generalization error developed in Suzuki (2018), we derive a fast learning rate for deep neural networks with more general activation …

This paper studies activation sparsity in large language models, finding key trends and implications.

The most widely used activation functions in current deep feed-forward neural networks are rectified linear units (ReLU), and many alternatives have been successfully applied, as well. However, none of the alternatives have managed to consistently outperform the rest and there is no unified theory connecting properties…

Develops active learning for scale-bridging simulations.

Efficiently selects nearest neighbors for labeling to speed up active learning.

Theory of MoE Transformers' generalization and scaling.

Improved MPC with neural networks and active sets for large-scale problems.

Leveraging the wealth of unlabeled data produced in recent years provides great potential for improving supervised models. When the cost of acquiring labels is high, probabilistic active learning methods can be used to greedily select the most informative data points to be labeled. However, for many large-scale problem…

A new framework scales active search for large datasets.

Annotating the right data for training deep neural networks is an important challenge. Active learning using uncertainty estimates from Bayesian Neural Networks (BNNs) could provide an effective solution to this. Despite being theoretically principled, BNNs require approximations to be applied to large-scale problems, …

Model shows loss curve with two distinct exponents due to sparse activations.

Proposes MscaleDNN for solving high-dimensional PDEs efficiently.

New findings connect shaped and unshaped neural networks using differential equations.

New algorithms speed up learning from large screens of proteins.

S2D selectively decays large singular values to improve quantization of neural activations.

Changing initialization scale affects deep model generalization, leading to memorization or improved performance.

Active testing for large language models is made more efficient and accurate.

Market activity scales near a constant of 0.632 in intrinsic time.

SAEs struggle with curved activation manifolds, revealing layer-dependent scaling laws.

Active inference implemented for high-dimensional tasks shows efficient exploration and improved sample efficiency.

Model captures neural activity related to behavior while separating internal computations.

Spike sorting is a fundamental preprocessing step in neuroscience that is central to access simultaneous but distinct neuronal activities and therefore to better understand the animal or even human brain. But numerical complexity limits studies that require processing large scale datasets in terms of number of electrod…

Complex systems comprise a large number of interacting elements, whose dynamics is not always a priori known. In these cases -- in order to uncover their key features -- we have to turn to empirical methods, one of which was recently introduced by Menezes and Barabasi. It is based on the observation that for the activi…

Several recent papers investigate Active Learning (AL) for mitigating the data dependence of deep learning for natural language processing. However, the applicability of AL to real-world problems remains an open question. While in supervised learning, practitioners can try many different methods, evaluating each agains…

We propose the point process model as the Poissonian-like stochastic sequence with slowly diffusing mean rate and adjust the parameters of the model to the empirical data of trading activity for 26 stocks traded on NYSE. The proposed scaled stochastic differential equation provides the universal description of the trad…

This paper investigates the scaling dependencies between measures of "activity" and of "size" for companies included in the FTSE 100. The "size" of companies is measured by the total market capitalization. The "activity" is measured with several quantities related to trades (transaction value per trade, transaction val…

Active Search has become an increasingly useful tool in information retrieval problems where the goal is to discover as many target elements as possible using only limited label queries. With the advent of big data, there is a growing emphasis on the scalability of such techniques to handle very large and very complex …

Dropout schedules can be optimized to significantly reduce model test loss.

Active learning method balances bias and variance under class imbalance.

Improved GP decoder training with SAS approximations.

Intensive development of urban systems creates a number of challenges for urban planners and policy makers in order to maintain sustainable growth. Running efficient urban policies requires meaningful urban metrics, which could quantify important urban characteristics including various aspects of an actual human behavi…

The paper improves Gaussian process models for efficient batch optimization.

Understanding the structure of financial markets deals with suitably determining the functional relation between financial variables. In this respect, important variables are the trading activity, defined here as the number of trades $N$, the traded volume $V$, the asset price $P$, the squared volatility $σ^2$, the bid…

New activation networks improve model efficiency and performance.

We show LLMs can be locally linear, enabling better control of activations.

New algorithms solve large-scale convex regression problems.

We propose a methodological framework to study the dynamics of inter-regional investment flow in Europe from a Complex Networks perspective, an approach with recent proven success in many fields including economics. In this work we study the network of investment stocks in Europe at two different levels: first, we comp…

We define a Hidden Markov Model (HMM) in which each hidden state has time-dependent $\textit{activity levels}$ that drive transitions and emissions, and show how to estimate its parameters. Our construction is motivated by the problem of inferring human mobility on sub-daily time scales from, for example, mobile phone …

Better use of unlabelled data improves Bayesian active learning models.

Recent years have witnessed amazing outcomes from "Big Models" trained by "Big Data". Most popular algorithms for model training are iterative. Due to the surging volumes of data, we can usually afford to process only a fraction of the training data in each iteration. Typically, the data are either uniformly sampled or…

Many neural network architectures rely on the choice of the activation function for each hidden layer. Given the activation function, the neural network is trained over the bias and the weight parameters. The bias catches the center of the activation, and the weights capture the scale. Here we propose to train the netw…

SAP corrects model for label noise by identifying and removing noisy samples.

Method scales up ML science by measuring multiple molecules at once.

Representational similarity analysis (RSA) has been shown to be an effective framework to characterize brain-activity profiles and deep neural network activations as representational geometry by computing the pairwise distances of the response patterns as a representational dissimilarity matrix (RDM). However, how to p…

Deep ResNets exhibit distinct scaling properties with depth, challenging neural ODE models.

We introduce the stochastic multiplicative point process modelling trading activity of financial markets. Such a model system exhibits power-law spectral density S(f) ~ 1/f**beta, scaled as power of frequency for various values of beta between 0.5 and 2. Furthermore, we analyze the relation between the power-law autoco…

This work provides a thorough study on how reward scaling can affect performance of deep reinforcement learning agents. In particular, we would like to answer the question that how does reward scaling affect non-saturating ReLU networks in RL? This question matters because ReLU is one of the most effective activation f…

Interactive machine learning improves learning efficiency with user input.