Efficiently classifies binary labels with XOR queries, even under noisy conditions.
problem Binary classification with unknown labels using XOR queries.
method Effective query type and an efficient inference algorithm for noisy conditions.
result Achieves information-theoretic limit on optimal number of queries.
This paper models the crowdsourced labeling/classification problem as a sparsely encoded source coding problem, where each query answer, regarded as a code bit, is the XOR of a small number of labels, as source information bits. In this paper we leverage the connections between this problem and well-studied codes with …
Study on neuron dynamics for XOR classification with zero-margin.
problem Understanding neural network training dynamics in zero-margin classification problems.
method Analysis of Gaussian XOR problem, focusing on neuron block dynamics and generalization without margin assumptions.
result Neurons cluster into four directions and block-level signals evolve coherently, essential for reliable prediction in the Gaussian setting.
SGD quickly learns a spurious XOR feature before the signal feature, revealing learning dynamics.
problem Over-reliance on spurious correlations in neural networks trained by SGD.
method Theoretical analysis of SGD on two-layer ReLU networks trained on XOR data.
result SGD learns the spurious feature first and exponentially fast, dominating the signal feature.
We show linear XOR classification is possible and propose equality separation for anomaly detection.
problem Linearly separating XOR data.
method Equality separation, adapting SVM objective for data within/outside margin.
result Equality separation can detect both seen and unseen anomalies.
FleXOR trains fractional quantization for neural networks, improving accuracy and size.
problem Quantization limits to integer bits restricts compression and accuracy.
method Encryption algorithm with XOR gates for fractional bits during inference.
result FleXOR achieves high accuracy with fractional sub-1-bit weights.
SGD efficiently learns the XOR function with near-optimal sample complexity.
problem Learning the XOR function with a 2-layer neural network.
method Minibatch SGD on a 2-layer neural network with ReLU activations, focusing on signal-finding and signal-heavy phases.
result Achieves population error o(1) with dextpolylog(d) samples. The exclusive or (xor) function is one of the simplest examples that illustrate why nonlinear feedforward networks are superior to linear regression for machine learning applications. We review the xor representation and approximation problems and discuss their solutions in terms of probabilistic logic and associative …
Neural networks learn to mimic brain neurons with two-input activation functions, improving performance and robustness.
problem Training neural networks to mimic the complex interactions of brain neurons.
method Developed a network-in-network architecture with two-input activation functions, optimized hyperparameters, and compared to conventional ReLU networks.
result Two-input activation functions can learn soft XOR functions, improving network performance and robustness.
Training an artificial neural network involves an optimization process over the landscape defined by the cost (loss) as a function of the network parameters. We explore these landscapes using optimisation tools developed for potential energy landscapes in molecular science. The number of local minima and transition sta…
When applying Machine Learning techniques to problems, one must select model parameters to ensure that the system converges but also does not become stuck at the objective function's local minimum. Tuning these parameters becomes a non-trivial task for large models and it is not always apparent if the user has found th…
Single neuron with ADA learns XOR and outperforms other functions.
problem Classifying linearly non-separable data.
method Proposed a new artificial neuron with apical dendrite activation.
result ADA function achieves 100% accuracy on XOR and superior performance on benchmark datasets.
Empirical evidence suggests that neural networks with ReLU activations generalize better with over-parameterization. However, there is currently no theoretical analysis that explains this observation. In this work, we provide theoretical and empirical evidence that, in certain cases, overparameterized convolutional net…
Neural networks can overfit perfectly to noisy data and then grok near-optimal generalization.
problem Neural networks' ability to overfit perfectly to noisy data and then generalize near-optimally.
method Two-layer ReLU networks trained by gradient descent on XOR cluster data.
result Neural networks can achieve perfect fit to noisy training data and then grok near-optimal generalization.
Study of eigenvalues in nonlinear kernels for classification of separable data.
problem Understanding the applicability of linear equivalents in nonlinearly separable data classification.
method Analysis of conjugate kernels and their quadratic equivalents for a canonical nonlinearly separable dataset (XOR problem).
result Identification of regimes where nonlinear kernels deviate from linear equivalents, leading to label-aligned eigenspaces.
Study of local optima in neural networks for feature interactions.
problem NNs struggle with local optima in feature interactions for small datasets.
method Proposed a node pruning and feature selection algorithm to improve NN performance.
result NNs have many non-equivalent local optima in XOR-like data with irrelevant variables.
New method trains Boltzmann machines without supervision.
problem Training unsupervised learning models.
method Mixed binary quadratic feasibility problem formulation.
result Theory validated on XOR patterns.
New method disentangles feature importance scores in machine learning.
problem Misinterpretation of feature importance scores due to interactions and dependencies.
method Derive DIP (Disentangled Importance) decomposition of feature importance scores.
result DIP decomposition uniquely separates standalone contributions from interactions and dependencies.
Model compression techniques, such as pruning and quantization, are becoming increasingly important to reduce the memory footprints and the amount of computations. Despite model size reduction, achieving performance enhancement on devices is, however, still challenging mainly due to the irregular representations of spa…
New measure LMN explains neural network grokking.
problem Delayed generalization after memorization in neural networks.
method Defined LMN to measure network complexity, showing LMN correlates with test losses linearly.
result LMN reveals intriguing XOR network behavior and is a promising complexity measure.
Overparameterisation can limit the benefits of curriculum learning in neural networks.
problem The ineffectiveness of curriculum learning in deep learning applications.
method Analytical study connecting curriculum learning and overparameterisation in an online learning setting for a 2-layer network in the XOR-like Gaussian Mixture problem.
result High degree of overparameterisation can limit the benefit from curricula.
Study on neural networks forgetting in continual learning.
problem Forgetting in neural networks during continual learning.
method Gradient descent analysis on XOR-cluster datasets.
result Explicit bounds on forgetting rate and generalization gap.
Paper proposes GrokTransfer to eliminate delayed generalization in neural networks.
problem Delayed generalization in neural networks, compromising predictability and efficiency.
method Trains a smaller, weaker model to reach a nontrivial test performance, then uses its learned input embedding to initialize the stronger model.
result GrokTransfer enables the target model to generalize directly without delay, across various tasks.
Explaining neural network computation in terms of probabilistic/fuzzy logical operations has attracted much attention due to its simplicity and high interpretability. Different choices of logical operators such as AND, OR and XOR give rise to another dimension for network optimization, and in this paper, we study the o…
For the tensor PCA (principal component analysis) problem, we propose a new hierarchy of increasingly powerful algorithms with increasing runtime. Our hierarchy is analogous to the sum-of-squares (SOS) hierarchy but is instead inspired by statistical physics and related algorithms such as belief propagation and AMP (ap…
A natural p-classes generalization of the eXclusive OR problem, the subtraction modulo p, where p is prime, is presented and solved using a single fully connected hidden layer with p-neurons. Although the problem is very simple, the landscape is intricate and challenging and represents an interesting benchmark for grad…
Reformulates binary classification on manifolds using Yang-Mills-Higgs theory.
problem Binary classification on non-contractible spaces.
method Formulates binary classification as a Yang-Mills-Higgs variational problem, encoding data as a functor.
result Reveals a geometric interpretation of binary classification and solves XOR on the torus.
Solves TOD systems' query annotation problem without explicit annotations.
problem Training TOD systems without explicit KB query annotation.
method Reinforcement learning (RL) and pipelined approach for query prediction and system training.
result Improved RL agent with modifications for TOD tasks.
The paper tackles sequential mode estimation with oracle queries.
problem Adaptively PAC-learning a probability distribution's mode.
method Two query models: index queries and pair queries. Sequential algorithms for mode estimation.
result Lower bounds on optimal query complexity for both models.
Study exact community recovery in noisy SBM with limited queries.
problem Community recovery in noisy stochastic block models with limited queries.
method Balanced uniform querying, two-stage adaptive strategy, sublinear queries, subsampled graph.
result Adaptive querying can improve exact recovery limits in noisy SBM.
Proposes a new query autocompletion method that maximizes retrieval performance.
problem Users often select suboptimal queries due to unknown best retrieval performance.
method Formulates query autocompletion as ranking item rankings, uses counterfactual learning.
result Empirical results show improved query suggestions for better retrieval performance.
Adapts large transformer model for search query intent understanding.
problem Understanding and predicting user intents from search queries.
method Adapts BERT-like architecture for search queries, accounts for query noisiness and sparseness.
result Builds a shareable deep learning model for query intent identification.
A new method for private query release using Johnson-Lindenstrauss projection.
problem Private release of query answers with minimal privacy loss.
method Random projection of query answers to a lower dimension, followed by noise addition.
result Optimal worst-case sample complexity for answering a workload of k queries.
A new random forest algorithm uncovers feature interdependencies better than traditional methods.
problem Tackles the sub-optimality of greedy decision tree implementations in random forests.
method Presented a 'stepwise lookahead' variation of random forests that considers multiple split nodes simultaneously.
result Significantly outperforms greedy random forests in uncovering feature interdependencies, especially in high-noise environments.
LAZO reduces query complexity and variance in ZO methods.
problem High query complexity and variance in zeroth-order optimization.
method LAZO uses adaptive lazy queries to reduce variance and save queries.
result LAZO achieves lower regret and query complexity compared to existing methods.
Estimates heavy hitters in data streams with queries, balancing accuracy and efficiency.
problem Identifying elements with high probability in i.i.d. samples.
method Sequential estimation algorithms for two query models: index and pair queries.
result Upper and lower bounds on query complexity for different distributions and noise models.
Query2box embeds complex queries as boxes to handle logical operations in large KGs.
problem Handling complex logical queries on large-scale incomplete knowledge graphs.
method Embed KG entities and queries into a vector space as boxes, handling conjunctions as intersections and disjunctions through Disjunctive Normal Form.
result Query2box achieves up to 25% relative improvement over state-of-the-art methods.
Study modular concept learning with different oracle interfaces.
problem Learning a concept that is a cross product of component concepts.
method Analyze different types of oracle interfaces and queries.
result Modular concept learning is easier with positive examples and membership queries.
Survey of statistical queries and their applications.
problem Understanding statistical queries and their applications.
method Exploration of statistical queries model, definitions, and connections to learnability.
result Connections to learnability and applications in optimization, evolvability, and differential privacy.
We study the query complexity of a learner-private sequential learning problem, motivated by the privacy and security concerns due to eavesdropping that arise in practical applications such as pricing and Federated Learning. A learner tries to estimate an unknown scalar value, by sequentially querying an external datab…
Source coding is the canonical problem of data compression in information theory. In a locally encodable source coding, each compressed bit depends on only few bits of the input. In this paper, we show that a recently popular model of semi-supervised clustering is equivalent to locally encodable source coding. In this …
Lower bounds on queries for Bayesian private learning.
problem Estimating a target within error ε while keeping it hidden from an adversary.
method Lower bound proof using Fano's inequality and proportional-sampling estimators.
result Query complexity is on the order of Llog(1/ε) for accurate estimation. A new query embedding method improves KB performance on complex queries.
problem QE systems disagree with deductive reasoning on some queries.
method A novel QE method more faithful to deductive reasoning.
result Better performance on complex queries to incomplete KBs.
In this paper we study the adaptive learnability of decision trees of depth at most d from membership queries. This has many applications in automated scientific discovery such as drugs development and software update problem. Feldman solves the problem in a randomized polynomial time algorithm that asks $\tilde O(2^…
We study black-box attacks on machine learning classifiers where each query to the model incurs some cost or risk of detection to the adversary. We focus explicitly on minimizing the number of queries as a major objective. Specifically, we consider the problem of attacking machine learning classifiers subject to a budg…
Fairly allocate items with noisy queries, reducing envy.
problem Fairly allocate indivisible goods with unknown valuations.
method Use Gaussian noisy queries to find an envy-free allocation.
result The optimal number of queries scales as \( \frac{m^{2.5}}{Δ^2} \) for large negative-envy.
In query learning, the goal is to identify an unknown object while minimizing the number of "yes" or "no" questions (queries) posed about that object. A well-studied algorithm for query learning is known as generalized binary search (GBS). We show that GBS is a greedy algorithm to optimize the expected number of querie…
We introduce new combinatorial quantities for concept classes, and prove lower and upper bounds for learning complexity in several models of query learning in terms of various combinatorial quantities. Our approach is flexible and powerful enough to enough to give new and very short proofs of the efficient learnability…