source: arxiv statistics ml: hedging on the frontier: learning new tasks with few samples
level: research
when a learner faces a new task with few samples, it must use any available side information. in practice, this often comes from model evaluations on related tasks in public benchmarks. a key question is how to model task relatedness so that it is realistic and the benchmark evaluations lead to provable gains. empirically, weak monotonicity is often approximately satisfied: if a model dominates another on many benchmarks, it also tends to outperform on the new task.
the paper explores the statistical complexity of learning under approximate weak monotonicity. it uses this property within two learning paradigms: transfer learning and model selection aggregation. the approach not only prunes the model class based on monotonicity but also adapts to the geometry of available trade-offs by hedging on the frontier. this means selecting models that are not dominated by others on the benchmarks, forming a frontier of non-dominated options.
the method provides theoretical guarantees for improved performance when few samples are available. by focusing on the frontier, the learner can combine models in a way that balances risk across different possible new tasks. this is especially useful when the relationship between benchmarks and the new task is only weakly monotonic, which is common in real-world scenarios. the work bridges the gap between empirical observations and rigorous learning theory.
why it matters: this method helps data scientists select and combine models for new tasks with limited data, using only public benchmark results.
source: arxiv statistics ml: hedging on the frontier: learning new tasks with few samples