Efficient coding is a widely established principle for how the brain allocates sensory resources in order to improve the encoding of perceptual (and other) variables. Typically, efficient coding assumes adaptation to long-run statistics of the environment. Yet, evidence suggests the brain also adapts efficiently and near-instantaneously to rapid changes in stimulus statistics. Here, to unify efficient coding at different time scales, we develop a model of context-dependent efficient coding (CoDEC) that combines previously suggested principles of Bayesian efficient coding and contextual inference. According to CoDEC, the brain uses trial-by-trial contextual inference to identify the current task context, which in turn guides the gradual refinement of context-dependent stimulus statistics on long time scales. Crucially, CoDEC learns the transition structure between task contexts from the training curriculum. Using this structure, CoDEC performs dynamical contextual inference to control both resource allocation through efficient coding (based on the learned stimulus distributions of each previously encountered context and their respective inferred probabilities in the current trial) and the updating of context-dependent stimulus distributions. We show that CoDEC accounts for a range of seemingly contradicting results from “roving” paradigms that revealed a strong dependence of perceptual learning (PL) on task structure, which have been difficult to reconcile with canonical, feed-forward “reweighting”-based models of PL. Beyond replicating classical PL and capturing all previously reported roving effects, CoDEC also makes novel predictions about the existence of “latent learning” under a random roving training curriculum that can be rapidly unmasked by introducing predictability during testing. By putting context-dependence center stage, CoDEC highlights the importance of feedback, as opposed to purely feed-forward, neural mechanisms of efficient coding and PL.