Recent work has established the importance of top-down influences on early sensory processing. These influences are alternatively taken as representing task context, attention, expectation, working memory and motor commands (Gilbert & Li, 2013). At the same time, top-down influences have been recognized as essential for supporting probabilistic inference (Lee & Mumford, 2003). Here, we combine the latter idea with the recent hypothesis that the brain implements probabilistic inference using a neural sampling-based representation (Fiser et al 2010) and show that a normative account can indeed explain several disparate empirical observations on the effect of task context, expectation and attention on neuronal response gain and interneuronal response correlation. In particular, we show for a classic 2AFC task, that neural sampling can explain the task-dependent correlations seen by Cohen & Newsome (2008) and the choice probability time-course observed by Nienborg & Cumming (2010). Furthermore, we propose that top-down attention due to unequal rewards acts as a loss-calibration of the sampling approximation to the true posterior and show that this hypothesis entails an increase in response magnitude for neurons at the attended location as well as a decrease in noise correlations (Cohen & Maunsell, 2009, Mitchell & Reynold, 2009). Unlike previous accounts which assumed these effects to be the source for an improved psychophysical performance at the attended location, in our account, they are a consequence of probabilistic inference with changing constraints.