Despite recent findings of sequential effects in perceptual serial decision making (SDM) (Chopin & Mamassian 2012; Fischer & Whitney 2014), SDM is typically investigated under the assumption that the decisions in the sequence are independent or at most, are influenced by a few previous trials. We set out to identify the true underlying internal model of event statistics that drives decision in SDM by investigating and modeling a set of novel sequential 2AFC visual discrimination tasks by humans and rats. Participants solved the same decision task across trials, but experienced one shift in baseline appearance probabilities of noisy stimuli during the experiment. We found non-trivial interactions between short- and equally strong long-term effects guiding evidence accumulation and decisions in such SDM. These interactions could elicit paradoxical and long-lasting net serial effects, for example, a counterintuitive negative decision bias towards the recently less frequent element. Our findings cannot be explained by previous models of SDM that either assume a sequential integration of prior evidence, presume an implicit compensation of discrepancies between recent and long-term summary statistics, or adjust learning rates of those statistics at change points. To provide a normative explanation for the empirical data, we developed a hierarchical Bayesian model that could simultaneously represent the priors over the appearance frequencies and a potentially non-uniform noise model over the different stimulus identities. The results of simulations with the model suggest that humans are more disposed to readjust their noise model instead of updating their priors on appearance probabilities when they observe sudden shifts in the input statistics of stimuli. In general, regardless of the simplicity of the decision task, humans automatically utilize a complex internal model during SDM and adaptively alter various components of this model when detecting sudden changes in the conditions of the task.