We investigated feature ensemble encoding at the lowest level of visual processing by focusing on contour encoding in natural images. In such images, the mean contour is not a single value, but it varies locally with spatial position, and variability of the contour can be quantified by the noisiness of the contour segments. We used a novel image decomposition/recomposition method, three different classes of images (circular patterns, object and fractal images), and two types of noise (orientation and position noise) to generate stimuli for a 2-AFC pedestal noise discrimination task. We found that humans readily encoded variability of contour ensembles, this encoding systematically varied with image classes, and it was distinctively different for orientation versus position noise despite participants not being able to reliably distinguish between the two types of noise. Moreover, JND obtained with mixed orientation and position noise followed the optimal maximum likelihood estimate, supporting a probabilitic coding of contours in humans.