Diffusion imaging enables assessment of human brain white matter (WM) in vivo. WM microstructural integrity is routinely quantified via fractional anisotropy (FA). However, FA is also influenced by the number of differentially oriented fiber populations per voxel. To date, the precise statistical relationship between FA and fiber populations has not been characterized, complicating microstructural integrity assessment. Here, we used 630 state-of-the-art diffusion datasets from the Human Con- nectome Project, which allowed us to infer the number of fiber populations per voxel in a model-free fashion. Beyond the known impact on mean FA, variance of anisotropy distributions was drastically impacted, not only for FA, but also the more recent anisotropy indices generalized FA and multidimensional anisotropy. To ameliorate this bias, we introduce a probabilistic WM atlas delineating the number of distinctly oriented fiber populations per voxel. Our atlas shows that the majority of WM voxels features two dif- ferentially directed fiber populations (44.7%) rather than unidirectional fibers (32.9%) and identified WM regions with high numbers of crossing fibers, referred to as cross- ing pockets. Compartmentalizing anisotropy drastically reduced variance in group comparisons ranging from the whole brain to a few voxels in a single slice. In summary, we demonstrate a systematic effect of intra-voxel diffusion inhomogeneity on anisotropy. Moreover, we introduce a potential solution: The provided probabilistic WM atlas can easily be used with any given diffusion dataset to enhance the statistical robustness of anisotropy measures and increase their neurobiological utility.