Individuals differ greatly in their ability to learn and navigate through environments. One potential source of this variation is “directional sense” or the ability to identify, maintain, and compare allocentric headings. Allocentric headings are facing directions that are fixed to the external environment, such as cardinal directions. Measures of the ability to identify and compare allocentric headings, using photographs of familiar environments, have shown significant individual and strategy differences; however, the neural basis of these differences is unclear. Forty-five college students, who were highly familiar with a campus environment and ranged in self-reported sense-of-direction, underwent fMRI scans while they completed the Relative Heading task, in which they had to indicate the direction of a series of photographs of recognizable campus buildings (i.e., “target headings”) with respect to initial “orienting headings.” Large individual differences were found in accuracy and correct decision latencies, with gender, self-reported sense-of-direction, and familiarity with campus buildings all predicting task performance. Using linear mixed models, the directional relationships between headings and the experiment location also impacted performance. Structural scans revealed that lateral orbitofrontal and superior parietal volume were related to task accuracy and decision latency, respectively. Bilateral hippocampus and right presubiculum volume were related to self-reported sense-of-direction. Meanwhile, functional results revealed clusters within the superior parietal lobule, supramarginal gyrus, superior frontal gyrus, lateral orbitofrontal cortex, and caudate among others in which the intensity of activation matched the linear magnitude of the difference between the orienting and target headings. While the retrosplenial cortex and hippocampus have previously been implicated in the coding of allocentric headings, this work revealed that comparing those headings additionally involved frontal and parietal regions. These results provide insights into the neural bases of the variation within human orientation abilities, and ultimately, human navigation.