To get to the village and fight the spirits, you need to receive three invitations. Invitations are granted for completing daily tasks on the global map. Look for an icon   over the location and head there.