Now archaeologists think they know after finding their nearby remains:
Remains found near Stonehenge are almost certainly of an ancient people who built the monument, excited archaeologists have revealed.
Researchers investigating the origins of the seven 4,500 year-old skeletons found buried on Salisbury Plain last year have run chemical tests to trace their origins and age.
They have concluded they are of people from south-west Wales - the same region from where bluestones forming the world-famous iconic monument originate.
Scientists seem dismissive of my theory, however:
Dr Andrew Fitzpatrick, of Wessex Archaeology, said: "In medieval times, people believed that the stones could only have been brought to Stonehenge by Merlin the Wizard.
I'll give it some thought.