St Mary's Chapel was first built about AD 1000 and rebuilt in the 12th century and again in the 14th century. It stands in the ruins of a Roman Signal Station built AD 370 on the castle headland, to give warning of Anglo-Saxon raiders off Scarborough.
Before this time the headland was occupied about 500BC by Iron Age settlers from the Low Countries or the Rhineland.
