It stands actually in a smaller valley off the main dale, and even without its roof and with its interior in ruins, it retains a most grim and depressing atmosphere: visitors are not surprised to learn that one of its earliest owners, a lord named de Soulis, was reputedly a magician who had sold his soul to the Devil, and was punished for his crimes by being boiled alive in a great cauldron.

In the sixteenth century, Hermitage was commanded by a Captain, who held the place for the Keeper of Liddesdale, and, since in addition to its moat system it was guarded by the Hermitage Water and by marshy approaches, it was a very strong place indeed . It needed to be, for it was supposed to dominate the most turbulent valley in the Borderland.

From the castle's guidebook: "...they shall deliver to the King's highnes of Englond ...the castell called the Hermitage with all the lordschipis and landes belonging to the same ..." -- from a secret agreement between King henry VII of England and Archibald Douglas, Earl of Angus, dated 1491

The isolated valley of the Hermitage Water, in Liddesdale, is said to have been the home long ago of a holy man of God. It is certainly a lonely spot. The feeling of foreboding is heightened by the presence of the awesome castle ruin, the inspiration of local legends, of the wicked Lord Soulis and of a giant with impregnable armour. In truth, though, Hermitage needs no myths. It already has a history of family murders, illicit affairs, romantic trysts, torture, treason and intrigue sufficient for a host of castles.

For most of its 400-year existence, Hermitage, "the strength of Liddesdale", was the key to control over much of the Border country, "the guardhouse of the bloodiest valley in Britain", and as such it was highly prized and much fought over. Even the building of it in 1242 brought England and Scotland to the brink of war. That first timber castle has gone, but the mighty earthwork defenses remain. The present stone castle took shape gradually over the course of the later fourteenth and fifteenth centuries. Begun by the English Lord Dacre, it was remodelled and greatly enlarged by the powerful Scottish noble family of Douglas and Angus. Later, the castle passed to the Earls of Bothwell, and it was during their ownership that Hermitage was the scene of one of the most well-known adventures of Mary Queen of Scots, in the course of which she almost died.

On 8 October 1566 Bothwell was wounded in a skirmish by Little John (also Jacques or Jock) Elliot of the Park, a noted reiver, and taken to Hermitage to recover. Two days later Mary, oblivious of this event, set out from Edinburgh on her annual progress which happened to be in the east Borders. Hearing of Bothwell's fate, and in the face of rumours which were already linking them romantically, Mary rode out from Jedburgh on 16 October to visit Bothwell. She was accompanied by several courtiers whose role was to protect not just the Queen but also her reputation. The tryst lasted two hours. Being married to Lord Darnley, Mary could not stay at the castle and so she and her party made the arduous journey back to Jedburgh, a 25-mile (40km) ride across difficult terrain. Mary stumbled into a bog en route, contracted a fever and was confined to her bed in Jedburgh for a week.

As the arrival of gun powdered artillery changed the face of warfare, so great castles like Hermitage lost their advantages. Attempts to modify defences, such as the great ravelin that dominates the castle's western approach, were never destined for success. The place was probably forsaken as a noble residence by the seventeenth century and gradually decayed until the haunting ruin in its lonely site beside Hermitage Water became a magnet for Sir Walter Scott and his contemporaries.