The first mention of a castle is in a document dating from about 1200, during the reign of William the Lion. This early castle stood about one mile inland from the point, and probably consisted of a wooden tower, possibly on a mound, surrounded by a palisade and ditch. The castle would later have been rebuilt in stone and probably changed in layout and expanded several times. We know little about the castle, but it is clearly shown on Timothy Pont’s map of about 1600, at “Innerbraky”.

From about 1500 or earlier, Inverbreakie was in the hands of the Innes family. It was sold to the Rosses of Morangie in the 1630s. In 1702, Sir William Gordon acquired the “lands of Inverbreckie, fishe boats, weekly mercats and faires” from George Ross. Sir William changed the name of the estate to Invergordon and made improvements to the castle and policies. After his death in 1742, his son, Sir John Gordon, set about building a new mansion in the classical style. This was designed by architect John Adam and was completed in about 1755, after which the medieval castle was left to fall into decay.

Sir John died in 1773. As he had no children, the estate passed to his nephew, Lord Macleod, who sold it to the Macleods of Cadboll. In 1805 the new mansion was badly damaged by fire, and for some time after this the Macleod family was only able to occupy one wing of the house. It was not until 1872 that any further major building work took place, when Robert Bruce Aeneas Macleod commissioned a grand new mansion house in Elizabethan style, reputedly with 48 bedrooms. Both the old castle and 18th century house were pulled down to make way for the new building.

Maps and records show that the castle policies were laid out from the 17th century with tree-lined avenues, formal gardens and planned woodland. From about 1860 the Hilton of Cadboll stone, a spectacular and important example of 9th century Pictish carving, stood in the American Gardens, now part of the golf course. East, South and West Lodges controlled the approaches to the castle. These were probably built in the mid-18th century at the same time as Sir John’s classical mansion. Only the West Lodge still survives.

In the 1920s the estate was sold to Sir William Martineau, who also owned nearby Kincraig House. The Hilton of Cadboll stone was sent to the British Museum, from where it was hurriedly restored to the Royal Museum of Scotland after a huge public outcry. The mansion of 1872 was demolished, bringing to an end the 800 year story of Invergordon Castle.