Brahan Castle was built by Colin Mackenzie, 1st Earl of Seaforth in 1611. Kenneth Mackenzie, a laborer on the estate, was a reputed seer who made a number of prophecies in the later 17th century. He is remembered as the Brahan Seer.

The Mackenzies were prominent Jacobites, and took part in the Jacobite risings of 1715, 1719 and 1745. The Siege of Brahan took place in November 1715. William Mackenzie, 5th Earl of Seaforth, was attainted, forfeiting the estate, which in 1725 became the headquarters of General Wade during his "pacification" of the Highlands. After the later Jacobite rising of 1745, the Mackenzies were the first clan to surrender, being forced to swear allegiance to the British Crown on the steps of the castle.

The estate was later sold back to the Mackenzie family, although the direct line of descent died out in 1781. The estate passed to Francis Mackenzie, 1st Baron Seaforth who carried out tree planting in the grounds. On his death in 1815, Brahan passed to the baron's eldest daughter Mary Elizabeth Frederica Mackenzie who married James Alexander Stewart of Glasserton in 1817. During the first half of the 19th century, the castle was rebuilt and extended as a large country house.

James Stewart-Mackenzie was created Baron Seaforth in 1921, but on his death without heir in 1923, he left the estate to a trust. Brahan Castle was briefly requisitioned during World War II, and after the war its condition deteriorated. In the early 1950s the building was demolished, leaving only the north wall of the 19th century building, which served as a garden ornament. The stable block survives, and is now known as Brahan House. Several heraldic panels and other decorative stones are preserved in the house.