There are actually two Ardross Castles – one was a c.14th century castle that was located in Elie and Earlsferry, Fife, Scotland, near the sea built by the Dishington family but sold to Sir William Scott of Elie in 1607. At the end of the 17th century is passed to Sir William Anstruther.  The castle is in ruins, with the vaulted basement visible above ground, along with the ruins of a later block.

The second is in a rural area in the Highland region of Scotland, 30 miles north of Inverness.

The 1st Duke of Sutherland bought Ardross in the late 1700s, and built a hunting lodge. In 1845, the 2nd Duke sold the estate to Alexander Matheson.

Sir Alexander Matheson (1805-86) born in Attadale, Ross-shire, the nephew of James Matheson (see Lews Castle) was a founder of Matheson & Co. which traded in tea and opium, and was a merchant bank with branches in India and China. Having amassed considerable capital from this successful business, he returned to Scotland in 1839 and purchased Ardross, amounting to 60,000 acres, for £90,000. He embarked on developing the estate, with the intention of attracting tenants to agricultural tenancies, under the supervision of William MacKenzie, an engineer who acted as factor. Between 1845-54 2,600 acres of land were 'reclaimed by means of trenching, draining, liming,' and '67 miles of dykes, and 11 miles of wire-fencing erected, 28 miles of roads made, and 3000 acres of ground enclosed and planted' (Gardeners Chronicle, 1875). Matheson improved estate workers' housing as well as reclaiming land, so that by 1875 agricultural tenants had increased from 109 to over 500, with an arable acreage of 1,200.

The architect Alexander Ross (1834-1925) was commissioned to re-design Ardross Castle in the Scots Baronial style. This incorporated the earlier mansion and added some 30 rooms, at a cost of c £7,000. Ross was supervisor for roads and buildings of the Highland Railway, of which Matheson was the first Chairman.

Matheson laid out pleasure grounds said to extend to 700 acres 'with the Alness River winding its way through the middle of them. The walks through the pleasure-grounds are upwards of 14 miles in length, their width varying from 5 to 6 feet. They have all been properly bottomed with stones, and finely covered over with gravel.' (Gardeners Chronicle, 1875). These walks along the Alness River and Tollie Burn, gave access on both banks for fishing and incorporated scenic views, pools and waterfalls. Flower gardens lay to the west of the Castle, between the Castle and the kitchen garden. These were arranged to either side of a broad walk with displays of ribbon bedding along the face of a 300 foot long embankment. East of the Castle were shrubberies and broad lawns, set with an oval pond and fountain enclosed by iron railings (Gardeners Chronicle, 1875). Ornamental tree planting started in the 1840s and continued through the latter part of the 19th century (Gardeners Chronicle, 1875; TROBI,1989). Elsewhere on the estate 2,020ha (5,000 acres) of plantation were laid out. The grounds were open to the public. Following Sir Alexander's death, his son, Sir Kenneth Matheson, sold the estate in 1898.

The new owner, also a successful business man, was C. W. Dyson Perrins (1864-1958), a Captain in the Highland Light Infantry, with interests in the Worcester Royal Porcelain Company and Lea & Perrins (Worcester sauce). The family spent several months annually at Ardross, with house parties enjoying the grouse moors, fishing and deer forests. Dyson Perrins continued Matheson's scheme of estate improvements: introducing electricity, purchasing additional lands at Glencalvie and Diebidale and modernizing the Castle. The East Lodge was built by Ross and MacBeth (1898) and the Pinetum extended.

A major addition was the Formal Garden, designed by Edward White (c 1873-1952) for the east front. A perspective of White's design drawn by C. E. Mallows in 1909 was exhibited at the Royal Academy. By 1903 White, a landscape gardener, married to the daughter of Henry Ernest Milner (c 1845-1906), was in charge of the landscape practice 'Milner, Son & White'. Following the Milners' tradition, White worked with the company Pulham & Son, who supplied rockwork and artificial stone features for Ardross. The Bromsgrove Guild of Applied Arts designed the statuary and ironwork for the gardens.

The estate was broken up and sold in 1937, although Perrins later bought back Achandunie, the former factor's house. Mr & Mrs Austin Mardon purchased Ardross Castle, Lealty Farm and over 80 acres and lived there until 1983, when the estate was sold.