The Barony of Blair was granted by King William of the Scots, surnamed the Lion, in the middle of the 12th century to one Jean François, a man of Norman descent. Jeanís son changed his name to Blair and he appears to have married a daughter of King John of England. He was called William and held the Barony in 1260. His successor, Sir Bryce de Blair was an adherent of Sir William Wallace and was executed by the English at the Barns of Ayr in 1296. Sir Bryce was succeeded by his brother David, whose son, Roger, was knighted by King Robert the Bruce for his services before and at Bannockburn. There is a stone on the old Keep which reads "Roger de Blair and Marie Mair, his spouse". This lady come from Rowallan and spelt her name either Muir or Mair. Her sister married King David II, a son of Robert the Bruce. From the fourteenth century onwards the Blairs continued to prosper at Blair and the estate descended in a direct line until 1752.

The house is difficult to date exactly. The oldest part is the quadroom tower, which was probably built before 1200. The next pole tower or keep was built in or about 1202. There are battlements under the high-pitched roof, over the two oldest towers but it is not known when the high roof was added: probably in the late 17th or early 18th century.

During the reign of Mary, Queen of Scots, John Blair was faithful to the Queen. He died during the reign of James VI. His grandson, Bryce, succeeded in 1610 and died in 1639. He and his son, William, were both adherents of Charles I and both were knighted.

William married Margaret, daughter of the 2nd Duke of Hamilton and they left Blair when Cromwell came to Scotland to deal with those of the Scots nobility and people who were loyal to the House of Stuart. They returned early in the 1660s and built the south wing of the house, whose gables show the lily of France and the initials W B and L M H. William Blair declared for King William and was captured by the Earl of Dundee; he died a prisoner on Skye. Her ladyship did not approve of her husband's disloyalty to the Stuarts. Their son, William, succeeded his father and was in Government service; he married a Campbell. Their daughter succeeded in due course and married William Scott who changed his name to Blair on marriage. The lady died at the birth of their son, who died unmarried before his father, who had married again and produced several sons, the eldest of whom succeeded to Blair: Hamilton Blair was a major in the Royal Scots Greys and carried out many improvements on the estate. His son, William, was MP for Ayrshire and planted many of the old trees still to be seen. He opposed the Reform Bill of 1832 and subsequently lost his seat. His son, William Fordyce Blair was a captain in the Royal Navy, and married a Miss Sprot from the Borders. The park and policies are due to this William: he it was who opened the policies to the people of Dalry. He built the Town Hall and subscribed largely to the rebuilding of St. Margaretís Church in the 1860s. He died in 1888 and was succeeded by the only surviving son, Colonel Fredrick Gordon Blair, the father of Miss Blair. Colonel Blair married Miss Mary Baird of Rosemount (near Ayr). Colonel and Mrs Blair modernised the house in 1893 and built a large addition to the west of the 1668 wing.