The Brodricks first occupied Ballyannan Castle, a fortified house in the townland of that name, which was in ruins by 1837. In 1786 Wilson refers to Ballyanan"belonging to Lord Viscount Midleton". On the first Ordnance Survey map there is a building named Ballyannan House (W867 715) right on the shore close to the castle and there is a house in the townland of Garryduff named Ballyannan. Ballyannan was occupied by Roger Adams in 1814 and by J. Adams in 1837. At the time of Griffith's Valuation the representatives of Viscount Midleton occupied a house and offices valued at £19 in the townland of Garryduff. The house in the townland of Ballyannan was valued at £9.10 shillings and was occupied by Daniel Twomey and held from the Viscount.