The original castle dates from the 11th century (1283) when it was built by Urse D'Abitot, Warden of the Welsh Marches. It was built in red sandstone in a pentagonal shape overlooking the river Severn. Little now remains of the original four round towers and square barbican, and the stonework is mainly 15th and 18th century. The tower itself dates from the 14th century.

With no male heir it passed through Urse's daughter into the ownership of the Beauchamp family where it remained until 1420. The Beauchamp family's interesting history includes honour at Crecy and the French Wars, as well as execution for treason.

Purchased by Sir John Bourne (who was Secretary of State between July 1553 and April 1558) from Beauchamp descendants, it was in turn sold to Sir Thomas Bromleigh [Bromley], Lord Chancellor of England, whose son Sir Henry, a member of the Essex circle, was heavily involved in the Essex Rebellion of 1601. In 1603 he was released and pardoned by James I and his estates were restored to him. Sir Henry Bromley, a staunch Puritan was constantly at odds with his neighbours, the Habington's of Hindlip who were devout Catholics and known to be harbourers of Jesuit priests.

As the local magistrate he was ordered by the government to search and apprehend the Jesuit fugitives Garnet and Oldcorne. They were taken back to Holt Castle where they were allowed to recuperate for a few days before being transported back to London.