Very little of the Castle survives today. It would have been a motte and bailey type and located at the top of a hill overlooking the land to the South and East. The site of the inner bailey is occupied by the 18th century circular bowling green. This is situated above the Castle Hotel, which occupies the site of the outer bailey and is believed, like many local houses, to have been contructed from materials salvaged from the stone fortress. Many large ornate timbers are to be found around the town, out of all proportion to the size and stature of the dwellings.

A short section of wall on land owned by the Old Castle Land Trust in Castle Street is all that can be seen of the castle today, although several of the houses situated around The Square also incorporate hidden parts of the original stone structure.

Like many castles in this area, it was built soon after the Norman conquest by the Bishops of Hereford who were appointed Marcher Lords to protect the troubled frontier of England, vulnerable to attacks by the Welsh. The first references to this particular castle can be found in records dated 1087, a year after the Domesday book. The Normans established a number of plantation towns in the Marches to encourage english settlement and to generate trade. This new town was established in 1127, and laid out as burgage plots either side of the line between the castle and the church now occupied by High Street and Church Street.

At this time, the castle was still constructed of earth and timber, but by 1167, a stone fortress was erected to protect the town, known at this time as Y Trefysgob (in Welsh - town of the bishops). The castle was completely refurbished in 1281, and was maintained until around 1570 when the town, castle, and surrounding lands were taken into possession by the Crown. Queen Elizabeth the First commissioned a survey of all crown possessions, and the illustration above dates from that time, although its accuracy is in some doubt. Elizabeth soon sold the estates and the castle was abandoned. By the time of the Civil War in 1645, the castle was ruined, and the stone reused.