It is strategically sited in a narrow neck of the Lower Usk Valley, where the river is crossed by a beautiful five arched stone bridge. A Norman Castle built on a small hill immediately north of the town, dominates views of the town and is said to have been built on the site of an earlier hill fort. Edward IV was born in the castle in 1460, fifty years after Owain Glyndwr's son Griffith had attacked it and paid for his folly with the loss of 1500 soldiers and his own life.

The Welsh name for Usk is Brynbuga which means "hill of Buga" and commemorates the legend of Buga the giant, who allegedly lived on the hill. In Roman times, the town was closely connected to Caerleon four miles to the south where there was a large Roman encampment said to accommodate a Roman Legion.

There are several early 14th Century buildings of historic and architectural interest within the town, notably the priory, old gaol house and the market hall.