Charlbury started life as an anglo-saxon settlement in a small clearing in the Wychwood forest on the eastern bank of the Evenload river.  'Bury' is an anglo-saxon word meaning "fortified place" and "Carl" was probably the name of a leader of the local people.

King Henry III granted Charlbury the right to hold a weekly market in 1256 making it a town rather than a village.  The town was controlled by the Abbey of Eynsham until the reformation.

After the reign of Henry VIII, the Lee family of Ditchley - whose primary residence can still be found a short walk along the "Ditchley Way' - took control of of the town for the next 200 years until the Churchill family (later the Dukes of Marlborough) replaced them.  Lee Place, the former dower house of Ditchley and now the home of Rosita Spencer-Churchill, Duchess of Marlborough can be found  a little further up Park Street from Grammar School Cottage.

Cornbury Park was originally a hunting lodge in the Royal Wychwood forest, probably built by Henry I in the early 12th century.  Most of what we can see today is 16th or 17th century.

Charlbury railway station was built by Isambard Kingdom Brunel in the chalet style and is the only remaining example of its type.