Old books describe clearly where Oxford Castle stood. It was close to St. George's Church, and not far from a water-mill; the stream that turned this mill flowed past the town, supplying water to the big moat which surrounded the castle, and which was crossed by a strong bridge. The most ancient form of the crest or coat-of-arms of Oxford shows a castle, a winding stream, and a bridge. There is a curious drawing of the castle, made by Ralph Agas, in 1538, during the reign of Henry VIII., though some people think he has put the round tower, or keep, in the wrong place. This keep is the last part of Oxford Castle to be left standing; the rest has gone.

It is difficult to find out when Oxford Castle was first built. It is certain that it dates from the time of the Saxons. There is a tradition that King Offa built the original castle, which would mean some date in the eighth century, and the great King Alfred was probably often at Oxford, staying at the castle. In the collections of Saxon coins, round in Oxford, there are some coins of his time. Then the son of Canute was crowned at Oxford, and lived for a while at the castle, but he reigned only four years. About 1791, the remains of old walls were found, immensely thick, with some remarkable wells. These walls were thought to be Saxon. Thus we pass on till the Normans conquered England, when there is proof that this castle was rebuilt by one Robert d'Oiley. The Conqueror divided the possessions of the Saxons freely among those who came over with him, and this man had Oxford Castle given to him. He rebuilt it in 1071, keeping, perhaps, some of the old fabric. In the year 1141, the Empress Maud, who had escaped from Devizes on a funeral bier, covered up as if dead, reached Oxford, and there she was again besieged. It seemed likely the castle would be taken, and she would be seized by her enemies, but we are told that she managed to escape again. Accompanied by three knights, she got out of Oxford to a place of safety.

At some date in the reign of Henry III., Oxford Castle had its walls strengthened, and the round tower was rebuilt. It was then, probably, that the towers were made along the embattled walls, and especially one of those peculiar towers called a barbican, contrived so as to give an outlook on approaching foes. These barbicans had a device by which hot water or stones could be flung down upon any enemy who succeeded in passing the bridge. King Charles I. was often a visitor to Oxford Castle, and after the wars between Parliament and King were over, some other changes were made in the defences of the castle. After the Revolution, it was allowed to decay gradually.