While both Pet and sari are right, I feel a point is being missed here.

The Pantheon, though a church, suffered greatly under the popes. It had the gilded roof tiles stripped from the exterior of the dome for their metal. The bronze roof beams of the portico were stolen and used by Bernini to make his baldacchino over the altar in St. Peter's. Crappy bell towers were erected on it. It is a testament to the Roman builders that it lasted despite the Church.

But why? One word: pozzolana. It was the magic stuff that made Roman concrete what it was, particularily of the early empire. It is a superb binder and also allows it to set up under water. Not only do we see its use in the Pantheon, but all over Ostia. The stuff comes from the Vesuvius region, ancient Putoli, modern Pozzoli. It is a relatively short sail up the coast to Rome with the stuff. Now, I would hazard a guess that it was not too terribly cost effective to drag it up to Londinium for much other than a few special cases, if at all. It would likely have been almost as expensive as marble. One would imagine the builders had to make do with local substitutes that were not quite up to the same quality as that in Rome and environs. Also, just how do we know that much of Roman Londinium was really brick and mortar and not some wood construction except for official buildings. Given the changes in economic times, those would have been the first to have been "quarried/harvested" for their structural elements, leaving scant behind.