So one thing that happens in history is, people stop paying to maintain things or governments don't care anymore, so people start taking them apart and using it for building materials.
Let's say you're a modern day contractor, and need $10,000 worth of brick... or you could go and uh just take some from that old library that no one uses anymore, except some homeless dudes camp in it. You could take down maybe just like one small part of it and not spend ANYTHING on brick. Other contractors are already doing this... No one will care.
Basically what happened to the library of alexandria if I remember correctly. It was less fire and more just hundreds of years of people stopping caring.