These stones were mostly removed from the great pyramid just 700 years ago. After an earthquake people began removing them for a few centuries to build other things until they were all gone. The great pyramid had its casing stones for 85% of its life.
Edit: I highly recommend "history for granite" on youtube.
