You will never be able to get everyone to agree on anything and you can't hold a referendum for every artifact.

So as far as responsibility goes, barring edge cases, it should be left upto the government to decide, as they represent the people.

And tbh, this feels like an argument made in bad faith, because this is such a rare case. No government is going to ask for an artifact back and then destroy it. What happened in afganistan and Syria was a tragedy (they didn't ask for those artifacts back, they were already there) But that only happened because the previous governments had been destabilized by Russian and American influences. (Iraq war - Isis, Afganistan war - alqaeda)

There's no clear 'owner' in many cases.

Just return it to the country where it was taken from. And I don't think there are many cases where ownership is vague, most are pretty plain and clear.

then we should prioritize saving the artifacts over the ones that seek to destroy them.

That's not on you, that's on their original keepers. Otherwise you are propagating colonial era crimes and justifying them by arguing in bad faith.

P.s.

• Museums have a notorious record when it comes to maintaining artifacts (they aren't shining beacons of humanity), especially the British museum.
• They also do less than what's needed to discourage artifact smuggling.
• watch: https://youtu.be/eJPLiT1kCSM