To me, that's not an argument for regulating AI, though, because most regulation we can come up with will benefit those with deep enough pockets to buy themselves out of the problem, while solving nothing.

E.g. as I've pointed out in other debates like this, Getty Images has a market cap of <$2bn. OpenAI may have had a valuation in the $90bn range. Google, MS, Adobe all also have shares prices that would trivially allow them to purchase someone like Getty to get ownership of a large training set of photos. Adobe already has rights to a huge selection via their own stock service.

Bertelsmann owns Penguin Random-House and a range ofter publishing subsidiaries. It's market cap is around 15 billion Euro. Also well within price for a large AI contender to buy to be able to insert clauses about AI rights. (You think authors will refuse to accept that? All but the top sellers will generally be unable to afford to turn down a publishing deal, especially if it's sugar-coated enough, but they also sit on a shit-ton of works where the source text is out-of-copyright but they own the right to the translations outright as works-for-hire)

That's before considering simply hiring a bunch of writers and artists to produce data for hire.

So any regulation you put in place to limit the use of copyrighted works only creates a "tax" effectively.

E.g. OpenAI might not be able to copy artist X's images, but they'll be able to hire artist Y on the cheap to churn out art in artist X's style for hire, and then train on that. They might not be able to use author Z's work, but they can hire a bunch of hungry writers (published books sells ca 200 copies on average; the average full time author in the UK earns below minimum wage from their writing) as a content farm.

The net result for most creators will be the same.

Even wonder why Sam Altmann of OpenAI has been lobbying about the dangers of AI? This is why. And its just the start. As soon as these companies have enough capital to buy themselves access for data, regulations preventing training on copyrighted data will be them pulling up the drawbridge and making it cost-prohibitive for people to build open, publicly accessible models in ways that can be legally used.

And in doing so they'll effectively get to charge an "AI tax" on everyone else.

If we're going to protect artists, we'd be far better off finding other ways of compensating them for the effects, not least because it will actually provide them some protection.