I can only talk for Germany:

Regional trains (in the 200km range) are usually commissioned by the municipalities and train companies then get paid for providing that service. Of course it depends on the route, but on average the municipalities pay around 50% of the cost and the other half is paid for by fares. They do that, because having a good train connection is good for the economy and lowers congestion, so they are obviously interested in having good tracks. There's a lot of smaller companies there, but the push comes from the municipalities. Also for one route there's usually one company to choose from, so no competition on the customer level.

Long Distance trains are for profit, but there are only very few companies competing with the state-owned DB Fernverkehr on a handful of routes.

So I don't think competition alone helps. The German rail network is in ruins btw. Maybe not in comparison to the US, but in comparison to the 1980s when there was no competition on the rail at all. The main reason for that is that we had incompetent carbrains as ministers of transportation for the majority of the last 30 years, leading to a massive underfunding.
That has slightly changed with the Ampel government in 2021 and we may hope that the new conservative minister keeps going in the same direction, as in times of war the rail network is valued even by conservatives. But fixing the errors of the past will take us at least a decade of construction work and delay.

That's a good point, the population in Europe is much more evenly spread. But that still doesn't explain why a train journey from San Francisco to Los Angeles has to be 12 hours long.

Also, I guess you guys do not regularly travel from New York to Los Angeles for a weekend trip, just as we Europeans don't usually do that with Stockholm and Barcelona (which is a distance the average European would also travel by plane).
We do however travel from Frankfurt to Strasbourg for a day trip, and of course we do that by train.

Lisbon to Tromsø ~3200 miles

Try Lisbon to Oulu: 4750km. I also explained why the land area is irrelevant when we talk about distances. If you're looking for a distance even closer to New York — Chicago (1271km) I could offer Vienna — Paris (1235km) which is also served by NightJet. Driving for one more hour doesn't really matter while you're asleep.

No, it is a 13 and a half hour drive according to Google, which would be according to the speed limit. We Americans would do it in a single day because hotels are expensive.