The problem is though that inference by itself is going to end up a low margin utility service that there will be loads of players offering, they'll never recoup their costs that way.

The only path to profitability I can imagine is to have a model that is vastly superior to what people can get elsewhere that they can somehow lock people into using and then charge them well over the cost of inference alone. None of which looks likely to happen.