I think that part of the problem with the RTS genre in a PvE mode is that, at least insofar as I've played the thing, it doesn't really lend itself to a long game life.

There are basically two ways that I've seen RTSes played in PvE mode.

The Campaign

You tend to have a campaign, which is really oriented around learning the concepts and units. These tend to have static maps. I enjoy those, but I don't know how many times you could play through a campaign. Maybe one could make a dynamic map generator, but I don't know how much it would alter gameplay from campaign to campaign -- you need to achieve the kind of roguelite situation, where the changing elements actually force you to change up gameplay in interesting ways, keep the play fresh and keeps the game replayable. A lot of what keeps the campaign being interesting is new units being introduced over time, and that is kind of a one-off thing. If you had other players creating new maps, I'm not sure how much ability they'd have to add interest, once you've learned the existing units.

Like, for an RTS campaign to be playable for the long run, I think that you'd need to have some kind of dynamic map generation that not just looks convincing, but also scales up difficulty in interesting ways. Maybe generate a story too. Maybe some way to introduce interesting mechanics with the map (and RTS campaigns certainly have had map formats that support scriptable events). But...I haven't seen anything that really aims at that. Against the Storm is sort of RTSy and is designed to have procedurally-generated maps, but while there's a campaign on the overmap, the actual in-game battles don't have much story or concept of a campaign.

The Skirmish

Then you have what I'll call "skirmish", because that's the term Total Annihilation used -- where you just play what is essentially a multiplayer game against the AI. That's also PvE in a sense. The limitation there is that RTS AIs have been kind of disappointing. It doesn't usually take too long to figure out the holes in the AI's logic. The kind of "meta" and bluffing that exists in multiplayer isn't something that the game AI can do. I think that one would really need to somehow change how things work to wind up with better AIs. And "better" isn't just "stronger" -- the AI can potentially do some things that a human might not be able to do, like micromanage many units effectively. It's gotta be fun.

Maybe it'd be possible to have a generic AI engine, like the AI equivalent of what Havok is for physics, if a lot of the work here is common across games. It'd have to be pretty flexible, too, since some AI is gonna be game-specific. That'd potentially let there be more work on a per-game basis on AI.

Another possibility might be making it the norm for the AI to be decoupled from the main game and the API to be exposed, maybe even just leave the AI's source open, and let modders and the like work on AIs. Haven't seen developers try that that I can recall.

Maybe it'd be possible to sell AI packages as DLC. There have been a few games I can think of with "enemy AI" that was kind of modular -- Rimworld has different "storytellers". That'd let ongoing work be done on the AI.

I like playing Wargame: Red Dragon (real time tactics, not real time strategy) single-player but the AI in that game is pretty horrendous and doesn't play at all the way a human would; I think that few people would want to play it single-player the way I do. So I hear you on that, wish that there were a way to come up with more-interesting AI behavior.

Other

I don't know where your specific concern is when it comes to casual players. Complicated concepts? One might be the kind of heavy micro required to play RTSes well.

I think that some of that focus on expert players might have been from the Starcraft world. Blizzard intentionally made a game that required heavy micromanagement by doing things like limiting selection group sizes -- I remember a developer talking about this. But that's not necessarily the route that RTSes had to go -- Total Annihilation went down a more-automated route, where units could be instructed to act more-autonomously. One could imagine RTSes that focused more on the high level, where control of individual units isn't all that useful.

Another "expert player" constraint might be the focus on highly-optimized build orders. Like, one typically needs to have a build that snowballs, and one needs to manually build things at precisely the right time to play optimally. I can imagine maybe setting up advancement to happen automatically, and just choosing in advance which direction one wants to go; that makes managing the build queue not a micro operation. Some games (Sins of a Solar Empire) have had pretty extensive queues for building and research. The focus isn't on remembering to build at the right times, but on choosing which direction to go.

I think that some of the question is also "what makes RTSes fun"? Like, does one enjoy playing through a long campaign? Is it that the element of bluffing or adaptation to one's tactics or some other human behavior isn't present in the AI? Following the "meta"? The feel of exploring a map (and if so, is the map needing to be interesting; is procedural generation practical?) Is it optimizing one's build order? Is it forcing one to micromanage well?