I feel that the biggest mistake of X11's protocol design is the idea of a "root window" that is supposed to cover the whole screen.
Perhaps that worked greatly in the 1990s, but it's just completely incompatible with multi-displays that we commonly see in modern setups. Hacks upon hacks were involved to make multi-displays a possibility on X11. The root window no longer corresponded to a single display. In heterogenous display setups, part of the root window is actually invisible.
Later on we decided to stack compositing on top of the already-hacky mess, and it was so bad that many opted to disable the compositor (no Martha, compositors are more than wobbly windows!).
And then there's the problem of sandboxing programs... Which is completely unmappable to X11 even with hacks.