shoulda makes vastly more sense, and is precisely the kind of thing that tends to become slang and eventually just the standard spelling, like "dunno" or "gotcha".

"should of" just feels like people know it should be two words, but don't know how it's spelled, and instead of just spelling it phonetically they make something half-phonetic up.