So 4.5 is bad in both of those scenarios? So how is the number of people having rated matter?

Personally I always look further than the AVG number anyway. As other commenters have said none of this means anything if you don't check negative reviews for reoccurring issues.