So even if it's 4mm away? 5mm? 6mm? 7mm? 8? 9?

If it were that far away, contact would not happen, and thus no injury. To be clear, a graze or tangential gunshot wound requires contact to occur. The hydrostatic shock occurs only when there is physical contact to allow a transfer of kinetic energy.

You argue as if you can be equally injured in a car crash by being a bystander because of the energy of a speeding car. Like, no?

A key part that you're missing in that comparison is the area over which the contact occurs. So, it would be roughly like a 1956 VW Beetle hitting someone in the ear at 60mph. However, this also isn't quite accurate or the full picture because it would require somehow getting an equivalent ratio of surface area contacting the ear, which would be much greater than ear-sized on the VW because a 5.56mm round is so small, as well as the additional forces exerted because of the sonic blast wave that the bullet causes.

Bullets work by transferring immense amounts of energy to a small surface area. High-velocity bullets (those flying at supersonic speeds, like all modern cartridges that are not specifically designed to be subsonic), have an additional effect of causing hydrostatic shock (some research suggests that this may, to a lesser degree, occur with subsonic rounds as well). What that means is that a component of the sonic blast wave participates in the transfer of energy to tissue (we're big bags of water), causing a radiating pressure wave that evidence shows can cause fractures in bone not impacted by the bullet, as well as damage to nearby internal organs and nerves.

Another great way of understanding the importance of surface area to the damage inflicted by rapid kinetic energy transfer would be Blendo. A battle robot built by the MythBusters guys that, as a "weapon" used a flywheel weighing roughly 100lbs (45kg) that was spun up to around 400RPM. The energy in that flywheel was transferred to the opposing robots in a very small surface area, causing such devastating effects that they were asked to withdraw from the competition.

This is very similar to how bullets work and one of a number of reasons that even a graze from 5.56mm bullet that hadn't first lost a significant amount of its energy is very unlikely. The wound being able to heal in a week with no visible scarring (not to mention suppression of any medical records from being used in the investigation or revealed to the public) makes that possibility even more vanishingly small.