they do interact with matter, just incredibly weakly

They do interact with other stuff a little bit. It's very difficult to detect them because they hardly interact with anything, but it's not absolutely nothing so it's not impossible to detect them. This is well beyond my level of physics knowledge, but apparently one such interaction is a process called inverse beta decay. High-energy antineutrinos that crash in to protons produce a pair of particles that is much more easily detectable. A rule of physics called lepton conservation, which is about the fundamental building blocks of particles involved in a reaction not changing, makes this pair of detectable particles identifiable as being caused by an incoming antineutrino.