I'd only ever say it while referring to myself, and when I do it's not of a consolation to myself or maybe as a way to tell the other person to not feel sorry about distracting me and making me forget. Is that the same way you interpret it?

Every single time that's happened to me and I later remembered what it was, it wasn't important.

Not important to them, very important to you.

I meean, if it was really important, it's very unlikely you would forget it. We use that saying a ton here