No, it's correct as it stand. I think you're having trouble reading it?

I turned away from beliefs without evidence IN SPITE OF the fact that I had a catholic upbringing.

Despite. That's how that is used.