Eh, living with themselves was punishment enough. I'm just sorry for the few level-headed outcasts who had to live thinking they were weird or pretending to fit in so they wouldn't be persecuted.

Well the plot twist is, they were generally exceptionally smart at what they needed to know to survive. It's easy to forget how difficult life was for average people up until fairly recently. Like less than a century ago. Education and literacy really weren't a priority.