I don't think there is any limit. The amount of nestings shouldn't even really be a database field. I think Lemmy just stores which comment was replied to and then has recursion logic to display it right.