Eldritch pacts are meant to be agreements between a master and an apprentice, kind of like the Sith, or tradesmen in real life. The Warlock receives knowledge and resources from the Patron, and the Patron gets the Warlock's service in return. If the pact is broken, the Warlock loses the ability to continue to learn from the Patron, leverage their resources and influence, etc., but they do not lose the knowledge they've already learned (unless that was a specific stipulation in the pact).