You're right, the first amendment wasn't about freedom of expression, it was about not having to pay for books.
Using the word free to describe something that doesn't restrict you has been a thing for centuries. "Free Software" has been the accepted term within the software world to denote freedom respecting, libre, and open source software since the 80's.
This isn't about because Richard Stallman said so. Its because its the definition pretty much everyone, especially those who've actually touched a compiler, uses.