Both have their roots in anarcho-capitalist movements and the idea that The State must be abolished.
The root anarchy and libertarian aspect of that is the idea that The State must be abolished. The difference is that (libertarian flavored) socialism is largely based on the idea of small communities to replace The State (and if that sounds contradictory...). Whereas (modern) libertarianism, is that the idea is that you replace The State with individual enclaves with the remnants existing solely to protect those rights. The "libertarian" aspect fundamentally boils down to the idea of individual liberty (hence the name) at the cost of The State and the distinctions between that and anarchy is, to put it bluntly, questionable.
It would be like insisting that those in favor of socialist democracies rebrand themselves as anarcho-communists because it sounds cooler and there IS a direct line between libertarian communists/anarcho-communists and modern socialist democratic thought. Which ignores that there is a reason that said thought evolved.