That is an interesting question. From what I know, youtube has every video in chunks that they serve to the client, and so server side ad injection is just serving some ad chunks before the video. I think you're right with the buffer thing, it seems to me like the only way to make sure the client can't skip it would be to make the buffer shorter, impacting some people (although seems like only really people with internet thats fast enough for streaming some seconds, but not other seconds, which is an odd catagory)

Ultimately it would be a tradeoff for youtube, but the fact that they put the effort into doing mass testing of the idea at all shows that clearly there are some good incentives, and it may eventually be implemented.

Don't worry, I have been using the powerful technology of "the mute button" and "doing other stuff" since before cable TV existed. We always have alternatives.