Oh no, filler is a good thing. Filler gives you time to know the characters, and adds depth and color to the world. Filler is where writers actually get to stretch and try out ideas. Filler is what makes a show feel full.
Imagine the X-Files with no filler. We'd lose the Jose Chung episodes, "Home," "the Post-Modern Prometheus," and so many other great episodes. Without the filler, it's just an endless slog through Chris Carter's poorly planned mythology. Just the smoking man and vanishing babies for eleven nine seasons.