You'll also want a root window and other essential features. This is provided by xfdesktop4 (or you can use an equivalent from another DE). You can use just the window manager if you want but you won't like it. Or you can use something like Openbox which includes everything needed (it's a tiny complete DEs not just a window manager).