There's a physical switch in the back to alternate between the two. Pretty sure you unknowingly changed it. The 2.4Ghz connection through the dock/dongle is much better than Bluetooth anyway.
To map extra buttons you need to switch the controller to d-input (turn it off, then restart it while holding B) to give Steam Input direct access to them. You need to be on one of the latest firmwares, so you need access to (a friend's) Mac or Window computer to update it.
https://gist.github.com/barraIhsan/783a82bcf32bed896c85d27dbb8018a5