We already do! And on the cheap! I have a Coral TPU running for presence detection on some security cameras, I'm pretty sure they can run LLMs but I haven't looked around.

GPT4ALL runs rather well on a 2060 and I would only imagine a lot better on newer hardware