I think you are describing some notion of a "will" or motive.. but also potentially describing an LLM's lack of temporal experience. I would argue that a human is constantly being "prompted" to react to things happening to them via sensory input. And adding that to an LLM is trivial. (Provided the input is of a modality that it can understand like text or image embeddings).

As far as will or motive to perform tasks goes, some think an AI agent could generate secondary sub-goals like a will to "survive" in order to carry out primary tasks like "make paperclips efficiently". This is called instrumental convergence and its speculative. I think what would really be scary is if someone explicitly optimized a model with billions of parameters to survive or carry out some specific task and they utilized online reinforcement learning. I dont think there is a big technical hurdle there.. you could imagine a sort of adversarial style training where one model predicts damage/danger/threats and the other attempts to avoid those. We could propagate rewards and punishment back over the sequence of actions that led to that state and train as the model is interacting with its environment.