A couple of reasons:
- Who would contribute? Banks are highly regulated and sometimes deal with complex products that most Devs don't have a background on. Even most Devs in banks rely on a team of business analysts, designers etc to shape the requirements. Add on top of that the general negative perception of banks, I can't think of a large open-source community forming.
- Competition. Bank's primarily compete with each other. They all offer very similar products, and any advantage they can gain by developing proprietary software will be explored.
- Third-party apps. Banks use a TON of third-party apps behind the scenes. A lot of times they will purchase licenses for existing products and then customise on top of that.
- Outsourcing. Even when they are building the app "in-house" they may have outsourced the development to another company, and will then just maintain the finished product.
- Banks move slooooowly. As it's a highly regulated industry, every deployment needs to go through a ton of red-tape. An exploit found in public might take weeks to be resolved internally.
- Reward is not worth risk. It simply isn't a priority and they can't see any benefit for doing it. It's more likely to cause a reputation risk than not.