This is the only real answer - it is not possible to do proper capacity planning without trying the same workload on similar hardware [1].
Some projects give an estimation of resource usage depending on a number of factors (simultaneous/total users...) but most don't, and even the estimations may be far from actual usage during peak load, with many concurrent services, etc.
The only real answer is close monitoring of resource usage and response times (possibly with alerting), and start adding resources or cutting down on resource-hungry features/programs if resource usage goes over a certain threshold (~80% is when you should start paying attention) and/or performance starts to degrade.
My general advice is to max out installed RAM from the start, virtualize your hosts (which make it easier to add/remove resources or migrate a hungry VM on more powerful hardware later), and watch out for disk I/O on certain workloads (databases... having db engines running off SSDs helps greatly).