The way I sorted it was to run nextcloud for a week, then run ps aux on the host and see what the memory use of a php process is.
The 5th column is the memory use of a process, divide the number into the amount of memory you want PHP to use. The number from ps is bytes, so you will need to use some maths to make it all fit
in Debian running PHP-FPM in /etc/php/{{ php_version }}/fpm/pool.d/www.conf
edit or add the below lines with the settings you need
pm = dynamic
pm.max_children = 8
pm.start_servers = 2
pm.min_spare_servers = 2
pm.max_spare_servers = 3
Also MySQL has some options you can change to use more or less memory, this handy tool MySQLTuner is your best way to get the options