You'll understand why this obscure setting is #1 on the list when you consider the benchmarks of drupal_load below (If you're not using APC, or have no idea what APC is I suggest you keep that fact a secret, and install it now):

Without APC: 286ms
APC with 32m shm_size: 700ms (default APC setting)
APC with 64m shm_size: 37ms!!!