Is it better to start multiple erlang nodes per machine, or just one per machine?

Question

Preface: When I say "machine" below, I mean either a physical dedicated server, or a virtual private server. When I say "node" I mean, an instance of the erlang virtual machine, of which there could be multiple running as separate processes under a single unix kernel.

I've got a project that involves multiple erlang/OTP applications. The applications will be running together and talking to each other on the same machine. They will all be hitting the disk, using memory and spawning erlang processes. They will also be using network resources because they will be talking to similar machines with the same set of applications running on them in a cluster.

Almost all of this communication is via HTTP. Thus I could separate each erlang OTP application into a separate instance of the erlang VM on the same machine and they could still talk to each other.

My question is: Is it better to have them running all under one erlang VM so that this erlang VM process can allocate access to resources among them, and schedule the execution of the various erlang processes.

Or is it better to have separate erlang nodes on a given server?

If one is better than the other, why?

I'm assuming running all of these apps in a single erlang vm which is given, essentially, full run of the server, will result in better performance. The OS is just managing the disk and ram at the low level, and only has one significant process (the erlang VM) to switch with... and the erlang VM is probably smarter about allocating resources when it has the holistic view of all the erlang processes.

This may be something that I need to test, but I'm not in a position to do so effectively in the near term.

Answer 1

The answer is: it depends.

Advantages of using a single node:

• Memory is controlled by a single Erlang VM. It is way easier.
• Inter-application communication (if using erlang-messaging) is faster.
• Less operating system context switches happens

Advantages of using multiple nodes:

• If the system is linking in C code to the VM, death of one node due to a bug in C will not kill the others.

Answer 2

Agree with @I GIVE CRAP ANSWERS

I would go with one VM. Here is why:

• dynamic handling of run time queues belonging to schedulers (with varied origin of CPU load its important)
• fewer VMs to monitor
• better understanding of memory allocation and easier to spot malicious process (can compare all of them at once)
• much easier inter app supervision

I wouldn't care about VM crash - you need to be prepared any way. Heart works especially well in the cluster of equal units.

Answer 3

We've always used one VM per application because it's easier to manage.

The scheduler and SMP support in Erlang have come a long way in the past few years, so there isn't as much reason as there used to be to run multiple VMs on the same node.

Answer 4

I Agree with previous answers but there is a case scenario where having multiple nodes per cpu is the answer: When a heavy task hits the node. A task may take multiple minutes to complete and in such case a gen server will hold the node until completion of the task.