Clusters

introduction vast is introducing support for several features (local vpns for client instances \[in beta], network volumes \[in alpha]) that depend on the host's lan configuration hosts can now register a set of machines they own sharing a lan as a cluster to allow clients renting their machines to access local network resources this will allow their machines to support use cases such as multi node training via nccl; as well, this will be a prerequisite to listing network volumes when they are released a registered cluster is associated with a machine that acts as the manager node and is responsible for dispatching cluster management commands from the vast server an ip subnet that machines in the cluster use to identify which network interface / which ip addresses for communication with other machines in the cluster a set of member machines the requirements for a set of machines to be registered as a cluster are that every member machine has a (non nated) ip address in the cluster's subnet to which any other machine in the cluster can communicate with on all ports cluster registration guide update to the newest version of the cli go to ( https //cloud vast ai/cli/)\[https //cloud vast ai/cli/ https //cloud vast ai/cli/)\[https //cloud vast ai/cli/ ] and copy+run the command starting with wget identify and test the subnet to register on the manager node run ip addr or ifconfig (the ip utility is part of the iproute2 package) identify which interface correspond's to their lan for most hosts this will be an ethernet interface, which have the naming format enp$buss$slot\[f$function]] in modern ubuntu hosts using mellanox devices for their main ethernet connection may instead see their interface show up as bond0 find the ipv4 subnet corresponding to that network interface in ip addr output, the third line for each interface usually starts with inet ipv4subnet where ipv4subnet has the format ipv4address/mask where mask is a non negative integer < 32 test that the other machines to be added to the cluster can reach the manager node on that subnet/address on the manager node run nc l ipv4address 2337 where ipv4address is the ipv4 address component of the chosen subnet on each other node run nc ipv4address 2337 type in some test text (i e , "hello") and press enter check that nc received and outputed the test text on the manager node run /vast py create cluster ipv4subnet machine id of manager node run /vast py show clusters to check the id of the cluster you just created run /vast py join cluster machine ids where machine ids is a space seperated list of the remaining machines to add to your cluster cluster management commands reference /vast create cluster subnet manager id initializes a cluster containing the machine with id manager id as its manager node and using the network interface corresponding to subnet /vast show clusters shows clusters, for each cluster showing its cluster id , associated subnet , manager node machine id, and list of member machines /vast join cluster cluster id machine ids takes machine ids as a space seperated list, and adds them to the cluster specified by cluster id /vast remove machine from cluster cluster id machine id \[new manager id] removes machine machine id from cluster cluster id if the machine is the only manager, another machine in the cluster new manager id must be specified so that the cluster still has a manager /vast delete cluster cluster id deletes cluster cluster id fails if cluster resources are currently in use by client instances