In my last blog post, Protect your VMs - The Cohesity way I showcased how easy its to use the Integrated Backup software or Cohesity UI to protect 40 VMs running in a VMware Vcenter. The VMs were protected at a VMware datacenter level and associated to a Policy which would ensure consistent backups of these VMs.
Allowing Policies to be tied to individual jobs provides a very versatile way to scale to upwards of protecting thousands of VMs easily and consistently.
Let's now dig a little bit deeper and look under the covers to understand exactly how the fully distributed Integrated backup software works and understand technical advantage of a fully distributed platform such as Cohesity.
When the Cohesity UI is used to fire off the the backup, in the background the job is divided into multiple tasks. These tasks are distributed across to all the nodes in the cluster for execution of the backup job.
As the Cohesity cluster is made up of multiple nodes (in this case 3 nodes), each of these nodes is a peer to each other and run all the same software components. This makes the cluster fully redundant from the software as well as hardware standpoint.
The Integrated Backup software on Cohesity is known as Magneto.
Magneto is a fully distributed software which runs on all the nodes in the cluster. On one of the nodes, the software would perform both the master and the slave functions. The master is responsible to assign to slaves in the cluster.
The master role is easily transferred to any other node in the cluster in case of SW or HW failures. In case of a failure the new master would communicate with Gandalf (distributed configuration manager) and pick up from where the last good master had left off and finish with the remaining tasks, no disruption to the existing job is experienced.
Here you can see the internal processing of the job we had created and how Magneto master distributes the job amongst the available nodes in the cluster.
Each of the nodes are assigned a constituent ID, which is used by Magneto to assign the tasks.
As seen below Magneto Master has been assigned a session Id 310 from Gandalf (similar to Google Chubby) and works in conjunction with a distributed journal which contains all the information about the tasks being executed by the master.
Below you can also see information about the backup job c2400-job which is running and has 9 total active and running tasks. As this job is the first run, its performing a full backup of each of the VMs.
Now lets also take a look at how the 40 VMs were backed up got distributed evenly across all the nodes in the cluster. Once the data comes onto the Cohesity platform the concept of replication factor (RF=2) ensures that the data protected on 2 separate nodes.
As seen below 40 of the tasks of backup job c2400-job were distributed across the 3 magneto slaves with constituent IDs 74,69,76. As this capture was taken during the run, we can see 9 jobs still running while the rest of the 31 tasks were completed successfully.
When new nodes are added to the cluster they start participating in the distributed ingest right away, and hence ingest performance increases and does not have a theoretical bottleneck as each node that is added comes with its own compute/network and storage capacities.
With this next generation hyper-converged architecture for data protection each of nodes in the cluster is a media server/catalog server and target and hence brings simplicity and manageability to the secondary storage environment.
Here is a recent video I did to show adding a source and protecting a vCenter, enabling automatic backup and then looking at the magneto service pages to see the distributed ingest happening on the system.