---
description: Learn how to run distributed workloads on Gradient
---

# Distributed Training

Gradient supports distributed training via two orchestration options: gRPC and Open-MPI to enable computation and communication at scale.

![Various Approaches for Distributed Training using TensorFlow](https://1320806315-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-LHZRFUkajubOAmgu6Rd%2F-LzgZzUJu1LW5yEPBoWn%2F-Lzgez__6D1fI1yTpKcT%2FScreenshot%202020-01-28%20at%2016.01.12.png?alt=media\&token=d1caa316-807f-4c03-a645-25d64e417840)

## What is gRPC?

gRPC is a modern, open source remote procedure call (RPC) framework that can run anywhere. It enables client and server applications to communicate transparently, and makes it easier to build connected systems. It is a generic point-to-point communication library and has no collective communication support. It works in a simple client/server style fashion and has been tightly integrated with [Tensorflow](https://www.tensorflow.org/) and Gradient.

You can learn how to launch and execute distributed experiment using Gradient and gRPC with Tensorflow. &#x20;

{% content-ref url="distributed-training/multi-node-training" %}
[multi-node-training](distributed-training/multi-node-training)
{% endcontent-ref %}

## What is MPI - [Message Passing Interface](http://www.mpi-forum.org/)

MPI stands for the Message Passing Interface. Written by the MPI Forum (a large committee comprised of a cross-section between industry and research representatives), MPI is a standardized API typically used for parallel and/or distributed computing.

All MPI specifications documents can be downloaded from the official MPI Forum web site: [http://www.mpi-forum.org/](http://www.mpi-forum.org/).

Message Passing Interface (MPI) is a de facto standard for  expressing distributed-memory programs. Several implementations of the MPI standard like MPICH, MVAPICH, OpenMPI  and CrayMPI have been developed and optimized over the period of several years for various processor architectures and high-performance interconnects like High-speed Ethernet (HSE) and InfiniBand.&#x20;

[Open MPI](https://www.open-mpi.org/) is an open source, freely available implementation of the MPI specifications. The Open MPI software achieves high performance and reliability and is used by a number of frameworks like [Tensorflow](https://www.tensorflow.org/), [Pytorch](https://pytorch.org/), [XGBoost](https://xgboost.ai/), [ChainerMN](https://docs.chainer.org/en/stable/chainermn/), [Horovod](https://github.com/horovod/horovod) and others.

{% content-ref url="distributed-training/distributed-machine-learning-with-mpi" %}
[distributed-machine-learning-with-mpi](distributed-training/distributed-machine-learning-with-mpi)
{% endcontent-ref %}

