Gradient Workflows

Gradient Workflows provides a simple way to automate machine learning tasks.

Overview

Gradient Workflows is a new and powerful way to build-out machine learning applications. Workflows utilize GitHub-action style syntax via YAML files to easily create powerful automation.

Workflows is based on the Argo runtime engine, which is a container-native continuous delivery tool for Kubernetes, and makes it easy to build complex and scalable projects with an arbitrary number of discrete steps.

Examining the logs from a simple workflow.

Workflows is used by ML Engineers to build-out deterministic machine learning pipelines.

Where to start

The best tutorial is the canonical Gradient Workflows Tutorial:

Gradient Workflows Tutorial

Gradient Workflows — Automate from idea to production

Key Terminology

• Workflow: a named or unnamed entity that belongs to a team and project

  • Named workflows can be re-run with a default workflow spec, or be passed a new spec every time

• Workflow Spec: a YAML list of jobs that is converted into an Argo template and run on the Gradient distributed runtime engine.

• Job: self-contained part of a workflow spec that is similar to an Argo step

  • Jobs can define inputs, outputs, and their own environment variables

  • Jobs can require other jobs via "needs" and collect/pass info between jobs

  • Jobs can be implemented with an action via "use"

• Action: a self-contained, composable set of code building blocks that can perform specific actions within a machine learning project.

  • Actions can receive parameters (e.g., args, image) within the job step via the "with" argument

  • E.g., container@v1 action = run a container, load inputs, and produce outputs

• Workflow Run: the implementation of a workflow

  • The most basic run requires a workflowId and clusterId - most will also include a workflowSpec, and the inputs to be passed into the workflow

  • The workflow run contains everything needed for the workflow to actually be executed, i.e., what (workflowId), where (clusterId), how (workflowSpec), with (inputs, etc.)