Gradient Workflows
Gradient Workflows provides a simple way to automate machine learning tasks.
Last updated
Gradient Next
Gradient Workflows provides a simple way to automate machine learning tasks.
Last updated
Gradient Workflows is a new and powerful way to build-out machine learning applications. Workflows utilize GitHub-action style syntax via YAML files to easily create powerful automation.
Workflows is based on the Argo runtime engine, which is a container-native continuous delivery tool for Kubernetes, and makes it easy to build complex and scalable projects with an arbitrary number of discrete steps.
Workflows is used by ML Engineers to build-out deterministic machine learning pipelines.
The best tutorial is the canonical Gradient Workflows Tutorial:
Gradient Workflows TutorialWorkflow: a named or unnamed entity that belongs to a team and project
Named workflows can be re-run with a default workflow spec, or be passed a new spec every time
Workflow Spec: a YAML list of jobs that is converted into an Argo template and run on the Gradient distributed runtime engine.
Job: self-contained part of a workflow spec that is similar to an Argo step
Jobs can define inputs, outputs, and their own environment variables
Jobs can require other jobs via "needs" and collect/pass info between jobs
Jobs can be implemented with an action via "use"
Action: a self-contained, composable set of code building blocks that can perform specific actions within a machine learning project.
Actions can receive parameters (e.g., args, image) within the job step via the "with" argument
E.g., container@v1 action = run a container, load inputs, and produce outputs
Workflow Run: the implementation of a workflow
The most basic run requires a workflowId and clusterId - most will also include a workflowSpec, and the inputs to be passed into the workflow
The workflow run contains everything needed for the workflow to actually be executed, i.e., what (workflowId), where (clusterId), how (workflowSpec), with (inputs, etc.)
