GradientCI
Set up continuous integration between your GitHub repository and Gradient
Last updated
Set up continuous integration between your GitHub repository and Gradient
Last updated
To create a Gradient Project with continuous integration powered by GradientCI and GitHub:
(GitHub Admin privilege is required.)
Click Create Project.
Select GitHub Project.
Grant Paperspace access to your GitHub repos via OAuth.
Confirm a repo with GradientCI installed for your new Gradient° Project.
To set up GradientCI, our continuous integration service, include a directory in your GitHub repository called .ps_project with a configuration file config.yaml, examples below.
You may additionally disable the builds of pull requests, enabled by default. Or enable builds of pull requests that originate from forked repositories, disabled by default to prevent unauthorized use of Gradient resources. Each of these options will allow configuration to be sourced from the relevant Git branch.
name: a string identifier for the step, often used in reporting results.
This name must be unique amongst the steps for a workflow.
command: what type of step should be executed.
These match the qualified names in the Paperspace SDK.
Supported commands:
experiment.run_single_node
experiment.run_multi_node
experiment.run_mpi_multi_node
One of params or paramsFile
paramsFile: a path to file contained in your project repository containing the yaml arguments for the step.
params this is a object with the parameters for the relevant step.
These are the same as the contents of a paramsFile inlined into your .ps_projects/config.yaml.
checks: these configure status checks on your GitHub pull requests.
The GradientCI service can help you from degrading your model. When you run an experiment based on a code change, GradientCI can automatically check properties of the experiment and forward those results to GitHub. GitHub can use these statuses to prevent pull requests merges that degrade your model. GradientCI will automatically report whether the experiment ran without error or which checks failed if there was an error. You may configure additional checks for metrics that are generated by your experiment under the checks key in your config.yaml. Currently, we only support scalar metrics coming from TensorFlow generated summaries.
In addition to the status checks, GradientCI writes a detailed summary of the experiment to a comment on the pull request so you have your critical data at a glance while reviewing code.
<identifier>s:<identifiers> are split into two parts: 1. the source of the metric and 2. the name of the metric, e.g. tensorflow:loss.
Supported <identifiers> are:
tensorflow
onnx
custom
These <identifiers> are case-sensitive
“defaults”: reserved for defaults to set for the <identifiers>.
Any of the above keys are valid within this block and will set the default behavior for all other <identifier> blocks.
If round is set to down all other metrics will be evaluated using the round down behavior.
If an <identifier> has a key specified underneath it it will override the value in the defaults block.
For instance, round: up under tensorflow:loss will override the round: down behavior in the defaults block.
<range>Ranges can appear in the following forms:
<number> or <number>.. this form allows you to specify that the metric must be greater than <number>.
..<number> this form allows you to specify that the metric must be less than <number>.
<left>..<right> this form allows you to specify that the metric must be less than <left> but greater than <right>.
Note these numbers are parsed as floats and relying on precise equality with the ends of the range is not recommended.
target: this is a <range> for the metric to appear in
aggregate: this is the aggregating function to evaluate the metric by.
These are objects emitted from your model summary, for tensorflow valid aggregates are:
mean
stddev
max
min
var
median
These paths must begin with /
For example the mean of loss in {"loss": { "mean": 0.5 }} would be accessed with /loss/mean
For example the 2nd element in the loss array in {"loss": [0.0, 0.5] } would be accessed with /loss/1
precision: how many decimal places to keep (default 2)
round: specifies rounding behavior (default “down”)
only-pulls: only perform this check on pull requests
if-not-found: return a default status if job has no data for <identifier>, defaults to “failure”
comment-on-pr: include this metric in a summary content if the metrics were generated from a pull-request
For some steps in your job GradientCI will automatically set environment variables so you can determine the triggering condition. These variables are currently set for experiment.* steps only.
PS_GRADIENT_CI_GIT_REF the branch or tag that triggered the build
PS_GRADIENT_CI_GIT_SHA the git commit currently being built
PS_GRADIENT_CI_GIT_REPO_URL the repository that received the trigger.
Note that builds from forks will appear to come from the target repository, not the fork.
Note this can only be done by an organization level administrator or on your personal repositories.
Navigate to the repository or organization that you wish to remove GradientCI from.
Click the "Settings" tab in the top row
Select "Integration & services" from the left menu, you should be presented with a list that includes "GradientCI" that looks like
Select "Configure" next to "GradientCI", you will be prompted to enter your password.
a. Uninstall the application from all repositories on the organization or personal account by clicking the red "Uninstall" button
b. Select the "Only select repositories" and choosing which repositories should have GradientCI from the dropdown, or unselecting them by clicking the "x" next to their name.
Install the GitHub app on your repository.
Navigate to the .
GradientCI supports building and sourcing project configuration from arbitrary branches or tags. By default we only build configuration sourced from your default branch (typically master). You can , if you only need the configuration from one branch. You can relax or tighten this rule by selecting "All" or "None" from the "Build Branches" dropdown in the project settings pane of the Gradient console. If you would like to build any tags or a subset of branches that are not the default branch, select "All" from this menu and provide filters in your config.yaml. To list the specific patterns of tags and branches to build, see .
Your CI configuration is separated into individual workflows. Workflows are named units of work that are executed in parallel. A workflow is comprised of a block and a series of to execute.
Steps are a series of sequentially executed actions. These generally correspond to actions available in the Paperspace CLI/SDK. Currently, only one step per workflow is supported. For complex pipelines you may run a single node experiment that contains a python script with a series of Paperspace SDK calls to orchestrate your pipeline. The SDK Pipeline style offers more flexibility than can be described by the yaml pipeline configuration as you can describe complex logic using plain python code. SDK pipelines offer the latest SDK functionality as soon as it becomes available, often before it can be integrated into the yaml pipeline syntax. For an example see
These are the same as those generated by the CLI, see the experiments for an example of one of these files and further documentation.
See for the full schema.
By default GradientCI will only build the default branch and no tags. If you would like to build additional non-pull request branches or tags you must select "All" from the "Build Branches" project configuration dropdown. This will build all branches and no tags. Once that is complete you can filter additional branches in your config.yaml by providing a triggers section. You can place the keys branches or tags to apply filters to the default. Under each key you can provide only or ignore fields, but not both, containing a or array of regex to match on. Branches or tags filtered by an only key must match one or more of the regex provided. Branches or tags filtered by an ignore key will be skipped if they match one or more of the regex provided.
For best results, especially on repositories with many contributors, we recommend configuring GitHub branch protections to prevent accidental merges of unintended pull requests. After your first build, statuses for the metrics will be reported back and you can make passing statuses required for the merge of your pull request. To do this follow
If the <identifier> is custom, the aggregate is evaluated as a
We highly recommend the use of secrets on datasets as these get stored as plain text. You may set secrets on Cluster, Project, and/or Team level. If the same secret name is created for more than one scope only one will be applied. Secrets have the following precedence: Cluster > Project > Team. You can learn more about setting and using secrets .
From the "GradientCI" settings menu, you can then either
This error occurs when GradientCI is unable to find the attached Paperspace project for your GitHub repository. The Paperspace project may have been deleted or you may have installed the app only through GitHub's interface. Follow the to ensure that your Paperspace project is properly configured.
