> For the complete documentation index, see [llms.txt](https://paperspace.gitbook.io/gradient/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://paperspace.gitbook.io/gradient/master/experiments/using-experiments/experiment-datasets.md).

# Experiment datasets

## About

When executing an experiment in Gradient you may optionally supply one or more datasets that will be downloaded into your experiment's environment prior to execution. These datasets can be downloaded from an S3 object or folder (including the full bucket). Gradient allows teams to run reproducible machine learning experiments by taking advantage of S3 ETags and Version IDs, which combine to allow you to be sure that datasets exactly match between training sets, and to be sure which version of a dataset you are using.

{% hint style="info" %}
Gradient private datasets with versioning are a Gradient Enterprise feature. [Contact Sales](https://info.paperspace.com/contact-sales) for inquiries!
{% endhint %}

### S3 Datasets

Datasets are downloaded and mounted readonly on `/data/DATASET` within your experiment jobs using the supplied AWS credentials. The credentials are optional for public buckets. The name of the dataset is the `basename` of the last item in the s3 path, e.g. `s3://my-bucket/mnist.zip` would have the name `mnist` and `s3://my-bucket` would have the name `my-bucket`. The name maybe overridden with the optional `name` parameter.

```
datasets: [
    {
        "uri": "s3://my-bucket/mnist-modified.zip",
        "awsSecretAccessKey": "secret:<some_secret_name>",
        "awsAccessKeyId": "secret:<some_other_secret_name>",
        "name": "mnist",
    },
]
```

{% hint style="info" %}
We highly recommend the use of the secrets feature on S3 datasets as the values get passed in as plain text. Using secrets as `secret:<some_secret_name` ensure that your credentials are encrypted and protected. You can learn more about using secrets [here](https://docs.paperspace.com/gradient/secrets/using-secrets).
{% endhint %}

{% tabs %}
{% tab title="CLI" %}
You can launch an experiment & specify the desired S3 dataset with e-tags using the CLI as follows.

```
$ gradient experiments run singlenode 
--projectId prda8mhcq 
--workspace https://github.com/Paperspace/mnist-sample.git
... 
--datasetAwsAccessKeyId secret:<some_secret_name> 
--datasetAwsSecretAccessKey secret:<some_other_secret_name>
--datasetName fashion
--datasetUri s3://my-bucket-name/fashion-mnist.zip
```

When launching an experiment using the config.yaml, pass in the multiple datasets using the following structure.

```
datasetUri:
  - "s3://some.dataset/uri"
  - "s3://some.other.dataset/uri"
datasetName:
  - "some dataset name"
  - null
datasetAwsAccessKeyId:
  - none
  - secret:<some_secret_name>
datasetAwsSecretAccessKey:
  -
  - secret:<some_other_secret_name>
datasetVersionId:
datasetEtag:
  - "some etag"
  - "some other etag"
```

{% endtab %}

{% tab title="SDK" %}

```
env = {
        "EPOCHS_EVAL":"10",
        "TRAIN_EPOCHS":"40",
        "MAX_STEPS":"50000",
        "EVAL_SECS":"600",
        "BATCH_SIZE":"100",

    }

single_node_parameters = { 
    "name": "dataset",
    "project_id": project,
    "container": "tensorflow/tensorflow:1.13.1-py3",
    "machine_type": "p2.xlarge",
    "command": "pip install -r requirements.txt && python mnist.py",
    "experiment_env": env,
    "workspace_url": "https://github.com/Paperspace/mnist-sample.git", #can be local directory, a git repo or commit, or s3 bucket
    "cluster_id" : cluster,
    "model_type" : "Tensorflow",
    "model_path" : "/artifacts",
    "datasets" : [{
         "uri": bucket,
         "aws_secret_access_key": os.getenv('ACCESS_KEY'),
         "aws_access_key_id": os.getenv('ACCESS_KEY_ID'),
         "etag": "ee1d4fd1b3a97b5384355941ee99d3e4",
         "name" : "fashion"
         }]
}

client.experiments.run_single_node(**single_node_parameters)
```

{% endtab %}
{% endtabs %}

The datasets will show up in the web interface in the environment tab of the experiment you launch.

![](https://1320806315-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-LHZRFUkajubOAmgu6Rd%2F-LygcWgRd7El-7C-aLc_%2F-Lygd4dzqCxLhOYxUnVD%2FScreen%20Shot%202020-01-15%20at%2011.37.06%20PM.png?alt=media\&token=c5c34ce8-eab5-4d45-973b-66a51640b17d)