dataset_shard
Creates a dataset that includes only 1 / num_shards of this dataset.
Description
This dataset operator is very useful when running distributed training, as it allows each worker to read a unique subset.
Usage
dataset_shard(dataset, num_shards, index) Arguments
| Arguments | Description |
|---|---|
| dataset | A dataset |
| num_shards | A integer representing the number of shards operating in parallel. |
| index | A integer, representing the worker index. |
Value
A dataset