R/dataset_methods.R

dataset_shard

Creates a dataset that includes only 1 / num_shards of this dataset.

Description

This dataset operator is very useful when running distributed training, as it allows each worker to read a unique subset.

Usage

 
dataset_shard(dataset, num_shards, index) 

Arguments

Arguments Description
dataset A dataset
num_shards A integer representing the number of shards operating in parallel.
index A integer, representing the worker index.

Value

A dataset