Selecting Storage in DevEnviron¶
Storage varies depending on performance, usability, and cost. No storage media can cover all scenarios. Learning about in-cloud storage application scenarios for better usage.
Storage | Application Scenario | Advantage | Disadvantage |
|---|---|---|---|
EFS (default storage) | Non-production scenarios such as exploration and experiment. One EFS device can be mounted to both a development environment and a training environment. In this way, you do not need to download data each time your training job starts. This type of storage is not suitable for heavy I/O training on more than 32 cards. | Free-of-charge 50 GB of shared network storage is available. EFS is implemented as NFS and can be shared between multiple development environments and between development and training environments. This type of storage is preferred for non-heavy-duty distributed training jobs, especially for the ones not requiring to download data additionally when the training jobs start. | Additional requests and configurations are required in production environments, and I/O bottlenecks may occur during heavy-duty training. |
EVS (block storage) | Data and algorithm exploration only in the development environment. | Block storage SSDs feature better overall I/O performance than NFS. The storage capacity can be dynamically expanded to up to 4096 GB. As persistent storage, EVS disks are mounted to /home/ma-user/work. The data in this directory is retained after the instance is stopped. The storage capacity can be expanded online based on demand. | This type of storage can only be used in a single development environment. |
OBS | Notebook of the new version cannot be stored in or mounted to OBS buckets. When uploading or downloading a large amount of data in the development environment, you can use OBS buckets to transfer data. | N/A | N/A |
Local storage | First choice for heavy-duty training jobs. | High-performance SSDs for the VM or BMS where the container is running, featuring high file I/O throughput. For heavy-duty training jobs, store data in the target directory and then start training. | The storage lifecycle is associated with the container lifecycle. Data needs to be downloaded each time the training job starts. |
Using the Storage¶
How do I use EFS (default storage) in a development environment?
When creating a notebook instance, select the default storage.
How do I use EVS in a development environment?
When creating a notebook instance, select the EVS storage. You can expand an EVS disk capacity on a running notebook instance. For details, see Dynamically Expanding EVS Disk Capacity.