agave-container-workshop-20180806

Working with Agave Storage

One of the strengths of the Agave API is its ability to interact with remote storage systems. Agave allows users to register and share (virtual) storage systems with other users, and interact with the associated files and folders on such systems.

In this section, we will focus on Agave’s files service, which allows us to interact with the files and folders that we have access to on the Agave system. There are four activities we will explore:

• Listing files and folders
• Creating directories
• Uploading and downloading files
• Transferring files between remote storage systems.

Working with the Files Service - Basics

All actions through the files service are in reference to a specific Agave storage system. One specifies the system one wishes to interact with by providing the system’s id. We will be using the storage systems you set up in the previous section that point to your VM and your UHM Lustre storage.

Make sure you have the id’s for your storage systems. If you have forgotten your system id’s, you can use the systems-list command to retrieve them.

We will be using the Agave CLI to interact with the Files service. We could also use curl or any other HTTP client. In fact, the Agave CLI is (currently) a thin wrapper on the curl program, and using the -V flag, you can actually see the curl commands being generated by the CLI.

Basic information about using the Agave Files service with the CLI:

• Pass the system id using the -S flag.
• Absolute paths in CLI commands are relative to the system’s rootDir attribute.
• Relative paths in CLI commands are relative to the system’s homeDir attribute.
• Pass the -L flag to the files-list command to see “verbose” output, including file metadata.
• Pass -V to see the full curl command generated.

Listing Files

To list files on a remote system, we use the files-list command.

# List the home directory on my VM:
files-list -S jfs-vm

# pass -L for "long format":
files-list -L -S jfs-vm

# provide a path -- relative to homeDir
files-list -S jfs-vm foo

# provide a path -- absolute, so relative to rootDir.
files-list -S jfs-vm /etc

# pass a -V to show the curl command and get the raw JSON response
files-list -V -S jfs-vm foo

Try the following exercise.

Exercise. Try listing directory content on your VM and your UHM Lustre storage.

Creating Directories

We can create directories on remote systems using Agave with the files-mkdir command:

# create a directory on the remote system -- relative path so it will be created in the home dir:
files-mkdir -S jfs-vm -N data

Try the following exercise.

Exercise. Create a test directory on both your VM and your UHM Lustre storage. Try using both relative and absolute
paths.

Uploading, Renaming and Downloading Files

Uploading files from your local machine to a remote storage system is done with the files-upload command. Use the-F flag, passing a relative or absolute path:

# Upload a file called test.txt in the current working directory to a directory called data relative to the homeDir of jfs-vm:
files-upload -S jfs-vm -F test.txt data/

Renaming files on remote storage is done with the files-move command. Use the -D flag to set the name:

# rename the file data/test.txt to data/test2.txt on the remote system
files-move -S jfs-vm -D data/test2.txt data/test.txt

Downloading files from remote storage to your local computer is done with the files-get command

# download a file, test2.txt, from the data directory within the homeDir of jfs-vm:
files-get -S jfs-vm data/test2.txt

Try the following exercise.

Exercise. Create a file in your Jupyter notebook. Upload the file to your VM. Rename the file on the remote storage
and then download the files with the new name.

Transferring Files

We can transfer files between two storage systems or between a URL on the public internet to a storage system as well. Transferring is done asynchornously, meaning the transfer will be queued and performed at some time in the future.

Transferring files is done using the files-import command. Keep in mind the following:

• The -S refers to the destination storage system id.
• The source file is specified using the -U flag and an agave-compliant URI.
• URIs can reference a file on the public internet or a file on another Agave storage system; for the latter, the following format should be used:
  • For paths within the home dir: agave:///
  • For paths within the root dir (note the two slashes after the system id): agave:////

# transfer the test2.txt file within the data directory of the homeDir on the jfs-vm to the homeDir on the jfs-hpc-stor system
files-import -U agave://jfs-vm/data/test2.txt -S jfs-hpc-stor .

# copy the classify_image.py file to our Lustre storage:
files-import -U https://raw.githubusercontent.com/tensorflow/models/master/tutorials/image/imagenet/classify_image.py -S jfs-hpc-stor

Try the following exercise.

Exercise. Transfer a file from your VM to your UHM Lustre storage and check that the transfer completed correctly.