The Pilot Command Line Interface (pilotcli) is a binary executable program that provides a convenient way for users to perform file actions and platform-related tasks programmatically. The Pilot Command Line Interface can be downloaded and run on on your local Linux and Linux subsystems (as Windows user, see for instance the Windows Subsystem for Linux installation guide) to support large file uploads. For convenience, pilotcli is also pre-deployed and installed in the users JupyterHub accessible via worskpace tools.

Data Stewardship

Users are reminded to abide by the Platform Terms of Use and any Project-specific restrictions when using the Command Line Interface to access data and code.

Using the Pilot Command Line Interface in JupyterHub

To get started with the Pilot Command Line Interface in the HDC JupyterHub workspace tool, see the article Using JupyterHub in HDC and refer to the remaining sections in this article for a full description of the pilotcli commands.

Downloading the Pilot Command Line Interface

If you wish to use the Pilot Command Line Interface to upload large files from your local environment, it is necessary to download Pilot Command Line Interface tool on your Linux system. The recommended Linux version is Ubuntu 20.04.

• Download the pilotcli tool here.

After download, please grant execution permission to the binary file so that Pilot Command Line Interface can be used. To grant permission to the binary file, use the following command in the downloaded pilotcli directory.

If you have sudo permission, you can make the Pilot Command Line Interface more accessible in any directory by moving it to the /usr/local/bin directory:

Without sudo permission, you can still use the Pilot Command Line Interface in a different directory by referring the Command Line Interface path. For example, use a command line Interface in the Downloads folder:

Pilot Command Line Interface Command Summary

Commands are organized into main groups:

1. User - User related actions such as login and logout
2. Project - Project related actions, including list Project Names and Codes for all Projects accessible by the logged-in user.
3. File - Actions on files. User could use file action commands to list files/folders, upload files/folders, list/export attribute, and download files as well.
4. Dataset - Actions related to datasets.

User Commands: pilotcli user

Help

Command: login

• Options: -u --username, -p --password
• Usage: Allows users to log into the platform through pilotcli. After login, the user must be actively using the Pilot Command Line Interface to remain logged in, or they will be logged out automatically after the session is idle for five minutes.
• Prerequisites: None.

Example

User login:

1) Use the output device validation link or QR code to launch authentication validation:

2) Grant access by clicking Yes:

3) Device Login Successful message will appear and your authentication browser window can be closed.

4) The output of the Command Line Tool will also be updated to notify you about the successful login:

Command: logout

• Options: -y --yes
• Usage: Allows users to log out of the platform through pilotcli.
• Prerequisites: User login is required, otherwise an error is returned: ‘The current login session is invalid. Please login to continue.'.

Examples

User logout:

User logout canceled:

User logout with option to skip additional confirmation:

Project Commands: pilotcli project

Help

Command: list

• Options: None
• Usage: List all Projects to which the user has access
• Prerequisite: User login is required, otherwise an error is returned: ‘The current login session is invalid. Please login to continue.'

Examples

Get a list of all Projects the user has access to:

By default, the table will only display the first ten elements. If the user has access to more than ten Projects, you will be prompted to use your arrow keys to select one of two options to either view the next ten items (next page), or to exit (exit). The selected option is indicated by two arrows and can be confirmed by hitting enter:

File Commands: pilotcli file

Help

Command: attribute-list

• Options: -p --project-code
• Usage: List all attributes from a specified Project.
• Prerequisite:

• User login is required, otherwise an error is returned: ‘The current login session is invalid. Please login to continue.’
• Requires the Project code, which can be retrieved via pilotcli project list.

Examples

List attributes from a given Project by providing the Project Code at the prompt. In this example, “cli” is the corresponding Project Code:

List attributes from given a Project and provide the Project Code “cli” as an option in the same line -p cli:

Command: attribute-export

• Options: -p --project-code, -n --attribute-name
• Usage: Export an attribute template as JSON-file from a given Project

• Prerequisite:

  • User login is required, otherwise an error is returned: "The current login session is invalid. Please login to continue."
  • Project must be accessible to the user, otherwise an error is returned: "Project Code not found in list. Please verify and try again."
  • Requires the Project code, which can be retrieved via pilotcli project list.
  • Requires the Attribute template name. If the attribute template name does not exist or is spelled incorrectly, an error is returned: "Attribute 'sample-template' not found in Project. Please verify and try again."
  • pilotcli must have the permission to write a file in the directory, otherwise an error is returned.

Examples

Export an attribute template from a given Project. The user is prompted to provide the Project Code (“cli”) and the Attribute Template Name (“Manifest1”):

Export an attribute template from a given Project using options to provide both the Project Code (-p cli) and the Attribute Template Name (-n Manifest1), “cli” and “Manifest1” in this case, respectively:

Command: list

• Options: -z --zone
• Usage: List files and folders in the specified directory within the Platform. Folders will be displayed in blue and files will be displayed in white (or black, depending on the background of your terminal). If no zone (i.e. Green Room or Core) is specified, Green Room will be used as default.

• Prerequisite:

  • User login is required, otherwise an error is returned: "The current login session is invalid. Please login to continue."
  • Project must be accessible to the user, otherwise an error is returned: "Project Code not found in list. Please verify and try again."
  • Requires the Project code, which can be retrieved via pilotcli project list.

Examples

List folders in the directory “admin” in the Project with the Project Code “indoctestproject”. Since no zone was specified, the corresponding directory in the Green Room will be queried:

To list the corresponding elements in the “indoctestproject/admin” directory in the Projects' Core, the zone needs to be specified (-z core):

Command: upload

• Options: -p --project, -a --attribute, -t --tag, -z --zone, -m --upload-message, -s --source-file, --zip, -td --thread, -o --output path
• Usage: Upload files/folders to the given project, with optional tags and attributes, to the target folder.

• Prerequisite:

  • User login is required, otherwise an error is returned: "The current login session is invalid. Please login to continue."
  • Project must be accessible to the user, otherwise an error is returned: "Project Code not found in list. Please verify and try again."
  • When uploading file to the Project Core, the user must have access to the Core (Project Contributors cannot upload files/folders to the Project Core).

Examples

Upload a file (or folder) to a specified directory of a Project’s Green Room, in this case the file located at ./sample_folder/sample_file1 will be uploaded to the folder “admin” in the Project with the Project Code “cli” (-p cli/admin):

In case the specified destination directory does not exist yet, you will be prompted to confirm its creation by entering “y”, or to abort the command by entering “N”:

In order to upload a file (or folder) to a destination directory in a Project's Core, core must be specified as corresponding zone (-z core) and you will be required to provide a message to that should describe the files` content and why you are uploading it directly into the Core (-m "sample message"). Moreover, you will be reminded about the Terms of Use of the Platform and additional considerations regarding the direct upload of data into the Core zone, and are prompted to confirm your intention to proceed with the upload by entering “y” or “Yes”:

Upload a folder to Project Green Room

Upload file to existing folder in Green Room

• Specify a target folder to upload files by inputting folder name after the Project code separated by the '/'.

Upload folder as zip to a specified directory of a Project's Green Room by adding the --zip option.

• The specified folder will then be compressed as a zip file and subsequently be uploaded to the specified destination. Once the upload is completed, the compressed zip file will be removed from the directory. Note: pilotcli must have the permission to create and delete zip files in the corresponding directory of your local machine:

Upload a file with tags and/or attributes.

• When entering the upload command, the user can add a maximum of ten tags to the file. For example, in order to add “sample_tag1” and “sample_tag2” to your file, simply add -t sample_tag1 -t sample_tag2 to the command (see example below using tag1 and tag2 as tags).
• In addition, you can also use attribute templates for a more enriched annotation of your files. Please note, that the corresponding attribute template file needs to be edited & saved prior to uploading. To get the blank template, see the attribute-export instructions above for more details. To attach a attribute JSON-file to your file upon uploading, use -a cli_Manifest1_template.json, where “cli_Manifest1_template.json" points to the location of the attributes JSON-file on your local machine.
• Upon uploading, pilotcli will also validate the content of the attached attribute JSON-file. Whenever a template is invalid, including missing non-optional fields, non-existing choice in multiple choice attribute, exceeding length text field and so on, pilotcli will raise error to user and user must update the attribute template and upload again:

Upload a file with specified number of threads, using the -td <thread_number> option.

• By default, the number of threads is 1. There will no UI difference when using threads. The maximal number of threads depends on your local machine and the recommend thread number is 3:

Upload with customized manifest name

When you upload a file using pilotcli, a JSON file with the details of your specific upload job will be created automatically and allows you to resume the upload at a later time, in case the process is aborted due to any reason. By default, this JSON file will be created as “manifest.json” in the current working directory (see Command: resume below for more details on how to resume the file upload). However, you can use the --output-path
option to specify the filepath and, thus, the filename of this JSON file:

Command: resume

• Options: -td --thread, -r --resumable-manifest
• Usage: Resume unfinished file uploading based on the manifest file.

• Prerequisite:

  • User login is required, otherwise an error is returned: "The current login session is invalid. Please login to continue."
  • Project must be accessible to the user, otherwise an error is returned: "Project Code not found in list. Please verify and try again."
  • The resumable manifest JSON file is required, which is generated upon executing the pilotcli file upload command.

Example:

For instance upon disconnecting from the internet, or by accidentally closing the terminal, a currently running uploading process can be aborted. You can use the pilotcli file resume command in order to resume the uploading process right where you left off. Simply point to the manifest JSON file that was created upon executing the original upload command. Please note that the source files still have to be in the same location, otherwise resuming the uploading process will fail.

Command: sync

• Usage: Download a file or a folder from a Project. If pilotcli is used outside of the Platform, e.g. the users local machine, users can only download from Project’s Core. When pilotcli is used within the Platform, e.g. from one of the workspace tools, users will also be able to download from the Green Room of a Project as well.

• Options: -z --zone, -i --geid, --zip

  • To look up a file’s geid, or unique file ID, open the Project File Explorer in the Portal and inspect the File Properties for the ID.

• Prerequisite:

  • User login is required, otherwise an error is returned: "The current login session is invalid. Please login to continue."
  • The specified file or folder must exist, otherwise an error is returned: “File does not exist or source file provided is invalid: folder_name/file_name.”
  • If the user has no access to the specified file or folder on the Portal, an error is returned: “Permission denied. Please verify that your role in the Project has permission to perform this action.”

Examples

Download a file or a folder from the Core zone of a Project:

Download multiple files or folders as a compressed zip by simply providing all source files or folders and adding the --zip
option:

Download files or folders by their geid, instead of their location. To enable this, add -i or --geid
to the command:

Download a file or folder from the Green Room of a Project. This will only be possible if pilotcli is used within the Platform, e.g. in one of the Workspace tools. It works the same way as downloading from Core, including all the additional options like zipping, by either adding -z greenroom or by omitting the -z  option entirely, as “greenroom” is the default value:

Dataset Commands: pilotcli dataset

Help

Command: list

• Usage: List all datasets for the logged in user

• Prerequisite:

  • User login is required, otherwise an error is returned: "The current login session is invalid. Please login to continue."

Examples

List all datasets that are accessible for the user:

Command: show-detail

• Usage: Show the detailed information of a dataset.

• Prerequisite:

  • User login is required, otherwise an error is returned: "The current login session is invalid. Please login to continue."

Examples

Show the details of the specified Dataset by providing the unique Dataset Code - “aug242021” in this case:

Command: download

• Usage: Download a particular version of a dataset.
• Options: -v --version

• Prerequisite:

  • User login is required, otherwise error is returned: "The current login session is invalid. Please login to continue."

Examples

Download a Dataset with Code “aug242021”:

Download the version 1.1 of a Dataset with Code “aug242021”:

Copyright © 2023 Indoc Systems.

HealthDataCloud is powered by Pilot technology, a product of Indoc Systems.