Working with HDC Project Files in the Command Line Interface
The Pilot Command Line Interface (pilotcli) is a binary executable program that provides a convenient way for users to perform file actions and platform-related tasks programmatically. The Pilot Command Line Interface can be downloaded and run on on your local Linux and Linux subsystems (as Windows user, see for instance the Windows Subsystem for Linux installation guide) to support large file uploads. For convenience, pilotcli is also pre-deployed and installed in the users JupyterHub accessible via worskpace tools.
Data Stewardship
Users are reminded to abide by the Platform Terms of Use and any Project-specific restrictions when using the Command Line Interface to access data and code.
Using the Pilot Command Line Interface in JupyterHub
To get started with the Pilot Command Line Interface in the HDC JupyterHub workspace tool, see the article Using JupyterHub in HDC and refer to the remaining sections in this article for a full description of the pilotcli commands.
Downloading the Pilot Command Line Interface
If you wish to use the Pilot Command Line Interface to upload large files from your local environment, it is necessary to download Pilot Command Line Interface tool on your Linux system. The recommended Linux version is Ubuntu 20.04.
- Download the pilotcli tool here.
After download, please grant execution permission to the binary file so that Pilot Command Line Interface can be used. To grant permission to the binary file, use the following command in the downloaded pilotcli directory.
If you have sudo permission, you can make the Pilot Command Line Interface more accessible in any directory by moving it to the /usr/local/bin directory:
Without sudo permission, you can still use the Pilot Command Line Interface in a different directory by referring the Command Line Interface path. For example, use a command line Interface in the Downloads folder:
Pilot Command Line Interface Command Summary
Commands are organized into main groups:
- User - User related actions such as login and logout
- Project - Project related actions, including list Project Names and Codes for all Projects accessible by the logged-in user.
- File - Actions on files. User could use file action commands to list files/folders, upload files/folders, list/export attribute, and download files as well.
- Dataset - Actions related to datasets.
User Commands: pilotcli user
Help
Usage: pilotcli user [OPTIONS] COMMAND [ARGS]...
Options:
--help Show this message and exit.
Commands:
login For user to login.
logout For user to logout.
Command: login
- Options: -u --username, -p --password
- Usage: Allows users to log into the platform through pilotcli. After login, the user must be actively using the Pilot Command Line Interface to remain logged in, or they will be logged out automatically after the session is idle for five minutes.
- Prerequisites: None.
Usage: pilotcli user login [OPTIONS]
For user to login.
Options:
--help Show this message and exit.
Example
User login:
1) Use the output device validation link or QR code to launch authentication validation:
Please, access https://iam.hdc.humanbrainproject.eu/realms/hdc/device?user_code=XXXX-XXXX to proceed
▄▄▄▄▄▄▄ ▄▄ ▄ ▄ ▄ ▄▄▄▄ ▄ ▄▄▄▄▄▄▄
█ ▄▄▄ █ ▄ ▄▄▀█ █▀▀▀▄▀█▀ ▀██▄ █ ▄▄▄ █
█ ███ █ ██▄ ▀▀▄▀▄▄█▄ ▄ ▀▀▀▄ █ ███ █
█▄▄▄▄▄█ █▀▄▀█▀▄▀█▀▄ █ ▄ ▄ █▀█ █▄▄▄▄▄█
▄▄▄▄▄▄▄ ▀█▀█▄▄█▄ █▀█ ▄▀▄▄ █ ▄ █▀▄▄▄
█ ▄▄▄ █ █▄█▄██ ▀ ▀█▀▀█ ▀▄█▄█▄▄▄█▀▄█
█ ███ █ █▀█ ▄▀██▄█▀▀█▄▀▀█▄▀█ ▀ ▀▄▀██
█▄▄▄▄▄█ ▄▀▄ ██▄▀▀▄ ▀▀▄ ▄▄▀▀▀▄ █▄▄▄█
Waiting validation finish...
2) Grant access by clicking Yes:
3) Device Login Successful message will appear and your authentication browser window can be closed.
4) The output of the Command Line Tool will also be updated (see line 14 below) to notify you about the successful login:
Please, access https://iam.hdc.humanbrainproject.eu/realms/hdc/device?user_code=XXXX-XXXX to proceed
▄▄▄▄▄▄▄ ▄▄ ▄ ▄ ▄ ▄▄▄▄ ▄ ▄▄▄▄▄▄▄
█ ▄▄▄ █ ▄ ▄▄▀█ █▀▀▀▄▀█▀ ▀██▄ █ ▄▄▄ █
█ ███ █ ██▄ ▀▀▄▀▄▄█▄ ▄ ▀▀▀▄ █ ███ █
█▄▄▄▄▄█ █▀▄▀█▀▄▀█▀▄ █ ▄ ▄ █▀█ █▄▄▄▄▄█
▄▄▄▄▄▄▄ ▀█▀█▄▄█▄ █▀█ ▄▀▄▄ █ ▄ █▀▄▄▄
█ ▄▄▄ █ █▄█▄██ ▀ ▀█▀▀█ ▀▄█▄█▄▄▄█▀▄█
█ ███ █ █▀█ ▄▀██▄█▀▀█▄▀▀█▄▀█ ▀ ▀▄▀██
█▄▄▄▄▄█ ▄▀▄ ██▄▀▀▄ ▀▀▄ ▄▄▀▀▀▄ █▄▄▄█
Waiting validation finish...
Welcome to Command Line Tool!
Command: logout
- Options: -y --yes
- Usage: Allows users to log out of the platform through pilotcli.
- Prerequisites: User login is required, otherwise an error is returned: ‘The current login session is invalid. Please login to continue.'.
Usage: pilotcli user logout [OPTIONS]
For user to logout.
Options:
-y, --yes Input Y/yes to confirm you want to logout, otherwise input N/no
to remain logged in.
--help Show this message and exit.
Examples
User logout:
Are you sure you want to logout? [y/N]: y
Logged out successfully. Bye!
User logout canceled:
Are you sure you want to logout? [y/N]: n
Logout canceled!
User logout with option to skip additional confirmation:
Logged out successfully. Bye!
Project Commands: pilotcli project
Help
Usage: pilotcli project [OPTIONS] COMMAND [ARGS]...
Options:
--help Show this message and exit.
Commands:
list List accessible projects.
Command: list
- Options: None
- Usage: List all Projects to which the user has access
- Prerequisite: User login is required, otherwise an error is returned: ‘The current login session is invalid. Please login to continue.'
Usage: pilotcli project list [OPTIONS]
List accessible projects.
Options:
--page INTEGER The page to be listed [default: 0]
--page-size INTEGER number of objects per page [default: 10]
--order [asc|desc] sorting order [default: desc]
--order-by [created_at|name|code]
sorting column [default: created_at]
-d, --detached whether run in detached mode
--help Show this message and exit.
Examples
Get a list of all Projects the user has access to:
Project Name Project Code
---------------------------------------------------------------------------
Test Project 02-13 | test230213byshan
Test1_Audit | test1audit
Test20221109 | test20220109
Indoc Test Project | indoctestproject
Page: 0, Number of projects: 4
By default, the table will only display the first ten elements. If the user has access to more than ten Projects, you will be prompted to use your arrow keys to select one of two options to either view the next ten items (next page), or to exit (exit). The selected option is indicated by two arrows and can be confirmed by hitting enter:
What do you want? (Use arrow keys)
» next page
exit
File Commands: pilotcli file
Help
Usage: pilotcli file [OPTIONS] COMMAND [ARGS]...
Options:
--help Show this message and exit.
Commands:
attribute-export Export attribute template from a given Project.
attribute-list List attribute templates of a given Project.
list List files and folders inside a given Project/folder.
resume Resume the upload process with given manifest file.
sync Download files/folders from a given Project/folder/file...
upload Upload files/folders to a given Project path.
Command: attribute-list
- Options: -p --project-code
- Usage: List all attributes from a specified Project.
Prerequisite:
- User login is required, otherwise an error is returned: ‘The current login session is invalid. Please login to continue.’
- Requires the Project code, which can be retrieved via `pilotcli project list.
-
Copyright © 2023 Indoc Research.
HealthDataCloud is powered by Pilot technology, a product of Indoc Research.