Last modified by Dennis Segebarth on 2024/10/02 18:38
From version 16.1
edited by Dennis Segebarth
on 2024/10/02 18:38
Change comment: There is no comment for this version
To version 1.8
edited by Susan Evans
on 2023/07/10 22:26
Change comment: (Autosaved)

Summary

Details

Page properties
Author
... ... @@ -1,1 +1,1 @@
1 -XWiki.dsegebarth
1 +XWiki.sgevans
Content
... ... @@ -1,7 +5,3 @@
1 -{{box cssClass="floatinginfobox" title="Table of Contents"}}
2 -{{toc depth="2"/}}
3 -{{/box}}
4 -
5 5  The Pilot Command Line Interface (pilotcli) is a binary executable program that provides a convenient way for users to perform file actions and platform-related tasks programmatically. The Pilot Command Line Interface can be downloaded and run on on your local Linux and Linux subsystems (as Windows user, see for instance the [[Windows Subsystem for Linux installation guide>>url:https://learn.microsoft.com/en-us/windows/wsl/install]]) to support large file uploads. For convenience, pilotcli is also pre-deployed and installed in the users JupyterHub accessible via worskpace tools.
6 6  
7 7  = Data Stewardship =
... ... @@ -8,18 +8,10 @@
8 8  
9 9  Users are reminded to abide by the Platform Terms of Use and any Project-specific restrictions when using the Command Line Interface to access data and code.
10 10  
11 -{{warning}}
12 -⚠ If you are using pilotcli in an environment or system where other users can access and read all your files, please make sure to always run {{code language="none"}}pilotcli user logout{{/code}} as soon as you are done using pilotcli. This could be, for instance, if you are using pilotcli on a shared computer in your local institute, with system administrators of your University or Hospital having superuser privileges. You should not use pilotcli in an environment where users with such privileges are from an entity that is not trusted.
13 -{{/warning}}
14 -
15 15  = Using the Pilot Command Line Interface in JupyterHub =
16 16  
17 17  To get started with the Pilot Command Line Interface in the HDC JupyterHub workspace tool, see the article //Using JupyterHub in HDC //and refer to the remaining sections in this article for a full description of the pilotcli commands.
18 18  
19 -= Using the Pilot Command Line Interface in Guacamole VMs =
20 -
21 -To get started with the Pilot Command Line Interface in the HDC Guacamole workspace tool, see //Using Guacamole in HDC// on how to launch a terminal, either in a Desktop VM or directly by establishing a connection to a Command Line VM. The Pilot Command Line Interface will be installed, configured, and ready to be used. Please refer to the remaining sections in this article for a full description of the pilotcli commands.
22 -
23 23  = Downloading the Pilot Command Line Interface =
24 24  
25 25  If you wish to use the Pilot Command Line Interface to upload large files from your local environment, it is necessary to download Pilot Command Line Interface tool on your Linux system. The recommended Linux version is Ubuntu 20.04.
... ... @@ -110,11 +110,11 @@
110 110  
111 111  2) Grant access by clicking Yes:
112 112  
113 -[[image:1723802541971-790.png||height="314" width="368"]]
101 +[[image:Pilot CLI authentication.png||height="34%" width="40%"]]
114 114  
115 115  3) Device Login Successful message will appear and your authentication browser window can be closed.
116 116  
117 -4) The output of the Command Line Tool will also be updated to notify you about the successful login:
105 +4) The output of the Command Line Tool will also be updated (see line 14 below) to notify you about the successful login:
118 118  
119 119  {{code language="none"}}
120 120  $ pilotcli user login
... ... @@ -350,7 +350,7 @@
350 350  
351 351  * User login is required, otherwise an error is returned: "The current login session is invalid. Please login to continue."
352 352  * Project must be accessible to the user, otherwise an error is returned: "Project Code not found in list. Please verify and try again."
353 -* Requires the Project code, which can be retrieved via {{code}}pilotcli project list{{/code}}.
341 +* Requires the Project code, which can be retrieved via pilotcli project list.
354 354  * Requires the Attribute template name. If the attribute template name does not exist or is spelled incorrectly, an error is returned: "Attribute 'sample-template' not found in Project. Please verify and try again."
355 355  * pilotcli must have the permission to write a file in the directory, otherwise an error is returned.
356 356  )))
... ... @@ -418,7 +418,7 @@
418 418  
419 419  * User login is required, otherwise an error is returned: "The current login session is invalid. Please login to continue."
420 420  * Project must be accessible to the user, otherwise an error is returned: "Project Code not found in list. Please verify and try again."
421 -* Requires the Project code, which can be retrieved via {{code}}pilotcli project list{{/code}}.
409 +* Requires the Project code, which can be retrieved via pilotcli project list.
422 422  )))
423 423  
424 424  {{code language="none"}}
... ... @@ -626,7 +626,7 @@
626 626  All uploading jobs have finished.
627 627  {{/code}}
628 628  
629 -Upload with customized manifest name
617 +(10) upload with customized manifest name
630 630  
631 631  When you upload a file using pilotcli, a JSON file with the details of your specific upload job will be created automatically and allows you to resume the upload at a later time, in case the process is aborted due to any reason. By default, this JSON file will be created as “manifest.json” in the current working directory (see //Command: **resume**// below for more details on how to resume the file upload). However, you can use the {{code}}--output-path{{/code}} option to specify the filepath and, thus, the filename of this JSON file:
632 632  
... ... @@ -634,285 +634,17 @@
634 634  $ pilotcli file upload -p <path> <file> --output-path test_manifest.json
635 635  {{/code}}
636 636  
637 -== Command: **resume** ==
638 638  
639 -* **Options**: -td ~-~-thread, -r ~-~-resumable-manifest
640 -* **Usage**: Resume unfinished file uploading based on the manifest file.
641 -* (((
642 -**Prerequisite**:
643 643  
644 -* User login is required, otherwise an error is returned: "The current login session is invalid. Please login to continue."
645 -* Project must be accessible to the user, otherwise an error is returned: "Project Code not found in list. Please verify and try again."
646 -* The resumable manifest JSON file is required, which is generated upon executing the {{code}}pilotcli file upload{{/code}} command.
647 -)))
648 648  
649 -=== Example: ===
628 +----
650 650  
651 -For instance upon disconnecting from the internet, or by accidentally closing the terminal, a currently running uploading process can be aborted. You can use the {{code}}pilotcli file resume{{/code}} command in order to resume the uploading process right where you left off. Simply point to the manifest JSON file that was created upon executing the original upload command. Please note that the source files still have to be in the same location, otherwise resuming the uploading process will fail.
630 +-
631 +{{code language="none"}}{{/code}}
652 652  
653 -{{code language="none"}}
654 -$ pilotcli file resume -r ./manifest.json
655 -Resume the upload for 10 files.
656 -Resumable upload check complete.
657 -Uploading aggregated.py: |██████████████████████████████ 100% 00:00
658 -Uploading __init__.py: |██████████████████████████████ 100% 00:00
659 -Uploading service_meta_class.py: |██████████████████████████████ 100% 00:00
660 -Uploading upload_form.py: |██████████████████████████████ 100% 00:00
661 -Uploading convert_type.py: |██████████████████████████████ 100% 00:00
662 -Uploading singleton.py: |██████████████████████████████ 100% 00:00
663 -Uploading __init__.py: |██████████████████████████████ 100% 00:00
664 -Uploading base_library.zip: |██████████████████████████████ 100% 00:00 0% ?
665 -Uploading EXE-00.toc: |██████████████████████████████ 100% 00:00
666 -Uploading xref-pilotcli_test20230227_3.html: |██████████████████████████████ 100% 00:00
667 -Upload Time: 11.13s for 10 files
668 -{{/code}}
669 -
670 -== Command: **sync** ==
671 -
672 -* **Usage**: Download a file or a folder from a Project. If pilotcli is used outside of the Platform, e.g. the users local machine, users can only download from Project’s Core. When pilotcli is used within the Platform, e.g. from one of the workspace tools, users will also be able to download from the Green Room of a Project as well.
673 -* (((
674 -**Options**: -z ~-~-zone, -i ~-~-geid, ~-~-zip
675 -
676 -* To look up a file’s geid, or unique file ID, open the Project File Explorer in the Portal and inspect the File Properties for the ID.
677 -)))
678 -* (((
679 -**Prerequisite**:
680 -
681 -* User login is required, otherwise an error is returned: "The current login session is invalid. Please login to continue."
682 -* The specified file or folder must exist, otherwise an error is returned: “File does not exist or source file provided is invalid: folder_name/file_name.”
683 -* If the user has no access to the specified file or folder on the Portal, an error is returned: “Permission denied. Please verify that your role in the Project has permission to perform this action.”
684 -)))
685 -
686 -{{code language="none"}}
687 -$ pilotcli file sync --help
688 -Usage: pilotcli file sync [OPTIONS] [PATHS]... OUTPUT_PATH
689 -
690 - Download files/folders from a given Project/folder/file in core zone.
691 -
692 -Options:
693 - -z, --zone TEXT Target Zone (i.e., core/greenroom)
694 - --zip Download files as a zip.
695 - -i, --geid Enable downloading by geid.
696 - --help Show this message and exit.
697 -{{/code}}
698 -
699 -=== Examples ===
700 -
701 -Download a file or a folder from the Core zone of a Project:
702 -
703 -{{code language="none"}}
704 -$ pilotcli file sync cli/admin/sample_file2 -z core ~/Downloads/
705 -Preparing status: READY_FOR_DOWNLOADING
706 -start downloading...
707 -Downloading sample_file2 |██████████████████████████████ 100% 00:00
708 -File has been downloaded successfully and saved to: /home/uname/Downloads/sample_file2
709 -{{/code}}
710 -
711 -Download multiple files or folders as a compressed zip by simply providing all source files or folders and adding the {{code}}--zip{{/code}} option:
712 -
713 -{{code language="none"}}
714 -$ pilotcli file sync cli/admin/sample_file2 cli/admin/sample_file7 -z core ~/Downloads/ --zip
715 -Preparing downloading
716 -Preparing status: READY_FOR_DOWNLOADING
717 -start downloading...
718 -Downloading cli_1632340658.768384.zip |██████████████████████████████ 100% 00:00
719 -File has been downloaded successfully and saved to: /home/uname/Downloads/cli_1632340658.768384.zip
720 -{{/code}}
721 -
722 -Download files or folders by their geid, instead of their location. To enable this, add {{code}}-i{{/code}} or {{code}}--geid{{/code}} to the command:
723 -
724 -{{code language="none"}}
725 -$ pilotcli file sync -i 6d89afd4-f86e-4df7-b6c5-01a9af4e828e-1627499957 7e8e2bac-97c6-4738-bc08-12139c9ba225-1627498854 ~/Downloads/
726 -Preparing status: READY_FOR_DOWNLOADING
727 -start downloading...
728 -Downloading sample_file7 |██████████████████████████████ 100% 00:00
729 -File has been downloaded successfully and saved to: /home/uname/Downloads/sample_file7
730 -Preparing status: READY_FOR_DOWNLOADING
731 -start downloading...
732 -Downloading sample_file2 |██████████████████████████████ 100% 00:00
733 -File has been downloaded successfully and saved to: /home/uname/Downloads/sample_file2
734 -{{/code}}
735 -
736 -Download a file or folder from the Green Room of a Project. This will only be possible if pilotcli is used within the Platform, e.g. in one of the Workspace tools. It works the same way as downloading from Core, including all the additional options like zipping, by either adding {{code}}-z greenroom{{/code}} or by omitting the {{code}}-z{{/code}} option entirely, as “greenroom” is the default value:
737 -
738 -{{code language="none"}}
739 -$ pilotcli file sync cli/admin/sample_file2 cli/admin/sample_file7 ~/Downloads
740 -Preparing status: READY_FOR_DOWNLOADING
741 -/home/uname/Downloads/sample_file2 already exist, file will be saved as /home/uname/Downloads/sample_file2 (1)
742 -start downloading...
743 -Downloading sample_file2 (1) |██████████████████████████████ 100% 00:00
744 -File has been downloaded successfully and saved to: /home/uname/Downloads/sample_file2 (1)
745 -Preparing status: READY_FOR_DOWNLOADING
746 -start downloading...
747 -Downloading sample_file7 |██████████████████████████████ 100% 00:00
748 -File has been downloaded successfully and saved to: /home/uname/Downloads/sample_file7
749 -{{/code}}
750 -
751 -= Dataset Commands: pilotcli **dataset** =
752 -
753 -== Help ==
754 -
755 -{{code language="none"}}
756 -$ pilotcli dataset --help
757 -Usage: pilotcli dataset [OPTIONS] COMMAND [ARGS]...
758 -
759 -Options:
760 - --help Show this message and exit.
761 -
762 -Commands:
763 - download Download a dataset or a particular version of a dataset.
764 - list List datasets belonging to logged in user.
765 - show-detail Show details of a dataset.
766 -{{/code}}
767 -
768 -== Command: **list** ==
769 -
770 -* **Usage**: List all datasets for the logged in user
771 -* (((
772 -**Prerequisite**:
773 -
774 -* User login is required, otherwise an error is returned: "The current login session is invalid. Please login to continue."
775 -)))
776 -
777 -{{code language="none"}}
778 -$ pilotcli dataset list --help
779 -Usage: pilotcli dataset list [OPTIONS]
780 -
781 - List datasets belonging to logged in user.
782 -
783 -Options:
784 - --all Show also datasets that belong to projects where user has admin role
785 -
786 - --page INTEGER The page to be listed [default: 0]
787 - --page-size INTEGER number of objects per page [default: 10]
788 - -d, --detached whether run in detached mode
789 - --help Show this message and exit.
790 -{{/code}}
791 -
792 -=== Examples ===
793 -
794 -List all datasets that are accessible for the user:
795 -
796 -{{code language="none"}}
797 -$ pilotcli dataset list
798 - Dataset Title Dataset Code
799 ----------------------------------------------------------------------------
800 - dataset2 aug24202102
801 - dataset-01 aug242021
802 -Page: 0, Number of datasets: 2
803 -{{/code}}
804 -
805 -If the user has the Project Administrator role in one or more Projects, they can also access all Datasets associated with those Projects, even if they were created by another user. To do so, use the "~-~-all" flag:
806 -
807 -{{code language="none"}}
808 -$ pilotcli dataset list --all
809 - Dataset Title Dataset Code
810 ----------------------------------------------------------------------------
811 - dataset2 aug24202102
812 - dataset-01 aug242021
813 - test-dataset-01 testdataset01
814 - test-dataset-02 testdataset02
815 -Page: 0, Number of datasets: 4
816 -{{/code}}
817 -
818 -== Command: **show-detail** ==
819 -
820 -* **Usage**: Show the detailed information of a dataset.
821 -* (((
822 -**Prerequisite**:
823 -
824 -* User login is required, otherwise an error is returned: "The current login session is invalid. Please login to continue."
825 -)))
826 -
827 -{{code language="none"}}
828 -$ pilotcli dataset show-detail --help
829 -Usage: pilotcli dataset show-detail [OPTIONS] CODE
830 -
831 - Show details of a dataset.
832 -
833 -Options:
834 - --page INTEGER The page to be listed [default: 0]
835 - --page-size INTEGER number of objects per page [default: 10]
836 - -d, --detached whether run in detached mode
837 - --help Show this message and exit.
838 -{{/code}}
839 -
840 -=== Examples ===
841 -
842 -Show the details of the specified Dataset by providing the unique Dataset Code - “aug242021” in this case:
843 -
844 -{{code language="none"}}
845 -$ pilotcli dataset show-detail aug242021
846 ---------------------------------------------------------------------------------
847 -| Title | dataset-01 |
848 ---------------------------------------------------------------------------------
849 -| Code | aug242021 |
850 ---------------------------------------------------------------------------------
851 -| Authors | uname, cli |
852 ---------------------------------------------------------------------------------
853 -| Type | GENERAL |
854 ---------------------------------------------------------------------------------
855 -| Modality | neuroimaging, microscopy, histological approach |
856 ---------------------------------------------------------------------------------
857 -| Collection_method | import, test, upload |
858 ---------------------------------------------------------------------------------
859 -| Tags | tag1, tag2, tag3 |
860 ---------------------------------------------------------------------------------
861 -| Versions | 1.0, 1.1, 1.2 |
862 ---------------------------------------------------------------------------------
863 -{{/code}}
864 -
865 -== Command: **download** ==
866 -
867 -* **Usage**: Download a particular version of a dataset.
868 -* **Options**: -v ~-~-version
869 -* (((
870 -**Prerequisite**:
871 -
872 -* User login is required, otherwise error is returned: "The current login session is invalid. Please login to continue."
873 -)))
874 -
875 -{{code language="none"}}
876 -$ pilotcli dataset download --help
877 -Usage: pilotcli dataset download [OPTIONS] [CODE]... OUTPUT_PATH
878 -
879 - Download a dataset or a particular version of a dataset.
880 -
881 -Options:
882 - -v, --version TEXT Download a particular version of a dataset.
883 - --help Show this message and exit.
884 -{{/code}}
885 -
886 -
887 -=== Examples ===
888 -
889 -Download a Dataset with Code “aug242021”:
890 -
891 -{{code language="none"}}
892 -$ pilotcli dataset download aug242021 ~/Downloads/
893 -Pre downloading dataset
894 -Preparing status: READY_FOR_DOWNLOADING
895 -start downloading...
896 -Downloading aug242021_1632342868.882398.zip |██████████████████████████████ 100% 00:00
897 -File has been downloaded successfully and saved to: /home/uname/Downloads/aug242021_163234
898 -2868.882398.zip
899 -{{/code}}
900 -
901 -Download the version 1.1 of a Dataset with Code “aug242021”:
902 -
903 -{{code language="none"}}
904 -$ pilotcli dataset download aug242021 ~/Downloads/ -v 1.1
905 -Current dataset version: 1.1
906 -Pre downloading dataset
907 -start downloading...
908 -Downloading aug242021_2021-08-24 11:10:00.042277.zip |██████████████████████████████ 100% 00:00
909 -File has been downloaded successfully and saved to: /home/uname/Downloads/aug242021_2021-08-24 11:10:00.042277.zip
910 -{{/code}}
911 -
912 -
913 -
914 914  ----
915 915  
916 -Copyright © 2023-2024 [[Indoc Systems>>url:https://www.indocsystems.com]].
635 +Copyright © 2023 [[Indoc Research>>url:https://www.indocresearch.org/]].
917 917  
918 -HealthDataCloud is powered by Pilot technology, a product of [[Indoc Systems>>url:https://www.indocsystems.com]].
637 +HealthDataCloud is powered by Pilot technology, a product of [[Indoc Research>>url:https://www.indocresearch.org/]].~{~{/code}}
638 +~)~)~)
1723802541971-790.png
Author
... ... @@ -1,1 +1,0 @@
1 -XWiki.dsegebarth
Size
... ... @@ -1,1 +1,0 @@
1 -22.2 KB
Content
Pilot CLI authentication.png
Author
... ... @@ -1,0 +1,1 @@
1 +XWiki.sgevans
Size
... ... @@ -1,0 +1,1 @@
1 +33.6 KB
Content