Last modified by Dennis Segebarth on 2024/10/02 18:38
From version 1.6
edited by Susan Evans
on 2023/07/10 22:22
Change comment: (Autosaved)
To version 2.1
edited by Susan Evans
on 2023/07/10 22:36
Change comment: There is no comment for this version

Summary

Details

Page properties
Content
... ... @@ -1,3 +1,7 @@
1 +{{box cssClass="floatinginfobox" title="Table of Contents"}}
2 +{{toc depth="2"/}}
3 +{{/box}}
4 +
1 1  The Pilot Command Line Interface (pilotcli) is a binary executable program that provides a convenient way for users to perform file actions and platform-related tasks programmatically. The Pilot Command Line Interface can be downloaded and run on on your local Linux and Linux subsystems (as Windows user, see for instance the [[Windows Subsystem for Linux installation guide>>url:https://learn.microsoft.com/en-us/windows/wsl/install]]) to support large file uploads. For convenience, pilotcli is also pre-deployed and installed in the users JupyterHub accessible via worskpace tools.
2 2  
3 3  = Data Stewardship =
... ... @@ -584,20 +584,308 @@
584 584  Upload a file with tags and/or attributes.
585 585  
586 586  * When entering the upload command, the user can add a maximum of ten tags to the file. For example, in order to add “sample_tag1” and “sample_tag2” to your file, simply add {{code}}-t sample_tag1{{/code}} {{code}}-t sample_tag2{{/code}} to the command (see example below using {{code}}tag1{{/code}} and {{code}}tag2{{/code}} as tags).
587 -* In addition, you can also use attribute templates for a more enriched annotation of your files. Please note, that the corresponding attribute template file needs to be edited & saved prior to uploading. To get the blank template, see the **attribute-export** instructions above for more details. To attach a attribute JSON-file to your file upon uploading, use -a cli_Manifest1_template.json, where “cli_Manifest1_template.json" points to the location of the attributes JSON-file on your local machine.
591 +* In addition, you can also use attribute templates for a more enriched annotation of your files. Please note, that the corresponding attribute template file needs to be edited & saved prior to uploading. To get the blank template, see the **attribute-export** instructions above for more details. To attach a attribute JSON-file to your file upon uploading, use {{code}}-a cli_Manifest1_template.json{{/code}}, where “{{code}}cli_Manifest1_template.json{{/code}}" points to the location of the attributes JSON-file on your local machine.
588 588  * Upon uploading, pilotcli will also validate the content of the attached attribute JSON-file. Whenever a template is invalid, including missing non-optional fields, non-existing choice in multiple choice attribute, exceeding length text field and so on, pilotcli will raise error to user and user must update the attribute template and upload again:
589 589  
594 +{{code language="none"}}
595 +$ pilotcli file upload ./sample_folder/sample_file6 -p cli/admin -t tag1 -t tag2 -a cli_Manifest1_template.json
596 +File attribute validation passed: True
597 +Starting upload of: ./sample_folder/sample_file6
598 +Pre-upload complete.
599 +uploader: admin container_id: 18061 total_size: 1048576 total_chunks: 1 uniq_identifier: bc47ec2a-1a4e-4f71-a37a-7ce6473e415a-1627499273 resumable_relative_path admin/
600 +
601 +Upload job is finalizing, please wait...
602 +Upload job complete.
603 +Attribute attached
604 +All files uploaded successfully.
605 +{{/code}}
590 590  
607 +Upload a file with specified number of threads, using the {{code}}-td <thread_number>{{/code}} option.
591 591  
609 +* By default, the number of threads is 1. There will no UI difference when using threads. The maximal number of threads depends on your local machine and the recommend thread number is 3:
592 592  
593 -----
611 +{{code language="none"}}
612 +$ pilotcli file upload -p indoctestproject/admin/test20230306_3 /home/color/indoc/script/large_folder_test_data/gentoo_root_100M.img -td 3
613 +Target folder does not exist. Would you like to create a new folder? [y/N]: y
614 +Starting upload of: /home/color/indoc/script/large_folder_test_data/gentoo_root_100M.img
615 +Pre-upload complete.
616 +Uploading gentoo_root_100M.img , resumable_id: c68752372b42c85d, job_id: 56de3632-58df-445c-bcb9-bddfe6be5bf7: |██████████████████████████████ 100% 00:00
617 +Upload Time: 79.77s for 1 files
618 +All uploading jobs have finished.
619 +{{/code}}
594 594  
595 --
596 -{{code language="none"}}{{/code}}
621 +(10) upload with customized manifest name
597 597  
623 +When you upload a file using pilotcli, a JSON file with the details of your specific upload job will be created automatically and allows you to resume the upload at a later time, in case the process is aborted due to any reason. By default, this JSON file will be created as “manifest.json” in the current working directory (see //Command: **resume**// below for more details on how to resume the file upload). However, you can use the {{code}}--output-path{{/code}} option to specify the filepath and, thus, the filename of this JSON file:
624 +
625 +{{code language="none"}}
626 +$ pilotcli file upload -p <path> <file> --output-path test_manifest.json
627 +{{/code}}
628 +
629 +== Command: **resume** ==
630 +
631 +* **Options**: -td ~-~-thread, -r ~-~-resumable-manifest
632 +* **Usage**: Resume unfinished file uploading based on the manifest file.
633 +* (((
634 +**Prerequisite**:
635 +
636 +* User login is required, otherwise an error is returned: "The current login session is invalid. Please login to continue."
637 +* Project must be accessible to the user, otherwise an error is returned: "Project Code not found in list. Please verify and try again."
638 +* The resumable manifest JSON file is required, which is generated upon executing the {{code}}pilotcli file upload{{/code}} command.
639 +)))
640 +
641 +=== Example: ===
642 +
643 +For instance upon disconnecting from the internet, or by accidentally closing the terminal, a currently running uploading process can be aborted. You can use the {{code}}pilotcli file resume{{/code}} command in order to resume the uploading process right where you left off. Simply point to the manifest JSON file that was created upon executing the original upload command. Please note that the source files still have to be in the same location, otherwise resuming the uploading process will fail.
644 +
645 +{{code language="none"}}
646 +$ pilotcli file resume -r ./manifest.json
647 +Resume the upload for 10 files.
648 +Resumable upload check complete.
649 +Uploading aggregated.py: |██████████████████████████████ 100% 00:00
650 +Uploading __init__.py: |██████████████████████████████ 100% 00:00
651 +Uploading service_meta_class.py: |██████████████████████████████ 100% 00:00
652 +Uploading upload_form.py: |██████████████████████████████ 100% 00:00
653 +Uploading convert_type.py: |██████████████████████████████ 100% 00:00
654 +Uploading singleton.py: |██████████████████████████████ 100% 00:00
655 +Uploading __init__.py: |██████████████████████████████ 100% 00:00
656 +Uploading base_library.zip: |██████████████████████████████ 100% 00:00 0% ?
657 +Uploading EXE-00.toc: |██████████████████████████████ 100% 00:00
658 +Uploading xref-pilotcli_test20230227_3.html: |██████████████████████████████ 100% 00:00
659 +Upload Time: 11.13s for 10 files
660 +{{/code}}
661 +
662 +== Command: **sync** ==
663 +
664 +* **Usage**: Download a file or a folder from a Project. If pilotcli is used outside of the Platform, e.g. the users local machine, users can only download from Project’s Core. When pilotcli is used within the Platform, e.g. from one of the workspace tools, users will also be able to download from the Green Room of a Project as well.
665 +* (((
666 +**Options**: -z ~-~-zone, -i ~-~-geid, ~-~-zip
667 +
668 +* To look up a file’s geid, or unique file ID, open the Project File Explorer in the Portal and inspect the File Properties for the ID.
669 +)))
670 +* (((
671 +**Prerequisite**:
672 +
673 +* User login is required, otherwise an error is returned: "The current login session is invalid. Please login to continue."
674 +* The specified file or folder must exist, otherwise an error is returned: “File does not exist or source file provided is invalid: folder_name/file_name.”
675 +* If the user has no access to the specified file or folder on the Portal, an error is returned: “Permission denied. Please verify that your role in the Project has permission to perform this action.”
676 +)))
677 +
678 +{{code language="none"}}
679 +$ pilotcli file sync --help
680 +Usage: pilotcli file sync [OPTIONS] [PATHS]... OUTPUT_PATH
681 +
682 + Download files/folders from a given Project/folder/file in core zone.
683 +
684 +Options:
685 + -z, --zone TEXT Target Zone (i.e., core/greenroom)
686 + --zip Download files as a zip.
687 + -i, --geid Enable downloading by geid.
688 + --help Show this message and exit.
689 +{{/code}}
690 +
691 +=== Examples ===
692 +
693 +Download a file or a folder from the Core zone of a Project:
694 +
695 +{{code language="none"}}
696 +$ pilotcli file sync cli/admin/sample_file2 -z core ~/Downloads/
697 +Preparing status: READY_FOR_DOWNLOADING
698 +start downloading...
699 +Downloading sample_file2 |██████████████████████████████ 100% 00:00
700 +File has been downloaded successfully and saved to: /home/uname/Downloads/sample_file2
701 +{{/code}}
702 +
703 +Download multiple files or folders as a compressed zip by simply providing all source files or folders and adding the {{code}}--zip{{/code}} option:
704 +
705 +{{code language="none"}}
706 +$ pilotcli file sync cli/admin/sample_file2 cli/admin/sample_file7 -z core ~/Downloads/ --zip
707 +Preparing downloading
708 +Preparing status: READY_FOR_DOWNLOADING
709 +start downloading...
710 +Downloading cli_1632340658.768384.zip |██████████████████████████████ 100% 00:00
711 +File has been downloaded successfully and saved to: /home/uname/Downloads/cli_1632340658.768384.zip
712 +{{/code}}
713 +
714 +Download files or folders by their geid, instead of their location. To enable this, add {{code}}-i{{/code}} or {{code}}--geid{{/code}} to the command:
715 +
716 +{{code language="none"}}
717 +$ pilotcli file sync -i 6d89afd4-f86e-4df7-b6c5-01a9af4e828e-1627499957 7e8e2bac-97c6-4738-bc08-12139c9ba225-1627498854 ~/Downloads/
718 +Preparing status: READY_FOR_DOWNLOADING
719 +start downloading...
720 +Downloading sample_file7 |██████████████████████████████ 100% 00:00
721 +File has been downloaded successfully and saved to: /home/uname/Downloads/sample_file7
722 +Preparing status: READY_FOR_DOWNLOADING
723 +start downloading...
724 +Downloading sample_file2 |██████████████████████████████ 100% 00:00
725 +File has been downloaded successfully and saved to: /home/uname/Downloads/sample_file2
726 +{{/code}}
727 +
728 +Download a file or folder from the Green Room of a Project. This will only be possible if pilotcli is used within the Platform, e.g. in one of the Workspace tools. It works the same way as downloading from Core, including all the additional options like zipping, by either adding {{code}}-z greenroom{{/code}} or by omitting the {{code}}-z{{/code}} option entirely, as “greenroom” is the default value:
729 +
730 +{{code language="none"}}
731 +$ pilotcli file sync cli/admin/sample_file2 cli/admin/sample_file7 ~/Downloads
732 +Preparing status: READY_FOR_DOWNLOADING
733 +/home/uname/Downloads/sample_file2 already exist, file will be saved as /home/uname/Downloads/sample_file2 (1)
734 +start downloading...
735 +Downloading sample_file2 (1) |██████████████████████████████ 100% 00:00
736 +File has been downloaded successfully and saved to: /home/uname/Downloads/sample_file2 (1)
737 +Preparing status: READY_FOR_DOWNLOADING
738 +start downloading...
739 +Downloading sample_file7 |██████████████████████████████ 100% 00:00
740 +File has been downloaded successfully and saved to: /home/uname/Downloads/sample_file7
741 +{{/code}}
742 +
743 += Dataset Commands: pilotcli **dataset** =
744 +
745 +== Help ==
746 +
747 +{{code language="none"}}
748 +$ pilotcli dataset --help
749 +Usage: pilotcli dataset [OPTIONS] COMMAND [ARGS]...
750 +
751 +Options:
752 + --help Show this message and exit.
753 +
754 +Commands:
755 + download Download a dataset or a particular version of a dataset.
756 + list List datasets belonging to logged in user.
757 + show-detail Show details of a dataset.
758 +{{/code}}
759 +
760 +== Command: **list** ==
761 +
762 +* **Usage**: List all datasets for the logged in user
763 +* (((
764 +**Prerequisite**:
765 +
766 +* User login is required, otherwise an error is returned: "The current login session is invalid. Please login to continue."
767 +)))
768 +
769 +{{code language="none"}}
770 +$ pilotcli dataset list --help
771 +Usage: pilotcli dataset list [OPTIONS]
772 +
773 + List datasets belonging to logged in user.
774 +
775 +Options:
776 + --page INTEGER The page to be listed [default: 0]
777 + --page-size INTEGER number of objects per page [default: 10]
778 + -d, --detached whether run in detached mode
779 + --help Show this message and exit.
780 +{{/code}}
781 +
782 +=== Examples ===
783 +
784 +List all datasets that are accessible for the user:
785 +
786 +{{code language="none"}}
787 +$ pilotcli dataset list
788 + Dataset Title Dataset Code
789 +---------------------------------------------------------------------------
790 + dataset2 aug24202102
791 + dataset-01 aug242021
792 +Page: 0, Number of datasets: 2
793 +{{/code}}
794 +
795 +== Command: **show-detail** ==
796 +
797 +* **Usage**: Show the detailed information of a dataset.
798 +* (((
799 +**Prerequisite**:
800 +
801 +* User login is required, otherwise an error is returned: "The current login session is invalid. Please login to continue."
802 +)))
803 +
804 +{{code language="none"}}
805 +$ pilotcli dataset show-detail --help
806 +Usage: pilotcli dataset show-detail [OPTIONS] CODE
807 +
808 + Show details of a dataset.
809 +
810 +Options:
811 + --page INTEGER The page to be listed [default: 0]
812 + --page-size INTEGER number of objects per page [default: 10]
813 + -d, --detached whether run in detached mode
814 + --help Show this message and exit.
815 +{{/code}}
816 +
817 +=== Examples ===
818 +
819 +Show the details of the specified Dataset by providing the unique Dataset Code - “aug242021” in this case:
820 +
821 +{{code language="none"}}
822 +$ pilotcli dataset show-detail aug242021
823 +--------------------------------------------------------------------------------
824 +| Title | dataset-01 |
825 +--------------------------------------------------------------------------------
826 +| Code | aug242021 |
827 +--------------------------------------------------------------------------------
828 +| Authors | uname, cli |
829 +--------------------------------------------------------------------------------
830 +| Type | GENERAL |
831 +--------------------------------------------------------------------------------
832 +| Modality | neuroimaging, microscopy, histological approach |
833 +--------------------------------------------------------------------------------
834 +| Collection_method | import, test, upload |
835 +--------------------------------------------------------------------------------
836 +| Tags | tag1, tag2, tag3 |
837 +--------------------------------------------------------------------------------
838 +| Versions | 1.0, 1.1, 1.2 |
839 +--------------------------------------------------------------------------------
840 +{{/code}}
841 +
842 +== Command: **download** ==
843 +
844 +* **Usage**: Download a particular version of a dataset.
845 +* **Options**: -v ~-~-version
846 +* (((
847 +**Prerequisite**:
848 +
849 +* User login is required, otherwise error is returned: "The current login session is invalid. Please login to continue."
850 +)))
851 +
852 +{{code language="none"}}
853 +$ pilotcli dataset download --help
854 +Usage: pilotcli dataset download [OPTIONS] [CODE]... OUTPUT_PATH
855 +
856 + Download a dataset or a particular version of a dataset.
857 +
858 +Options:
859 + -v, --version TEXT Download a particular version of a dataset.
860 + --help Show this message and exit.
861 +{{/code}}
862 +
863 +
864 +=== Examples ===
865 +
866 +Download a Dataset with Code “aug242021”:
867 +
868 +{{code language="none"}}
869 +$ pilotcli dataset download aug242021 ~/Downloads/
870 +Pre downloading dataset
871 +Preparing status: READY_FOR_DOWNLOADING
872 +start downloading...
873 +Downloading aug242021_1632342868.882398.zip |██████████████████████████████ 100% 00:00
874 +File has been downloaded successfully and saved to: /home/uname/Downloads/aug242021_163234
875 +2868.882398.zip
876 +{{/code}}
877 +
878 +Download the version 1.1 of a Dataset with Code “aug242021”:
879 +
880 +{{code language="none"}}
881 +$ pilotcli dataset download aug242021 ~/Downloads/ -v 1.1
882 +Current dataset version: 1.1
883 +Pre downloading dataset
884 +start downloading...
885 +Downloading aug242021_2021-08-24 11:10:00.042277.zip |██████████████████████████████ 100% 00:00
886 +File has been downloaded successfully and saved to: /home/uname/Downloads/aug242021_2021-08-24 11:10:00.042277.zip
887 +{{/code}}
888 +
889 +
890 +
598 598  ----
599 599  
600 600  Copyright © 2023 [[Indoc Research>>url:https://www.indocresearch.org/]].
601 601  
602 -HealthDataCloud is powered by Pilot technology, a product of [[Indoc Research>>url:https://www.indocresearch.org/]].~{~{/code}}
603 -~)~)~)
895 +HealthDataCloud is powered by Pilot technology, a product of [[Indoc Research>>url:https://www.indocresearch.org/]].