Changes for page Working with HDC Project Files in the Command Line Interface

Last modified by Dennis Segebarth on 2024/10/02 18:38

From 1.3 to 1.4 From 1.8 to 1.9

From version 1.4

edited by Susan Evans
on 2023/07/10 22:18

Change comment: (Autosaved)

To version 1.8

edited by Susan Evans
on 2023/07/10 22:26

Change comment: (Autosaved)

Raw
Rendered

Summary

Page properties (1 modified, 0 added, 0 removed)

Details

Page properties

Content

@@ -453,8 +453,178 @@
  * When uploading file to the Project Core, the user must have access to the Core (Project Contributors cannot upload files/folders to the Project Core).
  )))
++{{code language="none"}}
++$ pilotcli file upload --help
++Usage: pilotcli file upload [OPTIONS] [PATHS]...
++  Upload files/folders to a given Project path.
++Options:
++  -p, --project-path TEXT    Project folder path starting from Project code.
++                             (i.e., indoctestproject/user/folder)  [required]
++
++  -a, --attribute TEXT       File Attribute Template used for annotating files
++                             during upload.
++
++  -t, --tag TEXT             Add a tag to the file. This option could be used
++                             multiple times for adding multiple tags.
++
++  -z, --zone TEXT            Target Zone (i.e., core/greenroom)  [default:
++                             greenroom]  [default: greenroom]
++
++  -m, --upload-message TEXT  The message used to comment on the purpose of
++                             uploading your processed file  [default: ]
++
++  -s, --source-file TEXT     The Project path of the source file of your
++                             processed files.
++
++  --zip                      Upload folder as a compressed zip file.
++  -td, --thread INTEGER      The number of thread for upload a file  [default:
++                             1]
++
++  -o, --output-path TEXT     The output path for the manifest file of
++                             resumable upload  [default: ./manifest.json]
++
++  --help                     Show this message and exit.
++{{/code}}
++
++=== Examples ===
++
++Upload a file (or folder) to a specified directory of a Project’s Green Room, in this case the file located at {{code}}./sample_folder/sample_file1{{/code}} will be uploaded to the folder “admin” in the Project with the Project Code “cli” {{code}}(-p cli/admin){{/code}}:
++
++{{code language="none"}}
++$ pilotcli file upload ./sample_folder/sample_file1 -p cli/admin
++Starting upload of: ./sample_folder/sample_file1
++Pre-upload complete.
++uploader: admin container_id: 18061 total_size: 1048576 total_chunks: 1 uniq_identifier: e958247c-ce05-48b2-9038-78b774836256-1627498771 resumable_relative_path admin/
++
++Upload job is finalizing, please wait...
++Upload job complete.
++All files uploaded successfully.
++{{/code}}
++
++In case the specified destination directory does not exist yet, you will be prompted to confirm its creation by entering “y”, or to abort the command by entering “N”:
++
++{{code language="none"}}
++Target folder does not exist. Would you like to create a new folder? [y/N]: y
++{{/code}}
++
++In order to upload a file (or folder) to a destination directory in a Project's Core, core must be specified as corresponding zone {{code}}(-z core){{/code}} and you will be required to provide a message to that should describe the files` content and why you are uploading it directly into the Core {{code}}(-m "sample message"){{/code}}. Moreover, you will be reminded about the Terms of Use of the Platform and additional considerations regarding the direct upload of data into the Core zone, and are prompted to confirm your intention to proceed with the upload by entering “y” or “Yes”:
++
++{{code language="none"}}
++$ pilotcli file upload ./sample_folder/sample_file2 -p cli/admin -z core -m "sample message"
++You are about to transfer data directly to the Core! In accordance with the Terms of Use, please confirm that you
++have made your best efforts to pseudonymize or anonymize the data and that you have the legal authority to transfer and make this
++data available for dissemination and use within the Platform.To review the Terms of Use, please the Portal. If you
++need to process the data to remove sensitive identifiers, please cancel this transfer and upload the data to the Green Room to
++perform these actions.
++To cancel this transfer, enter [n/No]
++To confirm and proceed with the data transfer, enter [y/Yes]
++  [y/N]: y
++Starting upload of: ./sample_folder/sample_file2
++Pre-upload complete.
++uploader: admin container_id: 18061 total_size: 1048576 total_chunks: 1 uniq_identifier: fa807a0f-ceb0-4fff-9c9e-563bdcbb309f-1627498851 resumable_relative_path admin/
++
++Upload job is finalizing, please wait...
++Upload job complete.
++All files uploaded successfully.
++{{/code}}
++
++Upload a folder to Project Green Room
++
++{{code language="none"}}
++$ pilotcli file upload ./sample_folder/example_folder -p cli/admin
++Starting upload of: ./sample_folder/example_folder
++Current version does not support folder tagging, any selected tags will be ignored
++Pre-upload complete.
++uploader: admin container_id: 18061 total_size: 1048576 total_chunks: 1 uniq_identifier: 4f0eee2e-70f4-4b03-9e72-040544ec7d3a-1627498907 resumable_relative_path admin/example_folder/
++
++Starting finalization...
++uploader: admin container_id: 18061 total_size: 1048576 total_chunks: 1 uniq_identifier: 41ec70e2-0859-4aba-9009-06e700c7c79e-1627498907 resumable_relative_path admin/example_folder/
++
++Starting finalization...
++uploader: admin container_id: 18061 total_size: 1048576 total_chunks: 1 uniq_identifier: 28454c51-ec9a-4dc5-ab3c-0f7e80e3050d-1627498907 resumable_relative_path admin/example_folder/
++
++Upload job is finalizing, please wait...
++Upload job complete.
++All files uploaded successfully.
++{{/code}}
++
++Upload file to existing folder in Green Room
++
++* Specify a target folder to upload files by inputting folder name after the Project code separated by the '/'.
++
++{{code language="none"}}
++$ pilotcli file upload ./sample_folder/sample_file5 -p cli/admin/example_folder
++Starting upload of: ./sample_folder/sample_file5
++Pre-upload complete.
++uploader: admin container_id: 18061 total_size: 1048576 total_chunks: 1 uniq_identifier: 7ef58005-d408-4ad9-a741-fa938f58e67f-1627498977 resumable_relative_path admin/example_folder/
++
++Upload job is finalizing, please wait...
++Upload job complete.
++All files uploaded successfully.
++{{/code}}
++
++Upload folder as zip to a specified directory of a Project's Green Room by adding the ~-~-zip option.
++
++* The specified folder will then be compressed as a zip file and subsequently be uploaded to the specified destination. Once the upload is completed, the compressed zip file will be removed from the directory. Note: pilotcli must have the permission to create and delete zip files in the corresponding directory of your local machine:
++
++{{code language="none"}}
++$ pilotcli file upload ./sample_folder/example_folder -p cli/admin --zip
++Starting upload of: ./sample_folder/example_folder
++Started zipping files.
++Pre-upload complete.
++uploader: admin container_id: 18061 total_size: 3595 total_chunks: 1 uniq_identifier: 61a729ce-cdd6-4a87-82b5-3f8ea63e6c4a-1627499012 resumable_relative_path admin/
++
++Upload job is finalizing, please wait...
++Upload job complete.
++All files uploaded successfully.
++{{/code}}
++
++Upload a file with tags and/or attributes.
++
++* When entering the upload command, the user can add a maximum of ten tags to the file. For example, in order to add “sample_tag1” and “sample_tag2” to your file, simply add {{code}}-t sample_tag1{{/code}} {{code}}-t sample_tag2{{/code}} to the command (see example below using {{code}}tag1{{/code}} and {{code}}tag2{{/code}} as tags).
++* In addition, you can also use attribute templates for a more enriched annotation of your files. Please note, that the corresponding attribute template file needs to be edited & saved prior to uploading. To get the blank template, see the **attribute-export** instructions above for more details. To attach a attribute JSON-file to your file upon uploading, use {{code}}-a cli_Manifest1_template.json{{/code}}, where “{{code}}cli_Manifest1_template.json{{/code}}" points to the location of the attributes JSON-file on your local machine.
++* Upon uploading, pilotcli will also validate the content of the attached attribute JSON-file. Whenever a template is invalid, including missing non-optional fields, non-existing choice in multiple choice attribute, exceeding length text field and so on, pilotcli will raise error to user and user must update the attribute template and upload again:
++
++{{code language="none"}}
++$ pilotcli file upload ./sample_folder/sample_file6 -p cli/admin -t tag1 -t tag2 -a cli_Manifest1_template.json
++File attribute validation passed: True
++Starting upload of: ./sample_folder/sample_file6
++Pre-upload complete.
++uploader: admin container_id: 18061 total_size: 1048576 total_chunks: 1 uniq_identifier: bc47ec2a-1a4e-4f71-a37a-7ce6473e415a-1627499273 resumable_relative_path admin/
++
++Upload job is finalizing, please wait...
++Upload job complete.
++Attribute attached
++All files uploaded successfully.
++{{/code}}
++
++Upload a file with specified number of threads, using the {{code}}-td <thread_number>{{/code}} option.
++
++* By default, the number of threads is 1. There will no UI difference when using threads. The maximal number of threads depends on your local machine and the recommend thread number is 3:
++
++{{code language="none"}}
++$ pilotcli file upload -p indoctestproject/admin/test20230306_3 /home/color/indoc/script/large_folder_test_data/gentoo_root_100M.img -td 3
++Target folder does not exist. Would you like to create a new folder? [y/N]: y
++Starting upload of: /home/color/indoc/script/large_folder_test_data/gentoo_root_100M.img
++Pre-upload complete.
++Uploading gentoo_root_100M.img , resumable_id: c68752372b42c85d, job_id: 56de3632-58df-445c-bcb9-bddfe6be5bf7:  |██████████████████████████████ 100% 00:00
++Upload Time: 79.77s for 1 files
++All uploading jobs have finished.
++{{/code}}
++
++(10) upload with customized manifest name
++
++When you upload a file using pilotcli, a JSON file with the details of your specific upload job will be created automatically and allows you to resume the upload at a later time, in case the process is aborted due to any reason. By default, this JSON file will be created as “manifest.json” in the current working directory (see //Command: **resume**// below for more details on how to resume the file upload). However, you can use the {{code}}--output-path{{/code}} option to specify the filepath and, thus, the filename of this JSON file:
++
++{{code language="none"}}
++$ pilotcli file upload -p <path> <file> --output-path test_manifest.json
++{{/code}}
++
++
++
++
  ----
  -