Version 9.1 by Dennis Segebarth on 2023/09/13 18:09
Show last authors
1 {{box cssClass="floatinginfobox" title="Table of Contents"}}
2 {{toc depth="2"/}}
3 {{/box}}
4
5 The Pilot Command Line Interface (pilotcli) is a binary executable program that provides a convenient way for users to perform file actions and platform-related tasks programmatically. The Pilot Command Line Interface can be downloaded and run on on your local Linux and Linux subsystems (as Windows user, see for instance the [[Windows Subsystem for Linux installation guide>>url:https://learn.microsoft.com/en-us/windows/wsl/install]]) to support large file uploads. For convenience, pilotcli is also pre-deployed and installed in the users JupyterHub accessible via worskpace tools.
6
7 = Data Stewardship =
8
9 Users are reminded to abide by the Platform Terms of Use and any Project-specific restrictions when using the Command Line Interface to access data and code.
10
11 = Using the Pilot Command Line Interface in JupyterHub =
12
13 To get started with the Pilot Command Line Interface in the HDC JupyterHub workspace tool, see the article //Using JupyterHub in HDC //and refer to the remaining sections in this article for a full description of the pilotcli commands.
14
15 = Using the Pilot Command Line Interface in Guacamole VMs =
16
17 To get started with the Pilot Command Line Interface in the HDC Guacamole workspace tool, see //Using Guacamole in HDC// on how to launch a terminal, either in a Desktop VM or directly by establishing a connection to a Command Line VM. The Pilot Command Line Interface will be installed, configured, and ready to be used. Please refer to the remaining sections in this article for a full description of the pilotcli commands.
18
19 = Downloading the Pilot Command Line Interface =
20
21 If you wish to use the Pilot Command Line Interface to upload large files from your local environment, it is necessary to download Pilot Command Line Interface tool on your Linux system. The recommended Linux version is Ubuntu 20.04.
22
23 * Download the pilotcli tool [[here>>url:https://object.hdc.humanbrainproject.eu/public-resources/pilotcli]].
24
25 After download, please grant execution permission to the binary file so that Pilot Command Line Interface can be used. To grant permission to the binary file, use the following command in the downloaded pilotcli directory.
26
27 {{code language="none"}}
28 $ chmod +x ./pilotcli
29 {{/code}}
30
31
32 If you have sudo permission, you can make the Pilot Command Line Interface more accessible in any directory by moving it to the /usr/local/bin directory:
33
34 {{code language="none"}}
35 $ mv ./pilotcli /usr/local/bin
36 {{/code}}
37
38 Without sudo permission, you can still use the Pilot Command Line Interface in a different directory by referring the Command Line Interface path. For example, use a command line Interface in the Downloads folder:
39
40 {{code language="none"}}
41 $ ~/Downloads/pilotcli
42 {{/code}}
43
44 = Pilot Command Line Interface Command Summary =
45
46 Commands are organized into main groups:
47
48 1. **User**// - //User related actions such as login and logout
49 1. **Project** //- //Project related actions, including list Project Names and Codes for all Projects accessible by the logged-in user.
50 1. **File **- Actions on files. User could use file action commands to list files/folders, upload files/folders, list/export attribute, and download files as well.
51 1. **Dataset **- Actions related to datasets.
52
53 = User Commands: pilotcli **user** =
54
55 == Help ==
56
57 {{code language="none"}}
58 $ pilotcli user --help
59 Usage: pilotcli user [OPTIONS] COMMAND [ARGS]...
60
61 Options:
62 --help Show this message and exit.
63
64 Commands:
65 login For user to login.
66 logout For user to logout.
67 {{/code}}
68
69 == Command: **login** ==
70
71 * **Options**: -u ~-~-username, -p ~-~-password
72 * **Usage**: Allows users to log into the platform through pilotcli. After login, the user must be actively using the Pilot Command Line Interface to remain logged in, or they will be logged out automatically after the session is idle for five minutes.
73 * **Prerequisites**: None.
74
75 {{code language="none"}}
76 $ pilotcli user login --help
77 Usage: pilotcli user login [OPTIONS]
78
79 For user to login.
80
81 Options:
82 --help Show this message and exit.
83 {{/code}}
84
85 === Example ===
86
87 User login:
88
89 1) Use the output device validation link or QR code to launch authentication validation:
90
91 {{code language="none"}}
92 $ pilotcli user login
93 Please, access https://iam.hdc.humanbrainproject.eu/realms/hdc/device?user_code=XXXX-XXXX to proceed
94 ▄▄▄▄▄▄▄ ▄▄ ▄ ▄ ▄ ▄▄▄▄ ▄ ▄▄▄▄▄▄▄
95 █ ▄▄▄ █ ▄ ▄▄▀█ █▀▀▀▄▀█▀ ▀██▄ █ ▄▄▄ █
96 █ ███ █ ██▄ ▀▀▄▀▄▄█▄ ▄ ▀▀▀▄ █ ███ █
97 █▄▄▄▄▄█ █▀▄▀█▀▄▀█▀▄ █ ▄ ▄ █▀█ █▄▄▄▄▄█
98 ▄▄▄▄▄▄▄ ▀█▀█▄▄█▄ █▀█ ▄▀▄▄ █ ▄ █▀▄▄▄
99 █ ▄▄▄ █ █▄█▄██ ▀ ▀█▀▀█ ▀▄█▄█▄▄▄█▀▄█
100 █ ███ █ █▀█ ▄▀██▄█▀▀█▄▀▀█▄▀█ ▀ ▀▄▀██
101 █▄▄▄▄▄█ ▄▀▄ ██▄▀▀▄ ▀▀▄ ▄▄▀▀▀▄ █▄▄▄█
102
103
104 Waiting validation finish...
105 {{/code}}
106
107 2) Grant access by clicking Yes:
108
109 [[image:Pilot CLI authentication.png||height="34%" width="40%"]]
110
111 3) Device Login Successful message will appear and your authentication browser window can be closed.
112
113 4) The output of the Command Line Tool will also be updated to notify you about the successful login:
114
115 {{code language="none"}}
116 $ pilotcli user login
117 Please, access https://iam.hdc.humanbrainproject.eu/realms/hdc/device?user_code=XXXX-XXXX to proceed
118 ▄▄▄▄▄▄▄ ▄▄ ▄ ▄ ▄ ▄▄▄▄ ▄ ▄▄▄▄▄▄▄
119 █ ▄▄▄ █ ▄ ▄▄▀█ █▀▀▀▄▀█▀ ▀██▄ █ ▄▄▄ █
120 █ ███ █ ██▄ ▀▀▄▀▄▄█▄ ▄ ▀▀▀▄ █ ███ █
121 █▄▄▄▄▄█ █▀▄▀█▀▄▀█▀▄ █ ▄ ▄ █▀█ █▄▄▄▄▄█
122 ▄▄▄▄▄▄▄ ▀█▀█▄▄█▄ █▀█ ▄▀▄▄ █ ▄ █▀▄▄▄
123 █ ▄▄▄ █ █▄█▄██ ▀ ▀█▀▀█ ▀▄█▄█▄▄▄█▀▄█
124 █ ███ █ █▀█ ▄▀██▄█▀▀█▄▀▀█▄▀█ ▀ ▀▄▀██
125 █▄▄▄▄▄█ ▄▀▄ ██▄▀▀▄ ▀▀▄ ▄▄▀▀▀▄ █▄▄▄█
126
127
128 Waiting validation finish...
129 Welcome to Command Line Tool!
130 {{/code}}
131
132 == Command: **logout** ==
133
134 * **Options**: -y ~-~-yes
135 * **Usage**: Allows users to log out of the platform through pilotcli.
136 * **Prerequisites**: User login is required, otherwise an error is returned: ‘The current login session is invalid. Please login to continue.'.
137
138 {{code language="none"}}
139 $ pilotcli user logout --help
140 Usage: pilotcli user logout [OPTIONS]
141
142 For user to logout.
143
144 Options:
145 -y, --yes Input Y/yes to confirm you want to logout, otherwise input N/no
146 to remain logged in.
147
148 --help Show this message and exit.
149 {{/code}}
150
151 === Examples ===
152
153 User logout:
154
155 {{code language="none"}}
156 $ pilotcli user logout
157 Are you sure you want to logout? [y/N]: y
158 Logged out successfully. Bye!
159 {{/code}}
160
161 User logout canceled:
162
163 {{code language="none"}}
164 $ pilotcli user logout
165 Are you sure you want to logout? [y/N]: n
166 Logout canceled!
167 {{/code}}
168
169 User logout with option to skip additional confirmation:
170
171 {{code language="none"}}
172 $ pilotcli user logout -y
173 Logged out successfully. Bye!
174 {{/code}}
175
176 = Project Commands: pilotcli **project** =
177
178 == Help ==
179
180 {{code language="none"}}
181 $ pilotcli project --help
182 Usage: pilotcli project [OPTIONS] COMMAND [ARGS]...
183
184 Options:
185 --help Show this message and exit.
186
187 Commands:
188 list List accessible projects.
189 {{/code}}
190
191 == Command: **list** ==
192
193 * **Options**: None
194 * **Usage**: List all Projects to which the user has access
195 * **Prerequisite**: User login is required, otherwise an error is returned: ‘The current login session is invalid. Please login to continue.'
196
197 {{code language="none"}}
198 $ pilotcli project list --help
199 Usage: pilotcli project list [OPTIONS]
200
201 List accessible projects.
202
203 Options:
204 --page INTEGER The page to be listed [default: 0]
205 --page-size INTEGER number of objects per page [default: 10]
206 --order [asc|desc] sorting order [default: desc]
207 --order-by [created_at|name|code]
208 sorting column [default: created_at]
209 -d, --detached whether run in detached mode
210 --help Show this message and exit.
211 {{/code}}
212
213 === Examples ===
214
215 Get a list of all Projects the user has access to:
216
217 {{code language="none"}}
218 $ pilotcli project list
219 Project Name Project Code
220 ---------------------------------------------------------------------------
221 Test Project 02-13 | test230213byshan
222 Test1_Audit | test1audit
223 Test20221109 | test20220109
224 Indoc Test Project | indoctestproject
225
226 Page: 0, Number of projects: 4
227 {{/code}}
228
229 By default, the table will only display the first ten elements. If the user has access to more than ten Projects, you will be prompted to use your arrow keys to select one of two options to either view the next ten items (//next page//), or to exit (//exit//). The selected option is indicated by two arrows and can be confirmed by hitting enter:
230
231 {{code language="none"}}
232 Page: 0, Number of projects: 10
233
234 What do you want? (Use arrow keys)
235 » next page
236 exit
237 {{/code}}
238
239 = File Commands: pilotcli **file** =
240
241 == Help ==
242
243 {{code language="none"}}
244 $ pilotcli file --help
245 Usage: pilotcli file [OPTIONS] COMMAND [ARGS]...
246
247 Options:
248 --help Show this message and exit.
249
250 Commands:
251 attribute-export Export attribute template from a given Project.
252 attribute-list List attribute templates of a given Project.
253 list List files and folders inside a given Project/folder.
254 resume Resume the upload process with given manifest file.
255 sync Download files/folders from a given Project/folder/file...
256 upload Upload files/folders to a given Project path.
257 {{/code}}
258
259 == Command: **attribute-list** ==
260
261 * **Options**: -p ~-~-project-code
262 * **Usage**: List all attributes from a specified Project.
263 * **Prerequisite**:
264
265 * User login is required, otherwise an error is returned: ‘The current login session is invalid. Please login to continue.’
266 * Requires the Project code, which can be retrieved via {{code}}pilotcli project list{{/code}}.
267
268 {{code language="none"}}
269 $ pilotcli file attribute-list --help
270 Usage: pilotcli file attribute-list [OPTIONS]
271
272 List attribute templates of a given Project.
273
274 Options:
275 -p, --project-code TEXT Project Code
276 --help Show this message and exit.
277 {{/code}}
278
279 === Examples ===
280
281 List attributes from a given Project by providing the Project Code at the prompt. In this example, “cli” is the corresponding Project Code:
282
283 {{code language="none"}}
284 $ pilotcli file attribute-list
285 ProjectCode: cli
286
287 Manifest1
288 --------------------------------------------------------------------------
289 | Attribute Name | Type | Value | Optional |
290 --------------------------------------------------------------------------
291 | attr1 | multiple_choice | a1,a2,a3,a4,a5 | False |
292 --------------------------------------------------------------------------
293 | attr2 | text | None | False |
294 --------------------------------------------------------------------------
295 | attr3 | multiple_choice | t1,t2,t3,t4,t5 | True |
296 --------------------------------------------------------------------------
297
298 Manifest2
299 --------------------------------------------------------------------------
300 | Attribute Name | Type | Value | Optional |
301 --------------------------------------------------------------------------
302 | attr1 | multiple_choice | a1,a2,a3,a4 | True |
303 --------------------------------------------------------------------------
304 | attr2 | text | None | False |
305 --------------------------------------------------------------------------
306 | attr3 | multiple_choice | t1,t2,t3 | True |
307 --------------------------------------------------------------------------
308 All Attributes fetched successfully.
309 {{/code}}
310
311 List attributes from given a Project and provide the Project Code “cli” as an option in the same line {{code}}-p cli{{/code}}:
312
313 {{code language="none"}}
314 $ pilotcli file attribute-list -p cli
315
316 Manifest1
317 --------------------------------------------------------------------------
318 | Attribute Name | Type | Value | Optional |
319 --------------------------------------------------------------------------
320 | attr1 | multiple_choice | a1,a2,a3,a4,a5 | False |
321 --------------------------------------------------------------------------
322 | attr2 | text | None | False |
323 --------------------------------------------------------------------------
324 | attr3 | multiple_choice | t1,t2,t3,t4,t5 | True |
325 --------------------------------------------------------------------------
326
327 Manifest2
328 --------------------------------------------------------------------------
329 | Attribute Name | Type | Value | Optional |
330 --------------------------------------------------------------------------
331 | attr1 | multiple_choice | a1,a2,a3,a4 | True |
332 --------------------------------------------------------------------------
333 | attr2 | text | None | False |
334 --------------------------------------------------------------------------
335 | attr3 | multiple_choice | t1,t2,t3 | True |
336 --------------------------------------------------------------------------
337 All Attributes fetched successfully.
338 {{/code}}
339
340 == Command: **attribute-export** ==
341
342 * **Options**: -p ~-~-project-code, -n ~-~-attribute-name
343 * **Usage**: Export an attribute template as JSON-file from a given Project
344 * (((
345 **Prerequisite**:
346
347 * User login is required, otherwise an error is returned: "The current login session is invalid. Please login to continue."
348 * Project must be accessible to the user, otherwise an error is returned: "Project Code not found in list. Please verify and try again."
349 * Requires the Project code, which can be retrieved via {{code}}pilotcli project list{{/code}}.
350 * Requires the Attribute template name. If the attribute template name does not exist or is spelled incorrectly, an error is returned: "Attribute 'sample-template' not found in Project. Please verify and try again."
351 * pilotcli must have the permission to write a file in the directory, otherwise an error is returned.
352 )))
353
354 {{code language="none"}}
355 $ pilotcli file attribute-export --help
356 Usage: pilotcli file attribute-export [OPTIONS]
357
358 Export attribute template from a given Project.
359
360 Options:
361 -p, --project-code TEXT Project Code
362 -n, --attribute-name TEXT Attribute Template Name
363 --help Show this message and exit.
364 {{/code}}
365
366 === Examples ===
367
368 Export an attribute template from a given Project. The user is prompted to provide the Project Code (“cli”) and the Attribute Template Name (“Manifest1”):
369
370 {{code language="none"}}
371 $ pilotcli file attribute-export
372 ProjectCode: cli
373 AttributeName: Manifest1
374
375 Manifest1
376 --------------------------------------------------------------------------
377 | Attribute Name | Type | Value | Optional |
378 --------------------------------------------------------------------------
379 | attr1 | multiple_choice | a1,a2,a3,a4,a5 | False |
380 --------------------------------------------------------------------------
381 | attr2 | text | None | False |
382 --------------------------------------------------------------------------
383 | attr3 | multiple_choice | t1,t2,t3,t4,t5 | True |
384 --------------------------------------------------------------------------
385 Template saved successfully: cli_Manifest1_template.json
386 Attribute definition saved successfully: cli_Manifest1_definition.json
387 {{/code}}
388
389 Export an attribute template from a given Project using options to provide both the Project Code {{code}}(-p cli){{/code}} and the Attribute Template Name {{code}}(-n Manifest1){{/code}}, “cli” and “Manifest1” in this case, respectively:
390
391 {{code language="none"}}
392 $ pilotcli file attribute-export -p cli -n Manifest1
393
394 Manifest1
395 --------------------------------------------------------------------------
396 | Attribute Name | Type | Value | Optional |
397 --------------------------------------------------------------------------
398 | attr1 | multiple_choice | a1,a2,a3,a4,a5 | False |
399 --------------------------------------------------------------------------
400 | attr2 | text | None | False |
401 --------------------------------------------------------------------------
402 | attr3 | multiple_choice | t1,t2,t3,t4,t5 | True |
403 --------------------------------------------------------------------------
404 Template saved successfully: cli_Manifest1_template.json
405 Attribute definition saved successfully: cli_Manifest1_definition.json
406 {{/code}}
407
408 == Command: **list** ==
409
410 * **Options**: -z ~-~-zone
411 * **Usage**: List files and folders in the specified directory within the Platform. Folders will be displayed in blue and files will be displayed in white (or black, depending on the background of your terminal). If no zone (i.e. Green Room or Core) is specified, Green Room will be used as default.
412 * (((
413 **Prerequisite**:
414
415 * User login is required, otherwise an error is returned: "The current login session is invalid. Please login to continue."
416 * Project must be accessible to the user, otherwise an error is returned: "Project Code not found in list. Please verify and try again."
417 * Requires the Project code, which can be retrieved via {{code}}pilotcli project list{{/code}}.
418 )))
419
420 {{code language="none"}}
421 $ pilotcli file list --help
422 Usage: pilotcli file list [OPTIONS] PATHS
423
424 List files and folders inside a given Project/folder.
425
426 Options:
427 -z, --zone TEXT Target Zone (i.e., core/greenroom) [default:
428 greenroom] [default: greenroom]
429
430 --page INTEGER The page to be listed [default: 0]
431 --page-size INTEGER number of objects per page [default: 10]
432 -d, --detached whether run in detached mode
433 --help Show this message and exit.
434 {{/code}}
435
436 === Examples ===
437
438 List folders in the directory “admin” in the Project with the Project Code “indoctestproject”. Since no zone was specified, the corresponding directory in the Green Room will be queried:
439
440 {{code language="none"}}
441 $ pilotcli file list indoctestproject/admin
442 test_folder1 test_folder2 file1 file2 file3
443 {{/code}}
444
445 To list the corresponding elements in the “indoctestproject/admin” directory in the Projects' Core, the zone needs to be specified {{code}}(-z core){{/code}}:
446
447 {{code language="none"}}
448 $ pilotcli file list indoctestproject/admin -z core
449 folder1 folder2 test_folder.zip file.txt sample.txt large-file.json
450 {{/code}}
451
452 == Command: **upload** ==
453
454 * **Options**: -p ~-~-project, -a ~-~-attribute, -t ~-~-tag, -z ~-~-zone, -m ~-~-upload-message, -s ~-~-source-file, ~-~-zip, -td ~-~-thread, -o ~-~-output path
455 * **Usage**: Upload files/folders to the given project, with optional tags and attributes, to the target folder.
456 * (((
457 **Prerequisite**:
458
459 * User login is required, otherwise an error is returned: "The current login session is invalid. Please login to continue."
460 * Project must be accessible to the user, otherwise an error is returned: "Project Code not found in list. Please verify and try again."
461 * When uploading file to the Project Core, the user must have access to the Core (Project Contributors cannot upload files/folders to the Project Core).
462 )))
463
464 {{code language="none"}}
465 $ pilotcli file upload --help
466 Usage: pilotcli file upload [OPTIONS] [PATHS]...
467
468 Upload files/folders to a given Project path.
469
470 Options:
471 -p, --project-path TEXT Project folder path starting from Project code.
472 (i.e., indoctestproject/user/folder) [required]
473
474 -a, --attribute TEXT File Attribute Template used for annotating files
475 during upload.
476
477 -t, --tag TEXT Add a tag to the file. This option could be used
478 multiple times for adding multiple tags.
479
480 -z, --zone TEXT Target Zone (i.e., core/greenroom) [default:
481 greenroom] [default: greenroom]
482
483 -m, --upload-message TEXT The message used to comment on the purpose of
484 uploading your processed file [default: ]
485
486 -s, --source-file TEXT The Project path of the source file of your
487 processed files.
488
489 --zip Upload folder as a compressed zip file.
490 -td, --thread INTEGER The number of thread for upload a file [default:
491 1]
492
493 -o, --output-path TEXT The output path for the manifest file of
494 resumable upload [default: ./manifest.json]
495
496 --help Show this message and exit.
497 {{/code}}
498
499 === Examples ===
500
501 Upload a file (or folder) to a specified directory of a Project’s Green Room, in this case the file located at {{code}}./sample_folder/sample_file1{{/code}} will be uploaded to the folder “admin” in the Project with the Project Code “cli” {{code}}(-p cli/admin){{/code}}:
502
503 {{code language="none"}}
504 $ pilotcli file upload ./sample_folder/sample_file1 -p cli/admin
505 Starting upload of: ./sample_folder/sample_file1
506 Pre-upload complete.
507 uploader: admin container_id: 18061 total_size: 1048576 total_chunks: 1 uniq_identifier: e958247c-ce05-48b2-9038-78b774836256-1627498771 resumable_relative_path admin/
508
509 Upload job is finalizing, please wait...
510 Upload job complete.
511 All files uploaded successfully.
512 {{/code}}
513
514 In case the specified destination directory does not exist yet, you will be prompted to confirm its creation by entering “y”, or to abort the command by entering “N”:
515
516 {{code language="none"}}
517 Target folder does not exist. Would you like to create a new folder? [y/N]: y
518 {{/code}}
519
520 In order to upload a file (or folder) to a destination directory in a Project's Core, core must be specified as corresponding zone {{code}}(-z core){{/code}} and you will be required to provide a message to that should describe the files` content and why you are uploading it directly into the Core {{code}}(-m "sample message"){{/code}}. Moreover, you will be reminded about the Terms of Use of the Platform and additional considerations regarding the direct upload of data into the Core zone, and are prompted to confirm your intention to proceed with the upload by entering “y” or “Yes”:
521
522 {{code language="none"}}
523 $ pilotcli file upload ./sample_folder/sample_file2 -p cli/admin -z core -m "sample message"
524 You are about to transfer data directly to the Core! In accordance with the Terms of Use, please confirm that you
525 have made your best efforts to pseudonymize or anonymize the data and that you have the legal authority to transfer and make this
526 data available for dissemination and use within the Platform.To review the Terms of Use, please the Portal. If you
527 need to process the data to remove sensitive identifiers, please cancel this transfer and upload the data to the Green Room to
528 perform these actions.
529 To cancel this transfer, enter [n/No]
530 To confirm and proceed with the data transfer, enter [y/Yes]
531 [y/N]: y
532 Starting upload of: ./sample_folder/sample_file2
533 Pre-upload complete.
534 uploader: admin container_id: 18061 total_size: 1048576 total_chunks: 1 uniq_identifier: fa807a0f-ceb0-4fff-9c9e-563bdcbb309f-1627498851 resumable_relative_path admin/
535
536 Upload job is finalizing, please wait...
537 Upload job complete.
538 All files uploaded successfully.
539 {{/code}}
540
541 Upload a folder to Project Green Room
542
543 {{code language="none"}}
544 $ pilotcli file upload ./sample_folder/example_folder -p cli/admin
545 Starting upload of: ./sample_folder/example_folder
546 Current version does not support folder tagging, any selected tags will be ignored
547 Pre-upload complete.
548 uploader: admin container_id: 18061 total_size: 1048576 total_chunks: 1 uniq_identifier: 4f0eee2e-70f4-4b03-9e72-040544ec7d3a-1627498907 resumable_relative_path admin/example_folder/
549
550 Starting finalization...
551 uploader: admin container_id: 18061 total_size: 1048576 total_chunks: 1 uniq_identifier: 41ec70e2-0859-4aba-9009-06e700c7c79e-1627498907 resumable_relative_path admin/example_folder/
552
553 Starting finalization...
554 uploader: admin container_id: 18061 total_size: 1048576 total_chunks: 1 uniq_identifier: 28454c51-ec9a-4dc5-ab3c-0f7e80e3050d-1627498907 resumable_relative_path admin/example_folder/
555
556 Upload job is finalizing, please wait...
557 Upload job complete.
558 All files uploaded successfully.
559 {{/code}}
560
561 Upload file to existing folder in Green Room
562
563 * Specify a target folder to upload files by inputting folder name after the Project code separated by the '/'.
564
565 {{code language="none"}}
566 $ pilotcli file upload ./sample_folder/sample_file5 -p cli/admin/example_folder
567 Starting upload of: ./sample_folder/sample_file5
568 Pre-upload complete.
569 uploader: admin container_id: 18061 total_size: 1048576 total_chunks: 1 uniq_identifier: 7ef58005-d408-4ad9-a741-fa938f58e67f-1627498977 resumable_relative_path admin/example_folder/
570
571 Upload job is finalizing, please wait...
572 Upload job complete.
573 All files uploaded successfully.
574 {{/code}}
575
576 Upload folder as zip to a specified directory of a Project's Green Room by adding the ~-~-zip option.
577
578 * The specified folder will then be compressed as a zip file and subsequently be uploaded to the specified destination. Once the upload is completed, the compressed zip file will be removed from the directory. Note: pilotcli must have the permission to create and delete zip files in the corresponding directory of your local machine:
579
580 {{code language="none"}}
581 $ pilotcli file upload ./sample_folder/example_folder -p cli/admin --zip
582 Starting upload of: ./sample_folder/example_folder
583 Started zipping files.
584 Pre-upload complete.
585 uploader: admin container_id: 18061 total_size: 3595 total_chunks: 1 uniq_identifier: 61a729ce-cdd6-4a87-82b5-3f8ea63e6c4a-1627499012 resumable_relative_path admin/
586
587 Upload job is finalizing, please wait...
588 Upload job complete.
589 All files uploaded successfully.
590 {{/code}}
591
592 Upload a file with tags and/or attributes.
593
594 * When entering the upload command, the user can add a maximum of ten tags to the file. For example, in order to add “sample_tag1” and “sample_tag2” to your file, simply add {{code}}-t sample_tag1{{/code}} {{code}}-t sample_tag2{{/code}} to the command (see example below using {{code}}tag1{{/code}} and {{code}}tag2{{/code}} as tags).
595 * In addition, you can also use attribute templates for a more enriched annotation of your files. Please note, that the corresponding attribute template file needs to be edited & saved prior to uploading. To get the blank template, see the **attribute-export** instructions above for more details. To attach a attribute JSON-file to your file upon uploading, use {{code}}-a cli_Manifest1_template.json{{/code}}, where “{{code}}cli_Manifest1_template.json{{/code}}" points to the location of the attributes JSON-file on your local machine.
596 * Upon uploading, pilotcli will also validate the content of the attached attribute JSON-file. Whenever a template is invalid, including missing non-optional fields, non-existing choice in multiple choice attribute, exceeding length text field and so on, pilotcli will raise error to user and user must update the attribute template and upload again:
597
598 {{code language="none"}}
599 $ pilotcli file upload ./sample_folder/sample_file6 -p cli/admin -t tag1 -t tag2 -a cli_Manifest1_template.json
600 File attribute validation passed: True
601 Starting upload of: ./sample_folder/sample_file6
602 Pre-upload complete.
603 uploader: admin container_id: 18061 total_size: 1048576 total_chunks: 1 uniq_identifier: bc47ec2a-1a4e-4f71-a37a-7ce6473e415a-1627499273 resumable_relative_path admin/
604
605 Upload job is finalizing, please wait...
606 Upload job complete.
607 Attribute attached
608 All files uploaded successfully.
609 {{/code}}
610
611 Upload a file with specified number of threads, using the {{code}}-td <thread_number>{{/code}} option.
612
613 * By default, the number of threads is 1. There will no UI difference when using threads. The maximal number of threads depends on your local machine and the recommend thread number is 3:
614
615 {{code language="none"}}
616 $ pilotcli file upload -p indoctestproject/admin/test20230306_3 /home/color/indoc/script/large_folder_test_data/gentoo_root_100M.img -td 3
617 Target folder does not exist. Would you like to create a new folder? [y/N]: y
618 Starting upload of: /home/color/indoc/script/large_folder_test_data/gentoo_root_100M.img
619 Pre-upload complete.
620 Uploading gentoo_root_100M.img , resumable_id: c68752372b42c85d, job_id: 56de3632-58df-445c-bcb9-bddfe6be5bf7: |██████████████████████████████ 100% 00:00
621 Upload Time: 79.77s for 1 files
622 All uploading jobs have finished.
623 {{/code}}
624
625 Upload with customized manifest name
626
627 When you upload a file using pilotcli, a JSON file with the details of your specific upload job will be created automatically and allows you to resume the upload at a later time, in case the process is aborted due to any reason. By default, this JSON file will be created as “manifest.json” in the current working directory (see //Command: **resume**// below for more details on how to resume the file upload). However, you can use the {{code}}--output-path{{/code}} option to specify the filepath and, thus, the filename of this JSON file:
628
629 {{code language="none"}}
630 $ pilotcli file upload -p <path> <file> --output-path test_manifest.json
631 {{/code}}
632
633 == Command: **resume** ==
634
635 * **Options**: -td ~-~-thread, -r ~-~-resumable-manifest
636 * **Usage**: Resume unfinished file uploading based on the manifest file.
637 * (((
638 **Prerequisite**:
639
640 * User login is required, otherwise an error is returned: "The current login session is invalid. Please login to continue."
641 * Project must be accessible to the user, otherwise an error is returned: "Project Code not found in list. Please verify and try again."
642 * The resumable manifest JSON file is required, which is generated upon executing the {{code}}pilotcli file upload{{/code}} command.
643 )))
644
645 === Example: ===
646
647 For instance upon disconnecting from the internet, or by accidentally closing the terminal, a currently running uploading process can be aborted. You can use the {{code}}pilotcli file resume{{/code}} command in order to resume the uploading process right where you left off. Simply point to the manifest JSON file that was created upon executing the original upload command. Please note that the source files still have to be in the same location, otherwise resuming the uploading process will fail.
648
649 {{code language="none"}}
650 $ pilotcli file resume -r ./manifest.json
651 Resume the upload for 10 files.
652 Resumable upload check complete.
653 Uploading aggregated.py: |██████████████████████████████ 100% 00:00
654 Uploading __init__.py: |██████████████████████████████ 100% 00:00
655 Uploading service_meta_class.py: |██████████████████████████████ 100% 00:00
656 Uploading upload_form.py: |██████████████████████████████ 100% 00:00
657 Uploading convert_type.py: |██████████████████████████████ 100% 00:00
658 Uploading singleton.py: |██████████████████████████████ 100% 00:00
659 Uploading __init__.py: |██████████████████████████████ 100% 00:00
660 Uploading base_library.zip: |██████████████████████████████ 100% 00:00 0% ?
661 Uploading EXE-00.toc: |██████████████████████████████ 100% 00:00
662 Uploading xref-pilotcli_test20230227_3.html: |██████████████████████████████ 100% 00:00
663 Upload Time: 11.13s for 10 files
664 {{/code}}
665
666 == Command: **sync** ==
667
668 * **Usage**: Download a file or a folder from a Project. If pilotcli is used outside of the Platform, e.g. the users local machine, users can only download from Project’s Core. When pilotcli is used within the Platform, e.g. from one of the workspace tools, users will also be able to download from the Green Room of a Project as well.
669 * (((
670 **Options**: -z ~-~-zone, -i ~-~-geid, ~-~-zip
671
672 * To look up a file’s geid, or unique file ID, open the Project File Explorer in the Portal and inspect the File Properties for the ID.
673 )))
674 * (((
675 **Prerequisite**:
676
677 * User login is required, otherwise an error is returned: "The current login session is invalid. Please login to continue."
678 * The specified file or folder must exist, otherwise an error is returned: “File does not exist or source file provided is invalid: folder_name/file_name.”
679 * If the user has no access to the specified file or folder on the Portal, an error is returned: “Permission denied. Please verify that your role in the Project has permission to perform this action.”
680 )))
681
682 {{code language="none"}}
683 $ pilotcli file sync --help
684 Usage: pilotcli file sync [OPTIONS] [PATHS]... OUTPUT_PATH
685
686 Download files/folders from a given Project/folder/file in core zone.
687
688 Options:
689 -z, --zone TEXT Target Zone (i.e., core/greenroom)
690 --zip Download files as a zip.
691 -i, --geid Enable downloading by geid.
692 --help Show this message and exit.
693 {{/code}}
694
695 === Examples ===
696
697 Download a file or a folder from the Core zone of a Project:
698
699 {{code language="none"}}
700 $ pilotcli file sync cli/admin/sample_file2 -z core ~/Downloads/
701 Preparing status: READY_FOR_DOWNLOADING
702 start downloading...
703 Downloading sample_file2 |██████████████████████████████ 100% 00:00
704 File has been downloaded successfully and saved to: /home/uname/Downloads/sample_file2
705 {{/code}}
706
707 Download multiple files or folders as a compressed zip by simply providing all source files or folders and adding the {{code}}--zip{{/code}} option:
708
709 {{code language="none"}}
710 $ pilotcli file sync cli/admin/sample_file2 cli/admin/sample_file7 -z core ~/Downloads/ --zip
711 Preparing downloading
712 Preparing status: READY_FOR_DOWNLOADING
713 start downloading...
714 Downloading cli_1632340658.768384.zip |██████████████████████████████ 100% 00:00
715 File has been downloaded successfully and saved to: /home/uname/Downloads/cli_1632340658.768384.zip
716 {{/code}}
717
718 Download files or folders by their geid, instead of their location. To enable this, add {{code}}-i{{/code}} or {{code}}--geid{{/code}} to the command:
719
720 {{code language="none"}}
721 $ pilotcli file sync -i 6d89afd4-f86e-4df7-b6c5-01a9af4e828e-1627499957 7e8e2bac-97c6-4738-bc08-12139c9ba225-1627498854 ~/Downloads/
722 Preparing status: READY_FOR_DOWNLOADING
723 start downloading...
724 Downloading sample_file7 |██████████████████████████████ 100% 00:00
725 File has been downloaded successfully and saved to: /home/uname/Downloads/sample_file7
726 Preparing status: READY_FOR_DOWNLOADING
727 start downloading...
728 Downloading sample_file2 |██████████████████████████████ 100% 00:00
729 File has been downloaded successfully and saved to: /home/uname/Downloads/sample_file2
730 {{/code}}
731
732 Download a file or folder from the Green Room of a Project. This will only be possible if pilotcli is used within the Platform, e.g. in one of the Workspace tools. It works the same way as downloading from Core, including all the additional options like zipping, by either adding {{code}}-z greenroom{{/code}} or by omitting the {{code}}-z{{/code}} option entirely, as “greenroom” is the default value:
733
734 {{code language="none"}}
735 $ pilotcli file sync cli/admin/sample_file2 cli/admin/sample_file7 ~/Downloads
736 Preparing status: READY_FOR_DOWNLOADING
737 /home/uname/Downloads/sample_file2 already exist, file will be saved as /home/uname/Downloads/sample_file2 (1)
738 start downloading...
739 Downloading sample_file2 (1) |██████████████████████████████ 100% 00:00
740 File has been downloaded successfully and saved to: /home/uname/Downloads/sample_file2 (1)
741 Preparing status: READY_FOR_DOWNLOADING
742 start downloading...
743 Downloading sample_file7 |██████████████████████████████ 100% 00:00
744 File has been downloaded successfully and saved to: /home/uname/Downloads/sample_file7
745 {{/code}}
746
747 = Dataset Commands: pilotcli **dataset** =
748
749 == Help ==
750
751 {{code language="none"}}
752 $ pilotcli dataset --help
753 Usage: pilotcli dataset [OPTIONS] COMMAND [ARGS]...
754
755 Options:
756 --help Show this message and exit.
757
758 Commands:
759 download Download a dataset or a particular version of a dataset.
760 list List datasets belonging to logged in user.
761 show-detail Show details of a dataset.
762 {{/code}}
763
764 == Command: **list** ==
765
766 * **Usage**: List all datasets for the logged in user
767 * (((
768 **Prerequisite**:
769
770 * User login is required, otherwise an error is returned: "The current login session is invalid. Please login to continue."
771 )))
772
773 {{code language="none"}}
774 $ pilotcli dataset list --help
775 Usage: pilotcli dataset list [OPTIONS]
776
777 List datasets belonging to logged in user.
778
779 Options:
780 --page INTEGER The page to be listed [default: 0]
781 --page-size INTEGER number of objects per page [default: 10]
782 -d, --detached whether run in detached mode
783 --help Show this message and exit.
784 {{/code}}
785
786 === Examples ===
787
788 List all datasets that are accessible for the user:
789
790 {{code language="none"}}
791 $ pilotcli dataset list
792 Dataset Title Dataset Code
793 ---------------------------------------------------------------------------
794 dataset2 aug24202102
795 dataset-01 aug242021
796 Page: 0, Number of datasets: 2
797 {{/code}}
798
799 == Command: **show-detail** ==
800
801 * **Usage**: Show the detailed information of a dataset.
802 * (((
803 **Prerequisite**:
804
805 * User login is required, otherwise an error is returned: "The current login session is invalid. Please login to continue."
806 )))
807
808 {{code language="none"}}
809 $ pilotcli dataset show-detail --help
810 Usage: pilotcli dataset show-detail [OPTIONS] CODE
811
812 Show details of a dataset.
813
814 Options:
815 --page INTEGER The page to be listed [default: 0]
816 --page-size INTEGER number of objects per page [default: 10]
817 -d, --detached whether run in detached mode
818 --help Show this message and exit.
819 {{/code}}
820
821 === Examples ===
822
823 Show the details of the specified Dataset by providing the unique Dataset Code - “aug242021” in this case:
824
825 {{code language="none"}}
826 $ pilotcli dataset show-detail aug242021
827 --------------------------------------------------------------------------------
828 | Title | dataset-01 |
829 --------------------------------------------------------------------------------
830 | Code | aug242021 |
831 --------------------------------------------------------------------------------
832 | Authors | uname, cli |
833 --------------------------------------------------------------------------------
834 | Type | GENERAL |
835 --------------------------------------------------------------------------------
836 | Modality | neuroimaging, microscopy, histological approach |
837 --------------------------------------------------------------------------------
838 | Collection_method | import, test, upload |
839 --------------------------------------------------------------------------------
840 | Tags | tag1, tag2, tag3 |
841 --------------------------------------------------------------------------------
842 | Versions | 1.0, 1.1, 1.2 |
843 --------------------------------------------------------------------------------
844 {{/code}}
845
846 == Command: **download** ==
847
848 * **Usage**: Download a particular version of a dataset.
849 * **Options**: -v ~-~-version
850 * (((
851 **Prerequisite**:
852
853 * User login is required, otherwise error is returned: "The current login session is invalid. Please login to continue."
854 )))
855
856 {{code language="none"}}
857 $ pilotcli dataset download --help
858 Usage: pilotcli dataset download [OPTIONS] [CODE]... OUTPUT_PATH
859
860 Download a dataset or a particular version of a dataset.
861
862 Options:
863 -v, --version TEXT Download a particular version of a dataset.
864 --help Show this message and exit.
865 {{/code}}
866
867
868 === Examples ===
869
870 Download a Dataset with Code “aug242021”:
871
872 {{code language="none"}}
873 $ pilotcli dataset download aug242021 ~/Downloads/
874 Pre downloading dataset
875 Preparing status: READY_FOR_DOWNLOADING
876 start downloading...
877 Downloading aug242021_1632342868.882398.zip |██████████████████████████████ 100% 00:00
878 File has been downloaded successfully and saved to: /home/uname/Downloads/aug242021_163234
879 2868.882398.zip
880 {{/code}}
881
882 Download the version 1.1 of a Dataset with Code “aug242021”:
883
884 {{code language="none"}}
885 $ pilotcli dataset download aug242021 ~/Downloads/ -v 1.1
886 Current dataset version: 1.1
887 Pre downloading dataset
888 start downloading...
889 Downloading aug242021_2021-08-24 11:10:00.042277.zip |██████████████████████████████ 100% 00:00
890 File has been downloaded successfully and saved to: /home/uname/Downloads/aug242021_2021-08-24 11:10:00.042277.zip
891 {{/code}}
892
893
894
895 ----
896
897 Copyright © 2023 [[Indoc Systems>>url:https://www.indocsystems.com]].
898
899 HealthDataCloud is powered by Pilot technology, a product of [[Indoc Systems>>url:https://www.indocsystems.com]].