Last modified by Dennis Segebarth on 2024/10/02 18:38
Show last authors
1 {{box cssClass="floatinginfobox" title="Table of Contents"}}
2 {{toc depth="2"/}}
3 {{/box}}
4
5 The Pilot Command Line Interface (pilotcli) is a binary executable program that provides a convenient way for users to perform file actions and platform-related tasks programmatically. The Pilot Command Line Interface can be downloaded and run on on your local Linux and Linux subsystems (as Windows user, see for instance the [[Windows Subsystem for Linux installation guide>>url:https://learn.microsoft.com/en-us/windows/wsl/install]]) to support large file uploads. For convenience, pilotcli is also pre-deployed and installed in the users JupyterHub accessible via worskpace tools.
6
7 = Data Stewardship =
8
9 Users are reminded to abide by the Platform Terms of Use and any Project-specific restrictions when using the Command Line Interface to access data and code.
10
11 {{warning}}
12 ⚠ If you are using pilotcli in an environment or system where other users can access and read all your files, please make sure to always run {{code language="none"}}pilotcli user logout{{/code}} as soon as you are done using pilotcli. This could be, for instance, if you are using pilotcli on a shared computer in your local institute, with system administrators of your University or Hospital having superuser privileges. You should not use pilotcli in an environment where users with such privileges are from an entity that is not trusted.
13 {{/warning}}
14
15 = Using the Pilot Command Line Interface in JupyterHub =
16
17 To get started with the Pilot Command Line Interface in the HDC JupyterHub workspace tool, see the article //Using JupyterHub in HDC //and refer to the remaining sections in this article for a full description of the pilotcli commands.
18
19 = Using the Pilot Command Line Interface in Guacamole VMs =
20
21 To get started with the Pilot Command Line Interface in the HDC Guacamole workspace tool, see //Using Guacamole in HDC// on how to launch a terminal, either in a Desktop VM or directly by establishing a connection to a Command Line VM. The Pilot Command Line Interface will be installed, configured, and ready to be used. Please refer to the remaining sections in this article for a full description of the pilotcli commands.
22
23 = Downloading the Pilot Command Line Interface =
24
25 If you wish to use the Pilot Command Line Interface to upload large files from your local environment, it is necessary to download Pilot Command Line Interface tool on your Linux system. The recommended Linux version is Ubuntu 20.04.
26
27 * Download the pilotcli tool [[here>>url:https://object.hdc.humanbrainproject.eu/public-resources/pilotcli]].
28
29 After download, please grant execution permission to the binary file so that Pilot Command Line Interface can be used. To grant permission to the binary file, use the following command in the downloaded pilotcli directory.
30
31 {{code language="none"}}
32 $ chmod +x ./pilotcli
33 {{/code}}
34
35
36 If you have sudo permission, you can make the Pilot Command Line Interface more accessible in any directory by moving it to the /usr/local/bin directory:
37
38 {{code language="none"}}
39 $ mv ./pilotcli /usr/local/bin
40 {{/code}}
41
42 Without sudo permission, you can still use the Pilot Command Line Interface in a different directory by referring the Command Line Interface path. For example, use a command line Interface in the Downloads folder:
43
44 {{code language="none"}}
45 $ ~/Downloads/pilotcli
46 {{/code}}
47
48 = Pilot Command Line Interface Command Summary =
49
50 Commands are organized into main groups:
51
52 1. **User**// - //User related actions such as login and logout
53 1. **Project** //- //Project related actions, including list Project Names and Codes for all Projects accessible by the logged-in user.
54 1. **File **- Actions on files. User could use file action commands to list files/folders, upload files/folders, list/export attribute, and download files as well.
55 1. **Dataset **- Actions related to datasets.
56
57 = User Commands: pilotcli **user** =
58
59 == Help ==
60
61 {{code language="none"}}
62 $ pilotcli user --help
63 Usage: pilotcli user [OPTIONS] COMMAND [ARGS]...
64
65 Options:
66 --help Show this message and exit.
67
68 Commands:
69 login For user to login.
70 logout For user to logout.
71 {{/code}}
72
73 == Command: **login** ==
74
75 * **Options**: -u ~-~-username, -p ~-~-password
76 * **Usage**: Allows users to log into the platform through pilotcli. After login, the user must be actively using the Pilot Command Line Interface to remain logged in, or they will be logged out automatically after the session is idle for five minutes.
77 * **Prerequisites**: None.
78
79 {{code language="none"}}
80 $ pilotcli user login --help
81 Usage: pilotcli user login [OPTIONS]
82
83 For user to login.
84
85 Options:
86 --help Show this message and exit.
87 {{/code}}
88
89 === Example ===
90
91 User login:
92
93 1) Use the output device validation link or QR code to launch authentication validation:
94
95 {{code language="none"}}
96 $ pilotcli user login
97 Please, access https://iam.hdc.humanbrainproject.eu/realms/hdc/device?user_code=XXXX-XXXX to proceed
98 ▄▄▄▄▄▄▄ ▄▄ ▄ ▄ ▄ ▄▄▄▄ ▄ ▄▄▄▄▄▄▄
99 █ ▄▄▄ █ ▄ ▄▄▀█ █▀▀▀▄▀█▀ ▀██▄ █ ▄▄▄ █
100 █ ███ █ ██▄ ▀▀▄▀▄▄█▄ ▄ ▀▀▀▄ █ ███ █
101 █▄▄▄▄▄█ █▀▄▀█▀▄▀█▀▄ █ ▄ ▄ █▀█ █▄▄▄▄▄█
102 ▄▄▄▄▄▄▄ ▀█▀█▄▄█▄ █▀█ ▄▀▄▄ █ ▄ █▀▄▄▄
103 █ ▄▄▄ █ █▄█▄██ ▀ ▀█▀▀█ ▀▄█▄█▄▄▄█▀▄█
104 █ ███ █ █▀█ ▄▀██▄█▀▀█▄▀▀█▄▀█ ▀ ▀▄▀██
105 █▄▄▄▄▄█ ▄▀▄ ██▄▀▀▄ ▀▀▄ ▄▄▀▀▀▄ █▄▄▄█
106
107
108 Waiting validation finish...
109 {{/code}}
110
111 2) Grant access by clicking Yes:
112
113 [[image:1723802541971-790.png||height="314" width="368"]]
114
115 3) Device Login Successful message will appear and your authentication browser window can be closed.
116
117 4) The output of the Command Line Tool will also be updated to notify you about the successful login:
118
119 {{code language="none"}}
120 $ pilotcli user login
121 Please, access https://iam.hdc.humanbrainproject.eu/realms/hdc/device?user_code=XXXX-XXXX to proceed
122 ▄▄▄▄▄▄▄ ▄▄ ▄ ▄ ▄ ▄▄▄▄ ▄ ▄▄▄▄▄▄▄
123 █ ▄▄▄ █ ▄ ▄▄▀█ █▀▀▀▄▀█▀ ▀██▄ █ ▄▄▄ █
124 █ ███ █ ██▄ ▀▀▄▀▄▄█▄ ▄ ▀▀▀▄ █ ███ █
125 █▄▄▄▄▄█ █▀▄▀█▀▄▀█▀▄ █ ▄ ▄ █▀█ █▄▄▄▄▄█
126 ▄▄▄▄▄▄▄ ▀█▀█▄▄█▄ █▀█ ▄▀▄▄ █ ▄ █▀▄▄▄
127 █ ▄▄▄ █ █▄█▄██ ▀ ▀█▀▀█ ▀▄█▄█▄▄▄█▀▄█
128 █ ███ █ █▀█ ▄▀██▄█▀▀█▄▀▀█▄▀█ ▀ ▀▄▀██
129 █▄▄▄▄▄█ ▄▀▄ ██▄▀▀▄ ▀▀▄ ▄▄▀▀▀▄ █▄▄▄█
130
131
132 Waiting validation finish...
133 Welcome to Command Line Tool!
134 {{/code}}
135
136 == Command: **logout** ==
137
138 * **Options**: -y ~-~-yes
139 * **Usage**: Allows users to log out of the platform through pilotcli.
140 * **Prerequisites**: User login is required, otherwise an error is returned: ‘The current login session is invalid. Please login to continue.'.
141
142 {{code language="none"}}
143 $ pilotcli user logout --help
144 Usage: pilotcli user logout [OPTIONS]
145
146 For user to logout.
147
148 Options:
149 -y, --yes Input Y/yes to confirm you want to logout, otherwise input N/no
150 to remain logged in.
151
152 --help Show this message and exit.
153 {{/code}}
154
155 === Examples ===
156
157 User logout:
158
159 {{code language="none"}}
160 $ pilotcli user logout
161 Are you sure you want to logout? [y/N]: y
162 Logged out successfully. Bye!
163 {{/code}}
164
165 User logout canceled:
166
167 {{code language="none"}}
168 $ pilotcli user logout
169 Are you sure you want to logout? [y/N]: n
170 Logout canceled!
171 {{/code}}
172
173 User logout with option to skip additional confirmation:
174
175 {{code language="none"}}
176 $ pilotcli user logout -y
177 Logged out successfully. Bye!
178 {{/code}}
179
180 = Project Commands: pilotcli **project** =
181
182 == Help ==
183
184 {{code language="none"}}
185 $ pilotcli project --help
186 Usage: pilotcli project [OPTIONS] COMMAND [ARGS]...
187
188 Options:
189 --help Show this message and exit.
190
191 Commands:
192 list List accessible projects.
193 {{/code}}
194
195 == Command: **list** ==
196
197 * **Options**: None
198 * **Usage**: List all Projects to which the user has access
199 * **Prerequisite**: User login is required, otherwise an error is returned: ‘The current login session is invalid. Please login to continue.'
200
201 {{code language="none"}}
202 $ pilotcli project list --help
203 Usage: pilotcli project list [OPTIONS]
204
205 List accessible projects.
206
207 Options:
208 --page INTEGER The page to be listed [default: 0]
209 --page-size INTEGER number of objects per page [default: 10]
210 --order [asc|desc] sorting order [default: desc]
211 --order-by [created_at|name|code]
212 sorting column [default: created_at]
213 -d, --detached whether run in detached mode
214 --help Show this message and exit.
215 {{/code}}
216
217 === Examples ===
218
219 Get a list of all Projects the user has access to:
220
221 {{code language="none"}}
222 $ pilotcli project list
223 Project Name Project Code
224 ---------------------------------------------------------------------------
225 Test Project 02-13 | test230213byshan
226 Test1_Audit | test1audit
227 Test20221109 | test20220109
228 Indoc Test Project | indoctestproject
229
230 Page: 0, Number of projects: 4
231 {{/code}}
232
233 By default, the table will only display the first ten elements. If the user has access to more than ten Projects, you will be prompted to use your arrow keys to select one of two options to either view the next ten items (//next page//), or to exit (//exit//). The selected option is indicated by two arrows and can be confirmed by hitting enter:
234
235 {{code language="none"}}
236 Page: 0, Number of projects: 10
237
238 What do you want? (Use arrow keys)
239 » next page
240 exit
241 {{/code}}
242
243 = File Commands: pilotcli **file** =
244
245 == Help ==
246
247 {{code language="none"}}
248 $ pilotcli file --help
249 Usage: pilotcli file [OPTIONS] COMMAND [ARGS]...
250
251 Options:
252 --help Show this message and exit.
253
254 Commands:
255 attribute-export Export attribute template from a given Project.
256 attribute-list List attribute templates of a given Project.
257 list List files and folders inside a given Project/folder.
258 resume Resume the upload process with given manifest file.
259 sync Download files/folders from a given Project/folder/file...
260 upload Upload files/folders to a given Project path.
261 {{/code}}
262
263 == Command: **attribute-list** ==
264
265 * **Options**: -p ~-~-project-code
266 * **Usage**: List all attributes from a specified Project.
267 * **Prerequisite**:
268
269 * User login is required, otherwise an error is returned: ‘The current login session is invalid. Please login to continue.’
270 * Requires the Project code, which can be retrieved via {{code}}pilotcli project list{{/code}}.
271
272 {{code language="none"}}
273 $ pilotcli file attribute-list --help
274 Usage: pilotcli file attribute-list [OPTIONS]
275
276 List attribute templates of a given Project.
277
278 Options:
279 -p, --project-code TEXT Project Code
280 --help Show this message and exit.
281 {{/code}}
282
283 === Examples ===
284
285 List attributes from a given Project by providing the Project Code at the prompt. In this example, “cli” is the corresponding Project Code:
286
287 {{code language="none"}}
288 $ pilotcli file attribute-list
289 ProjectCode: cli
290
291 Manifest1
292 --------------------------------------------------------------------------
293 | Attribute Name | Type | Value | Optional |
294 --------------------------------------------------------------------------
295 | attr1 | multiple_choice | a1,a2,a3,a4,a5 | False |
296 --------------------------------------------------------------------------
297 | attr2 | text | None | False |
298 --------------------------------------------------------------------------
299 | attr3 | multiple_choice | t1,t2,t3,t4,t5 | True |
300 --------------------------------------------------------------------------
301
302 Manifest2
303 --------------------------------------------------------------------------
304 | Attribute Name | Type | Value | Optional |
305 --------------------------------------------------------------------------
306 | attr1 | multiple_choice | a1,a2,a3,a4 | True |
307 --------------------------------------------------------------------------
308 | attr2 | text | None | False |
309 --------------------------------------------------------------------------
310 | attr3 | multiple_choice | t1,t2,t3 | True |
311 --------------------------------------------------------------------------
312 All Attributes fetched successfully.
313 {{/code}}
314
315 List attributes from given a Project and provide the Project Code “cli” as an option in the same line {{code}}-p cli{{/code}}:
316
317 {{code language="none"}}
318 $ pilotcli file attribute-list -p cli
319
320 Manifest1
321 --------------------------------------------------------------------------
322 | Attribute Name | Type | Value | Optional |
323 --------------------------------------------------------------------------
324 | attr1 | multiple_choice | a1,a2,a3,a4,a5 | False |
325 --------------------------------------------------------------------------
326 | attr2 | text | None | False |
327 --------------------------------------------------------------------------
328 | attr3 | multiple_choice | t1,t2,t3,t4,t5 | True |
329 --------------------------------------------------------------------------
330
331 Manifest2
332 --------------------------------------------------------------------------
333 | Attribute Name | Type | Value | Optional |
334 --------------------------------------------------------------------------
335 | attr1 | multiple_choice | a1,a2,a3,a4 | True |
336 --------------------------------------------------------------------------
337 | attr2 | text | None | False |
338 --------------------------------------------------------------------------
339 | attr3 | multiple_choice | t1,t2,t3 | True |
340 --------------------------------------------------------------------------
341 All Attributes fetched successfully.
342 {{/code}}
343
344 == Command: **attribute-export** ==
345
346 * **Options**: -p ~-~-project-code, -n ~-~-attribute-name
347 * **Usage**: Export an attribute template as JSON-file from a given Project
348 * (((
349 **Prerequisite**:
350
351 * User login is required, otherwise an error is returned: "The current login session is invalid. Please login to continue."
352 * Project must be accessible to the user, otherwise an error is returned: "Project Code not found in list. Please verify and try again."
353 * Requires the Project code, which can be retrieved via {{code}}pilotcli project list{{/code}}.
354 * Requires the Attribute template name. If the attribute template name does not exist or is spelled incorrectly, an error is returned: "Attribute 'sample-template' not found in Project. Please verify and try again."
355 * pilotcli must have the permission to write a file in the directory, otherwise an error is returned.
356 )))
357
358 {{code language="none"}}
359 $ pilotcli file attribute-export --help
360 Usage: pilotcli file attribute-export [OPTIONS]
361
362 Export attribute template from a given Project.
363
364 Options:
365 -p, --project-code TEXT Project Code
366 -n, --attribute-name TEXT Attribute Template Name
367 --help Show this message and exit.
368 {{/code}}
369
370 === Examples ===
371
372 Export an attribute template from a given Project. The user is prompted to provide the Project Code (“cli”) and the Attribute Template Name (“Manifest1”):
373
374 {{code language="none"}}
375 $ pilotcli file attribute-export
376 ProjectCode: cli
377 AttributeName: Manifest1
378
379 Manifest1
380 --------------------------------------------------------------------------
381 | Attribute Name | Type | Value | Optional |
382 --------------------------------------------------------------------------
383 | attr1 | multiple_choice | a1,a2,a3,a4,a5 | False |
384 --------------------------------------------------------------------------
385 | attr2 | text | None | False |
386 --------------------------------------------------------------------------
387 | attr3 | multiple_choice | t1,t2,t3,t4,t5 | True |
388 --------------------------------------------------------------------------
389 Template saved successfully: cli_Manifest1_template.json
390 Attribute definition saved successfully: cli_Manifest1_definition.json
391 {{/code}}
392
393 Export an attribute template from a given Project using options to provide both the Project Code {{code}}(-p cli){{/code}} and the Attribute Template Name {{code}}(-n Manifest1){{/code}}, “cli” and “Manifest1” in this case, respectively:
394
395 {{code language="none"}}
396 $ pilotcli file attribute-export -p cli -n Manifest1
397
398 Manifest1
399 --------------------------------------------------------------------------
400 | Attribute Name | Type | Value | Optional |
401 --------------------------------------------------------------------------
402 | attr1 | multiple_choice | a1,a2,a3,a4,a5 | False |
403 --------------------------------------------------------------------------
404 | attr2 | text | None | False |
405 --------------------------------------------------------------------------
406 | attr3 | multiple_choice | t1,t2,t3,t4,t5 | True |
407 --------------------------------------------------------------------------
408 Template saved successfully: cli_Manifest1_template.json
409 Attribute definition saved successfully: cli_Manifest1_definition.json
410 {{/code}}
411
412 == Command: **list** ==
413
414 * **Options**: -z ~-~-zone
415 * **Usage**: List files and folders in the specified directory within the Platform. Folders will be displayed in blue and files will be displayed in white (or black, depending on the background of your terminal). If no zone (i.e. Green Room or Core) is specified, Green Room will be used as default.
416 * (((
417 **Prerequisite**:
418
419 * User login is required, otherwise an error is returned: "The current login session is invalid. Please login to continue."
420 * Project must be accessible to the user, otherwise an error is returned: "Project Code not found in list. Please verify and try again."
421 * Requires the Project code, which can be retrieved via {{code}}pilotcli project list{{/code}}.
422 )))
423
424 {{code language="none"}}
425 $ pilotcli file list --help
426 Usage: pilotcli file list [OPTIONS] PATHS
427
428 List files and folders inside a given Project/folder.
429
430 Options:
431 -z, --zone TEXT Target Zone (i.e., core/greenroom) [default:
432 greenroom] [default: greenroom]
433
434 --page INTEGER The page to be listed [default: 0]
435 --page-size INTEGER number of objects per page [default: 10]
436 -d, --detached whether run in detached mode
437 --help Show this message and exit.
438 {{/code}}
439
440 === Examples ===
441
442 List folders in the directory “admin” in the Project with the Project Code “indoctestproject”. Since no zone was specified, the corresponding directory in the Green Room will be queried:
443
444 {{code language="none"}}
445 $ pilotcli file list indoctestproject/admin
446 test_folder1 test_folder2 file1 file2 file3
447 {{/code}}
448
449 To list the corresponding elements in the “indoctestproject/admin” directory in the Projects' Core, the zone needs to be specified {{code}}(-z core){{/code}}:
450
451 {{code language="none"}}
452 $ pilotcli file list indoctestproject/admin -z core
453 folder1 folder2 test_folder.zip file.txt sample.txt large-file.json
454 {{/code}}
455
456 == Command: **upload** ==
457
458 * **Options**: -p ~-~-project, -a ~-~-attribute, -t ~-~-tag, -z ~-~-zone, -m ~-~-upload-message, -s ~-~-source-file, ~-~-zip, -td ~-~-thread, -o ~-~-output path
459 * **Usage**: Upload files/folders to the given project, with optional tags and attributes, to the target folder.
460 * (((
461 **Prerequisite**:
462
463 * User login is required, otherwise an error is returned: "The current login session is invalid. Please login to continue."
464 * Project must be accessible to the user, otherwise an error is returned: "Project Code not found in list. Please verify and try again."
465 * When uploading file to the Project Core, the user must have access to the Core (Project Contributors cannot upload files/folders to the Project Core).
466 )))
467
468 {{code language="none"}}
469 $ pilotcli file upload --help
470 Usage: pilotcli file upload [OPTIONS] [PATHS]...
471
472 Upload files/folders to a given Project path.
473
474 Options:
475 -p, --project-path TEXT Project folder path starting from Project code.
476 (i.e., indoctestproject/user/folder) [required]
477
478 -a, --attribute TEXT File Attribute Template used for annotating files
479 during upload.
480
481 -t, --tag TEXT Add a tag to the file. This option could be used
482 multiple times for adding multiple tags.
483
484 -z, --zone TEXT Target Zone (i.e., core/greenroom) [default:
485 greenroom] [default: greenroom]
486
487 -m, --upload-message TEXT The message used to comment on the purpose of
488 uploading your processed file [default: ]
489
490 -s, --source-file TEXT The Project path of the source file of your
491 processed files.
492
493 --zip Upload folder as a compressed zip file.
494 -td, --thread INTEGER The number of thread for upload a file [default:
495 1]
496
497 -o, --output-path TEXT The output path for the manifest file of
498 resumable upload [default: ./manifest.json]
499
500 --help Show this message and exit.
501 {{/code}}
502
503 === Examples ===
504
505 Upload a file (or folder) to a specified directory of a Project’s Green Room, in this case the file located at {{code}}./sample_folder/sample_file1{{/code}} will be uploaded to the folder “admin” in the Project with the Project Code “cli” {{code}}(-p cli/admin){{/code}}:
506
507 {{code language="none"}}
508 $ pilotcli file upload ./sample_folder/sample_file1 -p cli/admin
509 Starting upload of: ./sample_folder/sample_file1
510 Pre-upload complete.
511 uploader: admin container_id: 18061 total_size: 1048576 total_chunks: 1 uniq_identifier: e958247c-ce05-48b2-9038-78b774836256-1627498771 resumable_relative_path admin/
512
513 Upload job is finalizing, please wait...
514 Upload job complete.
515 All files uploaded successfully.
516 {{/code}}
517
518 In case the specified destination directory does not exist yet, you will be prompted to confirm its creation by entering “y”, or to abort the command by entering “N”:
519
520 {{code language="none"}}
521 Target folder does not exist. Would you like to create a new folder? [y/N]: y
522 {{/code}}
523
524 In order to upload a file (or folder) to a destination directory in a Project's Core, core must be specified as corresponding zone {{code}}(-z core){{/code}} and you will be required to provide a message to that should describe the files` content and why you are uploading it directly into the Core {{code}}(-m "sample message"){{/code}}. Moreover, you will be reminded about the Terms of Use of the Platform and additional considerations regarding the direct upload of data into the Core zone, and are prompted to confirm your intention to proceed with the upload by entering “y” or “Yes”:
525
526 {{code language="none"}}
527 $ pilotcli file upload ./sample_folder/sample_file2 -p cli/admin -z core -m "sample message"
528 You are about to transfer data directly to the Core! In accordance with the Terms of Use, please confirm that you
529 have made your best efforts to pseudonymize or anonymize the data and that you have the legal authority to transfer and make this
530 data available for dissemination and use within the Platform.To review the Terms of Use, please the Portal. If you
531 need to process the data to remove sensitive identifiers, please cancel this transfer and upload the data to the Green Room to
532 perform these actions.
533 To cancel this transfer, enter [n/No]
534 To confirm and proceed with the data transfer, enter [y/Yes]
535 [y/N]: y
536 Starting upload of: ./sample_folder/sample_file2
537 Pre-upload complete.
538 uploader: admin container_id: 18061 total_size: 1048576 total_chunks: 1 uniq_identifier: fa807a0f-ceb0-4fff-9c9e-563bdcbb309f-1627498851 resumable_relative_path admin/
539
540 Upload job is finalizing, please wait...
541 Upload job complete.
542 All files uploaded successfully.
543 {{/code}}
544
545 Upload a folder to Project Green Room
546
547 {{code language="none"}}
548 $ pilotcli file upload ./sample_folder/example_folder -p cli/admin
549 Starting upload of: ./sample_folder/example_folder
550 Current version does not support folder tagging, any selected tags will be ignored
551 Pre-upload complete.
552 uploader: admin container_id: 18061 total_size: 1048576 total_chunks: 1 uniq_identifier: 4f0eee2e-70f4-4b03-9e72-040544ec7d3a-1627498907 resumable_relative_path admin/example_folder/
553
554 Starting finalization...
555 uploader: admin container_id: 18061 total_size: 1048576 total_chunks: 1 uniq_identifier: 41ec70e2-0859-4aba-9009-06e700c7c79e-1627498907 resumable_relative_path admin/example_folder/
556
557 Starting finalization...
558 uploader: admin container_id: 18061 total_size: 1048576 total_chunks: 1 uniq_identifier: 28454c51-ec9a-4dc5-ab3c-0f7e80e3050d-1627498907 resumable_relative_path admin/example_folder/
559
560 Upload job is finalizing, please wait...
561 Upload job complete.
562 All files uploaded successfully.
563 {{/code}}
564
565 Upload file to existing folder in Green Room
566
567 * Specify a target folder to upload files by inputting folder name after the Project code separated by the '/'.
568
569 {{code language="none"}}
570 $ pilotcli file upload ./sample_folder/sample_file5 -p cli/admin/example_folder
571 Starting upload of: ./sample_folder/sample_file5
572 Pre-upload complete.
573 uploader: admin container_id: 18061 total_size: 1048576 total_chunks: 1 uniq_identifier: 7ef58005-d408-4ad9-a741-fa938f58e67f-1627498977 resumable_relative_path admin/example_folder/
574
575 Upload job is finalizing, please wait...
576 Upload job complete.
577 All files uploaded successfully.
578 {{/code}}
579
580 Upload folder as zip to a specified directory of a Project's Green Room by adding the ~-~-zip option.
581
582 * The specified folder will then be compressed as a zip file and subsequently be uploaded to the specified destination. Once the upload is completed, the compressed zip file will be removed from the directory. Note: pilotcli must have the permission to create and delete zip files in the corresponding directory of your local machine:
583
584 {{code language="none"}}
585 $ pilotcli file upload ./sample_folder/example_folder -p cli/admin --zip
586 Starting upload of: ./sample_folder/example_folder
587 Started zipping files.
588 Pre-upload complete.
589 uploader: admin container_id: 18061 total_size: 3595 total_chunks: 1 uniq_identifier: 61a729ce-cdd6-4a87-82b5-3f8ea63e6c4a-1627499012 resumable_relative_path admin/
590
591 Upload job is finalizing, please wait...
592 Upload job complete.
593 All files uploaded successfully.
594 {{/code}}
595
596 Upload a file with tags and/or attributes.
597
598 * When entering the upload command, the user can add a maximum of ten tags to the file. For example, in order to add “sample_tag1” and “sample_tag2” to your file, simply add {{code}}-t sample_tag1{{/code}} {{code}}-t sample_tag2{{/code}} to the command (see example below using {{code}}tag1{{/code}} and {{code}}tag2{{/code}} as tags).
599 * In addition, you can also use attribute templates for a more enriched annotation of your files. Please note, that the corresponding attribute template file needs to be edited & saved prior to uploading. To get the blank template, see the **attribute-export** instructions above for more details. To attach a attribute JSON-file to your file upon uploading, use {{code}}-a cli_Manifest1_template.json{{/code}}, where “{{code}}cli_Manifest1_template.json{{/code}}" points to the location of the attributes JSON-file on your local machine.
600 * Upon uploading, pilotcli will also validate the content of the attached attribute JSON-file. Whenever a template is invalid, including missing non-optional fields, non-existing choice in multiple choice attribute, exceeding length text field and so on, pilotcli will raise error to user and user must update the attribute template and upload again:
601
602 {{code language="none"}}
603 $ pilotcli file upload ./sample_folder/sample_file6 -p cli/admin -t tag1 -t tag2 -a cli_Manifest1_template.json
604 File attribute validation passed: True
605 Starting upload of: ./sample_folder/sample_file6
606 Pre-upload complete.
607 uploader: admin container_id: 18061 total_size: 1048576 total_chunks: 1 uniq_identifier: bc47ec2a-1a4e-4f71-a37a-7ce6473e415a-1627499273 resumable_relative_path admin/
608
609 Upload job is finalizing, please wait...
610 Upload job complete.
611 Attribute attached
612 All files uploaded successfully.
613 {{/code}}
614
615 Upload a file with specified number of threads, using the {{code}}-td <thread_number>{{/code}} option.
616
617 * By default, the number of threads is 1. There will no UI difference when using threads. The maximal number of threads depends on your local machine and the recommend thread number is 3:
618
619 {{code language="none"}}
620 $ pilotcli file upload -p indoctestproject/admin/test20230306_3 /home/color/indoc/script/large_folder_test_data/gentoo_root_100M.img -td 3
621 Target folder does not exist. Would you like to create a new folder? [y/N]: y
622 Starting upload of: /home/color/indoc/script/large_folder_test_data/gentoo_root_100M.img
623 Pre-upload complete.
624 Uploading gentoo_root_100M.img , resumable_id: c68752372b42c85d, job_id: 56de3632-58df-445c-bcb9-bddfe6be5bf7: |██████████████████████████████ 100% 00:00
625 Upload Time: 79.77s for 1 files
626 All uploading jobs have finished.
627 {{/code}}
628
629 Upload with customized manifest name
630
631 When you upload a file using pilotcli, a JSON file with the details of your specific upload job will be created automatically and allows you to resume the upload at a later time, in case the process is aborted due to any reason. By default, this JSON file will be created as “manifest.json” in the current working directory (see //Command: **resume**// below for more details on how to resume the file upload). However, you can use the {{code}}--output-path{{/code}} option to specify the filepath and, thus, the filename of this JSON file:
632
633 {{code language="none"}}
634 $ pilotcli file upload -p <path> <file> --output-path test_manifest.json
635 {{/code}}
636
637 == Command: **resume** ==
638
639 * **Options**: -td ~-~-thread, -r ~-~-resumable-manifest
640 * **Usage**: Resume unfinished file uploading based on the manifest file.
641 * (((
642 **Prerequisite**:
643
644 * User login is required, otherwise an error is returned: "The current login session is invalid. Please login to continue."
645 * Project must be accessible to the user, otherwise an error is returned: "Project Code not found in list. Please verify and try again."
646 * The resumable manifest JSON file is required, which is generated upon executing the {{code}}pilotcli file upload{{/code}} command.
647 )))
648
649 === Example: ===
650
651 For instance upon disconnecting from the internet, or by accidentally closing the terminal, a currently running uploading process can be aborted. You can use the {{code}}pilotcli file resume{{/code}} command in order to resume the uploading process right where you left off. Simply point to the manifest JSON file that was created upon executing the original upload command. Please note that the source files still have to be in the same location, otherwise resuming the uploading process will fail.
652
653 {{code language="none"}}
654 $ pilotcli file resume -r ./manifest.json
655 Resume the upload for 10 files.
656 Resumable upload check complete.
657 Uploading aggregated.py: |██████████████████████████████ 100% 00:00
658 Uploading __init__.py: |██████████████████████████████ 100% 00:00
659 Uploading service_meta_class.py: |██████████████████████████████ 100% 00:00
660 Uploading upload_form.py: |██████████████████████████████ 100% 00:00
661 Uploading convert_type.py: |██████████████████████████████ 100% 00:00
662 Uploading singleton.py: |██████████████████████████████ 100% 00:00
663 Uploading __init__.py: |██████████████████████████████ 100% 00:00
664 Uploading base_library.zip: |██████████████████████████████ 100% 00:00 0% ?
665 Uploading EXE-00.toc: |██████████████████████████████ 100% 00:00
666 Uploading xref-pilotcli_test20230227_3.html: |██████████████████████████████ 100% 00:00
667 Upload Time: 11.13s for 10 files
668 {{/code}}
669
670 == Command: **sync** ==
671
672 * **Usage**: Download a file or a folder from a Project. If pilotcli is used outside of the Platform, e.g. the users local machine, users can only download from Project’s Core. When pilotcli is used within the Platform, e.g. from one of the workspace tools, users will also be able to download from the Green Room of a Project as well.
673 * (((
674 **Options**: -z ~-~-zone, -i ~-~-geid, ~-~-zip
675
676 * To look up a file’s geid, or unique file ID, open the Project File Explorer in the Portal and inspect the File Properties for the ID.
677 )))
678 * (((
679 **Prerequisite**:
680
681 * User login is required, otherwise an error is returned: "The current login session is invalid. Please login to continue."
682 * The specified file or folder must exist, otherwise an error is returned: “File does not exist or source file provided is invalid: folder_name/file_name.”
683 * If the user has no access to the specified file or folder on the Portal, an error is returned: “Permission denied. Please verify that your role in the Project has permission to perform this action.”
684 )))
685
686 {{code language="none"}}
687 $ pilotcli file sync --help
688 Usage: pilotcli file sync [OPTIONS] [PATHS]... OUTPUT_PATH
689
690 Download files/folders from a given Project/folder/file in core zone.
691
692 Options:
693 -z, --zone TEXT Target Zone (i.e., core/greenroom)
694 --zip Download files as a zip.
695 -i, --geid Enable downloading by geid.
696 --help Show this message and exit.
697 {{/code}}
698
699 === Examples ===
700
701 Download a file or a folder from the Core zone of a Project:
702
703 {{code language="none"}}
704 $ pilotcli file sync cli/admin/sample_file2 -z core ~/Downloads/
705 Preparing status: READY_FOR_DOWNLOADING
706 start downloading...
707 Downloading sample_file2 |██████████████████████████████ 100% 00:00
708 File has been downloaded successfully and saved to: /home/uname/Downloads/sample_file2
709 {{/code}}
710
711 Download multiple files or folders as a compressed zip by simply providing all source files or folders and adding the {{code}}--zip{{/code}} option:
712
713 {{code language="none"}}
714 $ pilotcli file sync cli/admin/sample_file2 cli/admin/sample_file7 -z core ~/Downloads/ --zip
715 Preparing downloading
716 Preparing status: READY_FOR_DOWNLOADING
717 start downloading...
718 Downloading cli_1632340658.768384.zip |██████████████████████████████ 100% 00:00
719 File has been downloaded successfully and saved to: /home/uname/Downloads/cli_1632340658.768384.zip
720 {{/code}}
721
722 Download files or folders by their geid, instead of their location. To enable this, add {{code}}-i{{/code}} or {{code}}--geid{{/code}} to the command:
723
724 {{code language="none"}}
725 $ pilotcli file sync -i 6d89afd4-f86e-4df7-b6c5-01a9af4e828e-1627499957 7e8e2bac-97c6-4738-bc08-12139c9ba225-1627498854 ~/Downloads/
726 Preparing status: READY_FOR_DOWNLOADING
727 start downloading...
728 Downloading sample_file7 |██████████████████████████████ 100% 00:00
729 File has been downloaded successfully and saved to: /home/uname/Downloads/sample_file7
730 Preparing status: READY_FOR_DOWNLOADING
731 start downloading...
732 Downloading sample_file2 |██████████████████████████████ 100% 00:00
733 File has been downloaded successfully and saved to: /home/uname/Downloads/sample_file2
734 {{/code}}
735
736 Download a file or folder from the Green Room of a Project. This will only be possible if pilotcli is used within the Platform, e.g. in one of the Workspace tools. It works the same way as downloading from Core, including all the additional options like zipping, by either adding {{code}}-z greenroom{{/code}} or by omitting the {{code}}-z{{/code}} option entirely, as “greenroom” is the default value:
737
738 {{code language="none"}}
739 $ pilotcli file sync cli/admin/sample_file2 cli/admin/sample_file7 ~/Downloads
740 Preparing status: READY_FOR_DOWNLOADING
741 /home/uname/Downloads/sample_file2 already exist, file will be saved as /home/uname/Downloads/sample_file2 (1)
742 start downloading...
743 Downloading sample_file2 (1) |██████████████████████████████ 100% 00:00
744 File has been downloaded successfully and saved to: /home/uname/Downloads/sample_file2 (1)
745 Preparing status: READY_FOR_DOWNLOADING
746 start downloading...
747 Downloading sample_file7 |██████████████████████████████ 100% 00:00
748 File has been downloaded successfully and saved to: /home/uname/Downloads/sample_file7
749 {{/code}}
750
751 = Dataset Commands: pilotcli **dataset** =
752
753 == Help ==
754
755 {{code language="none"}}
756 $ pilotcli dataset --help
757 Usage: pilotcli dataset [OPTIONS] COMMAND [ARGS]...
758
759 Options:
760 --help Show this message and exit.
761
762 Commands:
763 download Download a dataset or a particular version of a dataset.
764 list List datasets belonging to logged in user.
765 show-detail Show details of a dataset.
766 {{/code}}
767
768 == Command: **list** ==
769
770 * **Usage**: List all datasets for the logged in user
771 * (((
772 **Prerequisite**:
773
774 * User login is required, otherwise an error is returned: "The current login session is invalid. Please login to continue."
775 )))
776
777 {{code language="none"}}
778 $ pilotcli dataset list --help
779 Usage: pilotcli dataset list [OPTIONS]
780
781 List datasets belonging to logged in user.
782
783 Options:
784 --all Show also datasets that belong to projects where user has admin role
785
786 --page INTEGER The page to be listed [default: 0]
787 --page-size INTEGER number of objects per page [default: 10]
788 -d, --detached whether run in detached mode
789 --help Show this message and exit.
790 {{/code}}
791
792 === Examples ===
793
794 List all datasets that are accessible for the user:
795
796 {{code language="none"}}
797 $ pilotcli dataset list
798 Dataset Title Dataset Code
799 ---------------------------------------------------------------------------
800 dataset2 aug24202102
801 dataset-01 aug242021
802 Page: 0, Number of datasets: 2
803 {{/code}}
804
805 If the user has the Project Administrator role in one or more Projects, they can also access all Datasets associated with those Projects, even if they were created by another user. To do so, use the "~-~-all" flag:
806
807 {{code language="none"}}
808 $ pilotcli dataset list --all
809 Dataset Title Dataset Code
810 ---------------------------------------------------------------------------
811 dataset2 aug24202102
812 dataset-01 aug242021
813 test-dataset-01 testdataset01
814 test-dataset-02 testdataset02
815 Page: 0, Number of datasets: 4
816 {{/code}}
817
818 == Command: **show-detail** ==
819
820 * **Usage**: Show the detailed information of a dataset.
821 * (((
822 **Prerequisite**:
823
824 * User login is required, otherwise an error is returned: "The current login session is invalid. Please login to continue."
825 )))
826
827 {{code language="none"}}
828 $ pilotcli dataset show-detail --help
829 Usage: pilotcli dataset show-detail [OPTIONS] CODE
830
831 Show details of a dataset.
832
833 Options:
834 --page INTEGER The page to be listed [default: 0]
835 --page-size INTEGER number of objects per page [default: 10]
836 -d, --detached whether run in detached mode
837 --help Show this message and exit.
838 {{/code}}
839
840 === Examples ===
841
842 Show the details of the specified Dataset by providing the unique Dataset Code - “aug242021” in this case:
843
844 {{code language="none"}}
845 $ pilotcli dataset show-detail aug242021
846 --------------------------------------------------------------------------------
847 | Title | dataset-01 |
848 --------------------------------------------------------------------------------
849 | Code | aug242021 |
850 --------------------------------------------------------------------------------
851 | Authors | uname, cli |
852 --------------------------------------------------------------------------------
853 | Type | GENERAL |
854 --------------------------------------------------------------------------------
855 | Modality | neuroimaging, microscopy, histological approach |
856 --------------------------------------------------------------------------------
857 | Collection_method | import, test, upload |
858 --------------------------------------------------------------------------------
859 | Tags | tag1, tag2, tag3 |
860 --------------------------------------------------------------------------------
861 | Versions | 1.0, 1.1, 1.2 |
862 --------------------------------------------------------------------------------
863 {{/code}}
864
865 == Command: **download** ==
866
867 * **Usage**: Download a particular version of a dataset.
868 * **Options**: -v ~-~-version
869 * (((
870 **Prerequisite**:
871
872 * User login is required, otherwise error is returned: "The current login session is invalid. Please login to continue."
873 )))
874
875 {{code language="none"}}
876 $ pilotcli dataset download --help
877 Usage: pilotcli dataset download [OPTIONS] [CODE]... OUTPUT_PATH
878
879 Download a dataset or a particular version of a dataset.
880
881 Options:
882 -v, --version TEXT Download a particular version of a dataset.
883 --help Show this message and exit.
884 {{/code}}
885
886
887 === Examples ===
888
889 Download a Dataset with Code “aug242021”:
890
891 {{code language="none"}}
892 $ pilotcli dataset download aug242021 ~/Downloads/
893 Pre downloading dataset
894 Preparing status: READY_FOR_DOWNLOADING
895 start downloading...
896 Downloading aug242021_1632342868.882398.zip |██████████████████████████████ 100% 00:00
897 File has been downloaded successfully and saved to: /home/uname/Downloads/aug242021_163234
898 2868.882398.zip
899 {{/code}}
900
901 Download the version 1.1 of a Dataset with Code “aug242021”:
902
903 {{code language="none"}}
904 $ pilotcli dataset download aug242021 ~/Downloads/ -v 1.1
905 Current dataset version: 1.1
906 Pre downloading dataset
907 start downloading...
908 Downloading aug242021_2021-08-24 11:10:00.042277.zip |██████████████████████████████ 100% 00:00
909 File has been downloaded successfully and saved to: /home/uname/Downloads/aug242021_2021-08-24 11:10:00.042277.zip
910 {{/code}}
911
912
913
914 ----
915
916 Copyright © 2023-2024 [[Indoc Systems>>url:https://www.indocsystems.com]].
917
918 HealthDataCloud is powered by Pilot technology, a product of [[Indoc Systems>>url:https://www.indocsystems.com]].