Version 1.7 by Susan Evans on 2023/07/10 22:24
Hide last authors
Susan Evans 1.1 1 The Pilot Command Line Interface (pilotcli) is a binary executable program that provides a convenient way for users to perform file actions and platform-related tasks programmatically. The Pilot Command Line Interface can be downloaded and run on on your local Linux and Linux subsystems (as Windows user, see for instance the [[Windows Subsystem for Linux installation guide>>url:https://learn.microsoft.com/en-us/windows/wsl/install]]) to support large file uploads. For convenience, pilotcli is also pre-deployed and installed in the users JupyterHub accessible via worskpace tools.
2
3 = Data Stewardship =
4
5 Users are reminded to abide by the Platform Terms of Use and any Project-specific restrictions when using the Command Line Interface to access data and code.
6
7 = Using the Pilot Command Line Interface in JupyterHub =
8
9 To get started with the Pilot Command Line Interface in the HDC JupyterHub workspace tool, see the article //Using JupyterHub in HDC //and refer to the remaining sections in this article for a full description of the pilotcli commands.
10
11 = Downloading the Pilot Command Line Interface =
12
13 If you wish to use the Pilot Command Line Interface to upload large files from your local environment, it is necessary to download Pilot Command Line Interface tool on your Linux system. The recommended Linux version is Ubuntu 20.04.
14
15 * Download the pilotcli tool [[here>>url:https://object.hdc.humanbrainproject.eu/public-resources/pilotcli]].
16
17 After download, please grant execution permission to the binary file so that Pilot Command Line Interface can be used. To grant permission to the binary file, use the following command in the downloaded pilotcli directory.
18
19 {{code language="none"}}
20 $ chmod +x ./pilotcli
21 {{/code}}
22
23
24 If you have sudo permission, you can make the Pilot Command Line Interface more accessible in any directory by moving it to the /usr/local/bin directory:
25
26 {{code language="none"}}
27 $ mv ./pilotcli /usr/local/bin
28 {{/code}}
29
30 Without sudo permission, you can still use the Pilot Command Line Interface in a different directory by referring the Command Line Interface path. For example, use a command line Interface in the Downloads folder:
31
32 {{code language="none"}}
33 $ ~/Downloads/pilotcli
34 {{/code}}
35
36 = Pilot Command Line Interface Command Summary =
37
38 Commands are organized into main groups:
39
40 1. **User**// - //User related actions such as login and logout
41 1. **Project** //- //Project related actions, including list Project Names and Codes for all Projects accessible by the logged-in user.
42 1. **File **- Actions on files. User could use file action commands to list files/folders, upload files/folders, list/export attribute, and download files as well.
43 1. **Dataset **- Actions related to datasets.
44
45 = User Commands: pilotcli **user** =
46
47 == Help ==
48
49 {{code language="none"}}
50 $ pilotcli user --help
51 Usage: pilotcli user [OPTIONS] COMMAND [ARGS]...
52
53 Options:
54 --help Show this message and exit.
55
56 Commands:
57 login For user to login.
58 logout For user to logout.
59 {{/code}}
60
61 == Command: **login** ==
62
63 * **Options**: -u ~-~-username, -p ~-~-password
64 * **Usage**: Allows users to log into the platform through pilotcli. After login, the user must be actively using the Pilot Command Line Interface to remain logged in, or they will be logged out automatically after the session is idle for five minutes.
65 * **Prerequisites**: None.
66
67 {{code language="none"}}
68 $ pilotcli user login --help
69 Usage: pilotcli user login [OPTIONS]
70
71 For user to login.
72
73 Options:
74 --help Show this message and exit.
75 {{/code}}
76
77 === Example ===
78
79 User login:
80
81 1) Use the output device validation link or QR code to launch authentication validation:
82
83 {{code language="none"}}
84 $ pilotcli user login
85 Please, access https://iam.hdc.humanbrainproject.eu/realms/hdc/device?user_code=XXXX-XXXX to proceed
86 ▄▄▄▄▄▄▄ ▄▄ ▄ ▄ ▄ ▄▄▄▄ ▄ ▄▄▄▄▄▄▄
87 █ ▄▄▄ █ ▄ ▄▄▀█ █▀▀▀▄▀█▀ ▀██▄ █ ▄▄▄ █
88 █ ███ █ ██▄ ▀▀▄▀▄▄█▄ ▄ ▀▀▀▄ █ ███ █
89 █▄▄▄▄▄█ █▀▄▀█▀▄▀█▀▄ █ ▄ ▄ █▀█ █▄▄▄▄▄█
90 ▄▄▄▄▄▄▄ ▀█▀█▄▄█▄ █▀█ ▄▀▄▄ █ ▄ █▀▄▄▄
91 █ ▄▄▄ █ █▄█▄██ ▀ ▀█▀▀█ ▀▄█▄█▄▄▄█▀▄█
92 █ ███ █ █▀█ ▄▀██▄█▀▀█▄▀▀█▄▀█ ▀ ▀▄▀██
93 █▄▄▄▄▄█ ▄▀▄ ██▄▀▀▄ ▀▀▄ ▄▄▀▀▀▄ █▄▄▄█
94
95
96 Waiting validation finish...
97 {{/code}}
98
99 2) Grant access by clicking Yes:
100
101 [[image:Pilot CLI authentication.png||height="34%" width="40%"]]
102
103 3) Device Login Successful message will appear and your authentication browser window can be closed.
104
105 4) The output of the Command Line Tool will also be updated (see line 14 below) to notify you about the successful login:
106
107 {{code language="none"}}
108 $ pilotcli user login
109 Please, access https://iam.hdc.humanbrainproject.eu/realms/hdc/device?user_code=XXXX-XXXX to proceed
110 ▄▄▄▄▄▄▄ ▄▄ ▄ ▄ ▄ ▄▄▄▄ ▄ ▄▄▄▄▄▄▄
111 █ ▄▄▄ █ ▄ ▄▄▀█ █▀▀▀▄▀█▀ ▀██▄ █ ▄▄▄ █
112 █ ███ █ ██▄ ▀▀▄▀▄▄█▄ ▄ ▀▀▀▄ █ ███ █
113 █▄▄▄▄▄█ █▀▄▀█▀▄▀█▀▄ █ ▄ ▄ █▀█ █▄▄▄▄▄█
114 ▄▄▄▄▄▄▄ ▀█▀█▄▄█▄ █▀█ ▄▀▄▄ █ ▄ █▀▄▄▄
115 █ ▄▄▄ █ █▄█▄██ ▀ ▀█▀▀█ ▀▄█▄█▄▄▄█▀▄█
116 █ ███ █ █▀█ ▄▀██▄█▀▀█▄▀▀█▄▀█ ▀ ▀▄▀██
117 █▄▄▄▄▄█ ▄▀▄ ██▄▀▀▄ ▀▀▄ ▄▄▀▀▀▄ █▄▄▄█
118
119
120 Waiting validation finish...
121 Welcome to Command Line Tool!
122 {{/code}}
123
124 == Command: **logout** ==
125
126 * **Options**: -y ~-~-yes
127 * **Usage**: Allows users to log out of the platform through pilotcli.
128 * **Prerequisites**: User login is required, otherwise an error is returned: ‘The current login session is invalid. Please login to continue.'.
129
130 {{code language="none"}}
131 $ pilotcli user logout --help
132 Usage: pilotcli user logout [OPTIONS]
133
134 For user to logout.
135
136 Options:
137 -y, --yes Input Y/yes to confirm you want to logout, otherwise input N/no
138 to remain logged in.
139
140 --help Show this message and exit.
141 {{/code}}
142
143 === Examples ===
144
145 User logout:
146
147 {{code language="none"}}
148 $ pilotcli user logout
149 Are you sure you want to logout? [y/N]: y
150 Logged out successfully. Bye!
151 {{/code}}
152
153 User logout canceled:
154
155 {{code language="none"}}
156 $ pilotcli user logout
157 Are you sure you want to logout? [y/N]: n
158 Logout canceled!
159 {{/code}}
160
161 User logout with option to skip additional confirmation:
162
163 {{code language="none"}}
164 $ pilotcli user logout -y
165 Logged out successfully. Bye!
166 {{/code}}
167
168 = Project Commands: pilotcli **project** =
169
170 == Help ==
171
172 {{code language="none"}}
173 $ pilotcli project --help
174 Usage: pilotcli project [OPTIONS] COMMAND [ARGS]...
175
176 Options:
177 --help Show this message and exit.
178
179 Commands:
180 list List accessible projects.
181 {{/code}}
182
183 == Command: **list** ==
184
185 * **Options**: None
186 * **Usage**: List all Projects to which the user has access
187 * **Prerequisite**: User login is required, otherwise an error is returned: ‘The current login session is invalid. Please login to continue.'
188
189 {{code language="none"}}
190 $ pilotcli project list --help
191 Usage: pilotcli project list [OPTIONS]
192
193 List accessible projects.
194
195 Options:
196 --page INTEGER The page to be listed [default: 0]
197 --page-size INTEGER number of objects per page [default: 10]
198 --order [asc|desc] sorting order [default: desc]
199 --order-by [created_at|name|code]
200 sorting column [default: created_at]
201 -d, --detached whether run in detached mode
202 --help Show this message and exit.
203 {{/code}}
204
205 === Examples ===
206
207 Get a list of all Projects the user has access to:
208
209 {{code language="none"}}
210 $ pilotcli project list
211 Project Name Project Code
212 ---------------------------------------------------------------------------
213 Test Project 02-13 | test230213byshan
214 Test1_Audit | test1audit
215 Test20221109 | test20220109
216 Indoc Test Project | indoctestproject
217
218 Page: 0, Number of projects: 4
219 {{/code}}
220
221 By default, the table will only display the first ten elements. If the user has access to more than ten Projects, you will be prompted to use your arrow keys to select one of two options to either view the next ten items (//next page//), or to exit (//exit//). The selected option is indicated by two arrows and can be confirmed by hitting enter:
222
223 {{code language="none"}}
224 Page: 0, Number of projects: 10
225
226 What do you want? (Use arrow keys)
227 » next page
228 exit
229 {{/code}}
230
231 = File Commands: pilotcli **file** =
232
233 == Help ==
234
235 {{code language="none"}}
236 $ pilotcli file --help
237 Usage: pilotcli file [OPTIONS] COMMAND [ARGS]...
238
239 Options:
240 --help Show this message and exit.
241
242 Commands:
243 attribute-export Export attribute template from a given Project.
244 attribute-list List attribute templates of a given Project.
245 list List files and folders inside a given Project/folder.
246 resume Resume the upload process with given manifest file.
247 sync Download files/folders from a given Project/folder/file...
248 upload Upload files/folders to a given Project path.
249 {{/code}}
250
251 == Command: **attribute-list** ==
252
253 * **Options**: -p ~-~-project-code
254 * **Usage**: List all attributes from a specified Project.
Susan Evans 1.2 255 * **Prerequisite**:
Susan Evans 1.1 256
257 * User login is required, otherwise an error is returned: ‘The current login session is invalid. Please login to continue.’
Susan Evans 1.2 258 * Requires the Project code, which can be retrieved via {{code}}pilotcli project list{{/code}}.
Susan Evans 1.1 259
Susan Evans 1.2 260 {{code language="none"}}
261 $ pilotcli file attribute-list --help
262 Usage: pilotcli file attribute-list [OPTIONS]
Susan Evans 1.1 263
Susan Evans 1.2 264 List attribute templates of a given Project.
Susan Evans 1.1 265
Susan Evans 1.2 266 Options:
267 -p, --project-code TEXT Project Code
268 --help Show this message and exit.
269 {{/code}}
Susan Evans 1.1 270
Susan Evans 1.3 271 === Examples ===
Susan Evans 1.2 272
Susan Evans 1.3 273 List attributes from a given Project by providing the Project Code at the prompt. In this example, “cli” is the corresponding Project Code:
Susan Evans 1.2 274
Susan Evans 1.3 275 {{code language="none"}}
276 $ pilotcli file attribute-list
277 ProjectCode: cli
Susan Evans 1.2 278
Susan Evans 1.3 279 Manifest1
280 --------------------------------------------------------------------------
281 | Attribute Name | Type | Value | Optional |
282 --------------------------------------------------------------------------
283 | attr1 | multiple_choice | a1,a2,a3,a4,a5 | False |
284 --------------------------------------------------------------------------
285 | attr2 | text | None | False |
286 --------------------------------------------------------------------------
287 | attr3 | multiple_choice | t1,t2,t3,t4,t5 | True |
288 --------------------------------------------------------------------------
Susan Evans 1.2 289
Susan Evans 1.3 290 Manifest2
291 --------------------------------------------------------------------------
292 | Attribute Name | Type | Value | Optional |
293 --------------------------------------------------------------------------
294 | attr1 | multiple_choice | a1,a2,a3,a4 | True |
295 --------------------------------------------------------------------------
296 | attr2 | text | None | False |
297 --------------------------------------------------------------------------
298 | attr3 | multiple_choice | t1,t2,t3 | True |
299 --------------------------------------------------------------------------
300 All Attributes fetched successfully.
301 {{/code}}
302
303 List attributes from given a Project and provide the Project Code “cli” as an option in the same line {{code}}-p cli{{/code}}:
304
305 {{code language="none"}}
306 $ pilotcli file attribute-list -p cli
307
308 Manifest1
309 --------------------------------------------------------------------------
310 | Attribute Name | Type | Value | Optional |
311 --------------------------------------------------------------------------
312 | attr1 | multiple_choice | a1,a2,a3,a4,a5 | False |
313 --------------------------------------------------------------------------
314 | attr2 | text | None | False |
315 --------------------------------------------------------------------------
316 | attr3 | multiple_choice | t1,t2,t3,t4,t5 | True |
317 --------------------------------------------------------------------------
318
319 Manifest2
320 --------------------------------------------------------------------------
321 | Attribute Name | Type | Value | Optional |
322 --------------------------------------------------------------------------
323 | attr1 | multiple_choice | a1,a2,a3,a4 | True |
324 --------------------------------------------------------------------------
325 | attr2 | text | None | False |
326 --------------------------------------------------------------------------
327 | attr3 | multiple_choice | t1,t2,t3 | True |
328 --------------------------------------------------------------------------
329 All Attributes fetched successfully.
330 {{/code}}
331
332 == Command: **attribute-export** ==
333
334 * **Options**: -p ~-~-project-code, -n ~-~-attribute-name
335 * **Usage**: Export an attribute template as JSON-file from a given Project
336 * (((
337 **Prerequisite**:
338
339 * User login is required, otherwise an error is returned: "The current login session is invalid. Please login to continue."
340 * Project must be accessible to the user, otherwise an error is returned: "Project Code not found in list. Please verify and try again."
341 * Requires the Project code, which can be retrieved via pilotcli project list.
342 * Requires the Attribute template name. If the attribute template name does not exist or is spelled incorrectly, an error is returned: "Attribute 'sample-template' not found in Project. Please verify and try again."
343 * pilotcli must have the permission to write a file in the directory, otherwise an error is returned.
344 )))
345
346 {{code language="none"}}
347 $ pilotcli file attribute-export --help
348 Usage: pilotcli file attribute-export [OPTIONS]
349
350 Export attribute template from a given Project.
351
352 Options:
353 -p, --project-code TEXT Project Code
354 -n, --attribute-name TEXT Attribute Template Name
355 --help Show this message and exit.
356 {{/code}}
357
358 === Examples ===
359
360 Export an attribute template from a given Project. The user is prompted to provide the Project Code (“cli”) and the Attribute Template Name (“Manifest1”):
361
362 {{code language="none"}}
363 $ pilotcli file attribute-export
364 ProjectCode: cli
365 AttributeName: Manifest1
366
367 Manifest1
368 --------------------------------------------------------------------------
369 | Attribute Name | Type | Value | Optional |
370 --------------------------------------------------------------------------
371 | attr1 | multiple_choice | a1,a2,a3,a4,a5 | False |
372 --------------------------------------------------------------------------
373 | attr2 | text | None | False |
374 --------------------------------------------------------------------------
375 | attr3 | multiple_choice | t1,t2,t3,t4,t5 | True |
376 --------------------------------------------------------------------------
377 Template saved successfully: cli_Manifest1_template.json
378 Attribute definition saved successfully: cli_Manifest1_definition.json
379 {{/code}}
380
Susan Evans 1.4 381 Export an attribute template from a given Project using options to provide both the Project Code {{code}}(-p cli){{/code}} and the Attribute Template Name {{code}}(-n Manifest1){{/code}}, “cli” and “Manifest1” in this case, respectively:
Susan Evans 1.3 382
Susan Evans 1.4 383 {{code language="none"}}
384 $ pilotcli file attribute-export -p cli -n Manifest1
Susan Evans 1.3 385
Susan Evans 1.4 386 Manifest1
387 --------------------------------------------------------------------------
388 | Attribute Name | Type | Value | Optional |
389 --------------------------------------------------------------------------
390 | attr1 | multiple_choice | a1,a2,a3,a4,a5 | False |
391 --------------------------------------------------------------------------
392 | attr2 | text | None | False |
393 --------------------------------------------------------------------------
394 | attr3 | multiple_choice | t1,t2,t3,t4,t5 | True |
395 --------------------------------------------------------------------------
396 Template saved successfully: cli_Manifest1_template.json
397 Attribute definition saved successfully: cli_Manifest1_definition.json
398 {{/code}}
399
400 == Command: **list** ==
401
402 * **Options**: -z ~-~-zone
403 * **Usage**: List files and folders in the specified directory within the Platform. Folders will be displayed in blue and files will be displayed in white (or black, depending on the background of your terminal). If no zone (i.e. Green Room or Core) is specified, Green Room will be used as default.
404 * (((
405 **Prerequisite**:
406
407 * User login is required, otherwise an error is returned: "The current login session is invalid. Please login to continue."
408 * Project must be accessible to the user, otherwise an error is returned: "Project Code not found in list. Please verify and try again."
409 * Requires the Project code, which can be retrieved via pilotcli project list.
410 )))
411
412 {{code language="none"}}
413 $ pilotcli file list --help
414 Usage: pilotcli file list [OPTIONS] PATHS
415
416 List files and folders inside a given Project/folder.
417
418 Options:
419 -z, --zone TEXT Target Zone (i.e., core/greenroom) [default:
420 greenroom] [default: greenroom]
421
422 --page INTEGER The page to be listed [default: 0]
423 --page-size INTEGER number of objects per page [default: 10]
424 -d, --detached whether run in detached mode
425 --help Show this message and exit.
426 {{/code}}
427
428 === Examples ===
429
430 List folders in the directory “admin” in the Project with the Project Code “indoctestproject”. Since no zone was specified, the corresponding directory in the Green Room will be queried:
431
432 {{code language="none"}}
433 $ pilotcli file list indoctestproject/admin
434 test_folder1 test_folder2 file1 file2 file3
435 {{/code}}
436
437 To list the corresponding elements in the “indoctestproject/admin” directory in the Projects' Core, the zone needs to be specified {{code}}(-z core){{/code}}:
438
439 {{code language="none"}}
440 $ pilotcli file list indoctestproject/admin -z core
441 folder1 folder2 test_folder.zip file.txt sample.txt large-file.json
442 {{/code}}
443
444 == Command: **upload** ==
445
446 * **Options**: -p ~-~-project, -a ~-~-attribute, -t ~-~-tag, -z ~-~-zone, -m ~-~-upload-message, -s ~-~-source-file, ~-~-zip, -td ~-~-thread, -o ~-~-output path
447 * **Usage**: Upload files/folders to the given project, with optional tags and attributes, to the target folder.
448 * (((
449 **Prerequisite**:
450
451 * User login is required, otherwise an error is returned: "The current login session is invalid. Please login to continue."
452 * Project must be accessible to the user, otherwise an error is returned: "Project Code not found in list. Please verify and try again."
453 * When uploading file to the Project Core, the user must have access to the Core (Project Contributors cannot upload files/folders to the Project Core).
454 )))
455
Susan Evans 1.5 456 {{code language="none"}}
457 $ pilotcli file upload --help
458 Usage: pilotcli file upload [OPTIONS] [PATHS]...
Susan Evans 1.4 459
Susan Evans 1.5 460 Upload files/folders to a given Project path.
Susan Evans 1.4 461
Susan Evans 1.5 462 Options:
463 -p, --project-path TEXT Project folder path starting from Project code.
464 (i.e., indoctestproject/user/folder) [required]
465
466 -a, --attribute TEXT File Attribute Template used for annotating files
467 during upload.
468
469 -t, --tag TEXT Add a tag to the file. This option could be used
470 multiple times for adding multiple tags.
471
472 -z, --zone TEXT Target Zone (i.e., core/greenroom) [default:
473 greenroom] [default: greenroom]
474
475 -m, --upload-message TEXT The message used to comment on the purpose of
476 uploading your processed file [default: ]
477
478 -s, --source-file TEXT The Project path of the source file of your
479 processed files.
480
481 --zip Upload folder as a compressed zip file.
482 -td, --thread INTEGER The number of thread for upload a file [default:
483 1]
484
485 -o, --output-path TEXT The output path for the manifest file of
486 resumable upload [default: ./manifest.json]
487
488 --help Show this message and exit.
489 {{/code}}
490
491 === Examples ===
492
493 Upload a file (or folder) to a specified directory of a Project’s Green Room, in this case the file located at {{code}}./sample_folder/sample_file1{{/code}} will be uploaded to the folder “admin” in the Project with the Project Code “cli” {{code}}(-p cli/admin){{/code}}:
494
495 {{code language="none"}}
496 $ pilotcli file upload ./sample_folder/sample_file1 -p cli/admin
497 Starting upload of: ./sample_folder/sample_file1
498 Pre-upload complete.
499 uploader: admin container_id: 18061 total_size: 1048576 total_chunks: 1 uniq_identifier: e958247c-ce05-48b2-9038-78b774836256-1627498771 resumable_relative_path admin/
500
501 Upload job is finalizing, please wait...
502 Upload job complete.
503 All files uploaded successfully.
504 {{/code}}
505
506 In case the specified destination directory does not exist yet, you will be prompted to confirm its creation by entering “y”, or to abort the command by entering “N”:
507
508 {{code language="none"}}
509 Target folder does not exist. Would you like to create a new folder? [y/N]: y
510 {{/code}}
511
512 In order to upload a file (or folder) to a destination directory in a Project's Core, core must be specified as corresponding zone {{code}}(-z core){{/code}} and you will be required to provide a message to that should describe the files` content and why you are uploading it directly into the Core {{code}}(-m "sample message"){{/code}}. Moreover, you will be reminded about the Terms of Use of the Platform and additional considerations regarding the direct upload of data into the Core zone, and are prompted to confirm your intention to proceed with the upload by entering “y” or “Yes”:
513
Susan Evans 1.6 514 {{code language="none"}}
515 $ pilotcli file upload ./sample_folder/sample_file2 -p cli/admin -z core -m "sample message"
516 You are about to transfer data directly to the Core! In accordance with the Terms of Use, please confirm that you
517 have made your best efforts to pseudonymize or anonymize the data and that you have the legal authority to transfer and make this
518 data available for dissemination and use within the Platform.To review the Terms of Use, please the Portal. If you
519 need to process the data to remove sensitive identifiers, please cancel this transfer and upload the data to the Green Room to
520 perform these actions.
521 To cancel this transfer, enter [n/No]
522 To confirm and proceed with the data transfer, enter [y/Yes]
523 [y/N]: y
524 Starting upload of: ./sample_folder/sample_file2
525 Pre-upload complete.
526 uploader: admin container_id: 18061 total_size: 1048576 total_chunks: 1 uniq_identifier: fa807a0f-ceb0-4fff-9c9e-563bdcbb309f-1627498851 resumable_relative_path admin/
527
528 Upload job is finalizing, please wait...
529 Upload job complete.
530 All files uploaded successfully.
531 {{/code}}
Susan Evans 1.5 532
Susan Evans 1.6 533 Upload a folder to Project Green Room
Susan Evans 1.5 534
Susan Evans 1.6 535 {{code language="none"}}
536 $ pilotcli file upload ./sample_folder/example_folder -p cli/admin
537 Starting upload of: ./sample_folder/example_folder
538 Current version does not support folder tagging, any selected tags will be ignored
539 Pre-upload complete.
540 uploader: admin container_id: 18061 total_size: 1048576 total_chunks: 1 uniq_identifier: 4f0eee2e-70f4-4b03-9e72-040544ec7d3a-1627498907 resumable_relative_path admin/example_folder/
541
542 Starting finalization...
543 uploader: admin container_id: 18061 total_size: 1048576 total_chunks: 1 uniq_identifier: 41ec70e2-0859-4aba-9009-06e700c7c79e-1627498907 resumable_relative_path admin/example_folder/
544
545 Starting finalization...
546 uploader: admin container_id: 18061 total_size: 1048576 total_chunks: 1 uniq_identifier: 28454c51-ec9a-4dc5-ab3c-0f7e80e3050d-1627498907 resumable_relative_path admin/example_folder/
547
548 Upload job is finalizing, please wait...
549 Upload job complete.
550 All files uploaded successfully.
551 {{/code}}
Susan Evans 1.5 552
Susan Evans 1.6 553 Upload file to existing folder in Green Room
Susan Evans 1.5 554
Susan Evans 1.6 555 * Specify a target folder to upload files by inputting folder name after the Project code separated by the '/'.
556
557 {{code language="none"}}
558 $ pilotcli file upload ./sample_folder/sample_file5 -p cli/admin/example_folder
559 Starting upload of: ./sample_folder/sample_file5
560 Pre-upload complete.
561 uploader: admin container_id: 18061 total_size: 1048576 total_chunks: 1 uniq_identifier: 7ef58005-d408-4ad9-a741-fa938f58e67f-1627498977 resumable_relative_path admin/example_folder/
562
563 Upload job is finalizing, please wait...
564 Upload job complete.
565 All files uploaded successfully.
566 {{/code}}
567
568 Upload folder as zip to a specified directory of a Project's Green Room by adding the ~-~-zip option.
569
570 * The specified folder will then be compressed as a zip file and subsequently be uploaded to the specified destination. Once the upload is completed, the compressed zip file will be removed from the directory. Note: pilotcli must have the permission to create and delete zip files in the corresponding directory of your local machine:
571
572 {{code language="none"}}
573 $ pilotcli file upload ./sample_folder/example_folder -p cli/admin --zip
574 Starting upload of: ./sample_folder/example_folder
575 Started zipping files.
576 Pre-upload complete.
577 uploader: admin container_id: 18061 total_size: 3595 total_chunks: 1 uniq_identifier: 61a729ce-cdd6-4a87-82b5-3f8ea63e6c4a-1627499012 resumable_relative_path admin/
578
579 Upload job is finalizing, please wait...
580 Upload job complete.
581 All files uploaded successfully.
582 {{/code}}
583
584 Upload a file with tags and/or attributes.
585
586 * When entering the upload command, the user can add a maximum of ten tags to the file. For example, in order to add “sample_tag1” and “sample_tag2” to your file, simply add {{code}}-t sample_tag1{{/code}} {{code}}-t sample_tag2{{/code}} to the command (see example below using {{code}}tag1{{/code}} and {{code}}tag2{{/code}} as tags).
Susan Evans 1.7 587 * In addition, you can also use attribute templates for a more enriched annotation of your files. Please note, that the corresponding attribute template file needs to be edited & saved prior to uploading. To get the blank template, see the **attribute-export** instructions above for more details. To attach a attribute JSON-file to your file upon uploading, use {{code}}-a cli_Manifest1_template.json{{/code}}, where “{{code}}cli_Manifest1_template.json{{/code}}" points to the location of the attributes JSON-file on your local machine.
Susan Evans 1.6 588 * Upon uploading, pilotcli will also validate the content of the attached attribute JSON-file. Whenever a template is invalid, including missing non-optional fields, non-existing choice in multiple choice attribute, exceeding length text field and so on, pilotcli will raise error to user and user must update the attribute template and upload again:
589
Susan Evans 1.7 590 {{code language="none"}}
591 $ pilotcli file upload ./sample_folder/sample_file6 -p cli/admin -t tag1 -t tag2 -a cli_Manifest1_template.json
592 File attribute validation passed: True
593 Starting upload of: ./sample_folder/sample_file6
594 Pre-upload complete.
595 uploader: admin container_id: 18061 total_size: 1048576 total_chunks: 1 uniq_identifier: bc47ec2a-1a4e-4f71-a37a-7ce6473e415a-1627499273 resumable_relative_path admin/
596
597 Upload job is finalizing, please wait...
598 Upload job complete.
599 Attribute attached
600 All files uploaded successfully.
601 {{/code}}
Susan Evans 1.6 602
Susan Evans 1.7 603 Upload a file with specified number of threads, using the {{code}}-td <thread_number>{{/code}} option.
Susan Evans 1.6 604
Susan Evans 1.7 605 * By default, the number of threads is 1. There will no UI difference when using threads. The maximal number of threads depends on your local machine and the recommend thread number is 3:
Susan Evans 1.6 606
Susan Evans 1.7 607 {{code language="none"}}
608 $ pilotcli file upload -p indoctestproject/admin/test20230306_3 /home/color/indoc/script/large_folder_test_data/gentoo_root_100M.img -td 3
609 Target folder does not exist. Would you like to create a new folder? [y/N]: y
610 Starting upload of: /home/color/indoc/script/large_folder_test_data/gentoo_root_100M.img
611 Pre-upload complete.
612 Uploading gentoo_root_100M.img , resumable_id: c68752372b42c85d, job_id: 56de3632-58df-445c-bcb9-bddfe6be5bf7: |██████████████████████████████ 100% 00:00
613 Upload Time: 79.77s for 1 files
614 All uploading jobs have finished.
615 {{/code}}
616
617 (10) upload with customized manifest name
618
619 When you upload a file using pilotcli, a JSON file with the details of your specific upload job will be created automatically and allows you to resume the upload at a later time, in case the process is aborted due to any reason. By default, this JSON file will be created as “manifest.json” in the current working directory (see //Command: **resume**// below for more details on how to resume the file upload). However, you can use the {{code}}--output-path{{/code}} option to specify the filepath and, thus, the filename of this JSON file:
620
621
622
623
624
Susan Evans 1.1 625 ----
626
627 -
628 {{code language="none"}}{{/code}}
629
630 ----
631
632 Copyright © 2023 [[Indoc Research>>url:https://www.indocresearch.org/]].
633
Susan Evans 1.2 634 HealthDataCloud is powered by Pilot technology, a product of [[Indoc Research>>url:https://www.indocresearch.org/]].~{~{/code}}
635 ~)~)~)