Update Gitlab sync + import to be modern (!208) · Merge requests · Eclipse Foundation / IT / Webdev / eclipsefdn-vc-tools

Martin Lowe requested to merge malowe/eclipsefdn-github-sync:malowe/master/update-gl into master Apr 12, 2022

Updates to npm packages including base GL binding. Some API changes, but overall mostly the same. Updated scripts where necessary, tested the import and sync with the following results.

Sync:

❯ node src/GitlabSync.js --devMode=true --host=http://localhost --secretLocation=$PWD/secrets -V --trace-warnings --tls-min-v1.0 --unhandled-rejections=strict
2022-04-12T13:30:11 [main] DEBUG GitlabSync:sanitizeGroupName(pid = eclipse)
2022-04-12T13:30:11 [main] DEBUG GitlabSync:sanitizeGroupName(pid = gitlab-instance-a8f39b1c)
2022-04-12T13:30:11 [main] DEBUG GitlabSync:sanitizeGroupName(pid = backup-20220411)
2022-04-12T13:30:11 [main] DEBUG GitlabSync:sanitizeGroupName(pid = github-backup)
2022-04-12T13:30:11 [main] DEBUG GitlabSync:sanitizeGroupName(pid = openhw-group)
2022-04-12T13:30:11 [main] INFO Starting sync
2022-04-12T13:30:11 [main] DEBUG GitlabSync:getGroup(name = Eclipse, path = eclipse, parent = undefined, visibility = public)
2022-04-12T13:30:11 [main] DEBUG GitlabSync:sanitizeGroupName(pid = eclipse)
2022-04-12T13:30:11 [main] INFO Processing 'spider.pig'
2022-04-12T13:30:11 [main] DEBUG GitlabSync:getGroup(name = Spider pig does what a spider pig does, path = spider.pig, parent = [object Object], visibility = public)
2022-04-12T13:30:11 [main] DEBUG GitlabSync:sanitizeGroupName(pid = spider.pig)
2022-04-12T13:30:11 [main] VERBOSE Creating new group with name 'Spider pig does what a spider pig does'
2022-04-12T13:30:11 [main] DEBUG Creating group with options: {"project_creation_level":"maintainer","visibility":"public","request_access_enabled":false,"parent_id":53}
2022-04-12T13:30:11 [main] DEBUG GitlabSync:sanitizeGroupName(pid = spider.pig)
2022-04-12T13:30:12 [main] DEBUG Created group: {"id":54,"web_url":"http://gitlab.dev.docker/groups/eclipse/spider.pig","name":"Spider pig does what a spider pig does","path":"spider.pig","description":"","visibility":"public","share_with_group_lock":false,"require_two_factor_authentication":false,"two_factor_grace_period":48,"project_creation_level":"maintainer","auto_devops_enabled":null,"subgroup_creation_level":"maintainer","emails_disabled":null,"mentions_disabled":null,"lfs_enabled":true,"default_branch_protection":2,"avatar_url":null,"request_access_enabled":false,"full_name":"Eclipse / Spider pig does what a spider pig does","full_path":"eclipse/spider.pig","created_at":"2022-04-12T17:30:11.846Z","parent_id":53,"shared_with_groups":[],"projects":[],"shared_projects":[]}
2022-04-12T13:30:12 [main] DEBUG GitlabSync:sanitizeGroupName(pid = spider.pig)
2022-04-12T13:30:12 [main] DEBUG GitlabSync:getUserList(project = {"project_id":"spider.pig","short_project_id":"spider.pig","name":"Spider pig does what a spider pig does","summary":"Can he fly? No, hes a pig. Look out, here comes the spider pig","logo":"","tags":["simpsons","doh","spider pig"],"github_repos":[{"url":"https://github.com/eclipsefdn-webdev/spider-pig"}],"gitlab_repos":[{"url":"https://gitlab.eclipse.org/eclipsefdn/webdev/gitlab-testing","org":"webdev","repo":"gitlab-testing"}],"contributors":[],"committers":[{"username":"malowe","url":"https://api.eclipse.org/account/profile/malowe"},{"username":"epoirier","url":"https://api.eclipse.org/account/profile/epoirier"}],"project_leads":[{"username":"malowe","url":"https://api.eclipse.org/account/profile/malowe"},{"username":"cguindon","url":"https://api.eclipse.org/account/profile/cguindon"}],"working_groups":[{"name":"Cloud Development Tools","id":"cloud-development-tools"}],"spec_project_working_group":[],"state":"Regular","pp_repos":["gitlab-testing"],"pp_orgs":["webdev"]})
2022-04-12T13:30:12 [main] DEBUG GitlabSync:getUser(uname = malowe, url = https://api.eclipse.org/account/profile/malowe)
2022-04-12T13:30:13 [main] VERBOSE Creating new user with name 'malowe'
2022-04-12T13:30:13 [main] DEBUG Creating user with options: {"username":"malowe","password":"redacted","force_random_password":true,"name":"Martin Lowe","email":"martin.lowe@eclipse-foundation.org","extern_uid":"473605","provider":"oauth2_generic","skip_confirmation":true}
2022-04-12T13:30:13 [main] ERROR Response code 409 (Conflict)
2022-04-12T13:30:13 [main] VERBOSE Could not retrieve user for UID 'malowe', skipping
2022-04-12T13:30:13 [main] DEBUG GitlabSync:getUser(uname = epoirier, url = https://api.eclipse.org/account/profile/epoirier)
2022-04-12T13:30:14 [main] VERBOSE Creating new user with name 'epoirier'
2022-04-12T13:30:14 [main] DEBUG Creating user with options: {"username":"epoirier","password":"redacted","force_random_password":true,"name":"Eric Poirier","email":"eric.poirier@eclipse-foundation.org","extern_uid":"2573","provider":"oauth2_generic","skip_confirmation":true}
2022-04-12T13:30:15 [main] DEBUG GitlabSync:addUserToGroup(user = [object Object], group = [object Object], perms = 30)
2022-04-12T13:30:15 [main] DEBUG GitlabSync:getGroupMembers(group = [object Object])
2022-04-12T13:30:15 [main] INFO Adding 'epoirier' to 'Spider pig does what a spider pig does' group
2022-04-12T13:30:15 [main] DEBUG GitlabSync:getUser(uname = cguindon, url = https://api.eclipse.org/account/profile/cguindon)
2022-04-12T13:30:16 [main] VERBOSE Creating new user with name 'cguindon'
2022-04-12T13:30:16 [main] DEBUG Creating user with options: {"username":"cguindon","password":"redacted","force_random_password":true,"name":"Christopher Guindon","email":"chris.guindon@eclipse-foundation.org","extern_uid":"9","provider":"oauth2_generic","skip_confirmation":true}
2022-04-12T13:30:17 [main] DEBUG GitlabSync:addUserToGroup(user = [object Object], group = [object Object], perms = 40)
2022-04-12T13:30:17 [main] DEBUG GitlabSync:getGroupMembers(group = [object Object])
2022-04-12T13:30:17 [main] INFO Adding 'cguindon' to 'Spider pig does what a spider pig does' group
2022-04-12T13:30:17 [main] DEBUG GitlabSync:removeAdditionalUsers(expectedUsers = [object Object], group = [object Object], projectID = spider.pig)
2022-04-12T13:30:17 [main] DEBUG GitlabSync:getGroupMembers(group = [object Object])
2022-04-12T13:30:17 [main] VERBOSE Checking user 'root' access to group 'Spider pig does what a spider pig does'
2022-04-12T13:30:17 [main] VERBOSE Checking user 'epoirier' access to group 'Spider pig does what a spider pig does'
2022-04-12T13:30:17 [main] VERBOSE Checking user 'cguindon' access to group 'Spider pig does what a spider pig does'
2022-04-12T13:30:17 [main] DEBUG Processing repo 'https://gitlab.eclipse.org/eclipsefdn/webdev/gitlab-testing'
2022-04-12T13:30:17 [main] DEBUG GitlabSync:getProject(name = gitlab-testing, parent = [object Object])
2022-04-12T13:30:17 [main] VERBOSE Creating new project with name 'gitlab-testing'
2022-04-12T13:30:17 [main] DEBUG Creating project with options: {"path":"gitlab-testing","visibility":"public","namespace_id":54}
2022-04-12T13:30:18 [main] DEBUG Created project: {"id":45,"description":null,"name":"gitlab-testing","name_with_namespace":"Eclipse / Spider pig does what a spider pig does / gitlab-testing","path":"gitlab-testing","path_with_namespace":"eclipse/spider.pig/gitlab-testing","created_at":"2022-04-12T17:30:17.724Z","default_branch":"main","tag_list":[],"topics":[],"ssh_url_to_repo":"git@gitlab.dev.docker:eclipse/spider.pig/gitlab-testing.git","http_url_to_repo":"http://gitlab.dev.docker/eclipse/spider.pig/gitlab-testing.git","web_url":"http://gitlab.dev.docker/eclipse/spider.pig/gitlab-testing","readme_url":null,"avatar_url":null,"forks_count":0,"star_count":0,"last_activity_at":"2022-04-12T17:30:17.724Z","namespace":{"id":54,"name":"Spider pig does what a spider pig does","path":"spider.pig","kind":"group","full_path":"eclipse/spider.pig","parent_id":53,"avatar_url":null,"web_url":"http://gitlab.dev.docker/groups/eclipse/spider.pig"},"_links":{"self":"http://gitlab.dev.docker/api/v4/projects/45","issues":"http://gitlab.dev.docker/api/v4/projects/45/issues","merge_requests":"http://gitlab.dev.docker/api/v4/projects/45/merge_requests","repo_branches":"http://gitlab.dev.docker/api/v4/projects/45/repository/branches","labels":"http://gitlab.dev.docker/api/v4/projects/45/labels","events":"http://gitlab.dev.docker/api/v4/projects/45/events","members":"http://gitlab.dev.docker/api/v4/projects/45/members"},"packages_enabled":true,"empty_repo":true,"archived":false,"visibility":"public","resolve_outdated_diff_discussions":false,"container_expiration_policy":{"cadence":"1d","enabled":false,"keep_n":10,"older_than":"90d","name_regex":".*","name_regex_keep":null,"next_run_at":"2022-04-13T17:30:17.802Z"},"issues_enabled":true,"merge_requests_enabled":true,"wiki_enabled":true,"jobs_enabled":true,"snippets_enabled":true,"container_registry_enabled":true,"service_desk_enabled":false,"service_desk_address":null,"can_create_merge_request_in":true,"issues_access_level":"enabled","repository_access_level":"enabled","merge_requests_access_level":"enabled","forking_access_level":"enabled","wiki_access_level":"enabled","builds_access_level":"enabled","snippets_access_level":"enabled","pages_access_level":"enabled","operations_access_level":"enabled","analytics_access_level":"enabled","container_registry_access_level":"enabled","security_and_compliance_access_level":"private","emails_disabled":null,"shared_runners_enabled":true,"lfs_enabled":true,"creator_id":1,"import_status":"none","import_error":null,"open_issues_count":0,"runners_token":"GR1348941sKFsyiGDjENDZiE5gfzd","ci_default_git_depth":20,"ci_forward_deployment_enabled":true,"ci_job_token_scope_enabled":false,"public_jobs":true,"build_git_strategy":"fetch","build_timeout":3600,"auto_cancel_pending_pipelines":"enabled","build_coverage_regex":null,"ci_config_path":null,"shared_with_groups":[],"only_allow_merge_if_pipeline_succeeds":false,"allow_merge_on_skipped_pipeline":null,"restrict_user_defined_variables":false,"request_access_enabled":true,"only_allow_merge_if_all_discussions_are_resolved":false,"remove_source_branch_after_merge":true,"printing_merge_request_link_enabled":true,"merge_method":"merge","squash_option":"default_off","suggestion_commit_message":null,"merge_commit_template":null,"squash_commit_template":null,"auto_devops_enabled":true,"auto_devops_deploy_strategy":"continuous","autoclose_referenced_issues":true,"repository_storage":"default","keep_latest_artifact":true,"runner_token_expiration_interval":null}
2022-04-12T13:30:18 [main] DEBUG GitlabSync:cleanUpProjectUsers(project = 45)

Import:

❯ npm run import-backup -- --verbose=true --host="http://localhost" --target=9 --organization=openhwgroup --secretLocation=/localdev/eclipsefdn-webdev-toolkit/projects-js/eclipsefdn-github-sync/secrets/import

> eclipsefdn-github-sync@1.0.0-BETA import-backup
> node src/auto_backup/Import.js "--verbose=true" "--host=http://localhost" "--target=9" "--organization=openhwgroup" "--secretLocation=/localdev/eclipsefdn-webdev-toolkit/projects-js/eclipsefdn-github-sync/secrets/import"

Importing with config given: {"target":9,"source":"openhwgroup","count":1,"host":"http://localhost","flags":{}}
createBackupGroup(backup-20220412, 9)
2022-04-12T13:47:52 [GitWrapper] DEBUG prefetchRepos(org = openhwgroup)
2022-04-12T13:47:52 [GitWrapper] VERBOSE Starting prefetch for repos in org=openhwgroup
2022-04-12T13:47:54 [GitWrapper] VERBOSE Finished prefetch for org=openhwgroup, got 42 repos
2022-04-12T13:47:54 [GitWrapper] DEBUG getDiscoveredRepos(org = openhwgroup)
importRepository(52028445,openhw-group/github-backup/backup-20220412,http://localhost)
importRepository(118652371,openhw-group/github-backup/backup-20220412,http://localhost)
importRepository(153259725,openhw-group/github-backup/backup-20220412,http://localhost)
importRepository(188901413,openhw-group/github-backup/backup-20220412,http://localhost)
importRepository(197242563,openhw-group/github-backup/backup-20220412,http://localhost)
importRepository(206841504,openhw-group/github-backup/backup-20220412,http://localhost)
importRepository(224885491,openhw-group/github-backup/backup-20220412,http://localhost)
importRepository(224904153,openhw-group/github-backup/backup-20220412,http://localhost)
importRepository(238013358,openhw-group/github-backup/backup-20220412,http://localhost)
importRepository(257965812,openhw-group/github-backup/backup-20220412,http://localhost)
importRepository(261766165,openhw-group/github-backup/backup-20220412,http://localhost)
importRepository(262127046,openhw-group/github-backup/backup-20220412,http://localhost)
importRepository(271641901,openhw-group/github-backup/backup-20220412,http://localhost)
importRepository(274802727,openhw-group/github-backup/backup-20220412,http://localhost)
importRepository(287073308,openhw-group/github-backup/backup-20220412,http://localhost)
importRepository(287224035,openhw-group/github-backup/backup-20220412,http://localhost)
importRepository(293831036,openhw-group/github-backup/backup-20220412,http://localhost)
importRepository(310614811,openhw-group/github-backup/backup-20220412,http://localhost)
importRepository(312659750,openhw-group/github-backup/backup-20220412,http://localhost)
importRepository(332727847,openhw-group/github-backup/backup-20220412,http://localhost)
importRepository(337370762,openhw-group/github-backup/backup-20220412,http://localhost)
importRepository(337477204,openhw-group/github-backup/backup-20220412,http://localhost)
importRepository(339651955,openhw-group/github-backup/backup-20220412,http://localhost)
importRepository(339652021,openhw-group/github-backup/backup-20220412,http://localhost)
importRepository(341497701,openhw-group/github-backup/backup-20220412,http://localhost)
importRepository(341502497,openhw-group/github-backup/backup-20220412,http://localhost)
importRepository(344473829,openhw-group/github-backup/backup-20220412,http://localhost)
importRepository(346717591,openhw-group/github-backup/backup-20220412,http://localhost)
importRepository(352510400,openhw-group/github-backup/backup-20220412,http://localhost)
importRepository(364263635,openhw-group/github-backup/backup-20220412,http://localhost)
importRepository(371703637,openhw-group/github-backup/backup-20220412,http://localhost)
importRepository(375289726,openhw-group/github-backup/backup-20220412,http://localhost)
importRepository(379653247,openhw-group/github-backup/backup-20220412,http://localhost)
importRepository(413464491,openhw-group/github-backup/backup-20220412,http://localhost)
importRepository(443370833,openhw-group/github-backup/backup-20220412,http://localhost)
importRepository(450620243,openhw-group/github-backup/backup-20220412,http://localhost)
importRepository(452067146,openhw-group/github-backup/backup-20220412,http://localhost)
importRepository(453158731,openhw-group/github-backup/backup-20220412,http://localhost)
importRepository(457069453,openhw-group/github-backup/backup-20220412,http://localhost)
importRepository(466032244,openhw-group/github-backup/backup-20220412,http://localhost)
importRepository(468498579,openhw-group/github-backup/backup-20220412,http://localhost)
importRepository(472438946,openhw-group/github-backup/backup-20220412,http://localhost)
Found 2 groups (with a keep count of 1)
Processing group backup-20220412 for potential pruning
Skipping group backup-20220412 for pruning
Processing group backup-20220411 for potential pruning
Successes:
[
  "Successfully imported repository 'openhwgroup.org' to openhw group backup / github backup / backup-20220412 / openhwgroup.org(54)",
  "Successfully imported repository 'core-v-freertos' to openhw group backup / github backup / backup-20220412 / core-v-freertos(48)",
  "Successfully imported repository 'pulp_soc' to openhw group backup / github backup / backup-20220412 / pulp_soc(46)",
  "Successfully imported repository 'aws-codebuild-run-build' to openhw group backup / github backup / backup-20220412 / aws-codebuild-run-build(55)",
  "Successfully imported repository 'core-v-docs' to openhw group backup / github backup / backup-20220412 / core-v-docs(52)",
  "Successfully imported repository 'aws-infra' to openhw group backup / github backup / backup-20220412 / aws-infra(47)",
  "Successfully imported repository 'cva5' to openhw group backup / github backup / backup-20220412 / cva5(58)",
  "Successfully imported repository 'cva6' to openhw group backup / github backup / backup-20220412 / cva6(56)",
  "Successfully imported repository 'cv32e40s' to openhw group backup / github backup / backup-20220412 / cv32e40s(65)",
  "Successfully imported repository 'corev-elf-psabi-doc' to openhw group backup / github backup / backup-20220412 / corev-elf-psabi-doc(53)",
  "Successfully imported repository 'embdebug-target-core-v' to openhw group backup / github backup / backup-20220412 / embdebug-target-core-v(64)",
  "Successfully imported repository 'riscv-ovpsim-corev' to openhw group backup / github backup / backup-20220412 / riscv-ovpsim-corev(61)",
  "Successfully imported repository 'core-v-verif' to openhw group backup / github backup / backup-20220412 / core-v-verif(57)",
  "Successfully imported repository 'corev-gcc' to openhw group backup / github backup / backup-20220412 / corev-gcc(63)",
  "Successfully imported repository 'OpenHW-Admin-Tools' to openhw group backup / github backup / backup-20220412 / OpenHW-Admin-Tools(74)",
  "Successfully imported repository 'core-v-ide-cdt' to openhw group backup / github backup / backup-20220412 / core-v-ide-cdt(60)",
  "Successfully imported repository 'corev-binutils-gdb' to openhw group backup / github backup / backup-20220412 / corev-binutils-gdb(59)",
  "Successfully imported repository 'apb_interrupt_cntrl' to openhw group backup / github backup / backup-20220412 / apb_interrupt_cntrl(69)",
  "Successfully imported repository 'linux' to openhw group backup / github backup / backup-20220412 / linux(50)",
  "Successfully imported repository 'timer_unit' to openhw group backup / github backup / backup-20220412 / timer_unit(71)",
  "Successfully imported repository 'core-v-sw' to openhw group backup / github backup / backup-20220412 / core-v-sw(49)",
  "Successfully imported repository 'riscv_vm' to openhw group backup / github backup / backup-20220412 / riscv_vm(62)",
  "Successfully imported repository 'cve2' to openhw group backup / github backup / backup-20220412 / cve2(77)",
  "Successfully imported repository 'cva6-sdk' to openhw group backup / github backup / backup-20220412 / cva6-sdk(70)",
  "Successfully imported repository 'core-v-mcu-devkit' to openhw group backup / github backup / backup-20220412 / core-v-mcu-devkit(66)",
  "Successfully imported repository 'corev-llvm-project' to openhw group backup / github backup / backup-20220412 / corev-llvm-project(68)",
  "Successfully imported repository 'osdforum.org' to openhw group backup / github backup / backup-20220412 / osdforum.org(67)",
  "Successfully imported repository 'cv32e41p' to openhw group backup / github backup / backup-20220412 / cv32e41p(51)",
  "Successfully imported repository 'core-v-mcu' to openhw group backup / github backup / backup-20220412 / core-v-mcu(73)",
  "Successfully imported repository 'u-boot' to openhw group backup / github backup / backup-20220412 / u-boot(72)",
  "Successfully imported repository 'cv32e40p' to openhw group backup / github backup / backup-20220412 / cv32e40p(76)",
  "Successfully imported repository 'core-v-cores' to openhw group backup / github backup / backup-20220412 / core-v-cores(75)",
  "Successfully imported repository 'experimental-repo' to openhw group backup / github backup / backup-20220412 / experimental-repo(81)",
  "Successfully imported repository 'infra' to openhw group backup / github backup / backup-20220412 / infra(79)",
  "Successfully imported repository 'marketing' to openhw group backup / github backup / backup-20220412 / marketing(80)",
  "Successfully imported repository 'core-v-freertos-kernel' to openhw group backup / github backup / backup-20220412 / core-v-freertos-kernel(78)",
  "Successfully imported repository 'core-v-trace' to openhw group backup / github backup / backup-20220412 / core-v-trace(84)",
  "Successfully imported repository 'core-v-mcu-cli-test' to openhw group backup / github backup / backup-20220412 / core-v-mcu-cli-test(82)",
  "Successfully imported repository 'force-riscv' to openhw group backup / github backup / backup-20220412 / force-riscv(83)",
  "Successfully imported repository 'cv32e40x' to openhw group backup / github backup / backup-20220412 / cv32e40x(86)",
  "Successfully imported repository 'core-v-xif' to openhw group backup / github backup / backup-20220412 / core-v-xif(85)",
  "Successfully imported repository 'downloads.openhwgroup.org' to openhw group backup / github backup / backup-20220412 / downloads.openhwgroup.org(87)"
]
Errors:
[]
Run complete!

Edited Apr 12, 2022 by Martin Lowe

Update Gitlab sync + import to be modern

Merge request reports