Complete deployment

This documentation is only visible to GBIF staff.

1. Create VMs

Set IP addresses and reverse DNS in Route 53
Find appropriate capacity on the VM hosts
Create the VMs from the base Ubuntu 24.04 (or 22.04) image

create-ubuntu-24.04-vm --hostname uatstatic-vh --domainname gbif-uat.org --ip 130.225.43.204 --disk 40G --ram 2048 --cpu 1
create-ubuntu-24.04-vm --hostname uatnode-vh --domainname gbif-uat.org --ip 130.225.43.208 --disk 20G --ram 4096 --cpu 1
create-ubuntu-24.04-vm --hostname uatnodews-vh --domainname gbif-uat.org --ip 130.225.43.123 --disk 50G --cpu 4 --ram 24576
create-ubuntu-24.04-vm --hostname uatcrawler1-vh --domainname gbif-uat.org --ip 130.225.43.110 --disk 50G --ram 32768 --cpu 4
create-ubuntu-24.04-vm --hostname uatcrawler3-vh --domainname gbif-uat.org --ip 130.225.43.116 --disk 50G --ram 65536 --cpu 18
create-ubuntu-24.04-vm --hostname uatcrawler4-vh --domainname gbif-uat.org --ip 130.225.43.114 --disk 50G --ram 65536 --cpu 18
create-ubuntu-24.04-vm --hostname uathpws-vh --domainname gbif-uat.org --ip 130.225.43.120 --disk 20G --ram 4096 --cpu 4
create-ubuntu-24.04-vm --hostname uatregistrysearch-vh --domainname gbif-uat.org --ip 130.225.43.209 --disk 20G --cpu 2 --ram 10240
create-ubuntu-24.04-vm --hostname uatcmssearch-vh --domainname gbif-uat.org --ip 130.225.43.206 --disk 10G --cpu 2 --ram 6144

create-ubuntu-22.04-vm --hostname uatapicache-vh --domainname gbif-uat.org --ip 130.225.43.201 --disk 20G --ram 8192 --cpu 4
create-ubuntu-22.04-vm --hostname uatportalcache-vh --domainname gbif-uat.org --ip 130.225.43.202 --disk 10G --ram 8192 --cpu 2
create-ubuntu-22.04-vm --hostname uattile-vh --domainname gbif-uat.org --ip 130.225.43.203 --disk 20G --ram 8192 --cpu 2
create-ubuntu-22.04-vm --hostname uatws-vh --domainname gbif-uat.org --ip 130.225.43.207 --disk 20G --ram 57344 --cpu 6
create-ubuntu-22.04-vm --hostname uatthumbor-vh --domainname gbif-uat.org --ip 130.225.43.211 --disk 30G --ram 4096 --cpu 2
create-ubuntu-22.04-vm --hostname uattomcat9-vh --domainname gbif-uat.org --ip 130.225.43.106 --disk 30G --ram 8192 --cpu 2

2. Run Ansible setup

With the SSH certificate, there shouldn’t be any SSH "unknown host" warnings when connecting.

ansible-playbook -i inventories/gbif -v site.yml -l '~.*.gbif-uat.org'

Deploy the TLS certificates from aino.gbif.org using the script /etc/letsencrypt/renewal-hooks/post/deploy.sh.

3. Run C-Deploy installations and deployment

Run a "Deploy/verify base system" job with the GBIF Deployment plugin.

Deploy registry-ws then the other services in turn, or everything together, as preferred. Expect the tile Varnish server to produce Ansible errors until raster-basemap-server is deployed; this doesn’t affect anything else.

Check all the webservices exist in http://ws.gbif.org/.

Deploy the Base Map UI:

checkout/cd gbif-basemaps/basemap-ui
./make.sh

4. Deploy CLIs

On cli1, 2, 3 and 4

ssh cli1.gbif-uat.org
mkdir -p /etc/hadoop/conf/
sudo ln -s /home/crap/config/hdfs-site.xml /etc/hadoop/conf/hdfs-site.xml
sudo ln -s /home/crap/config/core-site.xml /etc/hadoop/conf/core-site.xml
sudo -u crap -s

git config --global user.name "GBIF Developer"
git config --global user.email informatics@gbif.org
git clone git@github.com:gbif/gbif-configuration.git _github
for i in bin config lib logs; do ln -s _github/cli/uat/$i .; done; ln -s _github/cli/common/util .; ln -s /mnt/auto/crawler storage

cd ~/lib
./update-all.sh

cd ~/bin
./stop-all -d '1 day'

(Don’t start them yet. The final command is to prevent alerts from stopped services.)

5. Build indexes

Registry ElasticSearch index

ssh cli1.gbif-uat.org
sudo -u crap -s

cd ~/util
./registry-batch-indexer

Content ElasticSearch index

Needs to be filled. How is this done?

A new token is created like this:

curl -s -X POST -H "Content-Type: application/x-www-form-urlencoded" -u "7514:pYtoV1OFQe2uMmV5" -d "grant_type=refresh_token&refresh_token=MSw0NjkyMTkxLDc1MTQsYWxsLCwsLDlmOTQ2MzBhLTU1ODUtM2VjMy05ODBkLTY1MTczYzkxNmFhZixub3QtdXNlZCxub3QtdXNlZCwwOGJmNDk5OTQ2NjYzMTQzNWEwYTJlYjU3YWFkOTg5ZjYyNThneHJxYiwxNjA2MjkxOTMzMjMxLDkyRHpxQ09ZdXZSX2RhMEZSM3plcTgtVlRZMA&redirect_uri=http://localhost/ignored" https://api.mendeley.com/oauth/token

6. Deploy Node applications

Portal16

Deploy the Human Verification images:

git clone https://github.com/gbif/human-verification
rsync -av human-verification/ uatnode-vh.gbif-uat.org:/var/lib/human-verification/

Set logging directory

mkdir -p /var/www/portal16/log
chown portal16: /var/www/portal16/log

Then deploy portal16 from Jenkins.

gbif-web

Deploy gbif-web from Jenkins.

7. Create Thumbor table

See command at https://github.com/gbif/thumbor_hbase/blob/gbif/README_GBIF.md