Trimigi cloud migration

Trimigi cloud migration is a tool which key motivation is to migrate hetrogenous databases (i.e. MySql to Mongo). For every datasource there is specific agent which knows how to import and export data from it.

How it works

For more info go to wiki

Project structure

• trimigi-core - shared core for migration agents. It handles core busines logic like transformations, conversions, rule sets, beam creation and so on
• trimigi-agent - concreate agent implementations
  • trimigi-agent-elasticsearch - concrete agent implementation for elasticsearch data source
  • trimigi-agent-mongo - concrete agent implementation for mongo data source
  • trimigi-agent-sql - concrete agent implememtation for sql data sources
  • trimigi-agent-datastore - concrete agent implememtation for GCP datastore data sources

Docker swarm setup

Recommended way of running agents is via docker. To setup docker follow these steps:

• download docker machine - https://docs.docker.com/machine/install-machine/#install-machine-directly
• create docker machine - https://docs.docker.com/machine/reference/create/ - give machine name 'morty'
• start docker machine - to start docker machine run following command:

# docker-machine start morty

• add record to /etc/hosts for your docker machine - #172.16.139.211 morty - to find out what is ip of your docker machine run following command:

# docker-machine ls

• create network 'migration' in docker - to add network run following command:

# docker network create --scope swarm -d overlay migration

• add docker testing stacks - go to 'docker' directory and run one of the following commands (depending of migrating stack):

SQL (postgres) to elasticsearch migration stack:

# docker stack deploy -c docker-compose-postgres.yml,docker-compose-elasticsearch.yml,docker-compose-agent-sql.yml,docker-compose-agent-elasticsearch.yml migration 

SQL (postgres) to mongo migration stack:

# docker stack deploy -c docker-compose-postgres.yml,docker-compose-mongo.yml,docker-compose-agent-sql.yml,docker-compose-agent-agent-mongo.yml migration 

• To run database (optionaly with its UI) run following:

# docker stack deploy -c docker-compose-elasticsearch.yml migration
# docker stack deploy -c docker-compose-mongo.yml migration
# docker stack deploy -c docker-compose-mysql.yml migration
# docker stack deploy -c docker-compose-postgres.yml migration

• To setup apache spark processor engine run following command

# docker stack deploy -c docker-compose-spark.yml migration

Exposed docker ports

SQL setup (postgress)

• DjdbcUrl=jdbc:postgresql://morty:5432/root
• DjdbcUsername=root
• DjdbcPassword=admin123
• DjdbcDriver=org.postgresql.Driver
• DmigrationTargetAgent=ELASTICSEARCH
• DmigrationTargetAgentUrl=http://morty:8081

Elasticsearch setup

• DelasticsearchHosts=http://morty:9200
• Dmicronaut.server.port=8081

Mongo setup

• DmongoUri=mongodb://root:admin123@morty:27017
• Dmicronaut.server.port=8082

Datastore setup (env variables)

• GOOGLE_CLOUD_PROJECT=${gcp-project-name}
• GOOGLE_APPLICATION_CREDENTIALS=${path-to-service-account.json} (optional, if not set - well known locations will be used to get service-account.json)

Maven build

To run agents locally you need to build it with special maven profile. Each agent has its own profile:

# mvn clean install -P sql
# mvn clean install -P elasticsearch
# mvn clean install -P mongo
# mvn clean install -P datastore

Maven release

To relase into maven central run following commands:

mvn release:prepare -P sql,elasticsearch,mongo,datastore -DskipTests
mvn release:perform -P sql,elasticsearch,mongo,datastore -DskipTests -Darguments="-Dmaven.javadoc.skip=true"

Docker build image

To build docker images run following command:

# mvn clean install -P sql,elasticsearch,mongo,datstore,docker-build-image

Docker push

To push docker images into docker hub run following command:

# mvn clean install -P sql,elasticsearch,mongo,datastore,docker-push

Wiki

To learn more about trimigi visit wiki