Category: Data Migration
-
Accessing MySQL database through a SSH Tunnel
A SSH tunnel links a local port to a port on a remote host. Communication between local and remote host is passed through SSH Tunnel to the remote port. The added benefit of this setup is that the communications between your local machine and the remote host is encrypted by the SSH connection. In this…
-
Monitor Kubernetes cluster with Prometheus and Grafana
Suppose you have a Kubernetes cluster up and running. How do you monitor it? Well obvious choice is to use Prometheus (gather metrics data of cluster) and Grafana (for visualising metrics data in a dashbaord ). Prerequisites working kubernetes cluster with kubectl configured locally installed kubectl and helm commands RBAC authorisation already setup Setting up…
-
How to copy BigQuery tables between locations with PDI
Suppose you have accidentally created a dataset in US region instead of EU. Your dataset has few tables with large amount of data. How do you copy or move data from one region to another. You could use command line tool bq cp: Unfortunately the copy command does not support cross region copies. You can…
-
Creating Cloud MySQL instance with Terraform
“Terraform is a tool for building, changing, and versioning infrastructure safely and efficiently. Terraform can manage existing and popular service providers as well as custom in-house solutions.” In this post, will show how to use Terraform to mange Google Cloud SQL resources. Prerequisites: Google Cloud Platform account Terraform installed on your machine (you can find…
-
Copy data from local MySQL database to Google Cloud MySQL
The main job performs two tasks: insert latest records into the target database update the target database with delta records PDI crashes if it tries to load more than few millions records at a single run. The approach is taken to overcome this limitation by processing data in smaller data chunks. How it works get…
-
Reconciliation between source and target tables
MySQL INFORMATION_SCHEMA database offers useful statistics for basic data reconciliation. PDI transformation to check data between source and target tables
-
Move data from MySQL to Google Cloud MySQL
Suppose you have a MySQL database running on a hosted VM or local environment. You need to copy data from each table to Google Cloud MySQL. Some of the tables are quite large. Here are the few steps you need to consider: Pre-Requisites Connection to Source & Target – JDBC driver & connection settings User…