Using AWS data migration services,
how to migrate data to the cloud If you’re looking to move your data from an on-premises environment to the cloud, whether it’s data storage, processing, or analytics, AWS has several migration services that allow you to easily make this transition.
This guide will show you how to move all or part of your data from on-premises servers to AWS using AWS Data Migration Services and how to choose the right service for your needs.
For AWS Database Migration Service Set up your AWS account by following the steps
If you need help setting up your account, go through Setting up for AWS Database Migration Service. To use AWS DMS, you must set up an account on Amazon Web Services (AWS) and have a server available for migration.
Your source database instance will be created in that environment as part of your migration process. For best results when using AWS DMS, follow these guidelines:
Use at least two network interfaces in each instance; one interface should be dedicated to data transfer traffic only, while the other is used for normal operations or any necessary maintenance tasks.
This practice helps prevent interruptions of data transfer operations due to routine maintenance events on either interface.
To populate your source database and test replication Create your sample databases and an Amazon EC2 client
Create an Amazon VPC, create a Database in Your Amazon VPC, and Create an IAM User. Be sure that you can connect to your source database and that replication is working as expected.
Follow these steps:
1) Configure your data-migration client so that it can access a database on your local network.
2) Populate one of your test databases with sample data.
3) Make sure you can replicate data from one test database to another through MySQL replication.
4) Delete any existing tables in your migration target’s test database using this statement: DROP TABLE IF EXISTS `table name`;
5) Copy the entire contents of your source database’s table into the migration target’s table using this statement: COPY `source database`. `table name` (TRANSFORM DATABASE `target database` DEFAULT CHARSET UTF8mb4 USING latin1 COMMENT ‘test’)
6) Validate that replication worked by querying both databases and looking for differences.
7) Try inserting a row into both databases simultaneously to ensure they are synchronized.
Using a sample database creation script Populate your source database
The first step in migrating data is to get a copy of your source database. This should be relatively straightforward and, if you’re using a common source database platform (like MySQL or PostgreSQL), there are likely plenty of sample scripts available online that you can use.
The only real issue you might run into is with schemas. If your source database uses a different schema than your target database, you’ll need to develop a workaround plan here.
Sometimes this means exporting your data as JSON, doing some post-processing on it, and then importing it back into the new database.
Sometimes this also means writing custom migration code to make sure things work properly at all times.
To convert the schema from the source database to the target database Use the AWS Schema Conversion Tool (AWS SCT)
Migrating Schema and Use Cases. If you have large databases, use parallel queries to ensure that your workload is distributed across all of your available instances.
For information about parallel queries in Amazon Redshift, see Making Backups for Performance or Capacity Improvements.
After you migrate data from an on-premises database to Amazon Redshift, you should review any rows that were skipped due to errors.
The rows are typically skipped because they are longer than 16KB in size, they are larger than 2GB (in some cases), or they contain unsupported binary objects such as executables.
To perform all the processes for the migration, Create a replication instance.
Designing a Replication Solution and Creating a Replication Instance. You can use your existing instance as long as it is running Red Hat Enterprise Linux 5 or later.
For more information, see Requirements for Replication. Important: In most cases, Amazon S3 access credentials don’t change when you migrate data to another region through Data Migration services.
However, if you have configured your replication instance using IAM roles, you must modify your roles or create new ones before you begin migrating data.
Your applications can continue using IAM credentials while transferring data by creating temporary security credentials with limited permissions and associating them with an IAM role that has sufficient permissions for replication tasks.
Specify the source and target database endpoints.
Specify a port for each target endpoint. Note that if you are migrating data from a SQL Server database, you must use TCP/IP port 1433 for at least one of your targets.
Otherwise, you’ll receive an error during the migration setup. Specify the maximum number of concurrent migrations per endpoint.
The default is 4 concurrent migrations per endpoint. If you’re migrating multiple databases on a single instance, it may be helpful to set this higher. The maximum is 10 concurrent migrations per endpoint.:
Specify your credentials for connecting to the S3 bucket containing your data files:
Select the type of objects in your bucket: Objects with column-based storage or Objects with row-based storage
By running queries on the target database, verify that replication is working.
If you’ve set up replication using a read replica, run queries on your target database to ensure its receiving updates.
Query your source database directly and confirm that results return in an expected time frame. If you’re reading from an RDS instance, query any snapshots associated with that particular RDS instance; make sure these snapshots reflect recent data changes that took place after the initial setup.
If you have a recovery model other than a simple setting on your source database, or if you have multiple databases replicating from each other, verify that all schemas are replicated correctly and in their entirety.
If not, either adjust your replication settings or ask support for help troubleshooting any issues. Once verification is complete, migrate your source database.:
To do this, log into the console of your new database server and select Create Database from Scratch. You’ll be prompted to name the new DB (your choice) and to select one of two options:
Choose Full Recovery Model which will recreate all objects except those you manually specify as excluded; choose Simple Recovery Model which will create only those objects that were specified as excluded during setup.
Next, choose your desired collation (Latin1_General_CI_AS – default), language (US English – default), UTC Offset (Eastern Standard Time – default), and file format (.gdb).
You can also choose between UTF-8 or ASCII encoding schemes if necessary.