Empowering you with the knowledge to master Linux web hosting, DevOps and Cloud

 Linux Web Hosting, DevOps, and Cloud Solutions

Tag: Rsync

Limit /throttle rsync transfer speed

If you use the rsync utility to keep your backups synchronized between your servers or with a local machine, you might want to prevent the script from using too much bandwidth. However, rsync makes a lots of network I/O. The point of limiting bandwidth is to make sure your backup scripts don’t clog up the network connection.

Naturally, limiting the amount of bandwidth your backups are using is going to make them happen more slowly, but if you can deal with that, this is the way to do it.

 

Normal rsync command

rsync –avz -e ‘ssh’ /path/to/source user@remotehost:/path/to/dest/

What you’ll want to do is use the –bwlimit parameter with a KB/second value, like this:

rsync –bwlimit=<kb/second> –avz -e ‘ssh’ /path/to/source user@remotehost:/path/to/dest/

So if you wanted to limit transfer to around 10000KB/s (9.7MB/s), enter:

rsync –bwlimit=10000 –avz -e ‘ssh’ /path/to/source user@remotehost:/path/to/dest/

 

Example:-

rsync –bwlimit=10000 –avz -e ‘ssh’ /backup/ root@192.168.0.51:/backup/

Server migration

On occasion, you may want to move from one server to another. For example, you may be upgrading to a dedicated server or moving to a server with a different PHP deployment method. Server migration is a process in which data is moved from one server to another. Whatever the case, most people will agree that it is never a joyous event to switch server. Your sites will be down/unstable/inaccessible during the migration period and it could potentially lead to loss of sales. In this article, we are going to show you how you can migrate your site to a new server effortlessly and without any (or minimal) downtime. Below are the basic steps needs to be followed for a server migration with minimum downtime.

1. Prepare Your DNS

When you move your site(s) to a new server, you will likely get new IP addresses. The problem is that DNS servers cache domain name queries for as long as possible to provide the fastest response. Because of these DNS caching, once you change your IP, it can take up to three days before the changes propagate throughout the Internet. To keep this from happening, your first step in preparing for the migration is to change your DNS record TTL (Time To Live). This value designates how long your DNS entries should be cached. So the first thing you need to do before migrating is to lower the TTL to reasonable levels.

2. Set up Your New Server

Make sure your new server has the correct operating system installed and ready and that all hardware meets your applications specifications. Also setup the new IP addresses.

3. Tune Your Server

Check your file system and make sure your partitions are set up as you need them. Set up RAID if required. This is also the perfect time to implement any security practices within the OS and firewall (if installed). It’s important to complete these steps before you get too far because they’re much easier to do without content on the server.

4. Install Base Applications

Once you have installed the OS and setup the file system, you are ready to install the base applications, like the Web server, e-mail server, database server, PHP, etc.. Also verify the services are running fine on thr new server.

5. Begin Data Migration

Once the base applications are installed, you are ready to do the initial migration of your application data. This includes the database data, your application code files, mail server account configuration, logs, file caches, etc..

6. Bring your new server to life.

Once the data is migrated the server is configured to work everything correctly, you should change your sites domain DNS records to point to the new server address.

 

Data Migration

If you are using any Control Panels like cPanel, Plesk, etc., it rather easy to move data/accounts between servers as each control panel will give you option to move accounts.

cPanel Server to cPanel server migration

To migrate from one cPanel server to another, please do this:

1. Log into WHM as root on the new cPanel server

2. Click “Copy multiple accounts/packages from another server”

3. Fill in the required information like source server details and click “Fetch Account List”

You will then be given the option to choose which accounts and packages to transfer.

cPanel also does support migraring accounts from different control panel other than cPanel itself.

 

Server without a control panel

Rsync is a simple and very powerfull tools which can be used to transfer files between servers efficently.

First copy all the configurations files for the services like Webserver, Email server, etc. Then transfer the account data from old server.

Rsync example:-

#rsync -avz -e ssh /home root@remoteserverip:/home

-v: verbose, will tell you what file its on, how many left to check, etc.
-a: archive, will set most of the preferable options.
–progress: gives you per-file data transfer rates, and spinning progress bars if you’re into that sort of thing.
-z, –compress compress file data during the transfer

System files:-

Please note that we do not need to copy the server related files. We can exclude from the rync list. The most important excludes are the ones that reference hardware and system files that keep track of the server’s status while it’s running. That’s why directories like /boot, /proc, and /dev are essential excludes. The destination server should already have those directories and config files in place and they shouldn’t be overwritten.

Example
#rsync -avz -e ssh –exclude-from=”/home/exclude.txt” /home root@remoteserverip:/home

Finalizing the migration

Doing one pass got the more static files over to the new server. That’s good, but that may not be everything you needed to copy. An active database or a directory where a lot of files get created and deleted may have been copied but changed already. Or, more practically, they may have been in your exclude file.

To get those last files you’ll want to bring the services that use them down. That way you know the entire file will be migrated with no changes made after your last sync.

Edit your exclude file to remove references to the busy files you were holding off on then run the rsync command again.

Powered by WordPress & Theme by Anders Norén