To copy a large number of files, or have large files over Linux system (scp/rsync/nc/tar over ssh) | scp alternatives

To copy/send large number of files or to copy/send large files in Linux below are sample method that we can send over other linux server.

Here we have create dummy file of 5 GB.

fallocate -l 5G testfile.img

Method1:

scp testfile rupin@192.168.43.114:/tmp

Method2:

rsync -avzh --progress --stats testfile rupin@192.168.43.114:/var/tmp

Method3:

tar -c testfile | gzip -2 | ssh server2 "cat > ~/file.tar.gz"

Method4:

using nc command

Dest: nc -l 2342 | tar xz
Source: tar cvz testfile | nc 192.168.43.114 2342

COMPARISON:

1. SCP:

root@ubuntu:~# time scp tesfile rupin@192.168.43.114:/tmp
testfile 100% 5120MB 23.9MB/s 03:34

real 3m34.710s
user 0m5.108s
sys 3m14.012s

2. scp with advanced options:

root@ubuntu:~# time scp -p -C -o 'CompressionLevel 9' -o 'IPQoS throughput' testfile rupin@192.168.43.114:tmp
testfile 100% 5120MB 25.5MB/s 03:21

real 3m22.145s
user 0m9.352s
sys 2m57.064s
root@ubuntu:~#

3. rysnc:

root@ubuntu:~# time rsync -avzh --progress --stats testfile rupin@192.168.43.114:/var/tmp
sending incremental file list
testfile
 5.37G 100% 70.07MB/s 0:01:13 (xfr#1, to-chk=0/1)

Number of files: 1 (reg: 1)
Number of created files: 1 (reg: 1)
Number of deleted files: 0
Number of regular files transferred: 1
Total file size: 5.37G bytes
Total transferred file size: 5.37G bytes
Literal data: 5.37G bytes
Matched data: 0 bytes
File list size: 0
File list generation time: 0.001 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 5.22M
Total bytes received: 35

sent 5.22M bytes received 35 bytes 70.08K bytes/sec
total size is 5.37G speedup is 1,028.27

real 1m14.322s
user 0m14.080s
sys 0m46.612s
root@ubuntu:~#

4.tar over ssh:

root@ubuntu:~# time tar -c testfile | gzip -2 | ssh rupin@192.168.43.114 "cat > ~/file.tar.gz"
rupin@192.168.43.114's password:

real 1m52.354s
user 0m10.048s
sys 1m27.836s
root@ubuntu:~#

5.nc command:
Dest: nc -l 2342 | tar xz
Source: tar cvz testfile | nc 192.168.43.114 2342

root@ubuntu:~# time tar cvz testfile | nc 192.168.43.114 2342
testfile

real 1m54.216s
user 0m10.736s
sys 1m29.952s
root@ubuntu:~#

As per above stat rsync is faster than other method.


You can also compare if file is corrupted/broken in transfer using checksum.

server:1

root@ubuntu:~# cksum testfile
3128462852 5368709120 testfile
root@ubuntu:~#

server:2

rupin@L687:/tmp$ cksum testfile
3128462852 5368709120 testfile
rupin@L687:/tmp$

 

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s