AsperaDownloads

Aspera is a company that provides the fasp protocol which uses UDP for file transfer and is much faster than FTP over TCP. The protocol is proprietary and the server costs money. However, NCBI and EBI provide Aspera servers and thus the contents of their FTP servers is also available over the much faster FASP protocol.

This page describes how to use the ascp program for fast file transfer of NCBI FTP, SRA, and EBI ENA data. The instructions are written for Linux.

Overview

The steps are as follows:

1. install the ascp client
2. install some scripts for easier access
3. profit!

Installing the Client

The client is available as a DEB or RPM package but I found it more convenient to get the browser plugin which contains the actual ascp binary.

You can get the software from the Aspera Download Site. The installer is a script that extracts the program to /home/${USER}/.aspera/connect. Let's move it to somewhere more appropriate:

$ mkdir -p ~/local $ mv ~/.aspera/connect ~/local/aspera

Getting the Scripts

Get the scripts from the attachment of this wiki page. These scripts allow you to directly use FTP urls from NCBI/EBI.

• ascp_scripts.tar.gz: ascp_scripts.tar.gz

Copy them to ~/local/bin.

$ mkdir -p ~/local/bin
$ tar xzf ascp_scripts.tar.gz
$ mv ascp_scripts/*.sh ~/local/bin

That's it!

Examples

Now, we can quickly download the nt BLAST databse, for example:

$ ncbi-get.sh ftp://ftp.ncbi.nlm.nih.gov/blast/db/nt.{00..14}.tar.gz
EXECUTING /home/takifugu/holtgrew/local/aspera/bin/ascp -QT -l 300m -i /home/takifugu/holtgrew/local/aspera/etc/asperaweb_id_dsa.putty anonftp@ftp.ncbi.nlm.nih.gov:/blast/db/nt.00.tar.gz anonftp@ftp.ncbi.nlm.nih.gov:/blast/db/nt.01.tar.gz anonftp@ftp.ncbi.nlm.nih.gov:/blast/db/nt.02.tar.gz anonftp@ftp.ncbi.nlm.nih.gov:/blast/db/nt.03.tar.gz anonftp@ftp.ncbi.nlm.nih.gov:/blast/db/nt.04.tar.gz anonftp@ftp.ncbi.nlm.nih.gov:/blast/db/nt.05.tar.gz anonftp@ftp.ncbi.nlm.nih.gov:/blast/db/nt.06.tar.gz anonftp@ftp.ncbi.nlm.nih.gov:/blast/db/nt.07.tar.gz anonftp@ftp.ncbi.nlm.nih.gov:/blast/db/nt.08.tar.gz anonftp@ftp.ncbi.nlm.nih.gov:/blast/db/nt.09.tar.gz anonftp@ftp.ncbi.nlm.nih.gov:/blast/db/nt.10.tar.gz anonftp@ftp.ncbi.nlm.nih.gov:/blast/db/nt.11.tar.gz anonftp@ftp.ncbi.nlm.nih.gov:/blast/db/nt.12.tar.gz anonftp@ftp.ncbi.nlm.nih.gov:/blast/db/nt.13.tar.gz anonftp@ftp.ncbi.nlm.nih.gov:/blast/db/nt.14.tar.gz .
nt.00.tar.gz                   100%  805MB  174Mb/s    00:40
nt.01.tar.gz                   100%  833MB  261Mb/s    01:09
nt.02.tar.gz                   100%  772MB  283Mb/s    01:33
nt.03.tar.gz                   100%  779MB  242Mb/s    02:00
nt.04.tar.gz                   100%  695MB  269Mb/s    02:22
nt.05.tar.gz                   100%  779MB  242Mb/s    02:50
nt.06.tar.gz                   100%  750MB  278Mb/s    03:15
nt.07.tar.gz                   100%  655MB  254Mb/s    03:35
nt.08.tar.gz                   100%  628MB  282Mb/s    03:56
nt.09.tar.gz                   100%  546MB  291Mb/s    04:14
nt.10.tar.gz                   100%  784MB  246Mb/s    04:40
nt.11.tar.gz                   100%  764MB  193Mb/s    05:04
nt.12.tar.gz                   100%  847MB  268Mb/s    05:30
nt.13.tar.gz                   100% 1136MB  292Mb/s    06:05
nt.14.tar.gz                   100%  743MB  283Mb/s    06:27
Completed: 11800016K bytes transferred in 387 seconds
 (249454K bits/sec), in 15 files.

For comparison, I get an average of 10M/s using wget via the FTP protocol from NCBI.

Comments