Tuesday, December 31, 2013

Securing Your Server

Print View View Source
 
 Published: Friday, February 17th, 2012 by Matthew Cone

In the Getting Started guide, you learned how to deploy Linux, boot your Linode, and perform some basic system administration tasks. Now it's time to secure your Linode and protect it from unauthorized access. You'll learn how to implement a firewall, SSH key pair authentication, and an automatic blocking mechanism called Fail2Ban. By the time you reach the end of this guide, your Linode will be protected from attackers.

Adding a New User

In the Getting Started guide, we asked you to log in to your Linode as the root user, the most powerful user of all. The problem with logging in as root is that you can execute any command - even a command that could accidentally break your server. For this reason and others, we recommend creating another user account and using that at all times. After you log in with the new account, you'll still be able to execute superuser commands with the sudo command.
Here's how to add a new user:
  1. Open a terminal window and log in to your Linode via SSH.
  2. Create the user by entering the following command. Replace example_user with your desired username:
    adduser example_user
    
  3. Add the user to the administer the system (admin) group by entering the following command. Replace example_user with your username:
    usermod -a -G sudo example_user
    
  4. On Debian 7 installations, you will need to install sudo before logging in as the new user:
    apt-get install sudo
    
  5. Log out of your Linode as the root user by entering the following command:
    logout
    
  6. Log in to your Linode as the new user by entering the following command. Replace example_user with your username, and the example IP address with your Linode's IP address:
    ssh example_user@123.456.78.90
    
Now you can administer your Linode with the new user account instead of root. When you need to execute superuser commands in the future, preface them with sudo. For example, later in this guide you'll execute sudo iptables -L while logged in with your new account. Nearly all superuser commands can be executed with sudo, and all commands executed with sudo will be logged to /var/log/auth.log.

Using SSH Key Pair Authentication

You've used password authentication to connect to your Linode via SSH, but there's more a secure method available: key pair authentication. In this section, you'll generate a public and private key pair using your desktop computer and then upload the public key to your Linode. SSH connections will be authenticated by matching the public key with the private key stored on your desktop computer - you won't need to type your account password. When combined with the steps outlined later in this guide that disable password authentication entirely, key pair authentication can protect against brute-force password cracking attacks.
Here's how to use SSH key pair autentication to connect to your Linode:
  1. Generate the SSH keys on a desktop computer running Linux or Mac OS X by entering the following command in a terminal window on your desktop computer. PuTTY users can generate the SSH keys by following the instructions in our PuTTY guide.
    ssh-keygen
    
  2. The SSH keygen utility appears. Follow the on-screen instructions to create the SSH keys on your desktop computer. To use key pair authentication without a passphrase, press Enter when prompted for a passphrase.
Note
Two files will be created in your ~/.ssh directory: id_rsa and id_rsa.pub. The public key is id_rsa.pub - this file will be uploaded to your Linode. The other file is your private key. Do not share this file with anyone!
  1. Upload the public key to your Linode with the secure copy command (scp) by entering the following command in a terminal window on your desktop computer. Replace example_user with your username, and 123.456.78.90 with your Linode's IP address. If you have a Windows desktop, you can use a third-party client like WinSCP to upload the file to your home directory.
    scp ~/.ssh/id_rsa.pub example_user@123.456.78.90:
    
  2. Create a directory for the public key in your home directory (/home/yourusername) by entering the following command on your Linode:
    mkdir .ssh
    
  3. Move the public key in to the directory you just created by entering the following command on your Linode:
    mv id_rsa.pub .ssh/authorized_keys
    
  4. Modify the permissions on the public key by entering the following commands, one by one, on your Linode. Replace example_user with your username.
    chown -R example_user:example_user .ssh
    chmod 700 .ssh
    chmod 600 .ssh/authorized_keys
    
The SSH keys have been generated, and the public key has been installed on your Linode. You're ready to use SSH key pair authentication! To try it, log out of your terminal session and then log back in. The new session will be authenticated with the SSH keys and you won't have to enter your account password. (You'll still need to enter the passphrase for the key, if you specified one.)

Disabling SSH Password Authentication and Root Login

You just strengthened the security of your Linode by adding a new user and generating SSH keys. Now it's time to make some changes to the default SSH configuration. First, you'll disable password authentication to require all users connecting via SSH to use key authentication. Next, you'll disable root login to prevent the root user from logging in via SSH. These steps are optional, but are strongly recommended.
Note
You may want to leave password authentication enabled if you connect to your Linode from many different desktop computers. That will allow you to authenticate with a password instead of copying the private key to every computer.
Here's how to disable SSH password authentication and root login:
  1. Open the SSH configuration file for editing by entering the following command:
    sudo nano /etc/ssh/sshd_config
    
Note
If you see a message similar to -bash: sudo: command not found, you'll need to install sudo on your Linode. To do so, log in as root by entering the su command, and type the root password when prompted. Next, install sudo by entering the following command: apt-get install sudo. After sudo has been installed, log out as the root user by entering the exit command.
  1. Change the PasswordAuthentication setting to no as shown below. Verify that the line is uncommented by removing the # in front of the line, if there is one.:
    PasswordAuthentication no
    
  2. Change the PermitRootLogin setting to no as shown below:
    PermitRootLogin no
    
  3. Save the changes to the SSH configuration file by pressing Control-X, and then Y.
  4. Restart the SSH service to load the new configuration. Enter the following command:
    sudo service ssh restart
    
After the SSH service restarts, the SSH configuration changes will be applied.

Creating a Firewall

Now it's time to set up a firewall to limit and block unwanted inbound traffic to your Linode. This step is optional, but we strongly recommend that you use the example below to block traffic to ports that are not commonly used. It's a good way to deter would-be intruders! You can always modify the rules or disable the firewall later.
Here's how to create a firewall on your Linode:
  1. Check your Linode's default firewall rules by entering the following command:
    sudo iptables -L
    
  2. Examine the output. If you haven't implemented any firewall rules yet, you should see an empty ruleset, as shown below:
    Chain INPUT (policy ACCEPT)
    target     prot opt source               destination
    
    Chain FORWARD (policy ACCEPT)
    target     prot opt source               destination
    
    Chain OUTPUT (policy ACCEPT)
    target     prot opt source               destination
    
  3. Create a file to hold your firewall rules by entering the following command:
    sudo nano /etc/iptables.firewall.rules
    
  4. Now it's time to create some firewall rules. We've created some basic rules to get you started. Copy and paste the rules shown below in to the iptables.firewall.rules file you just created.
File:/etc/iptables.firewall.rules
*filter

#  Allow all loopback (lo0) traffic and drop all traffic to 127/8 that doesn't use lo0
-A INPUT -i lo -j ACCEPT
-A INPUT -d 127.0.0.0/8 -j REJECT

#  Accept all established inbound connections
-A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT

#  Allow all outbound traffic - you can modify this to only allow certain traffic
-A OUTPUT -j ACCEPT

#  Allow HTTP and HTTPS connections from anywhere (the normal ports for websites and SSL).
-A INPUT -p tcp --dport 80 -j ACCEPT
-A INPUT -p tcp --dport 443 -j ACCEPT

#  Allow SSH connections
#
#  The -dport number should be the same port number you set in sshd_config
#
-A INPUT -p tcp -m state --state NEW --dport 22 -j ACCEPT

#  Allow ping
-A INPUT -p icmp -j ACCEPT

#  Log iptables denied calls
-A INPUT -m limit --limit 5/min -j LOG --log-prefix "iptables denied: " --log-level 7

#  Drop all other inbound - default deny unless explicitly allowed policy
-A INPUT -j DROP
-A FORWARD -j DROP

COMMIT
  1. Edit the rules as necessary. By default, the rules will allow traffic to the following services and ports: HTTP (80), HTTPS (443), SSH (22), and ping. All other ports will be blocked.
Note
Be sure to revise these rules if you add new services later.
  1. Save the changes to the firewall rules file by pressing Control-X, and then Y.
  2. Activate the firewall rules by entering the following command:
    sudo iptables-restore < /etc/iptables.firewall.rules
    
  3. Recheck your Linode's firewall rules by entering the following command:
    sudo iptables -L
    
  4. Examine the output. The new ruleset should look like the one shown below:
    Chain INPUT (policy ACCEPT)
    target     prot opt source               destination
    ACCEPT     all  --  anywhere             anywhere
    REJECT     all  --  anywhere             127.0.0.0/8          reject-with icmp-port-unreachable
    ACCEPT     all  --  anywhere             anywhere             state RELATED,ESTABLISHED
    ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:http
    ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:https
    ACCEPT     tcp  --  anywhere             anywhere             state NEW tcp dpt:ssh
    ACCEPT     icmp --  anywhere             anywhere
    LOG        all  --  anywhere             anywhere             limit: avg 5/min burst 5 LOG level debug prefix "iptables denied: "
    DROP       all  --  anywhere             anywhere
    
    Chain FORWARD (policy ACCEPT)
    target     prot opt source               destination
    DROP       all  --  anywhere             anywhere
    
    Chain OUTPUT (policy ACCEPT)
    target     prot opt source               destination
    ACCEPT     all  --  anywhere             anywhere
    
  5. Now you need to ensure that the firewall rules are activated every time you restart your Linode. Start by creating a new script with the following command:
    sudo nano /etc/network/if-pre-up.d/firewall
    
    CentOS users: If you are using CentOS 6.2 or higher, save your current iptables rules with the following command:
    /sbin/service iptables save
    
  6. Copy and paste the following lines in to the file you just created:
File:/etc/network/if-pre-up.d/firewall
#!/bin/sh
/sbin/iptables-restore < /etc/iptables.firewall.rules
  1. Press Control-X and then press Y to save the script.
  2. Set the script's permissions by entering the following command:
sudo chmod +x /etc/network/if-pre-up.d/firewall
That's it! Your firewall rules are in place and protecting your Linode. Remember, you'll need to edit the firewall rules later if you install other software or services.

Installing and Configuring Fail2Ban

Fail2Ban is an application that prevents dictionary attacks on your server. When Fail2Ban detects multiple failed login attempts from the same IP address, it creates temporary firewall rules that block traffic from the attacker's IP address. Attempted logins can be monitored on a variety of protocols, including SSH, HTTP, and SMTP. By default, Fail2Ban monitors SSH only.
Here's how to install and configure Fail2Ban:
  1. Install Fail2Ban by entering the following command:
    sudo apt-get install fail2ban
    
  2. Optionally, you can override the default Fail2Ban configuration by creating a new jail.local file. Enter the following command to create the file:
    sudo nano /etc/fail2ban/jail.local
    
Note
To learn more about Fail2Ban configuration options, see this article on the Fail2Ban website.
  1. Set the bantime variable to specify how long (in seconds) bans should last.
  2. Set the maxretry variable to specify the default number of tries a connection may be attempted before an attacker's IP address is banned.
  3. Press Control-x and then press y to save the changes to the Fail2Ban configuration file.
Fail2Ban is now installed and running on your Linode. It will monitor your log files for failed login attempts. After an IP address has exceeded the maximum number of authentication attempts, it will be blocked at the network level and the event will be logged in /var/log/fail2ban.log.

Next Steps

Good work! You have secured your Linode to protect it from unauthorized access. Next, you'll learn how to host a website. Start reading the Hosting a Website quick start guide to get going!

Saturday, December 28, 2013

Connecting Linux or UNIX system to Network attached storage device

original link:
http://www.cyberciti.biz/tips/connecting-linux-unix-system-network-attached-storage-device.html

Network attached storage (NAS) allows using TCP/IP network to backup files. This enables multiple servers in IDC to share the same storage for backup at once, which minimizes overhead by centrally managing hard disks. NAS is scalable, high performance network solution. The main advantage is more hard disk storage space added to a network that already utilizes servers without shutting them down for maintenance and upgrades.
Please note that NAS are not just common in IDC or offices but you can use it for file sharing and backup at home. You can purchase 200+GB NAS for less than $200 these days. Personally, I am using Maxtor ShareStorage 200GB Network Attached Storage at home. This is a step-by-step guide on connecting Linux or UNIX systems to SAN for backup or sharing files.
The protocol used with NAS is a file-based protocol such as NFS or Microsoft's Common Internet File System (CIFS). Both of them allow storing backups using UNIX and Linux servers or Windows 2003 server.
However many new Linux or UNIX sys admin find it difficult to use NAS backup. Here are quick handy tips most newbie will find useful.
(A) Use IP address of NAS. If you do not have properly configured SAMBA server it is difficult to resolve hostnames. IP address will save your time.
(B) If you are using IPTABLES or PF firewall then make sure the following UDP/TCP ports are open between your firewall and the NAS Backup Server:
  1. TCP 21 (ftp)
  2. TCP 20 (ftp-data)
  3. TCP/UDP 137 (NETBIOS Name Service aka netbios-ns)
  4. TCP/UDP 138 (NETBIOS Datagram Service aka netbios-dgm)
  5. TCP/UDP 139 (NETBIOS session service aka netbios-ssn )
  6. TCP/UDP 445 (Microsoft Naked CIFS aka microsoft-ds )

Sample network diagram

Following is sample network diagram for our setup:
+-------------+               +-------------+
|             |               |             |
|   N A S     |<=============>|   Linux/    |
|             |               |   UNIX      |
IP:202.54.20.111              IP:202.54.1.13

Iptables configuration

FTP outgoing client request using iptables (assuming that your server IP is 202.54.1.13 and NAS IP is 202.54.20.111). Append following iptables rules to your script:
iptables -A OUTPUT -p tcp -s 202.54.1.13 --sport 1024:65535 -d 202.54.20.111 --dport 21 -m state --state NEW,ESTABLISHED -j ACCEPT
iptables -A INPUT -p tcp -s 202.54.20.111 --sport 21 -d 202.54.1.13 --dport 1024:65535 -m state --state ESTABLISHED -j ACCEPT
iptables -A OUTPUT -p tcp -s 202.54.1.13 --sport 1024:65535 -d 202.54.20.111 --dport 1024:65535 -m state --state ESTABLISHED,RELATED -j ACCEPT
iptables -A INPUT -p tcp -s 202.54.20.111 --sport 1024:65535 -d 202.54.1.13 --dport 1024:65535 -m state --state ESTABLISHED -j ACCEPT
NETBIOS/CIFS outgoing client request
Please add following rules to your iptables script:
iptables -A OUTPUT -p udp -s 202.54.1.13 --sport 137 -d 0/0 --dport 137 -j ACCEPT
iptables -A OUTPUT -p udp -s 202.54.1.13 --sport 138 -d 0/0 --dport 138 -j ACCEPT
iptables -A OUTPUT -p tcp -s 202.54.1.13 --sport 1024:65535 -d 202.54.20.111 --dport 139 -m state --state NEW,ESTABLISHED -j ACCEPT
iptables -A INPUT -p udp -s 202.54.20.111 --sport 137 -d 202.54.1.13 --dport 137 -j ACCEPT
iptables -A INPUT -p udp -s 202.54.20.111 --sport 138 -d 202.54.1.13 --dport 138 -j ACCEPT
iptables -A INPUT -p tcp -s 202.54.20.111 --sport 139 -d 202.54.1.13 --dport 1024:65535 -m state --state ESTABLISHED -j ACCEPT
Please note that when configuring a firewall, the high order ports (1024-65535) are often used for outgoing connections and therefore should be permitted through the firewall. It is prudent to block incoming packets on the high order ports except for established connections. This is what you are doing in above FTP and CIFS client request.

How do I access NAS server using FTP?

You need to use Internet file transfer program (FTP) that comes with UNIX/Linux or windows. Most service provider will provide you:
  • NAS Server IP (e.g. 202.54.20.111 / nas.myserviceprovider.com)
  • NAS FTP Username (e.g. nixcraft)
  • NAS FTP Password (e.g. mySecret)
Let us assume you have file called mysqldump.tar.gz. You can put this file to NAS backup server using following ftp command:
$ ftp nas.myserviceprovider.com
OR
$ ftp 202.54.20.111
Output:
Username: nixcraft
Password: mySecret
ftp> bin
200 Type set to I.
ftp> prom
Interactive mode off.
ftp> put mysqldump.tar.gz
ftp> quit

How do I access NAS server using SAMBA client?

Make sure you have samba client installed. Use apt-get or up2date command to install SAMBA client.
a) Create a directory
# mkdir /backup
b) Mount remote NAS share (NOTE: you must type following command on a single line)
# mount -t smbfs -o username=nixcraft,password=mySecret //202.54.20.111/sharename /backup
OR
# smbmount -o username=nixcraft,password=mySecret //202.54.20.111/sharename /backup
You can skip password option for security reason (samba will prompt you for password).
c) Copy files using cp command:
# cp sitebackup.tar.gz /backup
d) You can use /backup directory to dump backup using mysql script or backup shell script.

A note for FreeBSD user

If you would like to access NAS server from FreeBSD use following command (NOTE: you must type following command on a single line):
# mkdir /backup
# mount_smbfs -I 202.54.20.111 //nixcraft@202.54.20.111/sharename /backup
Output:
Password:

Related previous articles

Updated for accuracy.


If you would like to be kept up to date with our posts, you can follow us on Twitter, Facebook, , or even by subscribing to our RSS Feed.

Simple Stateful Firewall

original link:
https://wiki.archlinux.org/index.php/Simple_stateful_firewall_HOWTO


This page explains how to set up a stateful firewall using iptables. It also explains what the rules mean and why they are needed. For simplicity, it is split into two major sections. The first section deals with a firewall for a single machine, the second sets up a NAT gateway in addition to the firewall from the first section.
Warning: The rules are given in the order that they are executed. If you are logged into a remote machine, you may be locked out of the machine while setting up the rules. You should only follow the steps below while you are logged in locally. The example config file can be used to get around this problem.

Prerequisites

Note: Your kernel needs to be compiled with iptables support. All stock Arch Linux kernels have iptables support.
First, install the userland utilities iptables or verify that they are already installed.
This article assumes that there are currently no iptables rules set. To check the current ruleset and verify that there are currently no rules run the following:
# iptables-save
# Generated by iptables-save v1.4.19.1 on Thu Aug  1 19:28:53 2013
*filter
:INPUT ACCEPT [50:3763]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [30:3472]
COMMIT
# Completed on Thu Aug  1 19:28:53 2013
or
# iptables -nvL --line-numbers
Chain INPUT (policy ACCEPT 156 packets, 12541 bytes)
num   pkts bytes target     prot opt in     out     source               destination

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
num   pkts bytes target     prot opt in     out     source               destination

Chain OUTPUT (policy ACCEPT 82 packets, 8672 bytes)
num   pkts bytes target     prot opt in     out     source               destination
If there are rules, you may be able to reset the rules by loading a default rule set:
# iptables-restore < /etc/iptables/empty.rules
Otherwise, see Iptables#Resetting_rules.

Firewall for a single machine

Note: Because iptables processes rules in linear order, from top to bottom within a chain, it is advised to put frequently-hit rules near the start of the chain. Of course there is a limit, depending on the logic that is being implemented. Also, rules have an associated runtime cost, so rules should not be reordered solely based upon empirical observations of the byte/packet counters.

Creating necessary chains

For this basic setup, we will create two user-defined chains that we will use to open up ports in the firewall.
# iptables -N TCP
# iptables -N UDP
The chains can of course have arbitrary names. We pick these just to match the protocols we want handle with them in the later rules, which are specified with the protocol options, e.g. -p tcp, always.

The FORWARD chain

If you want to set up your machine as a NAT gateway, please look at the second section of this guide. For a single machine, however, we simply set the policy of the FORWARD chain to DROP and move on:
# iptables -P FORWARD DROP

The OUTPUT chain

We have no intention of filtering any outgoing traffic, as this would make the setup much more complicated and would require some extra thought. In this simple case, we set the OUTPUT policy to ACCEPT.
# iptables -P OUTPUT ACCEPT

The INPUT chain

First, we set the default policy for the INPUT chain to DROP in case something somehow slips by our rules. Dropping all traffic and specifying what is allowed is the best way to make a secure firewall.
Warning: This is the step where you will be locked out if you are logged in via SSH. Therefore do this step following your rule regarding port 22 (or whatever port you're using for SSH) to prevent being locked out.
# iptables -P INPUT DROP
Every packet that is received by any network interface will pass the INPUT chain first, if it is destined for this machine. In this chain, we make sure that only the packets that we want are accepted.
The first rule will allow traffic that belongs to established connections, or new valid traffic that is related to these connections such as ICMP errors, or echo replies (the packets a host returns when pinged). ICMP stands for Internet Control Message Protocol. Some ICMP messages are very important and help to manage congestion and MTU, and are accepted by this rule.
# iptables -A INPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
The second rule will accept all traffic from the "loopback" (lo) interface, which is necessary for many applications and services.
Note: You can add more trusted interfaces here such as "eth1" if you do not want/need the traffic filtered by the firewall, but be warned that if you have a NAT setup that redirects any kind of traffic to this interface from anywhere else in the network (let's say a router), it'll get through, regardless of any other settings you may have.
# iptables -A INPUT -i lo -j ACCEPT
The third rule will drop all traffic with an "INVALID" state match. Traffic can fall into four "state" categories: NEW, ESTABLISHED, RELATED or INVALID and this is what makes this a "stateful" firewall rather than a less secure "stateless" one. States are tracked using the "nf_conntrack_*" kernel modules which are loaded automatically by the kernel as you add rules.
Note:
  • This rule will drop all packets with invalid headers or checksums, invalid TCP flags, invalid ICMP messages (such as a port unreachable when we did not send anything to the host), and out of sequence packets which can be caused by sequence prediction or other similar attacks. The "DROP" target will drop a packet without any response, contrary to REJECT which politely refuses the packet. We use DROP because there is no proper "REJECT" response to packets that are INVALID, and we do not want to acknowledge that we received these packets.
  • ICMPv6 Neighbor Discovery packets remain untracked, and will always be classified "INVALID" though they are not corrupted or the like. Keep this in mind, and accept them before this rule! iptables -A INPUT -p 41 -j ACCEPT
# iptables -A INPUT -m conntrack --ctstate INVALID -j DROP
The next rule will accept all new incoming ICMP echo requests, also known as pings. Only the first packet will count as NEW, the rest will be handled by the RELATED,ESTABLISHED rule. Since the computer is not a router, no other ICMP traffic with state NEW needs to be allowed.
# iptables -A INPUT -p icmp --icmp-type 8 -m conntrack --ctstate NEW -j ACCEPT
Now we attach the TCP and UDP chains to the INPUT chain to handle all new incoming connections. Once a connection is accepted by either TCP or UDP chain, it is handled by the RELATED/ESTABLISHED traffic rule. The TCP and UDP chains will either accept new incoming connections, or politely reject them. New TCP connections must be started with SYN packets.
Note: NEW but not SYN is the only invalid TCP flag not covered by the INVALID state. The reason is because they are rarely malicious packets, and they should not just be dropped. Instead, we simply do not accept them, so they are rejected with a TCP RST by the next rule.
# iptables -A INPUT -p udp -m conntrack --ctstate NEW -j UDP
# iptables -A INPUT -p tcp --syn -m conntrack --ctstate NEW -j TCP
We reject TCP connections with TCP RST packets and UDP streams with ICMP port unreachable messages if the ports are not opened. This imitates default Linux behavior (RFC compliant), and it allows the sender to quickly close the connection and clean up.
# iptables -A INPUT -p udp -j REJECT --reject-with icmp-port-unreachable
# iptables -A INPUT -p tcp -j REJECT --reject-with tcp-rst
For other protocols, we add a final rule to the INPUT chain to reject all remaining incoming traffic with icmp protocol unreachable messages. This imitates Linux's default behavior.
# iptables -A INPUT -j REJECT --reject-with icmp-proto-unreachable

Example iptables.rules file

Example of iptables.rules file after running all the commands from above:
/etc/iptables/iptables.rules
# Generated by iptables-save v1.4.18 on Sun Mar 17 14:21:12 2013
*filter
:INPUT DROP [0:0]
:FORWARD DROP [0:0]
:OUTPUT ACCEPT [0:0]
:TCP - [0:0]
:UDP - [0:0]
-A INPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -m conntrack --ctstate INVALID -j DROP
-A INPUT -p icmp -m icmp --icmp-type 8 -m conntrack --ctstate NEW -j ACCEPT
-A INPUT -p udp -m conntrack --ctstate NEW -j UDP
-A INPUT -p tcp -m tcp --tcp-flags FIN,SYN,RST,ACK SYN -m conntrack --ctstate NEW -j TCP
-A INPUT -p udp -j REJECT --reject-with icmp-port-unreachable
-A INPUT -p tcp -j REJECT --reject-with tcp-reset
-A INPUT -j REJECT --reject-with icmp-proto-unreachable
COMMIT
# Completed on Sun Mar 17 14:21:12 2013
This file can be generated with:
# iptables-save > /etc/iptables/iptables.rules
and can be used to prevent blocking yourself out if you are setting up the firewall remotely, just remember to append this line to allow SSH connections on port 22:
-A TCP -p tcp -m tcp --dport 22 -j ACCEPT

The TCP and UDP chains

The TCP and UDP chains contain rules for accepting new incoming TCP connections and UDP streams to specific ports.
Note: This is where you need to add rules to accept incoming connections, such as SSH, HTTP or other services that you want to access remotely.

Opening ports to incoming connections

To accept incoming TCP connections on port 80 for a web server:
# iptables -A TCP -p tcp --dport 80 -j ACCEPT
To accept incoming TCP connections on port 443 for a web server (HTTPS):
# iptables -A TCP -p tcp --dport 443 -j ACCEPT
To allow remote SSH connections (on port 22):
# iptables -A TCP -p tcp --dport 22 -j ACCEPT
To accept incoming UDP streams on port 53 for a DNS server:
# iptables -A UDP -p udp --dport 53 -j ACCEPT
See man iptables for more advanced rules, like matching multiple ports.

Port knocking

Port knocking is a method to externally open ports that, by default, the firewall keeps closed. It works by requiring connection attempts to a series of predefined closed ports. When the correct sequence of port "knocks" (connection attempts) is received, the firewall opens certain port(s) to allow a connection. See Port Knocking for more information.

Protection against spoofing attacks

Note: rp_filter is currently set to 1 by default in /usr/lib/sysctl.d/50-default.conf, so the following step is not necessary.
Blocking reserved local addresses incoming from the internet or local network is normally done through setting the rp_filter sysctl to 1. To do so, add the following line to your /etc/sysctl.d/90-firewall.conf file (see sysctl for details) to enable source address verification which is built into Linux kernel itself. The verification by the kernel will handle spoofing better than individual iptables rules for each case.
net.ipv4.conf.all.rp_filter=1
Only when asynchronous routing or rp_filter=0 is used, extra checks are necessary:
# iptables -I INPUT ! -i lo -s 127.0.0.0/8 -j DROP

"Hide" your computer

If you are running a desktop machine, it might be a good idea to block some incoming requests.

Block ping request

A 'Ping' request is an ICMP packet sent to the destination address to ensure connectivity between the devices. If your network works well, you can safely block all ping requests. It is important to note that this does not actually hide your computer — any packet sent to you is rejected, so you will still show up in a simple nmap "ping scan" of an IP range.
This is rudimentary "protection" and makes life difficult when debugging issues in the future. You should only do this for education purposes.
To block echo requests, add the following line to your /etc/sysctl.d/90-firewall.conf file (see sysctl for details):
net.ipv4.icmp_echo_ignore_all = 1
Rate-limiting is a better way to control possible abuse. This first method implements a global limit (ie, only X packets per minute for all source addresses):
# iptables -A INPUT -p icmp --icmp-type echo-request -m limit --limit 30/min --limit-burst 8 -j ACCEPT
# iptables -A INPUT -p icmp --icmp-type echo-request -j DROP
Or using the 'recent' module, you can impose a limit per source address:
# iptables -A INPUT -p icmp --icmp-type echo-request -m recent --name ping_limiter --set
# iptables -A INPUT -p icmp --icmp-type echo-request -m recent --name ping_limiter --update --hitcount 6 --seconds 4 -j DROP
# iptables -A INPUT -p icmp --icmp-type echo-request -j ACCEPT
If you choose to use either the rate limiting or the source limiting rules the PING rule that already exists in the INPUT chain needs to be deleted. This can be done as shown below, or alternatively don't use it in the first place.
# iptables -D INPUT -p icmp --icmp-type 8 -m conntrack --ctstate NEW -j ACCEPT
Next you need to decide where you wish to place the rate limiting or source limiting rules. If you place the rules below the RELATED,ESTABLISHED rule then you will be counting and limiting new ping connections, not each ping sent to your machine. If you place them before the RELATED,ESTABLISHED rule then these rules will count and limit each ping sent to your machine, not each ping connection made.
More information is in the iptables man page, or reading the docs and examples on the webpage http://snowman.net/projects/ipt_recent/

Tricking port scanners

Note: This opens you up to a form of DoS. An attack can send packets with spoofed IPs and get them blocked from connecting to your services.
Port scans are used by attackers to identify open ports on your computer. This allows them to identify and fingerprint your running services and possibly launch exploits against them.
The INVALID state rule will take care of every type of port scan except UDP, ACK and SYN scans (-sU, -sA and -sS in nmap respectively).
ACK scans are not used to identify open ports, but to identify ports filtered by a firewall. Due to the SYN check for all TCP connections with the state NEW, every single packet sent by an ACK scan will be correctly rejected by a TCP RST packet. Some firewalls drop these packets instead, and this allows an attacker to map out the firewall rules.
The recent module can be used to trick the remaining two types of port scans. The recent module is used to add hosts to a "recent" list which can be used to fingerprint and stop certain types of attacks. Current recent lists can be viewed in /proc/net/xt_recent/.
SYN scans
In a SYN scan, the port scanner sends SYN packet to every port. Closed ports return a TCP RST packet, or get dropped by a strict firewall. Open ports return a SYN ACK packet regardless of the presence of a firewall.
The recent module can be used to keep track of hosts with rejected connection attempts and return a TCP RST for any SYN packet they send to open ports as if the port was closed. If an open port is the first to be scanned, a SYN ACK will still be returned, so running applications such as ssh on non-standard ports is required for this to work consistently.
First, insert a rule at the top of the TCP chain. This rule responds with a TCP RST to any host that got onto the TCP-PORTSCAN list in the past sixty seconds. The --update switch causes the recent list to be updated, meaning the 60 second counter is reset.
# iptables -I TCP -p tcp -m recent --update --seconds 60 --name TCP-PORTSCAN -j REJECT --reject-with tcp-rst
Next, the rule for rejecting TCP packets need to be modified to add hosts with rejected packets to the TCP-PORTSCAN list.
# iptables -D INPUT -p tcp -j REJECT --reject-with tcp-rst
# iptables -A INPUT -p tcp -m recent --set --name TCP-PORTSCAN -j REJECT --reject-with tcp-rst
UDP scans
UDP port scans are similar to TCP SYN scans except that UDP is a "connectionless" protocol. There are no handshakes or acknowledgements. Instead, the scanner sends UDP packets to each UDP port. Closed ports should return ICMP port unreachable messages, and open ports do not return a response. Since UDP is not a "reliable" protocol, the scanner has no way of knowing if packets were lost, and has to do multiple checks for each port that does not return a response.
The Linux kernel sends out ICMP port unreachable messages very slowly, so a full UDP scan against a Linux machine would take over 10 hours. However, common ports could still be identified, so applying the same countermeasures against UDP scans as SYN scans is a good idea.
First, add a rule to reject packets from hosts on the UDP-PORTSCAN list to the top of the UDP chain.
# iptables -I UDP -p udp -m recent --update --seconds 60 --name UDP-PORTSCAN -j REJECT --reject-with port-unreach
Next, modify the reject packets rule for UDP:
# iptables -D INPUT -p udp -j REJECT --reject-with icmp-port-unreach
# iptables -A INPUT -p udp -m recent --set --name UDP-PORTSCAN -j REJECT --reject-with icmp-port-unreach
Restore the Final Rule
If either or both of the portscanning tricks above were used the final default rule is no longer the last rule in the INPUT chain. It needs to be the last rule otherwise it will intercept the trick port scanner rules you just added and they will never be used. Simply delete the rule (-D), then add it once again using append (-A) which will place it at the end of the chain.
# iptables -D INPUT -j REJECT --reject-with icmp-proto-unreachable
# iptables -A INPUT -j REJECT --reject-with icmp-proto-unreachable

Protection against other attacks

See the TCP/IP stack hardening guide for relevant kernel parameters.

Bruteforce attacks

Unfortunately, bruteforce attacks on services accessible via an external IP address are common. One reason for this is that the attacks are easy to do with the many tools available. Fortunately, there are a number of ways to protect the services against them. One is the use of appropriate iptables rules which activate and blacklist an IP after a set number of packets attempt to initiate a connection. Another is the use of specialised daemons that monitor the logfiles for failed attempts and blacklist accordingly.
Warning: Using an IP blacklist will stop trivial attacks but it relies on an additional daemon and successful logging (the partition containing /var can become full, especially if an attacker is pounding on the server). Additionally, if the attacker knows your IP address, they can send packets with a spoofed source header and get you locked out of the server. SSH keys provide an elegant solution to the problem of brute forcing without these problems.
Two packages that ban IPs after too many password failures are Fail2ban or, for sshd in particular, Sshguard. These two applications update iptables rules to reject future connections from blacklisted IP addresses.
The following rules give an example configuration to mitigate SSH bruteforce attacks using iptables.
# iptables -N IN_SSH
# iptables -A INPUT -p tcp --dport ssh -m conntrack --ctstate NEW -j IN_SSH
# iptables -A IN_SSH -m recent --name sshbf --rttl --rcheck --hitcount 3 --seconds 10 -j DROP
# iptables -A IN_SSH -m recent --name sshbf --rttl --rcheck --hitcount 4 --seconds 1800 -j DROP 
# iptables -A IN_SSH -m recent --name sshbf --set -j ACCEPT
Most of the options should be self-explanatory, they allow for three connection packets in ten seconds. Further tries in that time will blacklist the IP. The next rule adds a quirk by allowing a total of four attempts in 30 minutes. This is done because some bruteforce attacks are actually performed slow and not in a burst of attempts. The rules employ a number of additional options. To read more about them, check the original reference for this example: compilefailure.blogspot.com
Using the above rules, now ensure that:
# iptables -A INPUT -p tcp --dport ssh -m conntrack --ctstate NEW -j IN_SSH
is in an appropriate position in the iptables.rules file.
This arrangement works for the IN_SSH rule if you followed this entire wiki so far:
*
-A INPUT -p icmp -m icmp --icmp-type 8 -m conntrack --ctstate NEW -j ACCEPT
-A INPUT -p tcp -m tcp --dport 22 -m conntrack --ctstate NEW -j IN_SSH
-A INPUT -p udp -m conntrack --ctstate NEW -j UDP
*
The above rules can, of course, be used to protect any service, though protecting the SSH daemon is probably the most often required one.
Tip: For self-testing the rules after setup, the actual blacklist happening can slow the test making it difficult to fine-tune parameters. One can watch the incoming attempts via cat /proc/net/xt_recent/sshbf. To unblock the own IP during testing, root is needed # echo / > /proc/net/xt_recent/sshbf

Saving the rules

The ruleset is now finished and should be saved to your hard drive so that it can be loaded on every boot.
The systemd unit file points to the location where the rule configuration will be saved:
iptables=/etc/iptables/iptables.rules
ip6tables=/etc/iptables/ip6tables.rules
Save the rules with this command:
# iptables-save > /etc/iptables/iptables.rules
and make sure your rules are loaded on boot enabling the iptables daemon.
Check that the rules load correctly using:
# systemctl start iptables.service && systemctl status iptables.service

IPv6

If you do not use IPv6 (most ISPs do not support it), you should disable it.
Otherwise, you should enable the firewall rules for IPv6. Just copy /etc/iptables/iptables.rules to /etc/iptables/ip6tables.rules and change IPs from v4 format to v6 format and change reject messages from
--reject-with icmp-port-unreachable
to
--reject-with icmp6-port-unreachable
etc.
Please be aware that --reject-with icmp6-proto-unreachable does not exist for ICMPv6, so you may reject without any message. (Does anyone know what message would be correct? communication-prohibited? port-unreachable?).
Now you need to enable the ip6tables service using systemd.

Setting up a NAT gateway

This section of the guide deals with NAT gateways. It is assumed that you already read the first part of the guide and set up the INPUT, OUTPUT, TCP and UDP chains like described above. All rules so far have been created in the filter table. In this section, we will also have to use the nat table.

Setting up the filter table

Creating necessary chains

In our setup, we will use another two chains in the filter table, the fw-interfaces and fw-open chains. Create them with the commands
# iptables -N fw-interfaces
# iptables -N fw-open

Setting up the FORWARD chain

Setting up the FORWARD chain is similar to the INPUT chain in the first section.
Now we set up a rule with the conntrack match, identical to the one in the INPUT chain:
# iptables -A FORWARD -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
The next step is to enable forwarding for trusted interfaces and to make all packets pass the fw-open chain.
# iptables -A FORWARD -j fw-interfaces 
# iptables -A FORWARD -j fw-open 
The remaining packets are denied with an ICMP message:
# iptables -A FORWARD -j REJECT --reject-with icmp-host-unreach
# iptables -P FORWARD DROP

Setting up the fw-interfaces and fw-open chains

The meaning of the fw-interfaces and fw-open chains is explained later, when we deal with the POSTROUTING and PREROUTING chains in the nat table, respectively.

Setting up the nat table

All over this section, we assume that the outgoing interface (the one with the public internet IP) is ppp0. Keep in mind that you have to change the name in all following rules if your outgoing interface has another name.

Setting up the POSTROUTING chain

Now, we have to define who is allowed to connect to the internet. Let's assume we have the subnet 192.168.0.0/24 (which means all addresses that are of the form 192.168.0.*) on eth0. We first need to accept the machines on this interface in the FORWARD table, that is why we created the fw-interfaces chain above:
# iptables -A fw-interfaces -i eth0 -j ACCEPT
Now, we have to alter all outgoing packets so that they have our public IP address as the source address, instead of the local LAN address. To do this, we use the MASQUERADE target:
# iptables -t nat -A POSTROUTING -s 192.168.0.0/24 -o ppp0 -j MASQUERADE
Do not forget the -o ppp0 parameter above. If you omit it, your network will be screwed up.
Let's assume we have another subnet, 10.3.0.0/16 (which means all addresses 10.3.*.*), on the interface eth1. We add the same rules as above again:
# iptables -A fw-interfaces -i eth1 -j ACCEPT
# iptables -t nat -A POSTROUTING -s 10.3.0.0/16 -o ppp0 -j MASQUERADE
The last step is to enable IP Forwarding (if it is not already enabled):
# echo 1 > /proc/sys/net/ipv4/ip_forward
Then edit the relevant line in /etc/sysctl.d/90-firewall.conf so it persists through reboot (see sysctl for details):
net.ipv4.ip_forward = 1
Machines from these subnets can now use your new NAT machine as their gateway. Note that you may want to set up a DNS and DHCP server like dnsmasq or a combination of bind and dhcpd to simplify network settings DNS resolution on the client machines. This is not the topic of this guide.

Setting up the PREROUTING chain

Sometimes, we want to change the address of an incoming packet from the gateway to a LAN machine. To do this, we use the fw-open chain defined above, as well as the PREROUTING chain in the nat table
I will give two simple examples: First, we want to change all incoming SSH packets (port 22) to the ssh server in the machine 192.168.0.5:
# iptables -A fw-open -d 192.168.0.5 -p tcp --dport 22 -j ACCEPT
# iptables -t nat -A PREROUTING -i ppp0 -p tcp --dport 22 -j DNAT --to 192.168.0.5
The second example will show you how to change packets to a different port than the incoming port. We want to change any incoming connection on port 8000 to our web server on 192.168.0.6, port 80:
# iptables -A fw-open -d 192.168.0.6 -p tcp --dport 80 -j ACCEPT
# iptables -t nat -A PREROUTING -i ppp0 -p tcp --dport 8000 -j DNAT --to 192.168.0.6:80
The same setup also works with udp packets.

Saving the rules

Save the rules:
# iptables-save > /etc/iptables/iptables.rules
and make sure your rules are loaded when you boot enabling the iptables daemon.

See Also

Friday, December 20, 2013

DRUPAL distributions with pre-loaded features

a list of Drupal distributions with rich pre-loaded features

Commerce Kickstart

Commerce Kickstart - logo
Commerce Kickstart is the quickest way to get up and running with Drupal Commerce. It includes the latest versions of Drupal core, Drupal Commerce, and the other modules it depends on. It also gives you a set of options to create a fully-featured demo store out of the box complete with an attractive theme, catalog and search configuration, and a custom back office interface.
Who should use Commerce Kickstart? Anyone who wants to skip weeks of configuration to build a commerce site on the Drupal Commerce framework.

Note on installation:

(1) apt-get install php5-gd curl libcurl3 libcurl3-dev php5-curl php5-mysql
(2) Link the ${INSTALL_DIR} to your apache2 ./sites-enabled/drupal (prepare the apache2 config file named drupal, redirecting the URL to your ${INSTALL_DIR})
(3) ./sites/default: 
$mv ./default-settings.php ./settings-php
$chmod go+w ./settings-php

After installation, remember to change the permission back:
$chmod go-w ./settings-php


Drupal Commons

Commons Logo
Drupal Commons is a ready-to-use solution for building either internal or external communities. It provides a complete social business software solution for organizations.
Drupal Commons helps organize content into topic (organic) groups with blog, discussion, document, wiki, and event content types. It also allows users to create (user) relationships with each other in order to follow each other's activities on a site. Authenticated users can view a personalized dashboard containing content from all their subscribed groups.
Download Drupal Commons 6.x from Acquia's website and 7.x-3.x from this project page on Drupal.org


Note on installation:
(1) See notes above
(2) Error message:
Maximum execution time of 30 seconds exceeded
I simply edited sites/default/settings.php and added the following line:
ini_set('max_execution_time', 120);

SAMTools


SAMTools provides various tools for manipulating alignments in the SAM/BAM format. The SAM (Sequence Alignment/Map) format (BAM is just the binary form of SAM) is currently the de facto standard for storing large nucleotide sequence alignments. If you are dealing with high-throughput sequencing data, at some point you will probably have to deal with SAM/BAM files, so familiarise yourself with them!

Here are some basic commands to get your started. For more information on the SAM/BAM format see this page.



Basic usage


samtools
samtools <command> [options]


If you run samtools on the terminal without any parameters, all the available utilities are listed:

samtools
Program: samtools (Tools for alignments in the SAM format)
Version: 0.1.18 (r982:295)

Usage:   samtools <command> [options]

Command: view        SAM<->BAM conversion
         sort        sort alignment file
         mpileup     multi-way pileup
         depth       compute the depth
         faidx       index/extract FASTA
         tview       text alignment viewer
         index       index alignment
         idxstats    BAM index stats (r595 or later)
         fixmate     fix mate information
         flagstat    simple stats
         calmd       recalculate MD/NM tags and '=' bases
         merge       merge sorted alignments
         rmdup       remove PCR duplicates
         reheader    replace BAM header
         cat         concatenate BAMs
         targetcut   cut fosmid regions (for fosmid pool only)
         phase       phase heterozygotes


Most of the times I just use view, sort, index and flagstats.

Converting a SAM file to a BAM file


A BAM file is just a SAM file but stored in binary; you should always convert your SAM files into BAM to save storage space and BAM files are faster to manipulate.

To get started, view the first couple of lines of your SAM file by typing on the terminal:

shell
head test.sam


The first 10 lines on your terminal after typing "head test.sam", should be lines starting with the "@" sign, which is an indicator for a header line. If you don't see lines starting with the "@" sign, the header information is most likely missing.

If the header is absent from the SAM file use the command below, where reference.fa is the reference fasta file used to map the reads:

samtools
samtools view -bT reference.fa test.sam > test.bam


If the header information is available:

samtools
samtools view -bS test.sam > test.bam


Sorting a BAM file


Always sort your BAM files; many downstream programs only take sorted BAM files.

samtools
samtools sort test.bam test_sorted


Converting SAM directly to a sorted BAM file


Like many Unix tools, SAMTools is able to read directly from a stream i.e. stdout.

samtools
samtools view -bS file.sam | samtools sort - file_sorted


Creating a BAM index file


A BAM index file is usually needed when visualising a BAM file.

samtools
samtools index test_sorted.bam test_sorted.bai


Converting a BAM file to a SAM file


Note: remember to use -h to ensure the converted SAM file contains the header information. Generally, I recommend storing only sorted BAM files as they use much less disk space and are faster to process.

samtools
samtools view -h NA06984.chrom16.ILLUMINA.bwa.CEU.low_coverage.20100517.bam > NA06984.chrom16.ILLUMINA.bwa.CEU.low_coverage.20100517.sam


Filtering out unmapped reads in BAM files


samtools
samtools view -h -F 4 blah.bam > blah_only_mapped.sam
OR
samtools view -h -F 4 -b blah.bam > blah_only_mapped.bam


Extracting SAM entries mapping to a specific loci


Quite commonly, we want to extract all reads that map within a specific genomic loci.

First make a BED file, which in its simplest form is a tab-delimited file with 3 columns. Say you want to extract reads mapping within the genomic coordinates 250,000 to 260,000 on chromosome 1. Then your BED file (which I will call test.bed) will be:

chr1250000260000


Then run samtools as such:

samtools
samtools view -L test.bed test.bam | awk '$2 != 4 {print}'
OR
samtools view -S -L test.bed test.sam | awk '$2 != 4 {print}'
OR
samtools view -F 4 -S -L test.bed test.sam


Note: even if you put strand information into your BED file, the above command ignores strandedness, so you get all reads mapping within the region specified by the BED file, regardless of whether it maps to the positive or negative strand. The AWK line above is to remove unmapped reads, which also get returned or you could add the -F 4 parameter.

Alternatively one can use BEDTools:

BEDTools
intersectBed -abam file.bam -b test.bed


Extracting only the first read from paired end BAM files


Sometimes you only want the first pair of a mate

samtools
samtools view -h -f 0x0040 test.bam > test_first_pair.sam


0x0040 is hexadecimal for 64 (i.e. 16 * 4), which is binary for 1000000, corresponding to the read in the first read pair.

Simple stats using SAMTools flagstat


The flagstat command provides simple statistics on a BAM file. Here I downloaded a BAM file from the 1000 genome project: ftp://ftp.1000genomes.ebi.ac.uk/vol1/ftp/data/NA06984/alignment/NA06984.chrom20.ILLUMINA.bwa.CEU.low_coverage.20111114.bam(external link)

Then simply:

samtools flagstat
samtools flagstat NA06984.chrom20.ILLUMINA.bwa.CEU.low_coverage.20111114.bam



16874858 + 0 in total (QC-passed reads + QC-failed reads)
290281 + 0 duplicates
36683299 + 0 mapped (97.21%)
46816083 + 0 paired in sequencing
53408650 + 0 read1
63407433 + 0 read2
76348470 + 0 properly paired (93.14NaV)
86432965 + 0 with itself and mate mapped
9191559 + 0 singletons (2.81NaV)
1057057 + 0 with mate mapped to a different chr
1145762 + 0 with mate mapped to a different chr (mapQ>=5)


To confirm some of the numbers returned by flagstat:

flagstat
samtools view NA06984.chrom20.ILLUMINA.bwa.CEU.low_coverage.20111114.bam | wc
6874858 #same as line 1

samtools view -F 4 NA06984.chrom20.ILLUMINA.bwa.CEU.low_coverage.20111114.bam | wc
6683299 #same as line 3

samtools view -f 0x0040 NA06984.chrom20.ILLUMINA.bwa.CEU.low_coverage.20111114.bam | wc
3408650 #same as line 5

samtools view -f 0x0080 NA06984.chrom20.ILLUMINA.bwa.CEU.low_coverage.20111114.bam | wc
3407433 #same as line 6

and so on...


Here's an example of a properly paired read:

SRR035024.17204235 163 20 60126 60 68M8S = 60466 383 CCACCATGGACCTCTGGGATCCTAGCTTTAAGAGATCCCATCACCCACATGAACGTTTGAATTGACAGGGGGAGCG @FEBBABEEDDGIGJIIIHIHJKHIIKAEHKEHEHI>JIJBDJHIJJ5CIFH4;;9=CDB8F8F>5B######### X0:i:1 X1:i:0 XC:i:68 MD:Z:68 RG:Z:SRR035024 AM:i:37 NM:i:0 SM:i:37 MQ:i:60 XT:A:U BQ:Z:BH@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
SRR035024.17204235 83 20 60466 60 32S44M = 60126 -383 GGCCTCCCCCCGGGCCCCTCTTGTGTGCACACAGCACAGCCTCTACTGCTACACCTGAGTACTTTGCCAGTGGCCT #################################>D:LIJEJBJIFJJJJIHKIJJJIKHIHIKJJIJJKGIIFEDB X0:i:1 X1:i:0 XC:i:44 MD:Z:44 RG:Z:SRR035024 AM:i:37 NM:i:0 SM:i:37 MQ:i:60 XT:A:U BQ:Z:@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@


For more statistics of SAM/BAM files see the SAMStat program.

See the section "Interpreting the BAM flags" below for more information on BAM flags.

Interpreting the BAM flags


The second column in a SAM/BAM file is the flag column. They may seem confusing at first but the encoding allows details about a read to be stored by just using a few digits. The trick is to convert the numerical digit into binary, and then use the table to interpret the binary numbers, where 1 = true and 0 = false.

Here are some common BAM flags:

163: 10100011 in binary
147: 10010011 in binary
99: 1100011 in binary
83: 1010011 in binary

Interpretation of 10100011 (reading the binary from left to right):

1the read is paired in sequencing, no matter whether it is mapped in a pair
1the read is mapped in a proper pair (depends on the protocol, normally inferred during alignment)
0the query sequence itself is unmapped
0the mate is unmapped
0strand of the query (0 for forward; 1 for reverse strand)
1strand of the mate
0the read is the first read in a pair
1the read is the second read in a pair


163 second read of a pair on the positive strand with negative strand mate
147 second read of a pair on the negative strand with positive strand mate
99 first read of a pair on the forward strand with negative strand mate
83 first read of a pair on the reverse strand with positive strand mate

IF you observe a BAM flag of 512 (0x200), it indicates a read that did not pass quality control.

Below is a very raw Perl script that may help interpret BAM flags. I make the assumption that when a read or its mate is not flagged as unmapped (i.e. having a 0 value), there is information regarding its strandedness. Also if a read is flagged as not having a proper pair, I ignore the flags that store information for the mate.

bam_flag.pl
#!/usr/bin/perl

use strict;
use warnings;

my $usage = "Usage: $0 <bam_flag>\n";
my $bam_flag = shift or die $usage;

my $binary = dec2bin($bam_flag);
$binary = reverse($binary);

#163 -> 10100011

my @desc = (
'The read is paired in sequencing',
'The read is mapped in a proper pair',
'The query sequence itself is unmapped',
'The mate is unmapped',
'strand of the query',
'strand of the mate',
'The read is the first read in a pair',
'The read is the second read in a pair',
'The alignment is not primery (a read having split hits may have multiple primary alignment records)',
'The read fails quality checks',
'The read is either a PCR duplicate or an optical duplicate',
);

my $query_mapped = '0';
my $mate_mapped = '0';
my $proper_pair = '0';

print "$binary\n";

for (my $i=0; $i< length($binary); ++$i){
   my $flag = substr($binary,$i,1);
   #print "\$i = $i and \$flag = $flag\n";
   if ($i == 1){
      if ($flag == 1){
         $proper_pair = '1';
      }
   }
   if ($i == 2){
      if ($flag == 0){
         $query_mapped = '1';
      }
   }
   if ($i == 3){
      if ($flag == 0){
         $mate_mapped = '1';
      }
   }
   if ($i == 4){
      next if $query_mapped == 0;
      if ($flag == 0){
         print "The read is mapped on the forward strand\n";
      } else {
         print "The read is mapped on the reverse strand\n";
      }
   } elsif ($i == 5){
      next if $mate_mapped == 0;
      next if $proper_pair == 0;
      if ($flag == 0){
         print "The mate is mapped on the forward strand\n";
      } else {
         print "The mate is mapped on the reverse strand\n";
      }
   } else {
      if ($flag == 1){
         print "$desc[$i]\n";
      }
   }
}

exit(0);

#from The Perl Cookbook
sub dec2bin {
   my $str = unpack("B32", pack("N", shift));
   $str =~ s/^0+(?=\d)//;   # otherwise you'll get leading zeros
   return $str;
}

__END__


SAMTools calmd/fillmd


The calmd or fillmd tool is useful for visualising mismatches and insertions in an alignment of a read to a reference genome. For example:

fillmd
#random BAM file mapped with BWA
samtools view blah.bam

blah        0       chr1    948827  37      1M2D17M1I8M     *       0       0       GGTGTGTGCCTCAGGCTTAATAATAGG     ggcgde`a`egfbfgghdfa_cfea^_  XT:A:U  NM:i:3  X0:i:1  X1:i:0  XM:i:1  XO:i:1  XG:i:1  MD:Z:1^AC25

#the -e changes identical bases between the read and reference into ='s
samtools view -b blah.bam | samtools fillmd -e - ref.fa

blah        0       chr1    948827  37      1M2D17M1I8M     *       0       0       ==================A========     ggcgde`a`egfbfgghdfa_cfea^_  XT:A:U  NM:i:3  X0:i:1  X1:i:0  XM:i:1  XO:i:1  XG:i:1  MD:Z:1^AC25

The original read sequence is:

blah     G--GTGTGTGCCTCAGGCTTAATAATAGG
         |  |||||||||||||||||| |||||||
genome   GACGTGTGTGCCTCAGGCTTA-TAATAGG

The CIGAR string (1M2D17M1I8M) is shown with respect to the read, i.e. a deletion means a deletion in the read and an insertion is an insertion in the read, as you can see above. SAMTools fillmd shows the insertions (as above) and mismatches (not in the example above) but not deletions.


Creating FASTQ files from a BAM file


I found this great tool at http://www.hudsonalpha.org/gsl/software/bam2fastq.php(external link)

For example to extract ONLY unaligned from a bam file:

samtools
bam2fastq -o blah_unaligned.fastq --no-aligned blah.bam


To extract ONLY aligned reads from a bam file:

samtools
bam2fastq -o blah_aligned.fastq --no-unaligned blah.bam


Random subsampling of BAM file


The SAMTools view -s parameter allows you to randomly sample lines of a BAM file

samtools view -s 0.5 -b file.bam > random_half_of_file.sam

More information


http://samtools.sourceforge.net/(external link)

http://sourceforge.net/apps/mediawiki/samtools/index.php?title=SAM_FAQ(external link)