Many websites or projects experiencing exponential growth face
distributed data storage issues. GlusterFS is an unified, poly-protocol,
scale-out filesystem, capable of serving PBs of data at lightning
speeds and turns common hardware into a high-performance scalable
storage solution.
In this tutorial, we will review a basic replication setup between two
nodes which allows instant synchronization of a specific directory, as
well their related content, permissions changes, etc. If certain terms
used are unfamiliar, please consult the the
GlusterFS documentation.
Disk Partitioning
For a replication setup, GlusterFS requires an identical disk partition present on each node. We will use
apollo and
chronos as nodes, with one GlusterFS volume and brick replicated across the nodes.
From my experience, the most important part of a GlusterFS setup is
planing ahead your disks partitioning. If you have a proper layout, it
will be very easy to create a designated GlusterFS volume group and
logical volume for each node.
We will use as example the following partitioning, present in both nodes:
# df -ah /mnt/gvol0
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/vg_gluster-lv_gvol0
10G 151M 9.2G 2% /mnt/gvol0
The GlusterFS volume naming conventions are:
- /<base directory>/<volume name>/<brick name>/brick
For Selinux compatibility, we will use
/mnt as base directory,
gvol0 as volume name and
brick0 as brick name. Create the volume path on each node:
# ls -lah /mnt/gvol0
total 28K
drwxr-xr-x. 4 root root 4.0K Sep 9 18:52 .
drwxr-xr-x. 3 root root 4.0K Sep 9 18:52 ..
drwx------. 2 root root 16K Sep 9 17:33 lost+found
# install -d -m 0755 /mnt/gvol0/brick0/brick
We are done with the initial setup, let's proceed to GlusterFS configuration.
GlusterFS Setup
Install GlusterFS into each node, by running the following commands:
# yum --enablerepo=axivo install glusterfs-server
# service rpcbind start
Enterprise Linux 6:
# chkconfig glusterd on
# service glusterd start
Enterprise Linux 7:
# systemctl enable glusterd.service
# systemctl start glusterd.service
Before probing the nodes, open the required firewall ports. GlusterFS uses the following ports:
- 111 (tcp and udp) - rpcbind
- 2049 (tcp) - nfs
- 24007 (tcp) - server daemon
- 38465:38469 (tcp) - nfs related services
- 49152 (tcp) - brick
We used the following
iptables configuration file, for each node:
# cat /etc/sysconfig/iptables-glusterfs
-A INPUT -m state --state NEW -m tcp -p tcp -s 192.168.1.0/24 --dport 111 -j ACCEPT
-A INPUT -m state --state NEW -m udp -p udp -s 192.168.1.0/24 --dport 111 -j ACCEPT
-A INPUT -m state --state NEW -m tcp -p tcp -s 192.168.1.0/24 --dport 2049 -j ACCEPT
-A INPUT -m state --state NEW -m tcp -p tcp -s 192.168.1.0/24 --dport 24007 -j ACCEPT
-A INPUT -m state --state NEW -m tcp -p tcp -s 192.168.1.0/24 --dport 38465:38469 -j ACCEPT
-A INPUT -m state --state NEW -m tcp -p tcp -s 192.168.1.0/24 --dport 49152 -j ACCEPT
If you use
Firewalld, add the following rules for each node:
# firewall-cmd --permanent --zone=public --add-rich-rule='rule family="ipv4" source address="192.168.1.0/24" port port="111" protocol="tcp" accept'
# firewall-cmd --permanent --zone=public --add-rich-rule='rule family="ipv4" source address="192.168.1.0/24" port port="111" protocol="udp" accept'
# firewall-cmd --permanent --zone=public --add-rich-rule='rule family="ipv4" source address="192.168.1.0/24" port port="2049" protocol="tcp" accept'
# firewall-cmd --permanent --zone=public --add-rich-rule='rule family="ipv4" source address="192.168.1.0/24" port port="24007" protocol="tcp" accept'
# firewall-cmd --permanent --zone=public --add-rich-rule='rule family="ipv4" source address="192.168.1.0/24" port port="38465-38469" protocol="tcp" accept'
# firewall-cmd --permanent --zone=public --add-rich-rule='rule family="ipv4" source address="192.168.1.0/24" port port="49152" protocol="tcp" accept'
You will need to add a port for each additional brick. Since we use only one brick, 49152 is sufficient.
Make sure each node name resolves properly in your DNS setup and probe each node:
[root@apollo ~]# gluster peer probe chronos
peer probe: success.
[root@chronos ~]# gluster peer probe apollo
peer probe: success.
On
apollo only, run the following command to create the replication volume:
[root@apollo ~]# gluster volume create gvol0 replica 2 {apollo,chronos}:/mnt/gvol0/brick0/brick
volume create: gvol0: success: please start the volume to access data
Breaking down the above command, we told GlusterFS to create a
replica
volume and keep a copy of the data on at least 2 bricks at any given
time. Since we only have two bricks, this means each server will house a
copy of the data. Lastly, we specify which nodes and bricks to use.
Verify the volume information and start the volume:
[root@apollo ~]# gluster volume info
Volume Name: gvol0
Type: Replicate
Volume ID: 2b9c2607-9569-48c3-9138-08fb5d8a213f
Status: Created
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: apollo:/mnt/gvol0/brick0/brick
Brick2: chronos:/mnt/gvol0/brick0/brick
[root@apollo ~]# gluster volume start gvol0
volume start: gvol0: success
We are done with the server setup, let's proceed to GlusterFS replication.
Replication Setup
We will use
/var/www/html as replication directory across the two nodes. Please make sure the directory
does not contain
any files. Once the directory is mounted as GlusterFS type, any
previous file present into directory will not be available anymore.
Execute the following commands, to mount
/var/www/html as GlusterFS type:
[root@apollo ~]# install -d -m 0755 /var/www/html
[root@apollo ~]# cat >> /etc/fstab << EOF
apollo:/gvol0 /var/www/html glusterfs defaults 0 0
EOF
[root@apollo ~]# mount -a
[root@apollo ~]# mount -l -t fuse.glusterfs
apollo:/gvol0 on /var/www/html type fuse.glusterfs (rw,default_permissions,allow_other,max_read=131072)
[root@chronos ~]# install -d -m 0755 /var/www/html
[root@chronos ~]# cat >> /etc/fstab << EOF
chronos:/gvol0 /var/www/html glusterfs defaults 0 0
EOF
[root@chronos ~]# mount -a
[root@chronos ~]# mount -l -t fuse.glusterfs
chronos:/gvol0 on /var/www/html type fuse.glusterfs (rw,default_permissions,allow_other,max_read=131072)
List the GlusterFS pool, to verify any node connectivity issues:
[root@apollo ~]# gluster pool list
UUID Hostname State
5cb470ae-3c88-46fa-8cb9-2dd29d26e104 chronos Connected
188975c7-6d69-472a-b421-641286551d28 localhost Connected
To test the GlusterFS replication, we installed Nginx into both nodes, created a file on
apollo and changed its content as well the file permissions randomly:
[root@apollo ~]# ls -lah /var/www/html
total 8K
drwxr-xr-x. 3 root root 4.0K Sep 9 19:38 .
drwxr-xr-x. 3 root root 4.0K Sep 9 19:38 ..
[root@apollo ~]# yum --enablerepo=axivo install nginx
[root@chronos ~]# yum --enablerepo=axivo install nginx
[root@apollo ~]# ls -lah /var/www/html
total 12K
drwxr-xr-x. 3 root root 4.0K Sep 9 20:01 .
drwxr-xr-x. 3 root root 4.0K Sep 9 19:38 ..
-rw-r--r--. 1 root root 535 Oct 11 2009 404.html
-rw-r--r--. 1 root root 543 Oct 11 2009 50x.html
-rw-r--r--. 1 root root 198 May 6 2006 favicon.ico
-rw-r--r--. 1 root root 528 Oct 11 2009 index.html
-rw-r--r--. 1 root root 377 May 6 2006 nginx.gif
[root@apollo ~]# vi /var/www/html/info.php
[root@chronos ~]# chown nginx /var/www/html/info.php
[root@apollo ~]# cat /var/www/html/info.php
<?php
phpinfo();
The file was instantly replicated on
chronos, with identical content and permissions on both nodes:
[root@apollo ~]# ls -lah /var/www/html
total 13K
drwxr-xr-x. 3 root root 4.0K Sep 9 20:06 .
drwxr-xr-x. 3 root root 4.0K Sep 9 19:38 ..
-rw-r--r--. 1 root root 535 Oct 11 2009 404.html
-rw-r--r--. 1 root root 543 Oct 11 2009 50x.html
-rw-r--r--. 1 root root 198 May 6 2006 favicon.ico
-rw-r--r--. 1 root root 528 Oct 11 2009 index.html
-rw-r--r--. 1 nginx root 18 Sep 9 20:06 info.php
-rw-r--r--. 1 root root 377 May 6 2006 nginx.gif
[root@chronos ~]# cat /var/www/html/info.php
<?php
phpinfo();
You are currently running a high-performance, scalable replication system.
Troubleshooting
The logs are the best place to start your troubleshooting, examine any encountered errors into
/var/log/glusterfs/var-www-html.log file. Please don't turn off your firewall just because you think it is blocking your setup, the firewall adds an
important security layer and should
never be disabled. Instead, study the logs and find the source of problems.
An easy way to determine which ports are used in GlusterFS is by running a volume status check:
# gluster volume status
Status of volume: gvol0
Gluster process Port Online Pid
------------------------------------------------------------------------------
Brick apollo:/mnt/gvol0/brick0/brick 49152 Y 1220
Brick chronos:/mnt/gvol0/brick0/brick 49152 Y 1363
NFS Server on localhost 2049 Y 1230
Self-heal Daemon on localhost N/A Y 1235
NFS Server on chronos 2049 Y 1371
Self-heal Daemon on chronos N/A Y 1375
Task Status of Volume gvol0
------------------------------------------------------------------------------
There are no active volume tasks
You can also run
netstat -tulpn | grep gluster to examine further the used ports.
This tutorial covered an infinitesimal part of GlusterFS capabilities.
You need to understand that rushing through the tutorial without proper
understanding or reading the
documentation
will result in a failure. Once you understand how GlusterFS works, you
are welcome to ask any related questions into our support forums.
Typically, an administrator will want to change the state of a node instead of remove it (for details, see Changing node state).
