Free Book: Pro Git

Book

The entire Pro Git book, written by Scott Chacon and published by Apress, is available here. All content is licensed under the Creative Commons Attribution Non Commercial Share Alike 3.0 license. Print versions of the book are available on Amazon.com.

1. 1. Getting Started

   1. 1.1 About Version Control
   2. 1.2 A Short History of Git
   3. 1.3 Git Basics
   4. 1.4 Installing Git
   5. 1.5 First-Time Git Setup
   6. 1.6 Getting Help
   7. 1.7 Summary

2. 2. Git Basics

   1. 2.1 Getting a Git Repository
   2. 2.2 Recording Changes to the Repository
   3. 2.3 Viewing the Commit History
   4. 2.4 Undoing Things
   5. 2.5 Working with Remotes
   6. 2.6 Tagging
   7. 2.7 Tips and Tricks
   8. 2.8 Summary

3. 3. Git Branching

   1. 3.1 What a Branch Is
   2. 3.2 Basic Branching and Merging
   3. 3.3 Branch Management
   4. 3.4 Branching Workflows
   5. 3.5 Remote Branches
   6. 3.6 Rebasing
   7. 3.7 Summary

4. 4. Git on the Server

   1. 4.1 The Protocols
   2. 4.2 Getting Git on a Server
   3. 4.3 Generating Your SSH Public Key
   4. 4.4 Setting Up the Server
   5. 4.5 Public Access
   6. 4.6 GitWeb
   7. 4.7 Gitosis
   8. 4.8 Gitolite
   9. 4.9 Git Daemon
   10. 4.10 Hosted Git
   11. 4.11 Summary

5. 5. Distributed Git

   1. 5.1 Distributed Workflows
   2. 5.2 Contributing to a Project
   3. 5.3 Maintaining a Project
   4. 5.4 Summary

6. 6. Git Tools

   1. 6.1 Revision Selection
   2. 6.2 Interactive Staging
   3. 6.3 Stashing
   4. 6.4 Rewriting History
   5. 6.5 Debugging with Git
   6. 6.6 Submodules
   7. 6.7 Subtree Merging
   8. 6.8 Summary

7. 7. Customizing Git

   1. 7.1 Git Configuration
   2. 7.2 Git Attributes
   3. 7.3 Git Hooks
   4. 7.4 An Example Git-Enforced Policy
   5. 7.5 Summary

8. 8. Git and Other Systems

   1. 8.1 Git and Subversion
   2. 8.2 Migrating to Git
   3. 8.3 Summary

9. 9. Git Internals

   1. 9.1 Plumbing and Porcelain
   2. 9.2 Git Objects
   3. 9.3 Git References
   4. 9.4 Packfiles
   5. 9.5 The Refspec
   6. 9.6 Transfer Protocols
   7. 9.7 Maintenance and Data Recovery
   8. 9.8 Summary

10. Index of Commands

rsync examples from samba.org

home FAQ resources features examples mailing lists		bug-tracking current issues and debugging download documentation search

rsync examples http://rsync.samba.org/examples.html If you have an interesting example of how you use rsync then please submit it to the rsync-bugs@samba.org for inclusion on this page. backup to a central backup server with 7 day incremental #!/bin/sh # This script does personal backups to a rsync backup server. You will end up # with a 7 day rotating incremental backup. The incrementals will go # into subdirectories named after the day of the week, and the current # full backup goes into a directory called "current" # tridge@linuxcare.com # directory to backup BDIR=/home/$USER # excludes file - this contains a wildcard pattern per line of files to exclude EXCLUDES=$HOME/cron/excludes # the name of the backup machine BSERVER=owl # your password on the backup server export RSYNC_PASSWORD=XXXXXX ######################################################################## BACKUPDIR=`date +%A` OPTS="--force --ignore-errors --delete-excluded --exclude-from=$EXCLUDES --delete --backup --backup-dir=/$BACKUPDIR -a" export PATH=$PATH:/bin:/usr/bin:/usr/local/bin # the following line clears the last weeks incremental directory [ -d $HOME/emptydir ] || mkdir $HOME/emptydir rsync --delete -a $HOME/emptydir/ $BSERVER::$USER/$BACKUPDIR/ rmdir $HOME/emptydir # now the actual transfer rsync $OPTS $BDIR $BSERVER::$USER/current backup to a spare disk I do local backups on several of my machines using rsync. I have an extra disk installed that can hold all the contents of the main disk. I then have a nightly cron job that backs up the main disk to the backup. This is the script I use on one of those machines. #!/bin/sh export PATH=/usr/local/bin:/usr/bin:/bin LIST="rootfs usr data data2" for d in $LIST; do mount /backup/$d rsync -ax --exclude fstab --delete /$d/ /backup/$d/ umount /backup/$d done DAY=`date "+%A"` rsync -a --delete /usr/local/apache /data2/backups/$DAY rsync -a --delete /data/solid /data2/backups/$DAY The first part does the backup on the spare disk. The second part backs up the critical parts to daily directories. I also backup the critical parts using a rsync over ssh to a remote machine. mirroring vger CVS tree The vger.rutgers.edu cvs tree is mirrored onto cvs.samba.org via anonymous rsync using the following script. #!/bin/bash cd /var/www/cvs/vger/ PATH=/usr/local/bin:/usr/freeware/bin:/usr/bin:/bin RUN=`lps x | grep rsync | grep -v grep | wc -l` if [ "$RUN" -gt 0 ]; then echo already running exit 1 fi rsync -az vger.rutgers.edu::cvs/CVSROOT/ChangeLog $HOME/ChangeLog sum1=`sum $HOME/ChangeLog` sum2=`sum /var/www/cvs/vger/CVSROOT/ChangeLog` if [ "$sum1" = "$sum2" ]; then echo nothing to do exit 0 fi rsync -az --delete --force vger.rutgers.edu::cvs/ /var/www/cvs/vger/ exit 0 Note in particular the initial rsync of the ChangeLog to determine if anything has changed. This could be omitted but it would mean that the rsyncd on vger would have to build a complete listing of the cvs area at each run. As most of the time nothing will have changed I wanted to save the time on vger by only doing a full rsync if the ChangeLog has changed. This helped quite a lot because vger is low on memory and generally quite heavily loaded, so doing a listing on such a large tree every hour would have been excessive. automated backup at home I use rsync to backup my wifes home directory across a modem link each night. The cron job looks like this #!/bin/sh cd ~susan { echo date dest=~/backup/`date +%A` mkdir $dest.new find . -xdev -type f \( -mtime 0 -or -mtime 1 \) -exec cp -aPv "{}" $dest.new \; cnt=`find $dest.new -type f | wc -l` if [ $cnt -gt 0 ]; then rm -rf $dest mv $dest.new $dest fi rm -rf $dest.new rsync -Cavze ssh . samba:backup } >> ~/backup/backup.log 2>&1 note that most of this script isn't anything to do with rsync, it just creates a daily backup of Susans work in a ~susan/backup/ directory so she can retrieve any version from the last week. The last line does the rsync of her directory across the modem link to the host samba. Note that I am using the -C option which allows me to add entries to .cvsignore for stuff that doesn't need to be backed up. Fancy footwork with remote file lists One little known feature of rsync is the fact that when run over a remote shell (such as rsh or ssh) you can give any shell command as the remote file list. The shell command is expanded by your remote shell before rsync is called. For example, see if you can work out what this does: rsync -avR remote:'`find /home -name "*.[ch]"`' /tmp/ note that that is backquotes enclosed by quotes (some browsers don't show that correctly).

A Tutorial on RSYNC

A Tutorial on Using

Last updated on November 20^th, 1999

Contents

1. Introduction
2. How Does it Work?
3. Setting up a Server
4. Using Rsync Itself
5. Rsync on the Net

Introduction

Rsync is a wonderful little utility that's amazingly easy to set up on your machines. Rather than have a scripted FTP session, or some other form of file transfer script -- rsync copies only the diffs of files that have actually changed, compressed and through ssh if you want to for security. That's a mouthful -- but what it means is:

• Diffs - Only actual changed pieces of files are transferred, rather than the whole file. This makes updates faster, especially over slower links like modems. FTP would transfer the entire file, even if only one byte changed.
• Compression - The tiny pieces of diffs are then compressed on the fly, further saving you file transfer time and reducing the load on the network.
• Secure Shell - The security concious of you out there would like this, and you should all be using it. The stream from rsync is passed through the ssh protocol to encrypt your session instead of rsh, which is also an option (and required if you don't use ssh - enable it in your /etc/inet.d and restart your inet daemon if you disabled it for security).

Rsync is rather versatile as a backup/mirroring tool, offering many features above and beyond the above. I personally use it to synchronize Website trees from staging to production servers and to backup key areas of the filesystems both automatically through cron and by a CGI script. Here are some other key features of rsync:

• Support for copying links, devices, owners, groups and permissions
• Exclude and exclude-from options similar to GNU tar
• A CVS exclude mode for ignoring the same files that CVS would ignore
• Does not require root privileges
• Pipelining of file transfers to minimize latency costs
• Support for anonymous or authenticated rsync servers (ideal for mirroring)

How does it work?

You must set up one machine or another of a pair to be an "rsync server" by running rsync in a daemon mode ("rsync --daemon" at the commandline) and setting up a short, easy configuration file (/etc/rsyncd.conf). Below I'll detail a sample configuration file. The options are readily understood, few in number -- yet quite powerful.

Any number of machines with rsync installed may then synchronize to and/or from the machine running the rsync daemon. You can use this to make backups, mirror filesystems, distribute files or any number of similar operations. Through the use of the "rsync algorithm" which transfers only the diffs between files (similar to a patch file) and then compressing them -- you are left with a very efficient system.

For those of you new to secure shell ("ssh" for short), you should be using it! There's a very useful and quite thourough Getting Started with SSH document available. You may also want to visit the Secure Shell Web Site. Or, just hit the Master FTP Site in Finland and snag it for yourself. It provides a secure, encrypted "pipe" for your network traffic. You should be using it instead of telnet, rsh or rlogin and use the replacement "scp" command instead of "rcp."

Setting up a Server

You must set up a configuration file on the machine meant to be a hosting server and run the rsync binary in daemon mode. Even your rsync client machines can run rsync in daemon mode for two-way transfers. You can do this automatically for each connection via the inet daemon or at the commandline in standalone mode to leave it running in the background for often repeated rsyncs. I personally use it in standalone mode, like Apache. I have a crontab entry that synchronizes a Web site directory hourly. Plus there is a CGI script that folks fire off frequently during the day for immediate updating of content. This is a lot of rsync calls! If you start off the rsync daemon through your inet daemon, then you incur much more overhead with each rsync call. You basically restart the rsync daemon for every connection your server machine gets! It's the same reasoning as starting Apache in standalone mode rather than through the inet daemon. It's quicker and more efficient to start rsync in standalone mode if you anticipate a lot of rsync traffic. Otherwise, for the occasional transfer follow the procedure to fire off rsync via the inet daemon. This way the rsync daemon, as small as it is, doesn't sit in memory if you only use it once a day or whatever. Your call.

Below is a sample rsync configuration file. It is placed in your /etc directory as rsyncd.conf.

motd file = /etc/rsyncd.motd
log file = /var/log/rsyncd.log
pid file = /var/run/rsyncd.pid
lock file = /var/run/rsync.lock

[simple_path_name]
   path = /rsync_files_here
   comment = My Very Own Rsync Server
   uid = nobody
   gid = nobody
   read only = no
   list = yes
   auth users = username
   secrets file = /etc/rsyncd.scrt

Various options that you would modify right from the start are the areas in italics in the sample above. I'll start at the top, line by line, and go through what you should pay attention to. What the sample above does is setup a single "path" for rsync transfers to that machine.

Starting at the top are four lines specifying files and their paths for rsync running in daemon mode. The first is a "message of the day" (motd) file like you would use for an FTP server. This file's contents get displayed when clients connect to this machine. Use it as a welcome, warning or simply identification. The next line specifies a log file to send diagnostic and norml run-time messages to. The PID file contains the "process ID" (PID) number of the running rsync daemon. A lock file is used to ensure that things run smoothly. These options are global to the rsync daemon.

The next block of lines is specific to a "path" that rsync uses. The options contained therein have effect only within the block (they're local, not global options). Start with the "path" name. It's somewhat confusing that rsync uses the term "path" -- as it's not necessarily a full pathname. It serves as an "rsync area nickname" of sorts. It's a short, easy to remember (and type!) name that you assign to a try filesystem path with all the options you specify. Here are the things you need to set up first and foremost:

• path - this is the actual filesystem path to where the files are rsync'ed from and/or to.
• comment - a short, descriptive explanation of what and where the path points to for listings.
• auth users - you really should put this in to restrict access to only a pre-defined user that you specify in the following secrets file - does not have to be a valid system user.
• secrets file - the file containing plaintext key/value pairs of usernames and passwords.

One thing you should seriously consider is the "hosts allow" and "hosts deny" options for your path. Enter the IPs or hostnames that you wish to specifically allow or deny! If you don't do this, or at least use the "auth users" option, then basically that area of your filesystem is wide open to the world by anyone using rsync! Something I seriously think you should avoid...

Check the rsyncd.conf man page with "man rsyncd.conf" and read it very carefully where security options are concerned. You don't want just anyone to come in and rsync up an empty directory with the "--delete" option, now do you?

The other options are all explained in the man page for rsyncd.conf. Basically, the above options specify that the files are chmod'ed to uid/gid, the filesystem path is read/write and that the rsync path shows up in rsync listings. The rsync secrets file I keep in /etc/ along with the configuration and motd files, and I prefix them with "rsyncd." to keep them together.

Using Rsync Itself

Now on to actually using, or initiating an rsync transfer with rsync itself. It's the same binary as the daemon, just without the "--daemon" flag. It's simplicity is a virtue. I'll start with a commandline that I use in a script to synchronize a Web tree below.

rsync --verbose  --progress --stats --compress --rsh=/usr/local/bin/ssh \
      --recursive --times --perms --links --delete \
      --exclude "*bak" --exclude "*~" \
      /www/* webserver:simple_path_name

Let's go through it one line at a time. The first line calls rsync itself and specifies the options "verbose," progress" and "stats" so that you can see what's going on this first time around. The "compress" and "rsh" options specify that you want your stream compressed and to send it through ssh (remember from above?) for security's sake.

The next line specifies how rsync itself operates on your files. You're telling rsync here to go through your source pathname recursively with "recursive" and to preserve the file timestamps and permissions with "times" and "perms." Copy symbolic links with "links" and delete things from the remote rsync server that are also deleted locally with "delete."

Now we have a line where there's quite a bit of power and flexibility. You can specify GNU tar-like include and exclude patterns here. In this example, I'm telling rsync to ignore some backup files that are common in this Web tree ("*.bak" and "*~" files). You can put whatever you want to match here, suited to your specific needs. You can leave this line out and rsync will copy all your files as they are locally to the remote machine. Depends on what you want.

Finally, the line that specifies the source pathname, the remote rsync machine and rsync "path." The first part "/www/*" specifies where on my local filesytem I want rsync to grab the files from for transmission to the remote rsync server. The next word, "webserver" should be the DNS name or IP address of your rsync server. It can be "w.x.y.z" or "rsync.mydomain.com" or even just "webserver" if you have a nickname defined in your /etc/hosts file, as I do here. The single colon specifies that you want the whole mess sent through your ssh tunnel, as opposed to the regular rsh tunnel. This is an important point to pay attention to! If you use two colons, then despite the specification of ssh on the commandline previously, you'll still go through rsh. Ooops. The last "www" in that line is the rsync "path" that you set up on the server as in the sample above.

Yes, that's it! If you run the above command on your local rsync client, then you will transfer the entire "/www/*" tree to the remote "webserver" machine except backup files, preserving file timestamps and permissions -- compressed and secure -- with visual feedback on what's happening.

Note that in the above example, I used GNU style long options so that you can see what the commandline is all about. You can also use abbreviations, single letters -- to do the same thing. Try running rsync with the "--help" option alone and you can see what syntax and options are available.

Rsync on the Net

You can find the rsync distribution at the rsync home page or hit the rsync FTP site.

There are also various pages of information on rsync out there, many of which reside on the rsync Web site. Below are three documents that you should also read thouroughly before using rsync so that you understand it well:

• Rsync man page
• Rsyncd.conf man page
• Rsync algorithm technical report

Samba, Nagios, xCAT, Git

Samba - opening windows to a wider world

Nagios - The Industry Standard in IT Infrastructure Monitoring

 

xCAT - Extreme Cloud Administration Toolkit

 

rsync examples from samba.org

 

Git

Provides documentation, tutorials and download links.

 

YoLinux LDAP Tutorial: Deploying OpenLDAP 2.x/1.2 LDAP Directory Server Installation and configuration http://www.yolinux.com/

---

Description: Lightweight Directory Access Protocol (LDAP) is a means of serving data on individuals, system users, network devices and systems over the network for e-mail clients, applications requiring authentication or information. The LDAP server is a means of providing a single directory source (with a redundant backup optional) for system information look-up and authentication. Using the LDAP server configuration example on this page will enable you to create an LDAP server to support email clients, web authentication, etc. We have many useful links for other LDAP deployments. LDAP can also be distributed in a hierarchical fashion but my examples refer to a single LDAP server. This tutorial will cover the setup and configuration of an LDAP server on Linux, the loading of data and use. Once configured, I recommend "gq" as an admin tool. (Note: Red Hat no longer ship with gq but it can still be downloaded and compiled manually.)
Simply put, this tutorial will enable you to create an LDAP server to which your e-mail clients (Outlook, Mozilla, Netscape, etc) can connect with their address books. It will allow one to search the LDAP database for people's e-mail addresses which are then pulled into the address list. Try it out with Thunderbird, Mozilla, Netscape or Outlook on our LDAP site ldap.yolinux.com for a demo. Cool eh! You can also try out authentication by pointing your application to authenticate at ldap.yolinux.com.

LDAP Server Tutorial Table of Contents: # LDAP e-Mail Clients: configuration etc # OpenLDAP Tutorial: Server Installation, Configuration, slapd, Loading LDIF data, Usage - (Creating a web accessible address book directory server.) # Quick Start Example and Test # Performance considerations # LDAP Database Backup # LDAP Manual pages # Berkeley BDB database # Links to other YoLinux LDAP tutorials # LDAP Links # LDAP Books

Securing and Optimizing Linux

RedHat Edition -A Hands on Guide

http://www.faqs.org/docs/securing/index.html

Gerhard Mourani

Open Network Architecture www.openna.com

             gmourani@openna.com
             gmourani@netscape.net
             

Madhu "Maddy"

Copyright © 2000 by Gerhard Mourani and OpenDocs, LLC.

Copyright © 2000 by Madhusudan (Madhu "Maddy") XML Source

---

Table of Contents

Preface

1. Why did i write this book?

2. Why fiddle?

3. DocBook !

4. DocBook/XML

1. Getting Started

1. Introduction

2. Installation

2. Overview of OS Linux

3. Installation of your Linux Server

4. Post-Install

3. Security, Optimization and Upgrade

5. General System Security

6. Linux General Optimization

7. Configuring and Building a Secure, Optimized Kernel

4. Networking -Management, Firewall, Masquerading and Forwarding

8. TCP/IP -Network Management

9. Files -Networking Functionality

10. Networking -Firewall

11. The firewall scripts files

12. Networking Firewall -Masquerading and Forwarding

5. Software -Security

13. Linux -The Compiler functionality

14. Software -Security/Monitoring

6. Software -Networking

15. Software -Securities

16. Software -Securties(commercial)

17. Software -Securities/System Integrity

18. Linux Tripwire ASR 1.3.1

19. Software -Securities/Management & Limitation

20. Set Limits using Qouta

21. Software -Networking

22. Software -Server/Mail Network

23. Linux IMAP & POP Server

24. Software -Networking/Encryption

25. Linux FreeS/WAN VPN

26. Linux OpenLDAP Server

27. Linux PostgreSQL Database Server

28. Software -Server/Proxy Network

29. Software -Network Server, web/Apache

30. Optional component to install with Apache

31. Software -Server/File Sharing-Network

32. Linux FTP Server

7. Backup and Restore

33. Why's and When's of Backup and Restore

I. Appendixes

A. Resources

B. Tweaks, Tips and Administration tasks

C. Obtaining Requests for Comments (RFCs)

List of Tables

3-1. Sample representaion of partitions

33-1. Dump scheme

List of Examples

3-1. Starting and Stopping various Daemon's

5-1. Export file systems using NFS

5-2. Disable console-equivalent access

5-3. Print log reports

5-4. Use man pages

5-5. Use find to find

6-1. For 128 MB of RAM

7-1. SMP support

8-1. Two ISA ethernet cards

12-1. rc.firewall.blocked

13-1. Using tar

15-1. Remote login using ssh

15-2. scp Secure Copy utility

15-3. local to remote

16-1. login to a remote using ssh2

16-2. sftp2, Secure File Transfer

18-1. Usage of Tripwire

19-1. Importing using gpg

19-2. Signing key

19-3. Encrypting

19-4. Decrypting

20-1. usrquota

20-2. grpquota

21-1. dnsquery

21-2. Look up host names

21-3. Using host

21-4. List a complete domain

22-1. Overriding RBL

22-2. Alternative names

22-3. sendmail.cf

26-1. my-data-file

26-2. LDMB backend

26-3. modifyentry

26-4. Address Book

30-1. Using Netscape browser

33-1. Backup directory of a week

33-2. scp SSH command

33-3. scp SSH command

 

solid2fastq convertors

A plethora of solid2fastq or csfasta convertors to fastq

I hadn't realised that there's an accumulation of prog/scripts to do the same task. Last count is 4 of these in my tool closet.


The C binary from bfast

solid2fastq 0.6.4a

Usage: solid2fastq [options]
        -c              produce no output.
        -n      INT     number of reads per file.
        -o      STRING  output prefix.
        -j              input files are bzip2 compressed.
        -z              input files are gzip compressed.
        -J              output files are bzip2 compressed.
        -Z              output files are gzip compressed.
        -t      INT     trim INT bases from the 3' end of the reads.
        -h              print this help message.

 send bugs to bfast-help@lists

solid2fastq.pl from bfast-0.6.4a
with notes in the script to refer to the above
# Author: Nils Homer
# Please see the C implementation of this script.


EDIT: THANKS to iceman for his reminder in the comments
"Make sure that you use the BWA's solid2fastq.pl if you are going to use BWA as it "double-encodes" the reads."

solid2fastq.pl from bwa-0.5.7
Usage: solid2fastq.pl

Note: is the string showed in the `# Title:' line of a
      ".csfasta" read file. Then F3.csfasta is read sequence
      file and F3_QV.qual is the quality file. If
      R3.csfasta is present, this script assumes reads are
      paired; otherwise reads will be regarded as single-end.

      The read name will be :panel_x_y/[12] with `1' for R3
      tag and `2' for F3. Usually you may want to use short
      to save diskspace. Long also causes troubles to maq.

# Author: lh3
# Note: Ideally, this script should be written in C. It is a bit slow at present.
# Also note that this script is different from the one contained in MAQ.

maq-0.7.1/scripts/solid2fastq.pl

Usage: solid2fastq.pl

Note: is the string showed in the `# Title:' line of a
      ".csfasta" read file. Then F3.csfasta is read sequence
      file and F3_QV.qual is the quality file. If
      R3.csfasta is present, this script assumes reads are
      paired; otherwise reads will be regarded as single-end.

      The read name will be :panel_x_y/[12] with `1' for F3
      tag and `2' for R3. Usually you may want to use short
      to save diskspace. Long also causes troubles to maq.

# Author: lh3
# Note: Ideally, this script should be written in C. It is a bit slow at present.