Saturday, August 25, 2012

NGS talks

Whole-Exome Sequencing to Identify Somatic Variants in Cancer - Yardena Samuels(from NHGRI)

Next Generation Sequencing

Determine genetic map position of SNP':
online tool
Conway Institute Bioinformatics Service

An overview of NGS technology and platforms:
http://www.seqwright.com

Blog on NGS technology (www.yuzuki.org) by Dale Yuzuki:

Next Generation Technologist 

How to remove duplicates reads using picard:


java -Xmx2g -jar /usr/local/picard-tools/MarkDuplicates.jar 
                            MAX_FILE_HANDLES_FOR_READ_ENDS_MAP=1000
                            METRICS_FILE=out.metrics 
                            REMOVE_DUPLICATES=true 
                            ASSUME_SORTED=true  
                            VALIDATION_STRINGENCY=LENIENT 
                            INPUT=input.sorted.bam 
                            OUTPUT=out.dedup.bam 
 
BAM file reheader trick (that works):
 
java -jar /home/zhengyuan/picard-tools-1.105/AddOrReplaceReadGroups.jar \
I=test.srt.bam O=test.srt.rhd.bam LB=wzy PL=illumina PU=wzy SM=wzy
 
samtools index  test.srt.rhd.bam
 
Then you can proceed for others tasks like GATK coverage analysis. 
 

NGS alignment software

http://lh3lh3.users.sourceforge.net/NGSalign.shtml

Software

Currently, this page only includes software I am familiar with. Most of them aim for aligning next-generation sequencing (NGS) data and were developed since 2007. I may extend the list when I have time. Several notes:
  • The programs are listed in the alphabet order in each category.
  • Features shown in brackets are optional and may affect efficiency.
  • The version number shown for each program is the one I have checked, but may not be the latest.

Indexing Reads with Hash Tables

  • CloudBurst [PMID:19357099]. RMAP-like algorithm that works in a cloud.
    • Platform: Illumina
    • Features: support cloud computation
    • Availability: open source
  • Cross_match [1.080730]. The latest cross_match has been substantially improved for short read alignment. Its speed is comparable to other aligners and might be the best choice for local alignment.
    • Platform: Illumina; 454
    • Features: gapped alignment (maximum 2 gaps in the fast mode); local alignment
    • Availability: academic free source codes
  • Eland [1.0]. Probably the first short read aligner. Eland substantially influences many aligners in this category and still outperforms many followers. Although it is not the fastest any more, it is close to the fastest and has the smallest memory footprint. Eland itself works for 32bp single-end reads only. Additional Perl scripts in GAPipeline extend its ability.
    • Platform: Illumina
    • Features: PET mapping; mapping quality; SNP caller; counting suboptimal occurrences.
    • Advantages: fast; light-weighted
    • Availability: free source codes for machine buyers.
  • MAQ [0.7.1, PMID: 18714091]. This is my program to align short reads and to call variants. It has been used in several high-profile papers.
    • Platform: Illumina; SOLiD (partial)
    • Features: PET mapping; quality aware; gapped alignment for PET; mapping quality; adapter trimming; partial occurrences counting; SNP caller
    • Advantages: feature rich; publication proved
    • Limitation: up to 128bp reads; no gapped alignment for single-end reads
    • Availability: GPL
  • mrFast/mrsFAST [0.5.1]. An aligner specifically designed for reporting all hits.
    • Platform: Illumina
    • Features: all hits; up to 3X faster than MAQ (not tested by myself); gapped alignment (mrFAST)
    • Availability: Free binary
  • RazerS [20081029, PMID: 19592482]. q-gram filteration; based on the SeqAn library.
    • Availability: free source codes
  • RMAP [0.41, PMID: 18307793;19736251]. One of the earliest short read aligners.
    • Platform: Illumina
    • Features: quality aware; [gapped alignment]; best unique hits
    • Availability: GPL
  • SeqMap [1.0.8, PMID: 18697769]. An Eland-like program.
    • Platform: Illumina
    • Features: [gapped alignment]
    • Limitation: not counting suboptimal hits
    • Availability: GPL
  • SHRiMP [1.10, PMID: 19461883]. Q-gram based algorithm.
    • Platform: SOLiD; Illumina; 454
    • Features: SOLiD mapping; gapped alignment; potential support for mapping quality
    • Limitations: a little slow
    • Availability: GPL
  • ZOOM [1.2.5, PMID: 18684737]. Eland-like algorithm with the improvement of using spaced seed. ZOOM supports longer reads and faster than Eland, although it uses more memory. ZOOM is feature rich, but some features may come at the cost of speed.
    • Platform: Illumina; SOLiD
    • Features: PET mapping; SOLiD mapping; [gapped alignment]; [mapping quality]; [quality aware]
    • Advantage: fast; feature rich
    • Limitation: up to 224bp reads; gapped alignment comes with cost
    • Availability: commercial

Indexing Genome with Hash Tables

  • BFAST [0.3.1, PMID: 19907642].
    • Platform: Illumina; SOLiD
    • Availability: open source
    • Comment on paper: evaluation for bowtie and bwa may be questionable.
  • gnumap [PMID: 19861355].
    • Platform: Illumina
    • Features: "Assigning a proportion of the read to relevant genomic matches based on the relative likelihood that the read maps to each location". (I do not know how this is compared with randomly distribute repetitive reads)
    • Comment on paper: It is possible to achieve paired-end mapping by indexing reads only. Maq does in this way.
    • Availability: open source (?)
  • MOM [0.1, PMID: 19228804].
    • Platform: Illumina; (?)
    • Features: counting suboptimal occurrences; local alignment
    • Availability: free
  • Mosaik [1.0]. Mosaik has been used in several high-profile publications and delivers good performance.
    • Platform: Illumina; 454; SOLiD
    • Advantages: long reads
    • Availability: open source
  • NovoAlign [2.0]. NovoAlign competes with MAQ on speed and feature set, and may be more accurate than MAQ. It also implements several important features missing in MAQ.
    • Platform: Illumina
    • Features: PET mapping; gapped alignment; mapping quality; quality aware; adapter trimming; MAQ format
    • Advantages: highly accurate; gapped alignment; feature rich
    • Requirements: >8GB RAM for paired-end mapping against the human genome.
    • Availability: proprietary; academic free binary (no multi-threading support)
  • PASS [0.5, PMID: 19218350].
    • Platform: Illumina; SOLiD; 454
    • Features: PET mapping
    • Advantages: long reads
    • Requirement: >15GB RAM against human genome
    • Availability: free source codes to academic users
  • PerM [0.1.0, PMID: 19675096]
    • Platform: Illumina; SOLiD
    • Advantages: fast
    • Availability: GPL
    • Limitation: no paired-end mapping apparently.
    • Requirement: 4.5 bytes per reference base
    • Comment on paper: PerM is very fast. The authors attribute its speed to the use of spaced seeds with higher weight. This is a reason, but to me, not the leading reason. I think PerM is fast mainly because in building index, it aligns the genome against itself under given a specified read length; in alignment, PerM aligns a repetitive read once rather than to each copy. The cost is a user needs to build a huge index for each read length.
  • SOAPv1 [1.11, PMID: 18227114]. The first published short read aligner.
    • Platform: Illumina
    • Features: PET mapping; adapter trimming; gapped alignment; SNP caller; counting occurrences
    • Advantages: feature rich
    • Requirements: >14GB RAM against human genome
    • Availability: GPL

Merge Sorting

  • Slider [0.6, PMID: 18974170]. A very clever short read aligner specifically designed for Illumina reads. It is able to use the second best base call, which potentially improves the accuracy on SNP finding.
    • Platform: Illumina
    • Features: Using second base
    • Advantages: fast; potentially more accurate on SNP discovery
    • Requirements: >160GB disk space
    • Availability: free source codes
  • Slider II [1.1].

Indexing Genome with Suffix Array/BWT

  • Bowtie [0.9.9, PMID: 19261174]. This is probably the fastest short read aligner to date. Although under the default option Bowtie does not guarantee to find the best hit or tell if the hit it finds is unique, it is possible to improve this behaviour at the cost of speed.
    • Platform: Illumina
    • Features: partial PET mapping; quality aware; [mapping quality]
    • Advantages: very fast
    • Availability: GPL
  • BWA [0.5.1, PMID: 19451168]. Another aligner written by me. Given high-quality reads, it is an order of magnitude faster than MAQ while achieving similar alignment accuracy.
    • Platform: Illumina; SOLiD; 454; Sanger
    • Features: PET mapping (short reads only); gapped alignment; mapping quality; counting suboptimal occurrences (short reads only); SAM output
    • Advantages: fast
    • Limitations: short read algorithm is slow for long reads and reads with high error rate
    • Availability: GPL
  • SOAPv2 [2.19, PMID: 19497933]. A marvelous program developed by the group who wrote BWT-SW.
    • Platform: Illumina
    • Features: PET mapping; mapping quality; counting occurrences
    • Advantages: fast
    • Availability: academic free binary
  • segemehl [0.0.7, PMID: 19750212].
    • Platform: Illumina
    • Features: accurate
    • Limitation: large memory requirement; no paired-end mapping
    • Comment on paper: the authors show maq and bwa are not as accurate probably because they were counting ambiguous alignments.
  • vmatch [SpringerLink].
    • Availability: academic free binary

Recommendation

First of all, as I am the key developer of two short read aligners (BWA and MAQ), it is really hard for me to give an unbiased evaluation. Please bear this fact in mind when reading through my comments below.
For Illumina reads, I would recommend my program BWA. BWA implements most of the major features of a practical aligner. It is relatively small in memory and highly efficient with little tradeoff on accuracy. BWA outputs alignment in the SAM format. Users may use SAMtools to sort/merge alignments and to make variants calls. One potential concern about BWA is it has not been widely used at the moment. It may be less robust than those publication-proved aligners such as Eland and MAQ.
[Update: With the help of paired-end reads, MAQ is able to find some SNPs at the edge of highly repetitive regions. However, BWA cannot. Nonetheless, I still prefer BWA given its speed and the fact that SNPs that can be called from repeats are rare and more likely to be false positives.]
Mapping inconsistent read pairs with NovoAlign is recommended for PET-based structural varition detection where alignment accuracy is the leading factor on reducing false positive calls. NovoAlign is the most accurate aligner to date.

Thursday, August 23, 2012

Learning Java

Java Programming: Lecture 1
Java Programming: Lecture 2

JEE programming: Lecture 1
JEE programming: Lecture2
JEE programming: Lecture 3 
JEE Programming: Lecture 4a
JEE programming: Lecture 5 
JEE programming: Lecture 6
JEE programming: Lecture 7
JEE programming: Lecture 8
JEE Programming: Lecture 9
JEE Programming: Lecture 10
JEE programming: Lecture 11


OOP in Java: Lecture 3
OOP in Java: Lecture 4
OOP in Java: Lecture 6
OOP in Java: Lecture 7
OOP in Java: Lecture 8
OOP in Java: Lecture 9
OOP in Java: Lecture 12

Client/Server Programming in Java: Lecture 30

Multi-threading in Java
Servlets tutorial: part 3
Data Structure: Lecture 2

 HTML/CSS programming Lecture
OO Programming in C++ (1)
OO Programming in C++ (2)


Wednesday, August 22, 2012

LoginPanel.java

import javax.swing.*;
import javax.swing.border.EmptyBorder;
import java.awt.*;
import java.awt.event.*;
import java.util.*;


/**
 * Title:        Login Panel
 * Description:  A simple yet complete login/logout panel with user callbacks
 *               for approving login attempts and getting notification of logouts.
 * Copyright:    Copyright (c) 2004
 * Company:      Superliminal Software
 * @author Melinda Green
 * @version 1.0
 */

public class LoginPanel extends JPanel {
    public final static String
        LOG_IN  = "Login",
        LOG_OUT = "Logout";
    protected JButton logButt;
    public JButton getLogButton() { return logButt; }
    private final static int DEFAULT_PSWD_CHARS = 10;
    private JTextField nameField = new JTextField(DEFAULT_PSWD_CHARS);
    public String getUserName() { return nameField.getText(); }

    /**
     * override this method to return true if approved, false otherwise.
     * default is true.
     */
    public boolean approveLogin(String uname, String pswd) {
        return true;
    }

    /**
     * override this method to learn about logout events.
     */
    public void loggedOut(String uname) {
    }

    public LoginPanel() {
        this(false);
    }

    public LoginPanel(final boolean clearPasswords) {
        this(clearPasswords, true, null, null);
    }

    /**
     * @param clearPasswords if true, clears password field on successful login.
     * @param initial_user optional default text to load into the 'user' type-in.
     * @param initial_password optional default text to load into the 'password' type-in.
     */
    public LoginPanel(final boolean clearPasswords, final boolean displayFailures, String initial_user, String initial_password) {
        final JPasswordField pswdField = new JPasswordField(DEFAULT_PSWD_CHARS);
        logButt = new JButton(LOG_IN);
        KeyListener quickLogin = new KeyAdapter() {
            public void keyTyped(KeyEvent ke) {
                if(ke.getKeyChar() == KeyEvent.VK_ENTER) {
                    logButt.doClick();
                    logButt.requestFocus();
                }
            }
        };
        nameField.setText(initial_user);
        pswdField.setText(initial_password);
        logButt.setName(LOG_IN);
        nameField.addKeyListener(quickLogin);
        pswdField.addKeyListener(quickLogin);
        // create the grid
        JPanel grid = new JPanel(new GridLayout(2, 2));
        grid.setBackground(new Color(255,255,255));
        grid.add(new JLabel("User Name"));
        grid.add(nameField);
        grid.add(new JLabel("Password"));
        grid.add(pswdField);

        // create login button row
        JPanel row = new JPanel();
        row.setBorder(new EmptyBorder(5, 0, 5, 0));
        row.setOpaque(false);
        row.setLayout(new BoxLayout(row, BoxLayout.X_AXIS));
        row.add(logButt);
        logButt.setBackground(new Color(220,220,220));

        logButt.addActionListener(new ActionListener() {
            public void actionPerformed(ActionEvent ae) {
                if(logButt.getText().equals(LOG_IN)) {
                    // seek login approval from derived class
                    if(approveLogin(nameField.getText(), new String(pswdField.getPassword()))) {
                        // note: must set logout text *before* clearing password
                        // otherwise component dependancy handler will disable the
                        // login button w/out password text before later setting logout text
                        // this closes bug #2336
                        logButt.setText(LOG_OUT);
                        if(clearPasswords)
                            pswdField.setText(null);
                        nameField.setEnabled(false);
                        pswdField.setEnabled(false);
                        fireLoginEvent(nameField.getText(), true);
                    }
                    else
                        if(displayFailures)
                            JOptionPane.showMessageDialog(LoginPanel.this, "Login Denied", "Login Error", JOptionPane.ERROR_MESSAGE);
                }
                else {
                    logButt.setText(LOG_IN);
                    loggedOut(nameField.getText());
                    nameField.setEnabled(true);
                    pswdField.setEnabled(true);
                    fireLoginEvent(nameField.getText(), false);
                }
            }
        });

        // implement component dependancies
        new ComponentDependencyHandler(nameField, pswdField) {
            public void dependencyNotification() {
                String
                    logtext = logButt.getText(),
                    nametext = nameField.getText(),
                    pswdtext = String.copyValueOf(pswdField.getPassword());
                boolean newstate = logtext.equalsIgnoreCase(LOG_OUT) ||
                    (nameField.getText() != null && nametext.length() > 0 // has login text?
                     && pswdtext.length() > 0);  // has password text?
                logButt.setEnabled(newstate);
            }
        };

        // construct final layout
        setLayout(new BoxLayout(this, BoxLayout.Y_AXIS));
        add(grid);
        add(row);
    }

    public interface LoginListener {
        void loggedIn(String uname);
        void loggedOut(String uname);
    }
    public static class LoginAdapter implements LoginListener {
        public void loggedIn(String uname){}
        public void loggedOut(String uname){}
    }
    private Vector loginListeners = new Vector();
    public void addLoginListener(LoginListener ll) { loginListeners.add(ll); }
    public void removeLoginListener(LoginListener ll) { loginListeners.remove(ll); }
    protected void fireLoginEvent(String uname, boolean in) {
        for(Enumeration e=loginListeners.elements(); e.hasMoreElements(); ) {
            LoginListener ll = (LoginListener)e.nextElement();
            if(in)
                ll.loggedIn(uname);
            else
                ll.loggedOut(uname);
        }
    }

    /**
     * simple example test program for LoginPanel class
     */
    public static void main(String[] args) {
        final String NOT_LOGGED_IN = "LoginPanel Test - Currently Logged Out";
        final JFrame frame = new JFrame(NOT_LOGGED_IN);
        frame.getContentPane().add(new LoginPanel() {
            public boolean approveLogin(String uname, String pswd) {
                // this is where to make the server call to approve or reject login attempt
                frame.setTitle("LoginPanel Test - Currently logged in as " + uname);
                return true;
            }
            public void loggedOut(String uname) {
                frame.setTitle(NOT_LOGGED_IN);
            }
        });
        frame.setDefaultCloseOperation(JFrame.EXIT_ON_CLOSE);
        frame.pack();
        frame.setSize(500, frame.getHeight());
        frame.setVisible(true);
    }
}

Tuesday, August 21, 2012

JAVA: Pop up a dialog box for user input

JTextField firstName = new JTextField();
JTextField lastName = new JTextField();
JPasswordField password = new JPasswordField();
final JComponent[] inputs = new JComponent[] {
                new JLabel("First"),
                firstName,
                new JLabel("Last"),
                lastName,
                new JLabel("Password"),
                password};
 
JOptionPane.showMessageDialog(null, inputs, "My custom dialog", JOptionPane.PLAIN_MESSAGE);
System.out.println("You entered " +
                firstName.getText() + ", " +
                lastName.getText() + ", " +
                password.getText());
 
 

Input Dialog Box Program

Here is example code showing a few ways of how to use the showInputDialog method of the JOptionPane class to make input dialog boxes. The program shows a series of input dialog boxes one after the other. The input method they ask the user to use varies from a text field, combo box and list box. The article relating to this code is Building an Input Dialog Box.
//This program shows a series of input dialog boxes.
//The next dialog is launched on the closing of the current dialog. 
//It provides examples of how to create dialog boxes
//with a text field, combo box and list box.

//Imports are listed in full to show what's being used
//could just import javax.swing.* and java.awt.* etc..
import javax.swing.JFrame;
import javax.swing.JTextArea;
import javax.swing.JOptionPane;
import javax.swing.UIManager;
import javax.swing.Icon;
import java.awt.EventQueue;
import java.awt.event.ActionListener;
import java.awt.event.ActionEvent;
import java.lang.reflect.Field;

public class InputDialogFrame extends JFrame{
    
    private JTextArea tracker;
    
    //Using a standard Java icon
    private Icon optionIcon = UIManager.getIcon("FileView.computerIcon");
    
    //Application start point   
    public static void main(String[] args) {
     
     //Use the event dispatch thread for Swing components
     EventQueue.invokeLater(new Runnable()
     {
         public void run()
         {
             //create GUI frame
             new InputDialogFrame().setVisible(true);          
         }
     });
              
    }
    
    public InputDialogFrame()
    {
        //make sure the program exits when the frame closes
        setDefaultCloseOperation(JFrame.EXIT_ON_CLOSE);
        setTitle("Dialog Box Example");
        setSize(500,300);
        
        //This will center the JFrame in the middle of the screen
        setLocationRelativeTo(null);
        
        
        //Using JTextArea to show clicks and responses
        tracker = new JTextArea("Click tracker:");
        add(tracker);
        setVisible(true);
        
        //Options for the combo box dialog
        String[] choices = {"Monday", "Tuesday"
                ,"Wednesday", "Thursday", "Friday"};
        
        //Options for the list dialog
        //There are more than 20 entries to make the showInputDialog method
        //choose to use a list box
        String[] jumboChoices = {"Abe", "Billy", "Colin", "Dexter"
                , "Edward", "Fred", "Gus", "Harry", "Ira", "Jeff"
                , "Kirk", "Larry", "Monty", "Nigel", "Orville", "Paul"
                , "Quint", "Richard", "Steve", "Tony", "Umberto", "Vinnie"
                , "Wade", "Xavier", "Yogi", "Zigmund"};
        
        //Input dialog with a text field
        String input =  JOptionPane.showInputDialog(this 
                ,"Enter in some text:");
        
        TrackResponse(input);
        
        //Input dialog with default text in the text field
        String defaultText =  JOptionPane.showInputDialog(this 
                ,"Enter in some text:","some text..");
        
        TrackResponse(defaultText);
        
        //Input dialog with a textfield, a message type and title
        String warningText =  JOptionPane.showInputDialog(this 
                ,"Erm, enter in a warning:" ,"Warning Message"
                ,JOptionPane.WARNING_MESSAGE);
        
        TrackResponse(warningText);
        
        //If an icon is used then it overrides the icon from the
        //message type. Likewise if a null is entered for the selection values
        //the dialog box will use a text field
        String entered = (String)JOptionPane.showInputDialog(this
                , "Enter a Day of the week:"
                , "Text Field Dialog", JOptionPane.QUESTION_MESSAGE
                , optionIcon, null, null);
        
        TrackResponse(entered);
        
        //If the String Array has more than 20 entries a JList is used
        //as the method the user gets to select a value
        String boyNames = (String)JOptionPane.showInputDialog(this, "Pick a Name:"
                , "ComboBox Dialog", JOptionPane.QUESTION_MESSAGE
                , optionIcon, jumboChoices, jumboChoices[0]);
        
        TrackResponse(boyNames);
        
        //Input dialog with a combo box created by
        //using a String array for the object message. Note how
        //a null icon for the icon results in the QUESTION_MESSAGE
        //message type being used.
        String picked = (String)JOptionPane.showInputDialog(this, "Pick a Day:"
                , "ComboBox Dialog", JOptionPane.QUESTION_MESSAGE
                , null, choices, choices[0]);
        
        TrackResponse(picked);      
    }
    
    //Append the picked choice to the tracker JTextArea
    public void TrackResponse(String response)
    {
        //showInputDialog method returns null if the dialog is exited
        //without an option being chosen
        if (response == null)
        {
            tracker.append("\nYou closed the dialog without any input..");
        }
        else
        {
            tracker.append("\nYou picked " + response + "..");
        }
    }
}

Monday, August 20, 2012

Java's Runtime.exec() and External Applications

Java was developed with an eye toward platform independence, freeing the software developer from the unpleasant and tedious task of cross-platform porting and testing of software. All too often, however, developers are not able to write completely new applications, but have to interface to existing legacy software and operating systems. This poses a problem as Java cannot make use of many utilities and system-dependent features that these programs have relied upon in the past.
There are two solutions to this problem, both of which sacrifice platform independence. The first is to make use of native methods in your Java code. You lose the portability, but achieve the result. Native methods can also be more difficult to implement, which can make them less desirable to use.
The second option is to satisfy the programming needs by executing external utilities and applications from within a Java program. In Java, this means employing the Runtime class exec() method. Like the fork(3) C function, Runtime.exec() allows you to execute a program; unlike fork(), it does not allow you to directly control the environment. Runtime.exec() provides a simple interface to platform-dependent utilities and applications, although at the expense of platform independence. In an environment where Java applications must coexist with other non-Java applications this will often be a valid trade-off.
In its simplest form, exec() is quite easy to use:
 Process p = Runtime.getRuntime().exec("/bin/ls");
The problem with this form is that it gets you nowhere. When Java forks a new process it redirects STDIN, STDOUT and STDERR. Therefore the results of "/bin/ls" do not go to the screen. Instead you must use the getInputStream(), getOutputStream() and getErrorStream() methods of the Process object to communicate with the program that you executed. Listing 1 shows an example using the Unix ls(1) command.
      Listing 1
import java.io.*;

class execInput {
    public static void main(String Argv[]) {
 try {
     String ls_str;

     Process ls_proc = Runtime.getRuntime().exec("/bin/ls -aFl");

     // get its output (your input) stream

     DataInputStream ls_in = new DataInputStream(
                                          ls_proc.getInputStream());

     try {
  while ((ls_str = ls_in.readLine()) != null) {
      System.out.println(ls_str);
  }
     } catch (IOException e) {
  System.exit(0);
     }
 } catch (IOException e1) {
     System.err.println(e1);
     System.exit(1);
 }

 System.exit(0);
    }
}
Some points to be aware of with this code. In the JDK1.0.2 implementation, you must supply a complete path to whatever it is that you are executing. There is no facility in that version of the JDK to get the shell environment that you are running in, hence no PATH variable is known. This is fixed in JDK1.1, however it is always good practice to fully qualify the path. Shell built-in commands will not work here. A prime example of this is the DOS "dir" command. This is a built-in and will not execute. You would need to use "command \c dir" as the command string.
If you pass the executable command as a single String, the exec() method will break the string into multiple strings, breaking the original string on white space. This will cause some trouble when trying to do things like I/O redirection.
Consider the following line of code:
 Process p = Runtime.getRuntime().exec("/bin/sh -c /bin/ls > ls.out");
This is intended to execute a Bourne shell and have the shell execute the ls command, redirecting the output of ls to the file ls.out. The reason for using /bin/sh is to get around the problem of having stdout redirected by the Java internals. Unfortunately, if you try this nothing will happen. When this command string is passed to the exec() method it will be broken into an array of Strings with the elements being "/bin/sh", "-c", "/bin/ls", ">", and "ls.out". This will fail, as sh expects only a single argument to the "-c" switch. To make this work try:
 String[] cmd = {"/bin/sh", "-c", "/bin/ls > out.dat"};
 Process p = Runtime.getRuntime().exec(cmd);
Since the command line is already a series of Strings, the strings will simply be loaded into the command array by the exec() method and passed to the new process as is. Thus the shell will see a "-c" and the command "/bin/ls > ls.out" and execute correctly. Using Runtime.exec() can make integration with system utilities, tools and legacy software achievable, although, as already noted, at the expense platform independence. If you must work with existing software and tools, it is often the easiest, and most straight-forward approach to the problem. But beware of the gotchas. They can often lead to unexpected results.

Friday, August 17, 2012

jSoapServer: sample server code

/*
 *  jSoapServer is a Java library implementing a multi-threaded
 *  soap server which can be easily integrated into java applications
 *  to provide a SOAP Interface for external programmers.
 *  
 *  Copyright (C) 2006 Martin Thelian
 *  
 *  This library is free software; you can redistribute it and/or
 *  modify it under the terms of the GNU Lesser General Public
 *  License as published by the Free Software Foundation; either
 *  version 2.1 of the License, or (at your option) any later version.
 *  
 *  This library is distributed in the hope that it will be useful,
 *  but WITHOUT ANY WARRANTY; without even the implied warranty of
 *  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
 *  Lesser General Public License for more details.
 *  
 *  You should have received a copy of the GNU Lesser General Public
 *  License along with this library; if not, write to the Free Software
 *  Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307  USA
 *  
 *  For more information, please email thelian@users.sourceforge.net
 */

/* =======================================================================
 * Revision Control Information
 * $Source: /cvsroot/jsoapserver/jSoapServer/test/java/source/org/jSoapServer/SoapServerTest.java,v $
 * $Author: thelian $
 * $Date: 2006/09/28 06:43:46 $
 * $Revision: 1.6 $
 * ======================================================================= */

package org.jSoapServer;
import java.io.ByteArrayInputStream;
import java.io.ByteArrayOutputStream;
import java.io.InputStream;
import java.net.MalformedURLException;
import java.net.URL;
import java.rmi.RemoteException;
import java.util.GregorianCalendar;
import java.util.zip.Deflater;
import java.util.zip.DeflaterOutputStream;
import java.util.zip.GZIPInputStream;
import java.util.zip.GZIPOutputStream;
import java.util.zip.ZipEntry;
import java.util.zip.ZipInputStream;
import java.util.zip.ZipOutputStream;

import javax.activation.DataHandler;
import javax.activation.DataSource;
import javax.xml.namespace.QName;
import javax.xml.rpc.ServiceException;
import javax.xml.soap.SOAPElement;
import javax.xml.soap.SOAPException;

import junit.framework.TestCase;

import org.apache.axis.AxisFault;
import org.apache.axis.Constants;
import org.apache.axis.attachments.Attachments;
import org.apache.axis.attachments.PlainTextDataSource;
import org.apache.axis.client.Call;
import org.apache.axis.client.Service;
import org.apache.axis.message.SOAPHeaderElement;
import org.apache.commons.httpclient.Header;
import org.apache.commons.httpclient.HttpClient;
import org.apache.commons.httpclient.NameValuePair;
import org.apache.commons.httpclient.methods.GetMethod;
import org.apache.commons.httpclient.methods.InputStreamRequestEntity;
import org.apache.commons.httpclient.methods.PostMethod;
import org.apache.commons.httpclient.methods.RequestEntity;
import org.apache.commons.httpclient.methods.StringRequestEntity;
import org.jSoapServer.http.HttpHeaders;
import org.jSoapServer.utils.FileUtils;

public class SoapServerTest extends TestCase {

    public static final String hostIp = "localhost";
    public static final int hostPort = 8090;
    
    private static SoapServer myServer = null;
    
    public static void main(String[] args) {
        junit.awtui.TestRunner.run(SoapServerTest.class);
    }

    protected void setUp() throws Exception {
        super.setUp();
//        
//        if (myServer == null) {
//            try {
//                // creating a new soap server
//                SoapServer myServer = new SoapServer();     
//                
//                // deploing all needed services
//                myServer.deployRpcSoapService("org.jSoapServer.SoapService","test");
//                
//                // configuring the server properties
//                String confFile = "conf" + File.separator + "jSoapServer.xml";                
//                if(myServer.initService(confFile)) {
//                    // TODO: reading out application specific configuration and process it ...
//                    // ApplicationConfiguration myConfig = myServer.getConfig().getApplicationConfiguration();
//                    
//                    // starting the soap server
//                    myServer.startServer();
//                    
//                    // starting the quickserver admin server if configured
//                    QSAdminServerConfig adminConfig = myServer.getConfig().getQSAdminServerConfig();
//                    if (adminConfig != null) myServer.startQSAdminServer();
//                }
//            } catch(Exception e) {
//                System.err.println("Error in server : "+e);
//                e.printStackTrace();
//                throw e;
//            }      
//        }
    }

    protected void tearDown() throws Exception {
        super.tearDown();
//        myServer.stopServer();
    }
    
    private String getNamespace(String portname) {
        return "http://" + SoapServerTest.hostIp + ":" + SoapServerTest.hostPort + "/" + portname;
    }
    
    private URL getWsdlURL(String portname) throws MalformedURLException {
        String wsdl = "http://" + SoapServerTest.hostIp + ":" + SoapServerTest.hostPort + "/" + portname;
        return new URL(wsdl);
    }
    
    public Object invokeServiceWithWSDL(URL wsdlURL, String Namespace, String serviceName, String portName, String operationName, Object[] params) throws ServiceException, RemoteException {
        if (params == null) params = new Object[]{};
        
        Service service = new Service(wsdlURL, new QName(Namespace,serviceName));
        Call call = (Call)service.createCall(new QName(Namespace,portName), new QName(Namespace,operationName));
        
        Object result = call.invoke(params);
        return result;
    }
    
    public Object invokeService(String endpoint, String operation, Object[] params, String[] usernamePwd, SOAPHeaderElement[] headers, String attachmentFormat, DataHandler[] attachments) throws ServiceException, MalformedURLException, RemoteException, SOAPException {
        if (params == null) params = new Object[]{};
        
        Service  service = new Service();
        Call call = (Call) service.createCall();
        
        call.setTargetEndpointAddress( new java.net.URL(endpoint));
        call.setOperationName(new QName(endpoint, operation));
        
        if (usernamePwd != null) {
            call.setUsername(usernamePwd[0]);
            call.setPassword(usernamePwd[1]);
        }
          
        if (headers != null) {
            for (int i=0; i < headers.length; i++) {
                call.addHeader(headers[i]);
            }
        }    
       
        if (attachmentFormat != null && attachments != null) {
            call.setProperty(Call.ATTACHMENT_ENCAPSULATION_FORMAT, attachmentFormat);
            for (int i=0; i < attachments.length; i++) {
                call.addAttachmentPart(attachments[i]);                 
            }
        }
           
        Object result = call.invoke(params);
        
        return result;
    }
    
    public Object execService(String serviceName, String methodName, Object[] params, String[] usernamePwd, SOAPHeaderElement[] headers, String attachmentFormat, DataHandler[] attachments) throws ServiceException, MalformedURLException, RemoteException, SOAPException {
        assertNotNull(serviceName);
        assertNotNull(methodName);
        
        String endpoint = "http://" + SoapServerTest.hostIp + ":" + SoapServerTest.hostPort + "/" + serviceName;
        Object result = invokeService(endpoint, methodName, params, usernamePwd, headers, attachmentFormat, attachments);    
        System.out.println("TEST " + serviceName + "." + methodName + (attachmentFormat !=null ? " with " + attachmentFormat + " attachment":"") + ": " + result);
        return result;
    }
    
    public void testDownloadServiceList() throws Exception {
        System.out.println("Downloading servicelist without content-encoding ...");        
        downloadServiceList(null);
    }
    
    public void testDownloadServiceListWithGzip() throws Exception {
        System.out.println("Downloading servicelist with gzip content-encoding ...");
        downloadServiceList("x-gzip");
    }    
    
    public void testDownloadServiceListWithZip() throws Exception {
        System.out.println("Downloading servicelist with compress content-encoding ...");
        downloadServiceList("x-compress");
    }      
    
    
    public void downloadServiceList(String contentEncoding) throws Exception {
        String serviceName = "test";
        String endpoint = "http://" + SoapServerTest.hostIp + ":" + SoapServerTest.hostPort + "/" + serviceName;
        
        HttpClient client = new HttpClient();
        GetMethod method = new GetMethod(endpoint);        
        method.setQueryString(new NameValuePair[]{new NameValuePair("list","")});
        
        if (contentEncoding != null) {
            method.setRequestHeader("Accept-Encoding",contentEncoding);
        }
        
        client.executeMethod(method);
        int    httpStatus    = method.getStatusCode();
        
        if (contentEncoding != null) {
            Header contentEncodingHeader = method.getResponseHeader("Content-Encoding");
            String usedEncoding = contentEncodingHeader.getValue();
            if (usedEncoding.startsWith("x-") && !contentEncoding.startsWith("x-")) contentEncoding = "x-" + contentEncoding;
            else if (!usedEncoding.startsWith("x-") && contentEncoding.startsWith("x-")) usedEncoding = "x-" + usedEncoding;
            assertEquals(usedEncoding, contentEncoding);
        }
        
        assertEquals(httpStatus, 200);
        
        StringBuffer list = new StringBuffer();
        if (contentEncoding != null) {
            Integer contentLength = Integer.valueOf(method.getResponseHeader("Content-Length").getValue());
            
            InputStream compressedIn = null;
            if (contentEncoding.equals("x-gzip")) {
                compressedIn = new GZIPInputStream(method.getResponseBodyAsStream());
            } else {
                compressedIn = new ZipInputStream(method.getResponseBodyAsStream());
                ((ZipInputStream)compressedIn).getNextEntry();
            }
            
            byte[] buffer = new byte[4096];
            
            int c;
            while ((c = compressedIn.read(buffer)) > 0) {
                list.append(new String(buffer,0,c));
            }
        } else {
          list.append(method.getResponseBodyAsString());  
        }
        
        method.releaseConnection();
    }        
    

    public void testDateChunk() throws Exception {
        transferAndContentEncodingTest(true, null);
    }
    
    public void testDateGzip() throws Exception {
        transferAndContentEncodingTest(false, "gzip");
    }   
    
    public void testDateCompress() throws Exception {
        transferAndContentEncodingTest(false, "compress");
    }   
    
    public void testDateDeflate() throws Exception {
        transferAndContentEncodingTest(false, "deflate");
    }       
    
    public void testDateChunkGzip() throws Exception {
        transferAndContentEncodingTest(true, "gzip");
    }
    
    public void testDateChunkCompress() throws Exception {
        transferAndContentEncodingTest(true, "compress");
    }
    
    public void testDateChunkDeflate() throws Exception {
        transferAndContentEncodingTest(true, "deflate");
    }
    
    public void transferAndContentEncodingTest(boolean transferEncoding, String contentEncoding) throws Exception {
        String serviceName = "test";
        String endpoint = "http://" + hostIp + ":" + hostPort + "/" + serviceName;
        String methodName = "testDate";
        
        HttpClient client = new HttpClient();
        PostMethod method = new PostMethod(endpoint); 
        
        if (transferEncoding) {
            method.setContentChunked(true);
        }
        
        StringBuffer fakeBody = new StringBuffer();        
        fakeBody.append("<?xml version=\"1.0\" encoding=\"UTF-8\"?>" +
                        "<SOAP-ENV:Envelope xmlns:SOAP-ENV=\"").append(Constants.URI_SOAP12_ENV).append("\">")
                    .append("<SOAP-ENV:Body>")
                        .append("<").append(methodName).append(">")
                            // TODO: what about additional params
                        .append("</").append(methodName).append(">")
                    .append("</SOAP-ENV:Body>")
                .append("</SOAP-ENV:Envelope>");        
        
        RequestEntity entity = null;
        if (contentEncoding == null) {
            entity = new StringRequestEntity(fakeBody.toString(),"text/xml","utf-8");
        } else {
            ByteArrayInputStream byteIn = new ByteArrayInputStream(fakeBody.toString().getBytes("UTF-8"));
            ByteArrayOutputStream byteOut = new ByteArrayOutputStream();
            if (contentEncoding.equals(HttpHeaders.HTTP_CONTENT_ENCODING_GZIP)) {
                GZIPOutputStream zippedOut = new GZIPOutputStream(byteOut);
                FileUtils.copy(byteIn, zippedOut);
                zippedOut.close();
            } else if (contentEncoding.equals(HttpHeaders.HTTP_CONTENT_ENCODING_COMPRESS)) {
                ZipOutputStream compressedOut = new ZipOutputStream(byteOut);
                compressedOut.putNextEntry(new ZipEntry("compressed output"));
                FileUtils.copy(byteIn, compressedOut);
                compressedOut.close();
            } else if (contentEncoding.equals(HttpHeaders.HTTP_CONTENT_ENCODING_DEFLATE)) {
                DeflaterOutputStream deflateOut = new DeflaterOutputStream(byteOut, new Deflater(9));
                FileUtils.copy(byteIn, deflateOut);
                deflateOut.close();                
            }
            byteIn = new ByteArrayInputStream(byteOut.toByteArray());
            entity = new InputStreamRequestEntity(byteIn);
            
            method.addRequestHeader(HttpHeaders.CONTENT_ENCODING, contentEncoding);
        }
        method.setRequestEntity(entity);
        
        client.executeMethod(method);
        int    httpStatus    = method.getStatusCode();
        String httpStatusTxt = method.getStatusText();
        
        assertEquals(httpStatus, 200);
        
        String response = method.getResponseBodyAsString();
        method.releaseConnection();
        
        System.out.println("TEST request with TransferEncoding=" + (transferEncoding?"ON ":"OFF") + 
                           " | ContentEncoding=" + (contentEncoding!=null?"ON (" + contentEncoding + ")":"OFF"));        
    }
    
    public void testString() throws Exception {
        String serviceName = "test";
        String methodName = "testString";
        Object[] params = null;
        
        Object result = execService(serviceName, methodName, params, null, null, null, null);
        assertTrue(result instanceof String);
    }
    
    public void testBoolean() throws Exception {
        String serviceName = "test";
        String methodName = "testBoolean";
        Object[] params = null;
        
        Object result = execService(serviceName, methodName, params, null, null, null, null);
        assertTrue(result instanceof Boolean);
    }
    
    public void testDate() throws Exception {
        String serviceName = "test";
        String methodName = "testDate";
        Object[] params = null;
        
        Object result = execService(serviceName, methodName, params, null, null, null, null);
        assertTrue(result instanceof GregorianCalendar);
    }    
    
    public void testIntAdd() throws Exception {
        String serviceName = "test";
        String methodName = "testIntAdd";
        Object[] params = new Object[]{new Integer(2),new Integer(3)};
        
        Object result = execService(serviceName, methodName, params, null, null, null, null);
        assertTrue(result instanceof Integer);
    }

    
    public void testUserNamePwd() throws Exception {
        String serviceName = "test";
        String methodName = "testUserNamePwd";
        Object[] params = null;
        
        String username = "testUser";
        String pwd = "testPwd";
        
        Object result = execService(serviceName, methodName, params, new String[]{username,pwd}, null, null, null);
        assertTrue(result instanceof String);  
        assertEquals(result, username + ":" + pwd);
    }
        
    public void testSoapHeader() throws Exception {
        String serviceName = "test";
        String methodName = "testSoapHeader";
        
        String username = "theUserName";
        String pwd = "thePwd";
        
        SOAPHeaderElement oHeaderElement = new SOAPHeaderElement("http://jSoapServer.org/securityTest", "securityHeader");
        oHeaderElement.setPrefix("sec");
        oHeaderElement.setMustUnderstand(false);
        SOAPElement oElement = oHeaderElement.addChildElement("username");
        oElement.addTextNode(username);
        oElement = oHeaderElement.addChildElement("password");
        oElement.addTextNode(pwd);
    
        Object result = execService(serviceName, methodName, null, null, new SOAPHeaderElement[]{oHeaderElement}, null, null);
        assertTrue(result instanceof String);  
        assertEquals(result, username + ":" + pwd);        
    }
    
    public void testException() throws Exception {
        String serviceName = "test";
        String methodName = "testException";
        try {
            execService(serviceName, methodName, null,null, null, null, null);
            
            assertTrue(false);
        } catch (AxisFault e) {
            System.out.println("TEST " + serviceName + "." + methodName + ": " + e.getMessage());
            assertTrue(e.getMessage().endsWith("TestException Text"));
        }
    }
    
    public void testReceiveAttachmentMime() throws Exception {
        sendAttachment(Call.ATTACHMENT_ENCAPSULATION_FORMAT_MIME);
    }
    
    public void testReceiveAttachmentDime() throws Exception {
        sendAttachment(Call.ATTACHMENT_ENCAPSULATION_FORMAT_DIME);
    }
    
    private void sendAttachment(String attachmentFormat)throws Exception {
        try {
            String serviceName = "test";
            String methodName = "testReceiveAttachment";
            String testData = "Testdata";
            
            DataSource data =  new PlainTextDataSource("Test.txt",testData);
            DataHandler attachmentFile = new DataHandler(data);        
            
            Object result = execService(serviceName, methodName, null,null, null, attachmentFormat, new DataHandler[]{attachmentFile});
            assertTrue(result instanceof String);  
            assertEquals(result,testData);  
        } catch (Exception e) {
            e.printStackTrace();
            throw e;
        }
    }    

    
    public void testSendAttachment() throws Exception {
        
        String serviceName = "test";
        String methodName = "testSendAttachment";
        
        String testMessageInput = "testdata";
        try {            
            // invoke the service
            Object result = execService(serviceName, methodName, new Object[]{new Integer(Attachments.SEND_TYPE_DIME),testMessageInput}, null, null, null, null);
            //Object result = invokeServiceWithWSDL(wsdlURL, namespace, serviceName, portName, methodName, new Object[]{new Integer(Attachments.SEND_TYPE_DIME),testMessageInput});
            
            // the output must be of type DataHandler
            assertTrue(result instanceof DataHandler);
            
            // get the content of the datahandler
            Object content = ((DataHandler)result).getContent();
            assertTrue(content instanceof String);
            
            // test if in and output are equal
            assertEquals(testMessageInput, (String)content);
            
        } catch (AxisFault e) {
            System.out.println("TEST " + serviceName + "." + methodName + ": " + e.getMessage());
            assertTrue(e.getMessage().endsWith("TestException Text"));
        }        
    }
}