Wednesday, November 12, 2014

Linux Netcat command – The swiss army knife of networking

Swiss Army Knife of networking netcat is a versatile tool that is able to read and write data across TCP and UDP network . Combined with other tools and redirection it can be used in number of ways in your scripts. You will be surprised to see what you can accomplish with Linux netcat command.

What netcat does it opens the connection between two machines and give back two streams. After that everything is up to your imagination. You can build a server, transfer files, chat with friends, stream media or use it as a standalone client for some other protocols.
Here are some of the usage of netcat.
[A(172.31.100.7) B(172.31.100.23)]

Linux netcat command examples

1. Port scanning

Port scanning is done by system admin and hackers to find the open ports at some machine. It helps them to identify the venerability in the system.
$nc -z -v -n 172.31.100.7 21-25
It can work in both TCP and UDP mode, default is TCP mode, to change to udp use -u option
z option tell netcat to use zero IO .i.e the connection is closed as soon as it opens and no actual data exchange take place.
v option is used for verbose option.
n option tell netcat not to use the DNS lookup for the address.
This command will print all the open ports between 21 to 25.
Banner is a text that services sends when you connects to them. Banner are very usefull when you are trying to velberability in the system as it identify the type and version of the services. NOTE not all services may send banner.
Once You have found the open ports you can easily grab the service banner by connecting to them using netcat.
$ nc -v 172.31.100.7 21
The Linux netcat command will connect to open port 21 and will print the banner of the service running at that port.

2. Chat Server

If you want to chat with your friend there are numerous software and messenger services available at your disposal.But what if you do not have that luxury anymore like inside your computer lab, where all outside connections are restricted, how will you communicate to your friend who is sitting in the next room. Don’t worry my friend because netcat has a solution for you just create a chat server and a predetermined port and he can connects to you.
Server
$nc -l 1567
The Linux netcat command starts a tcp server at port 1567 with stdout and stdin for input output stream i.e. The output is displayed at the shell and input is read from shell.
Client
$nc 172.31.100.7 1567
After this whatever you type on machine B will appear on A and vice-versa.

3. File transfer

Most of the time we are trying to transfer file over network and stumble upon the problem which tool to use. There are again numerous methods available like FTP, SCP, SMB etc. But is it really worth the effort to install and configure such complicated software and create a sever at your machine when you only need to transfer one file and only once.
Suppose you want to transfer a file “file.txt” from A to B
Anyone can be server or client, lets make A as server and B as client.
Server
$nc -l 1567 < file.txt
Client
$nc -n 172.31.100.7 1567 > file.txt
Here we have created a server at A at redirected the netcat input from file file.txt, So when any connection is successfull the netcat send the content of the file.
Again at the client we have redirect the output of netcat to file.txt. When B connects to A , A sends the file content and B save that content to file file.txt.
It is not necessary do create the source of file as server we can work in the eopposeit order also. Like in the below case we are sending file from B to A but server is created at A. This time we only need to redirect ouput of netcat at to file and input at B from file.
B as server
Server
$nc -l 1567 > file.txt
Client
$nc 172.31.100.23 1567 < file.txt

4. Directory transfer

Sending file is easy but what if we want to send more than one files, or a whole directory, its easy just use archive tool tar to archive the files first and then send this archive.
Suppose you want to transfer a directory over the network from A to B.
Server
$tar -cvf – dir_name | nc -l 1567
Client
$nc -n 172.31.100.7 1567 | tar -xvf -
Here at server A we are creating the tar archive and redirecting its outout at the console through -. Then we are piping it to netcat which is used to send it over network.
At Client we are just downloading the archive file from the server using the netcat and piping its output tar tool to extract the files.
Want to conserve bandwidth by compressing the archive, we can use bzip2 or other tool specific to content of files.
Server
$tar -cvf – dir_name| bzip2 -z | nc -l 1567
Compress the archive using the bzip2 utility.
Client
$nc -n 172.31.100.7 1567 | bzip2 -d |tar -xvf -
Decompress the archive using bzip2 archive

5. Encrypt your data when sending over the network

If you are worried about the security of data being sent over the network you can encrypt your data before sending using some tool like mcrypt.
Server
$nc localhost 1567 | mcrypt –flush –bare -F -q -d -m ecb > file.txt
Encrypt the data using the mcrypt tool.
Client
$mcrypt –flush –bare -F -q -m ecb < file.txt | nc -l 1567
Decrypt the data using the mcrypt tool.
Both the above commands will propmt for passowrd make sure to use the same password on both.
Here we have used mcrypt for encryption but any tool can be used.

6. Stream a video

Not the best method to stream but if the server doesn’t have the specific tools, then with netcat we still have hope.
Server
$cat video.avi | nc -l 1567
Here we are just reading the video file and redirecting its output to netcat
Client
$nc 172.31.100.7 1567 | mplayer -vo x11 -cache 3000 -
Here we are reading the data from the socket and redirecting it to mplayer.

7. Cloning a device

If you have just installed and configured a Linux machine and have to do the same to other machine too and do not want to do the configuration again. No need to repeat the process just boot the other machine with some boot-able pen drive and clone you machine.
Cloning a linux PC is very simple. Suppose your system disk is /dev/sda
Server
$dd if=/dev/sda | nc -l 1567
Client
$nc -n 172.31.100.7 1567 | dd of=/dev/sda
dd is a tool which reads the raw data from the disk, we are just redirecting its output stream through a netcat server to the other machine and writing it to the disk, it will copy everything along with the partition table. But if we have already done the partition and need to move only the root partition we can change sda with sda1, sda2 etc depending where out root is installed.

8. Opening a shell

We have used remote Shell using the telnet and ssh but what if they are not installed and we do not have the permission to install them, then we can create remote shell using netcat also.
If your netcat support -c and -e option (traditional netcat)
Server
$nc -l 1567 -e /bin/bash -i
Client
$nc 172.31.100.7 1567
Here we have created a netcat server and indicated it to run /bin/bash command when connection is successful.
If netcat doesn’t support -c or -e options(openbsd netcat) we can still crate remote shell.
Server
$mkfifo /tmp/tmp_fifo
$cat /tmp/tmp_fifo | /bin/sh -i 2>&1 | nc -l 1567 > /tmp/tmp_fifo
Here we have created a fifo. Then we have piped the content of this fifo file using pipe command to a shell 2>&1 is used to redirect stderr to same file where stdout is redirected which is piped to netcat server running at port 1567. Now here again we have redirected the output of netcat to fifo file.
Explanation:
The input received from network is written to fifo file.
The fifo file is read by cat command and it content is sent to sh command.
Sh command processes the received input and write it back to netcat.
Netcat send the output over the network to client.
All this is possible because pipe causes the command to run in parallel. The fifo file is used instead of regular file because the fifo causes the read to wait while if it was an ordinary file the cat command would have ended as soon as started reading an empty file.
At client is just as simple as conecting to server
Client
$nc -n 172.31.100.7 1567
And you will get a shell prompt at the client

9. Reverse Shell

Reverse shell are shell opened at the client side. Reverse shell are so named because unlike other configuration here server is using the services provided by the client.
Server
$nc -l 1567
At the client side simply tell netcat to execute the shell when connection is complete.
Client
$nc 172.31.100.7 1567 -e /bin/bash
Now what is so special about reverse shell.
Reverse shell is often used to bypass the firewall restrictions like blocked inbound connections. For example, I have a private IP address of 172.31.100.7 and I connect to outside network with a proxy server. If I want to access a shell at this machine from outside the network say 1.2.3.4, then I’ll use reverse shell for this purpose.

10. Specify Source Port

Suppose your firewall filters all ports but 25 then you need to specify the source port also with -p option
Server
$nc -l 1567
Client
$nc 172.31.100.7 1567 -p 25
You need root permissions to use port less than 1024.
This command will open the port 25 at the client which will be used for communication otherwise any random port can be used.

11. Specify Source Address

Suppose you have more than one addresses for your machine and you want to explicitly tell which address to use for outgoing data. Then we can specify the ip address with -s option in netcat
Server
$nc -u -l 1567 < file.txt
Client
$nc -u 172.31.100.7 1567 -s 172.31.100.5 > file.txt
This command will bind the address 172.31.100.5.
These are just some of the examples to play with netcat.
Some Other uses can be.
  •     Telnet client with -t option,
  •     HTTP client to download files,
  •     Check mail by connecting to mail server and using SMTP protocol,
  •     Stream your desktop using ffmpeg to grab our desktop and many more.
In short if you know the protocol you can implement any client using netcat as medium for network communication.
REFERNECES
Netcat Man Page

MPI example source codes in FORTRAN and C

Below are some excellent MPI example source codes in FORTRAN and C.

ex00 f_ex00.f  c_ex00.c 

This is a simple hello world program. Each processor prints out it's rank and the size of the current MPI run (Total number of processors).
  • MPI_Init
  • MPI_Comm_rank
  • MPI_Comm_size
  • MPI_Finalize

 

ex01 f_ex01.f  c_ex01.c 

A simple send/receive program in MPI
  • MPI_Init
  • MPI_Comm_rank
  • MPI_Comm_size
  • MPI_Send
  • MPI_Recv
  • MPI_Finalize

 

ex02 f_ex02.f  c_ex02.c 

Shows how to use probe and get_count to find the size of an incomming message.
  • MPI_Init
  • MPI_Comm_rank
  • MPI_Comm_size
  • MPI_SEND
  • MPI_Probe
  • MPI_get_count
  • MPI_recv
  • MPI_Finalize

 

ex03 f_ex03.f  c_ex03.c 

This is a simple isend/ireceive program in MPI.
  • MPI_Init
  • MPI_Comm_rank
  • MPI_Comm_size
  • MPI_Isend
  • MPI_Irecv
  • MPI_Wait
  • MPI_Finalize

 

ex04 f_ex04.f  c_ex04.c 

This is a simple broadcast program in MPI.
  • MPI_Init
  • MPI_Comm_rank
  • MPI_Comm_size
  • MPI_Bcast
  • MPI_Finalize

 

ex05 f_ex05.f  c_ex05.c 

This program shows how to use MPI_Scatter and MPI_Gather. Each processor gets different data from the root processor by way of mpi_scatter. The data is summed and then sent back to the root processor using MPI_Gather. The root processor then prints the global sum.
  • MPI_Init
  • MPI_Comm_size
  • MPI_Comm_rank
  • MPI_Scatter
  • MPI_Gather
  • MPI_Finalize

 

ex06 f_ex06.f  c_ex06.c 

This program shows how to use MPI_Scatter and MPI_Reduce. Each processor gets different data from the root processor by way of mpi_scatter. The data is summed and then sent back to the root processor using MPI_Reduce. The root processor then prints the global sum.
  • MPI_Init
  • MPI_Comm_size
  • MPI_Comm_rank
  • MPI_Scatter
  • MPI_Reduce
  • MPI_Finalize

 

ex07 f_ex07.f  c_ex07.c 

This program shows how to use MPI_Alltoall. Each processor send/rec a different random number to/from other processors.
  • MPI_Init
  • MPI_Comm_size
  • MPI_Comm_rank
  • MPI_alltoall
  • MPI_Finalize

 

ex08 f_ex08.f  c_ex08.c 

This program shows how to use MPI_Gatherv. Each processor sends a different amount of data to the root processor. We use MPI_Gather first to tell the root how much data is going to be sent.
  • MPI_Init
  • MPI_Comm_size
  • MPI_Comm_rank
  • MPI_Gather
  • MPI_Gatherv
  • MPI_Finalize

 

ex09 f_ex09.f  c_ex09.c 

This program shows how to use MPI_Alltoallv. Each processor send/rec a different and random amount of data to/from other processors. We use MPI_Alltoall to tell how much data is going to be sent.
  • MPI_Init
  • MPI_Comm_size
  • MPI_Comm_rank
  • MPI_alltoall
  • MPI_alltoallv
  • MPI_Finalize

 

ex10 f_ex10.f  c_ex10.c 

This program is designed to show how to set up a new communicator. We set up a communicator that includes all but one of the processors, The last processor is not part of the new communcator, TIMS_COMM_WORLD. We use the routine MPI_Group_rank to find the rank within the new connunicator. For the last processor the rank is MPI_UNDEFINED because it is not part of the communicator. For this processor we call get_input The processors in TIMS_COMM_WORLD pass a token between themselves in the subroutine pass_token. The remaining processor gets input, i, from the terminal and passes it to processor 1 of MPI_COMM_WORLD. If i > 100 the program stops.
  • MPI_Init
  • MPI_Comm_size
  • MPI_Comm_rank
  • MPI_COMM_GROUP
  • MPI_GROUP_INCL
  • MPI_COMM_CREATE
  • MPI_GROUP_RANK
  • MPI_Barrier
  • MPI_Finalize
  • MPI_COMM_DUP
  • MPI_Comm_rank
  • MPI_Comm_size
  • MPI_Barrier
  • MPI_Finalize
  • MPI_Iprobe
  • MPI_RECV
  • MPI_SEND
  • MPI_RECV
  • MPI_SEND
  • MPI_SEND

 

ex11 f_ex11.f  c_ex011.c 

Shows how to use MPI_Type_vector to send noncontiguous blocks of data and MPI_Get_count and MPI_Get_elements to see the number of elements sent.
  • MPI_Init
  • MPI_Comm_rank
  • MPI_Comm_size
  • MPI_Type_vector
  • MPI_Type_commit
  • MPI_Send
  • MPI_Recv
  • MPI_Get_count
  • MPI_Get_elements
  • MPI_Finalize

 

ex12 f_ex12.f  c_ex12.c 

Shows a short cut method to create a collection of communicators. All processors with the "same color" will be in the same communicator. In this case the color is either 0 or 1 for even or odd processors. Index gives rank in new communicator.
  • MPI_Init
  • MPI_Comm_size
  • MPI_Comm_rank
  • MPI_Comm_split
  • MPI_Comm_rank
  • MPI_Comm_rank
  • MPI_Bcast
  • MPI_Finalize

 

ex12 f_ex13.f  c_ex13.c 

This program shows how to use mpi_scatterv. Each processor gets a different amount of data from the root processor. We use mpi_Gather first to tell the root how much data is going to be sent.
  • MPI_Init
  • MPI_Comm_size
  • MPI_Comm_rank
  • MPI_comm_split
  • MPI_Comm_rank
  • MPI_Comm_size
  • MPI_bcast
  • MPI_Finalize
  • MPI_Init
  • MPI_Comm_size
  • MPI_Comm_rank
  • MPI_Gather
  • MPI_Scatterv
  • MPI_Finalize

 

makefile

mpi.tar

 

http://geco.mines.edu/workshop/class2/examples/mpi/index.html

Monday, November 10, 2014

MPI lectures

FIRST EXAMPLES IN MPI

Goals

The way to learn about the message passing interface (MPI) is to actually use it. Several examples are provided in the following that build gradually in their use of MPI commands. Copy these programs and compile them on the cluster, using either the Fortran or C versions. Make small changes and try to anticipate the consequences. Learn about the MPI commands:
  • MPI_Init
  • MPI_Finalize
  • MPI_Comm_rank
  • MPI_Comm_size
  • MPI_Send
  • MPI_Recv

First MPI Example

Consider the following program, called mpisimple1.c. This program is written in C with MPI commands included. It initializes MPI, executes a single print statement, then Finalizes (Quits) MPI.
In C:
#include < mpi.h>   /* PROVIDES THE BASIC MPI DEFINITION AND TYPES */
#include < stdio.h>

int main(int argc, char **argv) {
  
  MPI_Init(&argc, &argv); /*START MPI */
  printf("Hello world\n");
  MPI_Finalize();  /* EXIT MPI */
}
In Fortran:
 program mpisimple
      
      implicit none
      
      integer ierr

      include 'mpif.h'

      call mpi_init(ierr)

c     print message to screen

      write(6,*) 'Hello World!'

      call mpi_finalize(ierr)

      end


To compile this code, type:
 mpicc -o simple1 mpisimple1.c
or
 mpif77 -o simple1 mpisimple.f
To run this compiled code, type:
 mpirun -np 4 simple1
In the above example, the code "simple1" will execute on four processors (-np 4). Output printed to the screen will look like:
Hello world
Hello world
Hello world
Hello world
Discussion: The four processors each perform the exact same task. Each processor prints a single line. MPI_Init and MPI_Finalize take care of all of the details associated with running the code on multiple processors. MPI_Init must be called before any other MPI command. MPI_Finalize must be called last. So, these two commands form a wrapper around the body of the parallel code. Note that errors may result if more processors are specified than are available on the system. A typical (and uninformative!) error message looks like:
tsn004.acomp.usf.edu: No route to host
bm_list_15782:  p4_error: interrupt SIGINT: 2
rm_l_1_18402:  p4_error: interrupt SIGINT: 2
p0_15781:  p4_error: interrupt SIGINT: 2
rm_l_2_18291:  p4_error: interrupt SIGINT: 2
rm_18401:  p4_error: interrupt SIGINT: 2
rm_l_3_18120:  p4_error: interrupt SIGINT: 2
rm_18290:  p4_error: interrupt SIGINT: 2
rm_18119:  p4_error: interrupt SIGINT: 2

Example 2

Most parallel codes assign different tasks to different processors. For example, parts of an input data set might be divided and processed by different processors, or a finite difference grid might be divided among the processors available. This means that the code needs to identify processors. In this example, processors are identified by rank - an integer from 0 to total number of processors - 1.
In C:
#include < mpi.h>   /* PROVIDES THE BASIC MPI DEFINITION AND TYPES */
#include  < stdio.h>

int main(int argc, char **argv) {
  
  int my_rank; 
  int size;
  MPI_Init(&argc, &argv); /*START MPI */

/*DETERMINE RANK OF THIS PROCESSOR*/
  MPI_Comm_rank(MPI_COMM_WORLD, &my_rank); 

 /*DETERMINE TOTAL NUMBER OF PROCESSORS*/
  MPI_Comm_size(MPI_COMM_WORLD, &size);


  printf("Hello world! I'm rank (processor number) %d of size %d\n", my_rank, size);

  MPI_Finalize();  /* EXIT MPI */
  
}
in Fortran:
  program simple2
      
      implicit none
      
      integer ierr,my_rank,size

      include 'mpif.h'

      call mpi_init(ierr)

      call mpi_comm_rank(MPI_COMM_WORLD,my_rank,ierr)
      call mpi_comm_size(MPI_COMM_WORLD,size,ierr)

c     print rank and size to screen

      write(6,100) my_rank, size

      call mpi_finalize(ierr)

 100  format('Hello World! I am rank ', I2, ' of size ', I2)

      end


Compiling and executing the code using 4 processors yields this result:
Hello world! I'm rank 0 of size 4
Hello world! I'm rank 2 of size 4
Hello world! I'm rank 1 of size 4
Hello world! I'm rank 3 of size 4
Discussion: Two additional MPI commands appear in this code: MPI_Comm_rank(,) and MPI_Comm_size(,). MPI_Comm_rank(,) returns the processor id assigned to each processor during start-up. This rank is returned as an integer (in this case called my_rank). The first parameter (MPI_COMM_WORLD) is predefined in MPI, and includes information about all the processors started when the program execution begins. Similarly, MPI_Comm_size returns the total number of processors - in this case 4. It is important to know the total number of processors so the problem set (data for example) can be divided among all available. Note that the code does not print the output in any particular order. In this case, rank 2 prints before rank 1. This order changes depending on the vagaries of communication between the nodes. Additional MPI commands are needed to structure communication more effectively.

Example 3

A further example illustrates the need to control flow in the program. The following code is essentally the same a Example 2. An additional conditional ("If") statement has been added near the end of the code. If the rank of the processor is 0, let the world know that you are finished.
In C:
#include  < mpi.h>   /* PROVIDES THE BASIC MPI DEFINITION AND TYPES */
#include < stdio.h>

int main(int argc, char **argv) {
  
  int my_rank; 
  int size;
  MPI_Init(&argc, &argv); /*START MPI */
  MPI_Comm_rank(MPI_COMM_WORLD, &my_rank); /*DETERMINE RANK OF THIS PROCESSOR*/
  MPI_Comm_size(MPI_COMM_WORLD, &size); /*DETERMINE TOTAL NUMBER OF PROCESSORS*/


  printf("Hello world! I'm rank (processor number) %d of size %d\n", my_rank, size);

  if (rank == 0) printf("That is all for now!\n");
  MPI_Finalize();  /* EXIT MPI */
    
  exit(0);
  
}
In Fortran:

 program simple3
      
      implicit none
      
      integer ierr,my_rank,size

      include 'mpif.h'

      call mpi_init(ierr)

      call mpi_comm_rank(MPI_COMM_WORLD,my_rank,ierr)
      call mpi_comm_size(MPI_COMM_WORLD,size,ierr)

c     print message, rank and size to screen

      write(6,100) my_rank, size

      if(my_rank.eq.0) then
         write(6,*) 'That is all for now!'
      end if

      call mpi_finalize(ierr)

 100  format('Hello World! I am rank ', I2, ' of size ', I2)

      end


Output from this code will likely look like:
Hello world! I'm rank (processor number) 0 of size 4
That is all for now!
Hello world! I'm rank (processor number) 1 of size 4
Hello world! I'm rank (processor number) 2 of size 4
Hello world! I'm rank (processor number) 3 of size 4
Discussion: Note that the "If" statement does not have the desired effect. Although the if statement appears later in the code that earlier print statments, this step happens to execute faster on processor 0 than on the other processors. So the program claims it is finished before it should. This can lead to serious problems if, for example, subsequent calculations depend on prior results from multiple processors. One method of avoiding this problem is given in the next example.

Example 4

Two additional MPI commands may be used to direct traffic (message queuing) during the program execution: MPI_Send() and MPI_Recv(). It is safe to say these two commands are at the heart of MPI. Use of these statements makes the program appear more complicated, but it is well worth it if the flow of the program needs to be controlled. In C:
#include < mpi.h>   /* PROVIDES THE BASIC MPI DEFINITION AND TYPES */
#include < stdio.h>

int main(int argc, char **argv) {
  
  int my_rank; 
  int partner;
  int size, i,t;
  char greeting[100];
  MPI_Status stat;
  
  MPI_Init(&argc, &argv); /*START MPI */
  MPI_Comm_rank(MPI_COMM_WORLD, &my_rank); /*DETERMINE RANK OF THIS PROCESSOR*/
  MPI_Comm_size(MPI_COMM_WORLD, &size); /*DETERMINE TOTAL NUMBER OF PROCESSORS*/
  
  
  sprintf(greeting, "Hello world: processor %d of %d\n", my_rank, size);
  
  /* adding a silly conditional statement like the
     following graphically illustrates "blocking" and
     flow control during program execution
     
     if (my_rank == 1) for (i=0; i<1000000000; i++) t=i;
     
  */
  
  if (my_rank ==0) {
    fputs(greeting, stdout);
    for (partner = 1; partner < size; partner++){
      
      MPI_Recv(greeting, sizeof(greeting), MPI_BYTE, partner, 1, MPI_COMM_WORLD, &stat);
      fputs (greeting, stdout);
      
    }
  }
  else {
    MPI_Send(greeting, strlen(greeting)+1, MPI_BYTE, 0,1,MPI_COMM_WORLD);
  }
  
  
  
  if (my_rank == 0) printf("That is all for now!\n");
  MPI_Finalize();  /* EXIT MPI */
  
}
In Fortran:
  program simple4
      
      implicit none
      
      integer ierr,my_rank,size,partner
      CHARACTER*50 greeting

      include 'mpif.h'
      integer status(MPI_STATUS_SIZE)


      call mpi_init(ierr)

      call mpi_comm_rank(MPI_COMM_WORLD,my_rank,ierr)
      call mpi_comm_size(MPI_COMM_WORLD,size,ierr)

      write(greeting,100) my_rank, size


      if(my_rank.eq.0) then
         write(6,*) greeting
         do partner=1,size-1
         call mpi_recv(greeting, 50, MPI_CHARACTER, partner, 1, 
     &    MPI_COMM_WORLD, status, ierr)
            write(6,*) greeting
         end do
      else
         call mpi_send(greeting, 50, MPI_CHARACTER, 0, 1, 
     &    MPI_COMM_WORLD, ierr)
      end if

      if(my_rank.eq.0) then
         write(6,*) 'That is all for now!'
      end if

      call mpi_finalize(ierr)

 100  format('Hello World: processor ', I2, ' of ', I2)

      end

Output from this code looks like:
Hello world: processor 0 of 4
Hello world: processor 1 of 4
Hello world: processor 2 of 4
Hello world: processor 3 of 4
That is all for now!
Discussion: The structure of the program has been changed to assure that the output is in the proper order (the processors are now listed in ascending order). Furthermore, the statement "That is all for now!" prints last. Program execution was effectively blocked until all processors had the opportunity to print. This was done by communicating between processors. Rather than simply printing, most processors now send the greeting back to processor 0 - usually called the root of the master processor. This processor first prints its own greeting, then polls successive processors - waiting to receive a message from each one. Only when the message is received does proccessor 0 move on. Using the MPI_Send and MPI_Recv commands blocks program execution. This blocking is illustrated graphically by inserting a long loop in the code, causing one of the processors to take a long time to complete its tasks. The cost of this structure is added syntax.
Here is the syntax for MPI_Send and MPI_Recv:
int MPI_Send (
                      message ,    /* actual information sent */
                      length,         /* length of the message */
                      datatype,   /* MPI datatype of the message */
                      destination, /* rank of the processor getting the message */
                      tag,             /* tag helps sort messages - likely an int and the same as in MPI_Recv
                      MPI_Comm  /* almost alway MPI_Comm_World 
                      )   

int MPI_Recv (
                      message ,     /* actual information received*/
                      length,           /* length of the message */
                      datatype,     /* MPI datatype of the message */
                      source,         /* rank of the processor sending the message */
                      tag,               /* tag helps sort messages - likely an int and the same as in MPI_Recv* /
                      MPI_Comm,  /* almost alway MPI_Comm_World */
                      Status           /* a data structure that contains info on what was recv'd */
                      )   
A few additional notes:
  • MPI_Recv is "blocking" in the sense that when the process (in this case my_rank ==0) reaches the MPI_Recv statement, it will wait until it actually receives the message (another process sends it). If the other process is not ready to Send, then the process running on my_rank == 0 will simply remain idle. If the message is never sent, my_rank==0 will wait a very long time!
  • The message is of type "datatype" and is predefined in MPI. Possible datatypes are summarized in the following table.
    MPI Datatype C Datatype
    MPI_FLOAT float
    MPI_DOUBLE double
    MPI_LONG_DOUBLE long double
    MPI_INT signed int
    MPI_LONG signed long int
    MPI_SHORT signed short int
    MPI_CHAR signed char
    MPI_UNSIGNED unsigned int
    MPI_UNSIGNED_SHORT unsigned short int
    MPI_UNSIGNED_LONG unsigned long int
    MPI_UNSIGNED_CHAR unsigned char
    MPI_BYTE
    MPI_PACKED
  • The length of the Recv'd message does not need to be the same length as the Sent message. Typically the length of the sent message is known exactly, but the space allocated for the Recv'd message is larger, because it may not be known.

Monday, November 3, 2014

A short example MPI code in C

Here is a "Hello World" program in MPI written in C. In this example, we send a "hello" message to each processor, manipulate it trivially, return the results to the main process, and print the messages.

 /*
  "Hello World" MPI Test Program
 */
 #include <mpi.h>
 #include <stdio.h>
 #include <string.h>
 
 #define BUFSIZE 128
 #define TAG 0
 
 int main(int argc, char *argv[])
 {
   char idstr[32];
   char buff[BUFSIZE];
   int numprocs;
   int myid;
   int i;
   MPI_Status stat;
   /* MPI programs start with MPI_Init; all 'N' processes exist thereafter */
   MPI_Init(&argc,&argv);
   /* find out how big the SPMD world is */
   MPI_Comm_size(MPI_COMM_WORLD,&numprocs);
   /* and this processes' rank is */
   MPI_Comm_rank(MPI_COMM_WORLD,&myid);
 
   /* At this point, all programs are running equivalently, the rank
      distinguishes the roles of the programs in the SPMD model, with
      rank 0 often used specially... */
   if(myid == 0)
   {
     printf("%d: We have %d processors\n", myid, numprocs);
     for(i=1;i<numprocs;i++)
     {
       sprintf(buff, "Hello %d! ", i);
       MPI_Send(buff, BUFSIZE, MPI_CHAR, i, TAG, MPI_COMM_WORLD);
     }
     for(i=1;i<numprocs;i++)
     {
       MPI_Recv(buff, BUFSIZE, MPI_CHAR, i, TAG, MPI_COMM_WORLD, &stat);
       printf("%d: %s\n", myid, buff);
     }
   }
   else
   {
     /* receive from rank 0: */
     MPI_Recv(buff, BUFSIZE, MPI_CHAR, 0, TAG, MPI_COMM_WORLD, &stat);
     sprintf(idstr, "Processor %d ", myid);
     strncat(buff, idstr, BUFSIZE-1);
     strncat(buff, "reporting for duty", BUFSIZE-1);
     /* send to rank 0: */
     MPI_Send(buff, BUFSIZE, MPI_CHAR, 0, TAG, MPI_COMM_WORLD);
   }
 
   /* MPI programs end with MPI Finalize; this is a weak synchronization point */
   MPI_Finalize();
   return 0;
 } 
 
 
Compile and run:
#mpicc test_mpi.c
#mpirun -np 2 a.out

When run with two processors this gives the following output.

0: We have 2 processors 
0: Hello 1! Processor 1 reporting for duty 

 If you run it with 4 processors:
#mpirun -np 4 a.out

0: We have 4 processors 
0: Hello 1! Processor 1 reporting for duty 
0: Hello 2! Processor 2 reporting for duty
0: Hello 3! Processor 3 reporting for duty
 

Connect mysql DB in Python: example script

#!/usr/bin/python

import MySQLdb

# Open database connection
db = MySQLdb.connect("localhost","username","password","dbname" )

# prepare a cursor object using cursor() method
cursor = db.cursor()

# Prepare SQL query to INSERT a record into the database.
sql = "INSERT INTO EMPLOYEE(FIRST_NAME, \
       LAST_NAME, AGE, SEX, INCOME) \
       VALUES ('%s', '%s', '%d', '%c', '%d' )" % \
       ('Mac', 'Mohan', 20, 'M', 2000)
try:
   # Execute the SQL command
   cursor.execute(sql)
   # Commit your changes in the database
   db.commit()
except:
   # Rollback in case there is any error
   db.rollback()

# disconnect from server
db.close()

print 'done';