Prepare The Genome for BWA and GATK

Create BWA index for the genome:

>bwa index hg19.fa
 

Create the dict file for the genome: 

>java -jar CreateSequenceDictionary.jar R= hg19.fa O= hg19.dict

Create index for the genome:

>samtools faidx hg19.fa 

Submit Dependent Jobs Through qsub

$ qsub random.pbs 5594674.hpc0.local $ qsub -W depend=afterok:5594674.hpc0.local sort.pbs 5594675.hpc0.local $ qstat -u $USER   hpc0.local:                                                                          Req'd  Req'd   Elap Job ID               Username Queue    Jobname          SessID NDS   TSK Memory Time  S Time -------------------- -------- -------- ---------------- ------ ----- --- ------ ----- - ----- 5594674.hpc0.loc     manchu   ser2     random.pbs        18029     1   1    --  48:00 R 00:00 5594675.hpc0.loc     manchu   ser2     sort.pbs                    1   1    --  48:00 H   --

RVTools for Monitoring The Virtual Hosts

RVTools

RVTools is a windows .NET 2.0 application which uses the VI SDK to display information about your virtual machines and ESX hosts. Interacting with VirtualCenter 2.5, ESX Server 3.5, ESX Server 3i, VirtualCenter 4.x, ESX Server 4.x, VirtualCenter 5.0, VirtualCenter Appliance, ESX Server 5.0, VirtualCenter 5.1, ESX Server 5.1, VirtualCenter 5.5, ESX Server 5.5. RVTools is able to list information about VMs, CPU, Memory, Disks, Partitions, Network, Floppy drives, CD drives, Snapshots, VMware tools, Resource pools, Clusters, ESX hosts, HBAs, Nics, Switches, Ports, Distributed Switches, Distributed Ports, Service consoles, VM Kernels, Datastores, Multipath info and health checks. With RVTools you can disconnect the cd-rom or floppy drives from the virtual machines and RVTools is able to update the VMware Tools installed inside each virtual machine to the latest version.
More than 400,000 copies downloaded!

This application supports ESX Server 3.5, VirtualCenter 2.5, ESX Server 3i, ESX Server 4i, VirtualCenter 4.x, ESX 4.x, VirtualCenter 5.0, VirtualCenter Appliance, ESX 5.0, VirtualCenter 5.1, ESX 5.1, VirtualCenter 5.5 and ESX 5.5.

Latest Version: 3.6 | February, 2014
Download | Documentation


http://www.robware.net/

Source (or dot)

You need to source your script. If not it will be run in a separate subshell, changing the working directory of the subshell but not of the shell you run it in.

To source it :

. myfile.sh

or

source myfile.sh 

Multithreaded sorting of large bam files

samtools sort -@ 8 -m 4G

Note that each thread will use about 4-4.5GB of memory in this setting, so make sure you have enough memory when pushing up "-m" and "-@".

9 Must-Have Skills to Land Top Big Data Jobs in 2015

The secret is out, and the mad rush is on to leverage big data analytics tools and techniques for competitive advantage before they become commoditized. If you’re in the market for a big data job in 2015, these are the nine skills that will garner you a job offer.

1. Apache Hadoop

Sure, it’s entering its second decade now, but there’s no denying that Hadoop had a monstrous year in 2014 and is positioned for an even bigger 2015 as test clusters are moved into production and software vendors increasingly target the distributed storage and processing architecture. While the big data platform is powerful, Hadoop can be a fussy beast and requires care and feeding by proficient technicians. Those who know there way around the core components of the Hadoop stack–such as HDFS, MapReduce, Flume, Oozie, Hive, Pig, HBase, and YARN–will be in high demand.

2. Apache Spark

If Hadoop is a known quantity in the big data world, then Spark is a black horse candidate that has the raw potential to eclipse its elephantine cousin. The rapid rise of the in-memory stack is being proffered as a faster and simpler alternative to MapReduce-style analytics, either within a Hadoop framework or outside it. Best positioned as one of the components in a big data pipeline, Spark still requires technical expertise to program and run, thereby providing job opportunities for those in the know.

3. NoSQL

Source: Dice Tech 2014 Salary Survey

On the operational side of the big data house, distributed, scale-out NoSQL databases like MongoDB and Couchbase are taking over jobs previously handled by monolithic SQL databases like Oracle and IBM DB2. On the Web and with mobile apps, NoSQL databases are often the source of data crunched in Hadoop, as well as the destination for application changes put in place after insight is gleaned from Hadoop. In the world of big data, Hadoop and NoSQL occupy opposite sides of a virtuous cycle.

4. Machine Learning and Data Mining

People have been mining for data as long as they’ve been collecting it. But in today’s big data world, data mining has reached a whole new level. One of the hottest fields in big data last year is machine learning, which is poised for a breakout year in 2015. Big data pros who can harness machine learning technology to build and train predictive analytic apps such as classification, recommendation, and personalization systems are in super high demand, and can command top dollar in the job market.

5. Statistical and Quantitative Analysis

This is what big data is all about. If you have a background in quantitative reasoning and a degree in a field like mathematics or statistics, you’re already halfway there. Add in expertise with a statistical tool like R, SAS, Matlab, SPSS, or Stata, and you’ve got this category locked down. In the past, most quants went to work on Wall Street, but thanks to the big data boom, companies in all sorts of industries across the country are in need of geeks with quantitative backgrounds.

6. SQL

The data-centric language is more than 40 years old, but the old grandpa still has a lot of life yet in today’s big data age. While it won’t be used with all big data challenges (see: NoSQL above), the simplify of Structured Query Language makes it a no-brainer for many of them. And thanks to initiatives like Cloudera‘s Impala, SQL is seeing new life as the lingua franca for the next-generation of Hadoop-scale data warehouses.

7. Data Visualization

Big data can be tough to comprehend, but in some circumstances there’s no replacement for actually getting your eyeballs onto data. You can do multivariate or logistic regression analysis on your data until the cows come home, but sometimes exploring just a sample of your data in a tool like Tableau or Qlikview can tell you the shape of your data, and even reveal hidden details that change how you proceed. And if you want to be a data artist when you grow up, being well-versed in one or more visualization tools is practically a requirement.

8. General Purpose Programming Languages

Source: Wanted Analytics 2014 hiring survey

Having experience programming applications in general-purpose languages like Java, C, Python, or Scala could give you the edge over other candidates whose skill sets are confined to analytics. According to Wanted Analytics, there was a 337 percent increase in the number of job postings for “computer programmers” that required background in data analytics. Those who are comfortable at the intersection of traditional app dev and emerging analytics will be able to write their own tickets and move freely between end-user companies and big data startups.

9. Creativity and Problem Solving

No matter how many advanced analytic tools and techniques you have on your belt, nothing can replace the ability to think your way through a situation. The implements of big data will inevitably evolve and new technologies will replace the ones listed here. But if you’re equipped with a natural desire to know and a bulldog-like determination to find solutions, then you’ll always have a job offer waiting somewhere.

HaplotypeCaller failed to make a variant call

There are a few possibilities as to why HaplotypeCaller chose not to make a call for this SNP. 

1. HaplotypeCaller uses a more stringent mapping quality filter than UnifiedGenotyper (Is UnifiedGenotyper what you meant by “GenotypeCaller”?). Depending on what the mapping qualities in this area of reads is, the evidence that HaplotypeCaller is seeing may be very different than what UnifiedGenotyper is seeing to make the call.
2. HaplotypeCaller is not a locus walker as UnifiedGenotyper is. It doesn’t look at the pileup as presented to it. In areas of potential variation, it will gather all of the reads that pass its quality filters and perform a de novo assembly of these reads. This can potentially resolve false positives due to misalignment. And once the reassembly occurs, the BAM will not look like the original BAM that you are seeing in the browser picture.

One way to determine if the newly assembled BAM still shows this SNP is to use the “-bamout” argument in HaplotypeCaller. This will print out a new BAM file reflecting the reassembled reads that HaplotypeCaller is ultimately using to make its calls. As doing this for an entire BAM is quite compute intensive, we suggest writing out these small BAMs on test regions only, limiting your work using a “-L” argument to a region of ~1000 bp around the area you wish to examine. For example, you would run HaplotypeCaller as you previously did, but add “-bamout WhateverBAMNameYouLike.bam –L 19:38992600-38993600”. Make sure the chromosome designations match your reference genome designations. This would be –L 19 for GRCh37 and –L chr19 for hg19 reference.