Using Spot Instances in Amazon EMR without the risk of losing the job

All of us look for saving cost on machines running on the cloud and one such option provided by Amazon are the SPOT machines. But is it practical to use these for our EMR jobs?

You can bid on Spot instances on EMR for your Hadoop jobs but in this case there is always a risk of losing the machines and therefore the job failing. This is not entirely correct since EMR allows us to launch a job with a few spot nodes(task) and a few core nodes.

From: http://aws.typepad.com/aws/2011/08/run-amazon-elastic-mapreduce-on-ec2-spot-instances.html

The EC2 instances used to run an Elastic MapReduce job flow fall in to one of three categories or instance groups:

Master– The Master instance group contains a single EC2 instance. This instance schedules Hadoop tasks on the Core and Task nodes.

Core – The Core instance group contains one or more EC2 instances. These instances use HDFS to store the data for the job flow. They also run mapper and reducer tasks as specified in the job flow. This group can be expanded in order to accelerate a running job flow.

Task – The Task instance group contains zero or more EC2 instances and runs mapper and reduce tasks. Since they don’t store any data, this group can expand or contract during the course of a job flow.

You can choose to use either On-Demand or Spot Instances for each of your job flows. This is valid for all of the above types. However, from the definition above if you lose a master or core machine then your job is bound to fail. Theoretically, you can have something like:

elastic-mapreduce –create –alive –plain-output
…
–instance-group master –instance-type m1.small –instance-count 1 –bid-price 0.098 \
–instance-group core –instance-type m1.small –instance-count 10 –bid-price 0.028 \
–instance-group task –instance-type m1.small –instance-count 30 –bid-price 0.018

But realistically, as you know, if you request spot instances, keep in mind that if the current spot price exceeds your max bid, either instances will not be provisioned or will be removed from the current job flow. Thus, if at any time the bid price goes higher and you lose any of your CORE or MASTER node then the job will fail. Both CORE and TASKS nodes run TaskTrackers but only CORE nodes run DataNodes so you would need at least one CORE node.

To hedge the complete lose of a jobflow, multiple instance groups can be created where the `CORE` group is a smaller complement of traditional on-demand systems and the `TASK` group is the group of spot instances. In this configuration, the `TASK` group will only benefit the mapper phases of a job flow as work from the `TASK` group is “hand back up” to the `CORE` group for reduction.

So say if you have to run a job which would ideally need 40 slave machines, then you can have say 10 machines(CORE group) as the traditional instance while other 30 as spot instances(TASK group). The syntax for creating the multiple instance groups is below:

elastic-mapreduce –create –alive –plain-output
…
–instance-group master –instance-type m1.small –instance-count 1 \
–instance-group core –instance-type m1.small –instance-count 10 \
–instance-group task –instance-type m1.small –instance-count 30 –bid-price 0.018.

This will help you to save cost by running SPOT instances as your nodes and at the same time make sure that job does not fail. However, keep in mind that it is possible, depending upon your price and the time taken to complete the job, the SPOT instances may come and go so might in the worst case end up incurring the same cost and taking longer time to complete the job. It will all depend on your bid price so choose the price wisely.

Is Big Data processing just a buzzword?

Reading the paper today, I happened to stumble across an interview of Werner Vogels who is the CTO at Amazon. It reminded me of a question posed to me a journalist at the NASSCOM conference in October this year. The question being as simple as “What is really new in Big data. It seems like it is just a buzzword. We have been processing data from a long time”.

I gave her a simple reply by asking her whether given a choice she would choose a 1980 model of a car or a 2012 one. Well, both help you to travel from one place to another, isn’t it. The simple reply is that Big data is normal data which would take normal machines a long time to process but Distributed computing would help to do it in parallel and achieve the same result much faster, As Werner Vogels said, “Drug development is one area. Say you want to stop the behaviour of a cancer protein. You need a molecule to do that. But you do not know which of the 20 million molecules you have will work. With a company’s regular processors, it could take up to two months to find a potential molecule candidate. But with the cloud, you can get enough computing capacity to do it in an hour.” 

This is what it exactly is. With tools like Hadoop and Mapreduce it makes it simple to process huge amounts of data in a short period of time without having to go through the pain of setting it up. And with the availability of huge computing resources in the cloud which are available on demand, it is hardly a matter of minutes to setup a cluster without a huge setup cost.

Mark Beyer, research vice president at Gartner, said: “Despite the hype, big data is not a distinct, standalone market, but it represents an industry-wide market force which much be addressed in products, practices and solution delivery.” He adds further, ““By 2020, big data features and functionality will be non-differentiating and routinely expected from traditional enterprise vendors and part of their product offerings.”

So Big data and its related technologies is not something new but rather a set of tools that help you to process your data in a much faster and efficient way enabling you to achieve your result much faster which would be expected to be a standard pretty soon with the growing amount of data that needs to be processed more efficiently.

Debug Custom Java hadoop code in local environment

A debugger or debugging tool is a computer program that is used to test and debug other programs (the “target” program). It is greatly useful specially for a Hadoop environment wherein there is little room for error and one small error can cause a huge loss.

Debugging Custom Java code for Hadoop in your local eclipse environment is pretty straight forward and does not take much time to setup.

As you would know, Hadoop can be run in the local environment in 3 different modes :

1. Local Mode
2. Pseudo Distributed Mode
3. Fully Distributed Mode (Cluster)

Typically you will be running your local hadoop setup in Pseudo Distributed Mode to leverage HDFS and Map Reduce(MR). However you cannot debug MR programs in this mode as each Map/Reduce task will be running in a separate JVM process so you need to switch back to Local mode where you can run your MR programs in a single JVM process.

Here are the quick and simple steps to debug this in your local environment:

1. Run hadoop in local mode for debugging so mapper and reducer tasks run in a single JVM instead of separate JVMs. Below steps help you do it.

2. Configure HADOOP_OPTS to enable debugging so when you run your Hadoop job, it will be waiting for the debugger to connect. Below is the command to debug the same at port 8080.

(export HADOOP_OPTS=”-agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=8008“)

3. Configure fs.default.name value in core-site.xml to file:/// from hdfs://. You won’t be using hdfs in local mode.

4. Configure mapred.job.tracker value in mapred-site.xml to local. This will instruct Hadoop to run MR tasks in a single JVM.

5. Create debug configuration for Eclipse and set the port to 8008 – typical stuff. For that go to the debugger configurations and create a new Remote Java Application type of configuration and set the port as 8080 in the settings.

7. Run your hadoop job (it will be waiting for the debugger to connect) and then launch Eclipse in debug mode with the above configuration. Do make sure to put a break-point first.

That is all you need to do.

Challenges and Best practices when working with location based data

This is a presentation I gave at a maps conference regarding the challenges when working with Location based data.

It is very interesting on how Location based data affects so many aspects of our lives today. In this presentation I talk about how we map locations to tiles for our maps and then every add or log that we receive is mapped to a certain tile at a specific time-period and how this helps us aggregating the data received.

It also talks about how we optimized hadoop to give us huge savings.

What is Big Data?

In information technology, big data is a collection of data sets so large and complex that it becomes difficult to process using on-hand database management tools and conventional programming methodology. Simply storing, understanding, curating and making sense of this data becomes a challenge. Processing and visualizing this data in a reasonable time-frame becomes an enromous task in itself. Verifying the result after finally processing it, is yet another story. In 2012, Gartner updated its definition of Big Data as follows: “Big Data are high-volume, high-velocity, and/or high-variety information assets that require new forms of processing to enable enhanced decision making, insight discovery and process optimization.”

Meteorology, Genomics, Connectomics, Complex physics simulations,Biological and environmental research, Prevention of diseases, Legal citations, Targeted advertising, Combatting crime and maps are a few simple examples where data is becoming big. Companies are now having huge data stores but have no clue how to process this information.

Data is being continuously gathered by ubiquitous information-sensing mobile devices, aerial sensory technologies (remote sensing), software logs, cameras, microphones, radio-frequency identification readers and wireless sensor networks among others.

All this data needs to be processed in parallel to make any sense out of it in a reasonable time-frame. Massive prallel processing is required to process it and the conventional databases are no longer suitable. Several tools in this domain have surfaced which can help you in make senses of this data, an example of which is Apache Hadoop. Additional technologies being applied to big data include massively parallel-processing (MPP) databases, search-based applications, data-mining grids, distributed file systems, distributed databases, cloud computing platforms, the Internet, and scalable storage systems.

Many people feel that only big companies today can afford clusters and harware to process this big data but today with the affordable and scalable cloud infrastructure it is possible for smaller companies to solve their Big Data problems.

You can read more on this at:
http://en.wikipedia.org/wiki/Big_data