Hadoop Example: WordCount v1.0 Part-2

Usage

Compile WordCount.java:

$ mkdir wordcount_classes $ javac -cp classpath -d wordcount_classes WordCount.java

where classpath is:

• CDH4 - /usr/lib/hadoop/*:/usr/lib/hadoop/client-0.20/*
• CDH3 - /usr/lib/hadoop-0.20/hadoop-0.20.2-cdh3u4-core.jar

Create a JAR:

$ jar -cvf wordcount.jar -C wordcount_classes/ .

Assuming that:

• /user/cloudera/wordcount/input - input directory in HDFS
• /user/cloudera/wordcount/output - output directory in HDFS

Create sample text files as input and move to HDFS:

$ echo "Hello World Bye World" > file0
$ echo "Hello Hadoop Goodbye Hadoop" > file1
$ hadoop fs -mkdir /user/cloudera /user/cloudera/wordcount /user/cloudera/wordcount/input
$ hadoop fs -put file* /user/cloudera/wordcount/input

Run the application:

$ hadoop jar wordcount.jar org.myorg.WordCount /user/cloudera/wordcount/input /user/cloudera/wordcount/output

Output:

$ hadoop fs -cat /user/cloudera/wordcount/output/part-00000
Bye 1
Goodbye 1
Hadoop 2
Hello 2
World 2

Applications can specify a comma-separated list of paths that would be present in the current working directory of the task using the option -files. The -libjars option allows applications to add JARs to the classpaths of the maps and reduces. The -archives allows them to pass archives as arguments that are unzipped/unjarred and a link with name of the zip/JAR are created in the current working directory of tasks. More details about the command line options are available at Hadoop Command Guide.

Running wordcount example with -libjars and -files:

hadoop jar hadoop-examples.jar wordcount -files cachefile.txt -libjars mylib.jar input output