Hadoop - MapReduce eclipse plugin not pointing to hdfs

-

MapReduce Eclipse plugin not pointing to local file system rather hdfs.


On your first run of WordCount from eclipse -> run on hadoop you may notice the java program uses local file system instead hdfs.

Adding hdfs location in your Configuration object solves this issue. In case of WordCount.java add the below into your main method.


public static void main(String[] args) throws Exception {
Configuration conf = new Configuration();

conf.addResource(new Path(HADOOP_INSTALL+"/conf/core-site.xml"));
conf.addResource(new Path(HADOOP_INSTALL+"/conf/hdfs-site.xml"));

}

Comments

Post a Comment

Popular posts from this blog

Log4j multiple WAR files in single EAR configuration

-
Log4j multiple WAR files in single EAR configuration Problem: When you have multiple WAR files under the same EAR with multiple log4j config files in each of the war files, the logging API will pickup the first available config file and ignores the list. i.e, only a single log4j config file applies to entire EAR. Consider the below app app.ear |__admin.war    |__WEB-INF/classes/log4.xml |__web.war    |__WEB-INF/classes/log4j.xml When you deploy this app to the server all logs of two war files admin.war and web.war will go to admin.war' log. The log4j.xml in web.war has no effect here. There are some complex solutions for this like adding a logback, specifying package level log files in log4j configuration etc. Solution The easiest way is configuring the context param in web.xml In your web.xml add the following context param will send the log4j config locations specific to the contexts and fixes this issue. web.war/WEB-INF/classes/log4j.xml <conte
More posts »

Java NIO2 - Watching a directory for changes. FileWatcherService.

-
Image
File Watch Service The NIO2 WatchService API provides the ability for java code to watch for file modifications in a directory in the filesystem. A WatchService object watches the folder it is registered with and signals the changes detected through event notifications.  Steps to create a WatchService Create a WatchService object (FileSystem.newWatchService()) Register the directories to be watched with the WatchService (WatchService.register()) Watch for file change events (poll()/take() methods) Retrieve the event keys and process them Rest the key and got to step 3 to watch again. Example code Below is a simple working code. import java.io.IOException; import java.nio.file.Path; import java.nio.file.Paths; import java.nio.file.StandardWatchEventKinds; import java.nio.file.WatchEvent; import java.nio.file.WatchKey; import java.nio.file.WatchService; import java.util.List; import java.util.concurrent.TimeUnit; public class FileWatcherTest { private static
More posts »

Ubuntu Add programs to launcher (Desktop)

-
Ubuntu Add programs to launcher (Desktop) Manual Create .Dekstop file  Move the file to ~/.local/share/applications (for local user) or /usr/share/applications folder (for global) S ample desktop file for dbeaver [Desktop Entry] Version=1.0 Type=Application Terminal=false Name=DBeaver Community GenericName=UniversaL Database Manager Comment=Universal Database Manager and SQL Client. Path=/usr/share/dbeaver-ce/current/ Exec=/usr/share/dbeaver-ce/current/dbeaver Icon=/usr/share/dbeaver-ce/current/dbeaver.png Categories=IDE;Development WM_CLASS=DBeaver StartupWMClass=DBeaver StartupNotify=true Keywords=Database;SQL;IDE;JDBC;ODBC;MySQL;PostgreSQL MimeType=application/sql   Alacarte Alacarte is a menu editing tool. sudo apt install alacarte will install alacarte. Open alacarte Create new menu entry Enter the command to start the item
More posts »