Hadoop temp direcotry issue - namenode disappear while restarting.

-

Hadoop temp direcotry issue 


While running hadoop in psudo-distributed mode  you may see the error

'directory /tmp/hadoop-{user}/dfs/name is inconsistent state'.

This happens because each system shutdown cleans up unwanted temp folders and files from /tmp folder.So pointing your hadoop dfs location other than default /tmp folder will solve this issue.

You can edit onf/hdfs-site.xml and set dfs location specific to your box

<property>
  <name>dfs.name.dir</name>
  <value>/hadoopstorage/name/</value>
</property>

 <property>
  <name>dfs.data.dir</name>
  <value>/hadoopstorage/data/</value>
</property>

This will remove namenode, datanode errors. But you may get errors related jobtracker and other process still the most other files pointing to the default /tmp directory.

The default value for all of the hadoop related directories will be like below.

  • dfs.name.dir -> ${hadoop.tmp.dir}/dfs/name.
  • dfs.data.dir -> ${hadoop.tmp.dir}/dfs/data.
  • fs.checkpoint.dir:-> ${hadoop.tmp.dir}/dfs/namesecondary.


One quick solution is to change hadoop tmp directory to point folder of your interest.

Edit hdfs-site.xml as below

<configuration>
     <property>
         <name>dfs.replication</name>
         <value>1</value>
     </property>
     <property>
          <name>hadoop.tmp.dir</name>
          <value>/hadoopstorage/hadoop-${user.name}</value>
     </property>
</configuration>




Comments

  1. rmouniakNovember 20, 2018 at 10:16 PM

    It'sVery informative blog and useful article thank you for sharing with us , keep posting learn more
    Hadoop admin Online Course

    ReplyDelete
  2. mahaDecember 23, 2020 at 11:20 PM

    Thanks for one marvelous posting! I enjoyed reading it; you are a great
    author. I will make sure to bookmark your blog and may come back
    someday. I want to encourage that you continue your great posts, have
    a nice weekend!


    Digital Marketing Training in Chennai

    Digital Marketing Course in Chennai


    ReplyDelete

Post a Comment

Popular posts from this blog

Log4j multiple WAR files in single EAR configuration

-
Log4j multiple WAR files in single EAR configuration Problem: When you have multiple WAR files under the same EAR with multiple log4j config files in each of the war files, the logging API will pickup the first available config file and ignores the list. i.e, only a single log4j config file applies to entire EAR. Consider the below app app.ear |__admin.war    |__WEB-INF/classes/log4.xml |__web.war    |__WEB-INF/classes/log4j.xml When you deploy this app to the server all logs of two war files admin.war and web.war will go to admin.war' log. The log4j.xml in web.war has no effect here. There are some complex solutions for this like adding a logback, specifying package level log files in log4j configuration etc. Solution The easiest way is configuring the context param in web.xml In your web.xml add the following context param will send the log4j config locations specific to the contexts and fixes this issue. web.war/WEB-INF/classes/log4j.xml <conte
More posts »

Ubuntu / kubuntu - Laptop key board cursor jumping issue

-
Image
Ubuntu cursor jumping problem Issue : When you type in Ubuntu/Kubuntu you may get annoyed by cursor jumping here and there in the screen. This happens across all the applications. You might have tried adjusting key board settings, key stroke speed etc. But this is not related to your key board, but the touch pad! Cause: The cause of this issue is your touch pad. Yes, while typing parts of your hand/fingers often touch the keyboard. In windows this touch will not be deducted while typing. But for Ubuntu it is.  So when you type something and un-intentionally unknowingly touch your touch pad will take your cursor to the appropriate mouse position with respect to your touch forgetting you are typing something!!! Solution: Simple. Say your touch pad not to care the touches while I am eagerly typing something important.. How to say that? Open Synaptix Touch pad manager  Check Automatically switch of touch pad on keyboard activity Time to wait before switching touc
More posts »

Java NIO2 - Watching a directory for changes. FileWatcherService.

-
Image
File Watch Service The NIO2 WatchService API provides the ability for java code to watch for file modifications in a directory in the filesystem. A WatchService object watches the folder it is registered with and signals the changes detected through event notifications.  Steps to create a WatchService Create a WatchService object (FileSystem.newWatchService()) Register the directories to be watched with the WatchService (WatchService.register()) Watch for file change events (poll()/take() methods) Retrieve the event keys and process them Rest the key and got to step 3 to watch again. Example code Below is a simple working code. import java.io.IOException; import java.nio.file.Path; import java.nio.file.Paths; import java.nio.file.StandardWatchEventKinds; import java.nio.file.WatchEvent; import java.nio.file.WatchKey; import java.nio.file.WatchService; import java.util.List; import java.util.concurrent.TimeUnit; public class FileWatcherTest { private static
More posts »