Hadoop temp direcotry issue - namenode disappear while restarting.


Hadoop temp direcotry issue 


While running hadoop in psudo-distributed mode  you may see the error

'directory /tmp/hadoop-{user}/dfs/name is inconsistent state'.

This happens because each system shutdown cleans up unwanted temp folders and files from /tmp folder.So pointing your hadoop dfs location other than default /tmp folder will solve this issue.

You can edit onf/hdfs-site.xml and set dfs location specific to your box

<property>
  <name>dfs.name.dir</name>
  <value>/hadoopstorage/name/</value>
</property>

<property>
  <name>dfs.data.dir</name>
  <value>/hadoopstorage/data/</value>
</property>

This will remove namenode, datanode errors. But you may get errors related jobtracker and other process still the most other files pointing to the default /tmp directory.

The default value for all of the hadoop related directories will be like below.

  • dfs.name.dir -> ${hadoop.tmp.dir}/dfs/name.
  • dfs.data.dir -> ${hadoop.tmp.dir}/dfs/data.
  • fs.checkpoint.dir:-> ${hadoop.tmp.dir}/dfs/namesecondary.


One quick solution is to change hadoop tmp directory to point folder of your interest.

Edit hdfs-site.xml as below

<configuration>
     <property>
         <name>dfs.replication</name>
         <value>1</value>
     </property>
     <property>
          <name>hadoop.tmp.dir</name>
          <value>/hadoopstorage/hadoop-${user.name}</value>
     </property>
</configuration>




Comments

  1. It'sVery informative blog and useful article thank you for sharing with us , keep posting learn more
    Hadoop admin Online Course

    ReplyDelete
  2. Thanks for one marvelous posting! I enjoyed reading it; you are a great
    author. I will make sure to bookmark your blog and may come back
    someday. I want to encourage that you continue your great posts, have
    a nice weekend!


    Digital Marketing Training in Chennai

    Digital Marketing Course in Chennai


    ReplyDelete

Post a Comment

Popular posts from this blog

Log4j multiple WAR files in single EAR configuration

Ubuntu / kubuntu - Laptop key board cursor jumping issue

Java NIO2 - Watching a directory for changes. FileWatcherService.