Hadoop temp direcotry issue - namenode disappear while restarting.


Hadoop temp direcotry issue 


While running hadoop in psudo-distributed mode  you may see the error

'directory /tmp/hadoop-{user}/dfs/name is inconsistent state'.

This happens because each system shutdown cleans up unwanted temp folders and files from /tmp folder.So pointing your hadoop dfs location other than default /tmp folder will solve this issue.

You can edit onf/hdfs-site.xml and set dfs location specific to your box

<property>
  <name>dfs.name.dir</name>
  <value>/hadoopstorage/name/</value>
</property>

<property>
  <name>dfs.data.dir</name>
  <value>/hadoopstorage/data/</value>
</property>

This will remove namenode, datanode errors. But you may get errors related jobtracker and other process still the most other files pointing to the default /tmp directory.

The default value for all of the hadoop related directories will be like below.

  • dfs.name.dir -> ${hadoop.tmp.dir}/dfs/name.
  • dfs.data.dir -> ${hadoop.tmp.dir}/dfs/data.
  • fs.checkpoint.dir:-> ${hadoop.tmp.dir}/dfs/namesecondary.


One quick solution is to change hadoop tmp directory to point folder of your interest.

Edit hdfs-site.xml as below

<configuration>
     <property>
         <name>dfs.replication</name>
         <value>1</value>
     </property>
     <property>
          <name>hadoop.tmp.dir</name>
          <value>/hadoopstorage/hadoop-${user.name}</value>
     </property>
</configuration>




Comments

  1. It'sVery informative blog and useful article thank you for sharing with us , keep posting learn more
    Hadoop admin Online Course

    ReplyDelete
  2. Thanks for one marvelous posting! I enjoyed reading it; you are a great
    author. I will make sure to bookmark your blog and may come back
    someday. I want to encourage that you continue your great posts, have
    a nice weekend!


    Digital Marketing Training in Chennai

    Digital Marketing Course in Chennai


    ReplyDelete
  3. A common playing card game in casinos, where the item is to get as near 온라인카지노 21 with out going over. When saved correctly the Running Count will start at zero and, if all the cards have been to be played out, would end at zero. This is outcome of|as a result of} there are an equal number of high cards and low cards. The HiLo depend is due to this fact referred to as a ‘Balanced Counting System’. Card Counting supplies the participant a mathematically provable opportunity to realize an advantage over the home. It should be understood that this doesn't guarantee that the participant will win.

    ReplyDelete

Post a Comment

Popular posts from this blog

SVN File Creation Date

Ubuntu / kubuntu - Laptop key board cursor jumping issue

Log4j multiple WAR files in single EAR configuration