'Hadoop configuration object not pointing to hdfs file system

I am trying to create small Spark program in Java. I am creating a Hadoop configuration object as show below:

Configuration conf = new Configuration(false);
conf.addResource(new Path("/dir/core-site.xml"));
conf.addResource(new Path("/dir/hdfs-site.xml"));
conf.addResource(new Path("/dir/yarn-site.xml"));

When I call the following I get file system but it points to local file system and not the hdfs file system

FileSystem fs = FileSystem.get(conf);

I am new to Hadoop.

javahadoophdfs


Solution 1:[1]

I usually keep the core-site.xml (along with the other files) in the classpath and get the configuration as follows:

Configuration conf = new Configuration();
FileSystem localfs = FileSystem.getLocal(conf);
FileSystem hdfs = FileSystem.get(conf);

Here is the content of my core-site.xml:

<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
  <property>
    <name>fs.default.name</name>
    <value>hdfs://{hadoop-server-addr}</value>
  </property>
</configuration>

Solution 2:[2]

Add your hdfs-site.xml and core-site.xml to class path. And use following code.

Configuration conf = new Configuration();
FileSystem hdfs = FileSystem.newInstance(conf);

It will pick the configs mentioned in the xmls.

Solution 3:[3]

To debug the issue better, add this line after creating a new Configuration instance. This will give you a better idea.

Configuration conf = new Configuration(false);
conf.setQuietMode(false);

In most cases, defaulting to local filesystem instead of hdfs occurs when Configuration is not able to load resources specified by the Path strings. In such cases, the default behaviour of Configuration is to fail quietly and return null for requested key. In this case, the requested key is fs.defaultFS , the namenode host. FileSystem instance then defaults to the file:/// scheme which is the local file system.

Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source
Solution 1 bachr
Solution 2 Suraj Nayak
Solution 3 shunya

Related Questions