Hive Error : FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask

Question

I have got twitter data using flume on HDFS. Have 3 node cluster and MySQL Metastore for hive.

When i execute below query

select user_name.screen_name, user_name.followers_count from BBMP_election_tweets limit 10;

It is giving proper results.

But if i run below query

select user_name.screen_name, user_name.followers_count c from BBMP_election_tweets order by c desc;

It throws below error

Query ID = hduser_20150827102628_15a405ae-59a4-4c90-aa47-045d58c62f02 Total jobs = 1 Launching Job 1 out of 1 Number of reduce tasks determined at compile time: 1 In order to change the average load for a reducer (in bytes): set hive.exec.reducers.bytes.per.reducer= In order to limit the maximum number of reducers: set hive.exec.reducers.max= In order to set a constant number of reducers: set mapreduce.job.reduces= Starting Job = job_1439458788229_0016, Tracking URL = http://hdmaster:8088/proxy/application_1439458788229_0016/ Kill Command = /usr/local/hadoop/bin/hadoop job -kill job_1439458788229_0016 Hadoop job information for Stage-1: number of mappers: 2; number of reducers: 1 2015-08-27 10:27:33,567 Stage-1 map = 0%, reduce = 0% 2015-08-27 10:27:43,908 Stage-1 map = 50%, reduce = 0% 2015-08-27 10:27:45,015 Stage-1 map = 0%, reduce = 0% 2015-08-27 10:28:31,011 Stage-1 map = 100%, reduce = 100% Ended Job = job_1439458788229_0016 with errors Error during job, obtaining debugging information... Examining task ID: task_1439458788229_0016_m_000001 (and more) from job job_1439458788229_0016

Task with the most failures(4): ----- Task ID: task_1439458788229_0016_m_000001

URL:
http://hdmaster:8088/taskdetails.jsp?jobid=job_1439458788229_0016&tipid=task_1439458788229_0016_m_000001 ----- Diagnostic Messages for this Task: Error: java.lang.RuntimeException: Error in configuring object at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109) at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:75) at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:446) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:106) ... 9 more Caused by: java.lang.RuntimeException: Error in configuring object at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109) at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:75) at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133) at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:38) ... 14 more Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:106) ... 17 more Caused by: java.lang.RuntimeException: Map operator initialization failed at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.configure(ExecMapper.java:140) ... 22 more Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.ClassNotFoundException: Class com.cloudera.hive.serde.JSONSerDe not found at org.apache.hadoop.hive.ql.exec.MapOperator.getConvertedOI(MapOperator.java:328) at org.apache.hadoop.hive.ql.exec.MapOperator.setChildren(MapOperator.java:338) at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.configure(ExecMapper.java:109) ... 22 more Caused by: java.lang.ClassNotFoundException: Class com.cloudera.hive.serde.JSONSerDe not found at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1980) at org.apache.hadoop.hive.ql.plan.PartitionDesc.getDeserializer(PartitionDesc.java:141) at org.apache.hadoop.hive.ql.exec.MapOperator.getConvertedOI(MapOperator.java:302) ... 24 more

FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask MapReduce Jobs Launched: Stage-Stage-1: Map: 2 Reduce: 1 HDFS Read: 0 HDFS Write: 0 FAIL Total MapReduce CPU Time Spent: 0 msec

When checked the logs created. It gives below message

2015-08-27 10:28:48,852 INFO [main] org.apache.hadoop.mapred.YarnChild: Kind: mapreduce.job, Service: job_1439458788229_0016, Ident: (org.apache.hadoop.mapreduce.security.token.JobTokenIdentifier@988145f) 2015-08-27 10:28:49,023 INFO [main] org.apache.hadoop.mapred.YarnChild: Sleeping for 0ms before retrying again. Got null now. 2015-08-27 10:28:49,627 INFO [main] org.apache.hadoop.mapred.YarnChild: mapreduce.cluster.local.dir for child: /tmp/hadoop-hduser/nm-local-dir/usercache/hduser/appcache/application_1439458788229_0016 2015-08-27 10:28:50,363 INFO [main] org.apache.hadoop.conf.Configuration.deprecation: session.id is deprecated. Instead, use dfs.metrics.session-id 2015-08-27 10:28:51,186 INFO [main] org.apache.hadoop.mapred.Task: Using ResourceCalculatorProcessTree : [ ] 2015-08-27 10:28:51,534 INFO [main] org.apache.hadoop.mapred.MapTask: Processing split: Paths:/user/flume/tweets/BBMP_elections/2015/08/25/11/FlumeData.1440484157685:0+52974,/user/flume/tweets/BBMP_elections/2015/08/25/11/FlumeData.1440484188899:0+20119InputFormatClass: org.apache.hadoop.mapred.TextInputFormat

2015-08-27 10:28:51,701 INFO [main] org.apache.hadoop.hive.ql.exec.Utilities: PLAN PATH = hdfs://hdmaster:8020/tmp/hive/hduser/55daf291-b44e-414b-b3a5-e326134f9f24/hive_2015-08-27_10-26-28_312_5076847637673296334-1/-mr-10004/6e310852-ea63-4789-93f1-754147923e7f/map.xml 2015-08-27 10:28:51,701 INFO [main] org.apache.hadoop.hive.ql.exec.Utilities: ***************non-local mode*************** 2015-08-27 10:28:51,701 INFO [main] org.apache.hadoop.hive.ql.exec.Utilities: local path = hdfs://hdmaster:8020/tmp/hive/hduser/55daf291-b44e-414b-b3a5-e326134f9f24/hive_2015-08-27_10-26-28_312_5076847637673296334-1/-mr-10004/6e310852-ea63-4789-93f1-754147923e7f/map.xml 2015-08-27 10:28:51,701 INFO [main] org.apache.hadoop.hive.ql.exec.Utilities: Open file to read in plan: hdfs://hdmaster:8020/tmp/hive/hduser/55daf291-b44e-414b-b3a5-e326134f9f24/hive_2015-08-27_10-26-28_312_5076847637673296334-1/-mr-10004/6e310852-ea63-4789-93f1-754147923e7f/map.xml 2015-08-27 10:28:51,719 INFO [main] org.apache.hadoop.hive.ql.log.PerfLogger: 2015-08-27 10:28:51,720 INFO [main] org.apache.hadoop.hive.ql.exec.Utilities: Deserializing MapWork via kryo 2015-08-27 10:28:52,043 INFO [main] org.apache.hadoop.hive.ql.log.PerfLogger: 2015-08-27 10:28:52,080 INFO [main] org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader: Processing file hdfs://hdmaster:8020/user/flume/tweets/BBMP_elections/2015/08/25/11/FlumeData.1440484157685 2015-08-27 10:28:52,080 INFO [main] org.apache.hadoop.conf.Configuration.deprecation: map.input.file is deprecated. Instead, use mapreduce.map.input.file 2015-08-27 10:28:52,080 INFO [main] org.apache.hadoop.conf.Configuration.deprecation: map.input.start is deprecated. Instead, use mapreduce.map.input.start 2015-08-27 10:28:52,080 INFO [main] org.apache.hadoop.conf.Configuration.deprecation: map.input.length is deprecated. Instead, use mapreduce.map.input.length 2015-08-27 10:28:52,080 INFO [main] org.apache.hadoop.mapred.MapTask: numReduceTasks: 1 2015-08-27 10:28:52,228 INFO [main] org.apache.hadoop.mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584) 2015-08-27 10:28:52,228 INFO [main] org.apache.hadoop.mapred.MapTask: mapreduce.task.io.sort.mb: 100 2015-08-27 10:28:52,228 INFO [main] org.apache.hadoop.mapred.MapTask: soft limit at 83886080 2015-08-27 10:28:52,228 INFO [main] org.apache.hadoop.mapred.MapTask: bufstart = 0; bufvoid = 104857600 2015-08-27 10:28:52,228 INFO [main] org.apache.hadoop.mapred.MapTask: kvstart = 26214396; length = 6553600

2015-08-27 10:28:52,243 INFO [main] org.apache.hadoop.hive.ql.exec.MapOperator: Initializing operator MAP[3] 2015-08-27 10:28:52,243 INFO [main] org.apache.hadoop.hive.ql.exec.MapOperator: Initialization Done 3 MAP 2015-08-27 10:28:52,243 INFO [main] org.apache.hadoop.hive.ql.exec.MapOperator: Operator 3 MAP initialized 2015-08-27 10:28:52,262 WARN [main] org.apache.hadoop.mapred.YarnChild: Exception running child : java.lang.RuntimeException: Error in configuring object at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109) at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:75) at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:446) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:106) ... 9 more Caused by: java.lang.RuntimeException: Error in configuring object at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109) at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:75) at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133) at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:38) ... 14 more Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:106) ... 17 more Caused by: java.lang.RuntimeException: Map operator initialization failed at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.configure(ExecMapper.java:140) ... 22 more Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.ClassNotFoundException: Class com.cloudera.hive.serde.JSONSerDe not found at org.apache.hadoop.hive.ql.exec.MapOperator.getConvertedOI(MapOperator.java:328) at org.apache.hadoop.hive.ql.exec.MapOperator.setChildren(MapOperator.java:338) at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.configure(ExecMapper.java:109) ... 22 more Caused by: java.lang.ClassNotFoundException: Class com.cloudera.hive.serde.JSONSerDe not found at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1980) at org.apache.hadoop.hive.ql.plan.PartitionDesc.getDeserializer(PartitionDesc.java:141) at org.apache.hadoop.hive.ql.exec.MapOperator.getConvertedOI(MapOperator.java:302) ... 24 more

2015-08-27 10:28:52,269 INFO [main] org.apache.hadoop.mapred.Task: Runnning cleanup for the task 2015-08-27 10:28:52,278 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping MapTask metrics system... 2015-08-27 10:28:52,278 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: MapTask metrics system stopped. 2015-08-27 10:28:52,279 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: MapTask metrics system shutdown complete.

Please suggest. How can this issue be resolved. and why is it occuring.

Thanks

Answer 1

It is a classpath problem Root cause is java.lang.ClassNotFoundException: Class com.cloudera.hive.serde.JSONSerDe

you can have a look here to add hive-serdes: https://github.com/cloudera/cdh-twitter-example