11/15/2016

Hadoop Config Setting "yarn.acl.enable" Solves Hive Issue

The Visual Studio 2015 components stopped working.  They sent the jobs off to Hadoop, no return.  And the Hive command prompt didn't open when instantiated.  Just hung there.

After some research, decided to try Map Reduce as default instead of Tez. Hive started.  VS components did not.

Troubleshot some more.  Had to set it down to get other work done.

So then I had to prepare for the demo on Polybase and sure enough my Hadoop sandbox stopped working the day prior to the demo.

So actually read my own blog and blog to redo the steps to get Hyper-V Sandbox working.  While there I poked around at some of the settings.  Actually, all the Hive settings, Pig, Tez and Yarn.

Decided to change default back to Tez.

Also noticed the /user/yarn/ folder was missing in HDFS, added that.


Looking at the resources, HDFS was filling up throwing an alert, for no reason.  Looking at the jobs, there were 171 pending jobs in the queue.







It was in the Yarn settings something didn't look correct.  Did some research online, although the default setting for yarn.acl.enable = false, didn't seem correct as that schedules the jobs in the queue for Yarn.  Restarted the services, no luck.

 
So I sent the command "yarn application -kill 171 times to flush the queue.  Typed "hive" from command prompt, sure enough, Hive opened under Tez. 


Ran Visual Studio 2015 Hadoop HDFS, Pigg and Hive scripts, all ran.



This was a tough one.  Hadoop has SO many configuration settings.  And everything is interconnected.  And I wasn't supposed to be solving this issue at the moment.

Back in business.

And there you have it~!

No comments:

Post a Comment

We Interrupt this Broadcast