9/21/2016

Visual Studio 2015 Hadoop Component ETL using Hortonworks Hyper-V Sandbox [Part-1]

Today I am going to get a project working to move files from a particular location on a local laptop up to a Hortonworks Hadoop Linux Virtual Machine running on Hyper-V on Windows.

The first step, download Visual Studio 2015.  Clicked on "Select All" and started the installation process:



Load the Hyper-V instance of Hortonworks Hadoop.  You can find my previous post on how to do this here.


Starting Hyper-V, we see the "New Virtual Machine":



First, check the assigned ram for the Virtual Switch, it defaults to 1024, I changed it to 2048, although they recommend minimum 4gig ram:



We first click the "Start" button, then click the "Connect" button.  The Virtual Machine begins its process to spin up the machine:



Prompts to log in:



username: root

password (first time, it prompts to change it): hadoop

remember what the new password is, can costs a bit of time trying to remember, wink, wink!


In order to get the IP Address, mine is not connected to the Internet, type "ifconfig":




What's interesting is now that I'm trying to work with my sandbox, it didn't run in the Hyper-V environment.  So basically have to troubleshoot from start.


I wasn't able to connect via internet explorer, so tried Chrome. 


I was able to load the VM in Hyper-V, using username and new password, but it indicated a "network connection issue":



So digging in a little deeper, you have to modify your Hosts file in "C:\Windows\System32\drivers\etc", the Hosts file.  I created a link between the Hyper-V IP address, 192.168.0.27 to Sandbox.Hortonworks.com:



The thing to remember, you must open the file as "administrator" or it won't let you save your changes, I used Notepad++.

Next, you'll want to go to Hadoop VM, and update your Hosts file and modify using "vi hosts":


Next, you have to allow permission to access the Hyper-V VM from your laptop, which hosts the VM, in the "hosts.allow" file:


Remember, to exit from VI editor in Linux, you press Esc, then colon ":" then "q". or if you want to saving change, Esc, then colon ":" then "wq".


In the hosts.allow file, I have added the "All" feature, followed by colon ":" and the prefix of the calling IP address of my local laptop:



This set of IP address' have access to the VM.


There's also a Hosts.Deny file, which I didn't modify:




The hosts, hosts.allow and hosts.deny files reside within the "etc" folder on Hadoop.


While booting up, because the Virtual Machine image was previously an Oracle VM, the "eth0" was not able to resolve.  After a bit of research, it turns out we need to tweak this setting.


So we drill into our ETC folder -->; udev -->; rules.d folder, to a file called:  "70-persistent-net.rules.  This file translates the "eth0" or "eth1" or eth2" to the mac address.  I found a link that said to remove the file using the "rm filename" command and it removes the file.  It rebuilds the file after restarting the VM.


Here's the contents of the file:



But first, we need to visit another file in another folder, the etc -->; sysconfig -->; "network-scripts" -->; "ifcfg-eth0", we view its contents:


I commented out the DHCP stuff by prefixing with "#" along with a few other entries not needed and added HWADDR="the mac address" from the other file:


 
You can see, I changed the IP address (static) to 192.168.0.27, that way it's on the same gateway of 192.168.0.1, otherwise it doesn't work.




The entries with # indicate "commented out" and are not read as part of the file.  The HWADDR is how the VM translates the newly assigned IP Address with the assigned Mac address.

To restart the service, type: "Service Network Restart" and it flushes and rechecks the "eth0" setting:



Next, we do an "ifconfig" to see the IP address assigned to "eth0":


We see the IP address is 192.168.0.27 and Mask is 255.255.255.0 which is what we are expecting.


So that appears to be fixed.  Here's a Hortonworks URL for reference: https://community.hortonworks.com/articles/49082/network-issues-after-cloning-a-virtual-machine-wit.html


Now, we still have the issue of connecting to the VM from the host laptop.  So we verify our Hyper-V adapter, in this case, we chose "internal":




We assigned the Internal Switch to our VM in the Settings of Hyper-V VM:



Next, we check the Network Adapter under the Network settings of the host laptop, we assigned a static IP in the 192.168.0.10:



We also need to share our internal network device, found this great post to  assist: https://blogs.msdn.microsoft.com/virtual_pc_guy/2008/01/09/using-hyper-v-with-a-wireless-network-adapter/


We also need to modify our WiFi adapter settings, can't use Obtain an IP address automatically, because it must be on the same 192.168. range:



So now we start up our Hyper-V Sandbox for Hortonworks VM and we see the IP address finally updates:



We also ping the Hyper-V IP to make sure we have round trip access:


We then check that the IP configurations were set properly on the host local laptop:



I noticed the DHCP was not enabled, but it seemed to be okay.


Next, our Hyper-V Hadoop24 VM loads:



We ignore the 127.0.0.1:8888 entry as we manually changed IP in the ifcfg-eth0 file to 192.168.0.27 earlier...attempt to load page via IP Address:



Next, we attempt to load using the assigned name in the local hosts file, sandbox.hortonworks.com:




And finally, we attempt to load Ambari, port 8080 a
nd it displays the Login credential info:


And we log in:



One other thing to mention, during the process, I had to set a rule in the Firewall to open up some ports.

Next, we'll connect with our Hyper-V Sandbox environment via  Visual Studio 2015 in Part 2, click the following link...


Visual Studio 2015 Hadoop Component ETL using Hortonworks Hyper-V Sandbox [Part-2]


Thanks for reading~!

No comments:

Post a Comment

Bloom Consulting Since Year 2000