This is part 2 of setting up a single-node Hadoop Yarn system for sandbox use. Part 1 was here, or for the series for using MapReduceV1, go here. I’m hoping to keep this series in a similar order as the original set of articles, and will deviate only when necessary. All the content here is based on the Cloudera documentation, but I’ve modified it to be easier to follow for setting up a pseudo cluster and added additional content where necessary.
Please be careful when copying lines from these articles to paste into Hadoop config files or a terminal window. I have found that the double hyphen characters used in the comment lines may copy over as a long hyphen instead. This is likely to cause issues when attempting to run the various components.
Continue reading “Setup a Single-node Hadoop Yarn machine using CDH5 – Part 2”
Previously I posted a series of articles that walked through installing a single-node Hadoop machine using version 1 of MapReduce. Since that series went live, a good deal of development has moved on to MRv2 aka YARN. I won’t go into the intricacies of the new architecture, suffice to say that it aims to be more efficient than the older MRv1. This article and any following it are based on the web documentation from Cloudera. While their information is very thorough, it often jumps around making it difficult for someone new to Hadoop to follow. Hopefully the instructions I will be providing here will make it easier for others to setup as well.
Please be careful if you are copying lines from these articles to paste into your Hadoop config files. I have found that the double hyphen characters used in the comment lines may copy over as a long hyphen instead. This is likely to cause issues when attempting to run the various components.
Some may ask, “What’s the point?” Cloudera, Hortonworks and other vendors offer Sandbox VM’s of their distribution that you can download and play with so why install it yourself? My answer to that is, by installing it myself, I can learn more about how it all fits together, and gives me a better understanding of the whole Hadoop ecosystem.
WARNING: This is not meant to be a production system, and I provide no warranties as to the usability of the system once you are complete. If you are setting up a production system and/or one that will exposed outside your firewall, please make sure you enable adequate security measures!
Continue reading “Setup a Single-node Hadoop Yarn machine using CDH5 – Part 1”
I’ve been experimenting and working around the fringes of Hadoop for over a year now, and seen lots of books on how to use Hadoop – programming through MapReduce and other applications and using HDFS as a backend data store. But until now there has been almost nothing released on how to administer and maintain a Hadoop cluster system. Packt Publishing has released a new book to help fill that void, Cloudera Administration Handbook by Rohut Menon. Continue reading “Review: Cloudera Administrators Guide”
SysAdm purists often look down on people who use a GUI to handle tasks on their servers, but having worked for several years on Novell Netware at the beginning of my career (shudder), give me a GUI over a command line every time! On my home CentOS servers, I have the GNOME desktop environment loaded, and it makes me a lot more productive, because I don’t have to remember the locations of many scripts, or the various command line switches to run various applications.
Recently, I was installing a replacement server for my Hadoop cluster, and I found that the Services GUI option was not present under the System – Administration menu. A little hunting turned up that the system-config-services package wasn’t installed. If this happens to you, here’s a quick way to get it back. Open up a terminal – kidding!!!
- Start the Add/Remove Software application from the Administration menu under System.
- Search for system-config-services and check the two options that should appear. One is the application, the other the documentation.
- Click Apply down in the lower right corner, and authenticate as Root.
- Wait a few second, then check under System – Administration. Service should be back right above Software Update.