I installed HortonWorks HDP 2.0. As part of the installation, the main server also installs Ambari – an open source server application that is used to monitor and manage Hadoop clusters (hence my reason for adding a fourth server with more memory than the other three) and Nagios – a server monitoring application. Continue reading “Set up a Hadoop Cluster – Part 4”
Third part of this series, and we haven’t even gotten to installing Hadoop yet! This is a quick one though.
One of the things you need to make Hadoop work is to be able to use SSH across all of the servers in your cluster. To enable it, log into each box, and go to System in the menu, Administration from the menu and over to Services. Click on it. Continue reading “Set up a Hadoop Cluster – Part 3”
In part 1 of this series, I talked about starting a project to set up a cluster of inexpensive refurbished PCs (cheap) to act as a test Hadoop cluster, and some of the issues I encountered in getting the operating system installed. With this part, I’ll talk a little bit about configuring these workstations to get them setup for Hadoop. (And I did want to add, I introduced a fourth PC into the mix, to act as the main Hadoop server – a Zotac Zbox PC with 4GB of memory.)
On all of the PCs, I added a static IP, using the DHCP assigned address. Before you start changing your connection info, I recommend opening a terminal window and entering the “ifconfig” command. This will show you the existing address for your PC, the gateway, the net mask and a lot of other info you probably won’t care about. Continue reading “Set Up a Hadoop Cluster – Part 2”
I have posted a few times about various aspects of my experimenting with Hadoop and other Big Data platforms. Between Thanksgiving and Christmas, I even maxed out the memory of my desktop PC (a Mac Mini) so I could run multiple virtual machines to act as a Hadoop cluster. It worked, but performance was poor.
Since then, I’ve been looking into getting a small number of cheap PCs to setup as a physical cluster. Continue reading “Set up a Hadoop Cluster – Part 1”