BryanHalfpap, Author at CTOvision.com

Choosing Your First Programming Language

BryanHalfpap November 23, 2013

Many new programmers struggle when it comes to selecting their first language to learn. Which is easiest, the most professionally useful, the most newbie-friendly? Lets find out by showing you a range of options, their strengths and weaknesses, and some information about the most common programming languages in demand today. At the end, we’ll make some recommendations and you’ll understand a bit more about why they were chosen.

The Task At Hand:

Every job requires the right kind of tool, and each programming language can be thought of as a separate tool. Just like some tools are good for hammering nails, driving screws, and leveling screws, some programming languages are innately more suitable than others for tasks like designing a website, interpreting text, or reacting to user input.

Lets go through a few common use cases:

Language	Typical Usage
Java	Thick-client applications, large web apps, web applets, cross-platform
C# / .Net	Windows applications
Objective C	iPhone/Mac Apps
PHP	Web Application back-ends
Javascript	User interaction through web browser
Python	Science apps, computer admin tools, automating repetitive tasks quickly, can run on Linux and Windows (like java)
Ruby	Robust & Large web applications
Perl	Working with lots of text
Bash	Automating Linux Activities (like starting/stopping processes)
C/C++	Medium to Large projects, Embedded development, applications where speed is a factor

All of these languages can perform the same (or nearly the same) tasks as all the others, but the amount of effort can vary widely from language to language. You’ll see that in some of the examples in the next section…

Detail-Oriented

All programming languages rely on structure to perform their tasks. This structure is known as Syntax. The syntax for English and Japanese are night and day, but the syntax for French and Spanish are very similar (if you remember your language classes from high school!). The same applies to computer languages. Lets look at some examples:

Java	C#
public class HelloWorld { public static void main(String[] args) { System.out.println(“Hello, World”); } }	public class Hello1 { public static void Main() { System.Console.WriteLine(“Hello, World!”); } }

Perl

Python

#!/usr/bin/perl
print “Hello World.\n”;

#!/usr/bin/python

print “Hello World\n”

You can see that Perl and Python are nearly as identical to each other as Java and C# are!

This makes sense because many languages are modeled after one another. By choosing a language that is syntactically similar to many other languages, you can practically learn multiple languages at once!

Hey, by the way, if you looked closely at those examples, you’ll notice some are simple, others are complex, and some require semicolons at the ends of lines while others don’t. If you’re just getting started in programming, sometimes it’s best to choose languages without many syntactical (or logical) rules because it allows the language to “Get out of its own way”. If you’ve tried one language and really struggled with it, try a simpler one!

Setup:

If you’ve seen your use case mentioned in the table above, or have decided on a language already, you’ll need a way to run it. Generally, scripting languages require something called an Interpreter whereas C and C++ require compilers. Almost all of the languages mentioned in this article (with the exception of C#) are easiest to set up with any flavor of Linux — it really doesn’t matter which kind. C# is a special case because you need Visual Studio and Windows to use that.

PHP really requires a lot of work to set up in tandem with Apache/IIS, so we’ll save it for your own Googling or another article. The languages and links below are the easiest to set up.

Language	Windows	Ubuntu Linux Console Command
Ruby	http://rubyinstaller.org/	apt-get install ruby
Python 2.7	http://www.activestate.com/activepython/downloads	apt-get install python
C#/.Net	http://www.microsoft.com/visualstudio/eng/downloads#d-2012-express	N/A
Java	http://www.oracle.com/technetwork/java/javase/downloads/index.html http://www.eclipse.org/downloads/	apt-get install openjdk
C++	http://www.microsoft.com/visualstudio/eng/downloads#d-2012-express	apt-get install build-essential gcc

You can see that things on Linux can be quite easy to set up, but of course requires some amount of familiarity with the operating system. If you’re not scared to jump into Linux feet-first and take some time to Google your issues, I highly recommend that you use Linux. Either way, you’ll be using the command line in order to debug your programs anyways.

Popularity Contest:

TIOBE.com maintains a who’s-who popularity list for all the most popular computer programming languages. It’s created by crawling various search engines and scraping the results. You can view all of their charts and graphs at http://www.tiobe.com/index.php/content/paperinfo/tpci/index.html

These are the top ten:

1	1	*	C	16.975%	-2.32%	A
2	2	*	Java	16.154%	-0.11%	A
3	4	*	C++	8.664%	-0.48%	A
4	3	*	Objective-C	8.561%	-1.21%	A
5	6	*	PHP	6.430%	+0.82%	A
6	5	*	C#	5.564%	-1.03%	A
7	7	*	(Visual) Basic	4.837%	-0.69%	A
8	8	*	Python	3.169%	-0.69%	A
9	11	*	JavaScript	2.015%	+0.69%	A
10	14	*	Transact-SQL	1.997%	+1.12%	A

So All This Information Is Nice But…

You need to make a decision, right. Well, lets make a couple more charts and then get you started:

1. I’m a Windows User and I want to Get Into Programming

If you’re making something simple and want some experience that will translate into real-world usefulness, start with C#/.Net — setup is easy, the Visual Studio really helps you out, and there are plenty of resources to help you, including technet and MSDN directly from Microsoft!

Those languages aren’t as easy as Python, but Python isn’t going to help you out as much if you’re looking to make a simple graphical application. That being said, if you just need a command-line utility, take a close look at Python.

2. I’m familiar with Linux and I want to make/automate something

Python. Simple, fast, incredibly powerful, easy to use and experience with it translates fairly well into other languages. Absolutely a must for beginners willing to get their feet wet with Linux.

3. I want to build a website

Javascript and PHP — These languages require the most amount of know-how to set up (you have to install a webserver and make sure it’s configured to run PHP) but aren’t too complicated once you get that ball rolling. PHP isn’t going to get you as much experience working with modern programming languages unless you use it in an Object-Oriented style though.

Javascript is very useful to know and is easy to learn. You’ll get a lot of use out of it, even without PHP. In fact, you can make and run Javascript programs without any setup, right in your browser.

Just give me a Language!

Python. It’s easy, already used by thousands, very extendable, doesn’t require as much attention to detail, and allows beginners to learn logic basics before going into object-oriented programming.

5 Webapps to Add to Your Security Tool Arsenal

BryanHalfpap November 12, 2013

When you’re mobile, away from the office, or just caught off-guard without your tools and security setup, you need a way to analyze, assess, and interpret emerging threats on-the-fly. That’s why these 5 free web applications are absolute must-haves for your Favorites Folder.

Site: http://urlquery.net/index.php

Purpose: Visits URLs so you don’t have to.

About: Scans whatever URL you give it, applies some snort to it, gives you a screenshot, and displays all HTTP transactions that page caused to run. It allows you to visit a site that might be malicious without actually exposing your web browser to any funk, then tells you if it encounters any exploit kits/snort rules and tells you what those are.

Site: http://robtex.com/

Purpose: Looking up Domain Name information, A must-have for malware analysis.

About: Lets you know if a given domain name is in a blacklist, tells you all the other records that map to the same IP. Also has awesome graphs and meta-info about a target domain.

Site: http://onlinedisassembler.com/odaweb/

Purpose: Disassembling binaries

About: This is probably one of the coolest and most-technical. Forget IDA pro at home? No problem. Upload your binary and it’ll spit out formatted assembly for you to look at, including sectioned formatting and hex dumps. It’ll also tell you if it’s known good or known bad (unknown how it’s determining that thought) Sweet.

Site: https://www.virustotal.com/

Purpose: Scans uploaded files or URLs for malware

About: You should already know about this one, but if you don’t, this is a great tool for using when you don’t quite trust a file. While it doesn’t run the file (which is important since many engines have run-time behavior checking) it will cross-reference the definitions of more than 40 antivirus vendors. There’s also a voting system and comments section that’ll give you more info if the results are mixed. Results show up in Google too.

Site: https://malwr.com/

Purpose: Runs your malware, for you

About: Upload malware, and malwr.com will run it inside of the Cuckoo Sandbox, an open-source malware analysis program built for the express purpose of compartmentalizing and analyzing malware samples. It’ll break everything down from registry changes, network traffic, ASCII strings dump, static and live analysis, the whole enchilada! Seriously powerful stuff if you know what you’re looking at/for. This free tool does as much as the $50,000 Fireeye Malware Analysis Tool.

The Problem With PEAP

BryanHalfpap September 24, 2013

In previous posts (The Wireless Attack Toolkit and What’s Trending For Hackers) I highlighted some common wireless security attacks. If you believe your corporate network has some additional security on top of it, well, this post is aimed directly at you.

Modern wireless security protocols have evolved a great deal since their inception with WEP (wired equivalent protections) and WPA (Wireless Protected Access). Now we have a bevy of wireless security protocols for corporate use and pretty good security for home users, all based off of WPA version 2. In this blog post, we’ll tackle the corporate side’s most typical implementation of WPA2 — PEAP (PEAP stands for Protected Extensible Authentication Protocol).

WPA2 PEAP is a wireless security protocol that implements 802.1X network access control mechanisms — you can’t connect to the network without an authorized username and password. Usernames and passwords are stored and checked by another portion of the corporate network, the Active Directory server (or any other LDAP server that you use if you’re not a Windows shop). This authentication actually uses the same pieces and parts as an HTTPS session to send your passwords and establish an encrypted tunnel. Pretty neat, right?

The problem that exists within this security protocol is in the implementation of the wireless standard and in the per-company implementation of the network.

SSL Certificates provide the underlying security mechanisms for many security protocols, including PEAP

Accepts all connections:

Some PEAP implementations apparently lack the ability to check that the encrypted tunnel is to a legitimate computer. So what happens when this is the case? Take some time to review how the “Evil Twin” attack is performed: I create a wireless network with the same name and characteristics as your intended network and I walk around outside your building advertising this network to any and everyone that will connect with me. Eventually someone tries to connect and establish an encrypted tunnel with me.

When they do this, I hand them an encryption certificate for their session that I control. If they are unable to verify the certificate is correct either because they have nothing to compare it to (no certificate installed) or they are unable to (the implementation doesn’t support certificate checking) then they will accept my certificate as valid and I’ll be able to witness their username and hashed password being transmitted! It happens that Active Directory tends to use a very weak cryptographic hash which can be cracked in under 24 hours for $100 using cloud-based services.

Now, not only is the client under my control and subject to a variety of attacks and sniffing, but I also have a way to sign into the corporate VPN for $100 because a certificate wasn’t deployed or checked.

This is how much it would cost me to get your password. — This is how much it would cost me to get your password, even if it’s long, in 24 hours.

Locking it down:

To get to the wireless settings, it’s somewhat circuitous.

If you don’t have a WPA2-Enterprise network, use this : Click on Network and Sharing Center > Set Up A New Connection > Manually Connect > Create a new network called ‘test’ and select WPA2-Enterprise from the security settings. Create the network and then click “Change connection settings” > click the security Tab > Settings

To properly secure an implementation, you could create your own CA, deploy the root cert to all machines, and then use the root CA to sign the radius server’s certificates. Then you can modify the settings for each client’s wireless setup to verify the cert. This is a bit overkill unless you need to have your own root CA (for stuff like monitoring SSL connections for malware, 802.1x, ect…) or you already have one.

If you want a more light-weight solution, you can get a certificate issued from a already-trusted Root CA for your radius server’s name (provided you have a valid domain and TLD that the certificate authority can issue a cert for). Then you can deploy this certificate on the radius server. Then configure your clients to validate the certificate against the certificate authority and put in the name of the radius server. The image below from the University of Texas Dallas does it right:

Validate Certs? Check. Radius Server Defined? Check. Lookin' Good! — Validate Certs? Check. Radius Server Defined? Check. Lookin’ Good!

They could go one step further (if this is a machine that the users do not have local admin on) and check the “Do not prompt user to authorize new servers or trusted certification authorities” box for additional security that won’t let users click through security prompts.

In closing, PEAP is actually secure when you deploy it correctly — with certificates created and deployed by your Public Key Infrastructure. Create your own root certificate for your network, add it to the certificate store in your device, and then enable certificate checking with that certificate authority. Prevent users from adding servers to the certificate authority list as well if possible. Then enable this across all phones, laptops, and tablets.

For more info on setting up an Active Directory PKI see : http://technet.microsoft.com/en-us/library/cc772393(v=ws.10).aspx

The Wireless Attack Toolkit

BryanHalfpap August 15, 2013

Wireless security is all about implementation. The vendors implementation of the stack, the implementation of the security protocol standards in your OS that you have deployed to your users, the ways that the radios work, and the settings you’ve chosen for your network. Unfortunately, wireless security is generally weak (unless we’re talking about a properly-deployed WPA2-PEAP setup, which we’ll go over in another post) because of the way network associations are performed in the protocol standard, the way networks are discovered, and the fact that management frames are not encrypted.

Your Previous Connection History Leaked:

When you first decide to turn on your wifi, you start leaking information about yourself. How much and what you leak depends on your wireless driver, operating system, and the networks that you’ve connected to in the past. In general, you leak two pieces of information: your uniquely assigned network card number (MAC Address) and the names of networks you’ve previously connected to.

Why do you leak the names of networks you’ve connected to? It’s supposed to be a feature — the operating system is attempting to see if you’re around any of those networks by sending out a beacon request to see who responds to them. If someone, anyone responds to a beacon, well, that must be a network you know! Connect! It works this way by design to allow you to roam between multiple access points for the same network.

capture_240 — HPSetup is an old holdover from the more insecure wireless days of early windows XP and is a good network to spoof for older computers, whereas the default linksys and belkin names are good gambles for SSID spoofing. Free Public Wifi also adds a good “Human Hack” element.

Attack 1 – Karma:

There’s a problem with this leakage — what happens when someone starts responding to any/every network beacon request? That’s when you start seeing your home network name at Starbucks, ten miles away! This attack is sometimes known as Karma.

If you’re thinking “The network i’m connected to uses encryption so I’m safe”, you’re wrong. Potentially any of your past network connections that weren’t encrypted are targets if they are beaconed for by your hardware. That’s why I delete wireless network connections that are unencrypted when I am done using them (if I use them at all).

It seems that Apple hardware is most susceptible to attacks of this nature. They are chatty by nature and almost always query the wireless around you looking for previous connections. Of Apple products, iPhones are consistently the worst and are well-known in the wireless security community for their promiscuousness. Older versions of XP, Android, and many embedded devices are also good targets for this kind of attack. Generally, up-to-date Windows and Linux aren’t as vulnerable to this, but everyone is vulnerable to another kind of attack against unsecured wireless connections — the Evil Twin.

This is your face when you realize your iPhones are getting pwn'd — This is your face when you realize your iPhones are getting pwn’d

Attack 2 – The Evil Twin:

The evil twin attack requires a two strong radio signal cards and a wireless network with the same name as a nearby but preferably weaker-signal wireless network that you want to attack.

You utilize one radio card to find clients of the target wireless network and remove them by abusing the WiFi connection protocol. Specifically, your wireless card talks on the same radio frequency channel impersonating the access point of the target network and tells its clients that they are disconnected.

Confused, the disconnected clients start looking for the wireless network again, but this time, your radio responds to the request louder and more quickly than the other network — forcing victims to connect to you. If you provide internet to these clients and the exchange is quick enough they may never even notice that they have been connected to your malicious access point.

The disconnection attack works even if your targets are connected to an encrypted wireless network, and may cause them to leap from that encrypted network onto an unprotected network that they’ve previously connected to and are now looking for since they’ve been disconnected. This happens because the management frames (the parts of the wifi protocol that allow you to connect and disconnect from a network) are not encrypted in (most) wireless encryption standards.

How do I test my vulnerability to this?

I’ve developed a software package to make testing wireless vulnerability to these well-known attacks very easy for someone acquainted with Linux. If you understand how to get your network interfaces working under linux, you should be able to use this software package.

WAT : Wireless Attack Toolkit

The Wireless Attack Toolkit automates the Karma and Evil Twin attacks and throws in a few other attacks for good measure. Normally you’d have to set up IPTables, compile the hostapd-karma software, configure a DHCP server, and a dozen other small things, but these aspects are all taken care of for you — you just specify which interfaces will be performing which tasks (connecting you to the internet, acting as the fake access point, attacking the victims) and you’re off to the races. You can watch people come in via the DHCP server, watch where they’re browsing via the DNS server, and attack their web browsers via some built-in software.

One feature of this toolkit is that this software can run on a Raspberry Pi for ~$260 in equipment costs. With a battery pack providing power and a mobile phone providing internet, the attack package can hack people onto your wireless connection and place them under your control for over 8 hours on a single charge. That’s a lot of iPhones.

Download the Wireless Attack Toolkit here: https://sourceforge.net/projects/piwat/

Check out my Defcon Presentation about WAT

Note: To be most effective with the Wireless Attack Toolkit, you should change the static Malicious Access Point Name. I’m working on making that easier.

Legal Note: It may be of questionable legality to run this software in public locations.

What’s Trending For Hackers?

BryanHalfpap August 13, 2013

While at Defcon (the largest computer security conference in the United States) I overheard it described as a sort of “Hacker New-Years holiday”. If that’s true, then maybe it’s time to go over one of the trends I’ve seen popping up for a while that has gained a lot of momentum this past year: Wireless hacking.

Wireless isn’t just about WiFi — there’s all sorts of protocols, specifications, and frequencies zipping through the air at any given moment, especially in the ISM bands. The ISM bands are a section of wireless spectrums which the Federal Communications Commission has designated as “Free Use”. The only limitation (generally speaking) is transmitting power. WiFi, RFID, Z-Wave, Zigbee, Bluetooth, cordless phones, and many, many more applications and appliances operate on these frequency bands. Many of the protocols in the ISM bands and especially those I mentioned are starting to get a lot of attention from hackers. This is because they can do such things as track (or spoof) planes in the air for ~$20…and much more.

The HackRF Jawbreaker, a $275 software defined radio that's shaking up the hacker community — The HackRF Jawbreaker, a $275 software defined radio that’s shaking up the hacker community

Helping hackers get into the radio scene are several low-cost software-defined-radios which have been developed and released to the public either for free (as circuit board designs and specifications) or at low cost. In addition to the hardware, very effective and fairly robust software has been released as well (GnuRadio). Some of these radios and their software can perform tasks which used to cost thousands or even hundreds of thousands of dollars in software and radio equipment. The best and cheapest example of this is probably the newly-released HackRF, an open-sourced radio design that can perform between 100 Mhz to 6 Ghz and is designed to operate with GNU radio. It costs around $275 and is currently outperforming its Kickstarter.com goal by leaps and bounds.

Hackers can use hardware like the HackRF in conjunction with signals processing software such as GNURadio to build interpretations of physical-layer radio signal protocols like your car key fob or building Zigbee sniffers, Bluetooth security tools and sniffers, or even pager networks with low-cost hardware and free software.

As the cost of radio hardware falls and more players enter the radio arena, you can expect more attention to be paid to these (comparatively) little-known and poorly-secured protocols. Perhaps the attention from the sort of people that attend security conferences like Blackhat and Defcon will force better protocol design and encryption implementation for things like home automation systems, power company meter infrastructure, and industrial automation equipment. If you utilize lesser-known radio protocols and protocol stacks in your workplace, it will be worth keeping tabs on this research.

Bug Bounty Programs : Encourage Responsible Disclosure

BryanHalfpap April 17, 2012

Ladybugs are still Bugs — Bug Bounty Programs Encourage Responsible Disclosure From Hackers

The idea that you might pay someone else to keep quiet a vulnerability while you fix it may seem a bit backward to some in computer security. It would also seem to invite attacks on infrastructure. It’s no surprise, then, that many companies with technological products don’t have bug bounties.

A bug bounty is a fee that is paid out whenever a hacker discovers a legitimate bug and co-ordinates with the company in order to recieve the fee rather than selling the exploit or announcing it to the world. It’s a technique that has come into the public spotlight by being implemented at such prestiegeous places as Facebook and Google.

But WHY? Why does it work?

For many hackers, the security vulnerability is the means and the end — that is, they hack in order to find a security vulnerability and once one is found, they know that the service is vulnerable. Some hackers inevitably choose to sell these or trade these in underground markets, but many wish to see them fixed (perhaps the hacker uses the service or understands the potential for damage should it be abused). Many will then attempt to contact the company and disclose the vulnerability. Should the company drag its feet or if the hacker is feeling particularly onery, the vulnerability will be publicly disclosed (typically referred to as a “grey hat” method of disclosure).

Having a bug bounty program lets computer security folks know:

You understand security processes and the motives behind finding vulnerabilities
That your company is dedicated to fixing the problem
That you wont shoot the messenger or try to stifle them
That they can earn more respect and money by working WITH you than AGAINST you

It’s a psychological change that can help garner you more security and more respect in the security community as an organization that is dedicated to secure software/hardware. That way you won’t end up with anonymous trying to wage Internet battle with your company.

Below is a list of bug bounties offered by various companies:

http://en.wikipedia.org/wiki/Pwn2Own
http://www.mozilla.org/security/bug-bounty-faq-webapp.html
http://www.mozilla.org/security/bug-bounty-faq.html
http://www.tarsnap.com/bugbounty.html
http://www.facebook.com/whitehat/bounty/
http://www.google.com/about/company/rewardprogram.html
http://www.chromium.org/Home/chromium-security/vulnerability-rewards-program

In the end it’s easier to pony up cash for vulnerabilities and fix them before they become security headaches or,worse, go unnoticed for long periods of time while they are used in advanced attacks. Encourage the behavior you want to see in the security community by implementing a bounty program of your own. Companies like Facebook and Google have already handed out tens of thousands of dollars for their bug bounty programs at between $500 and $1337 dollars a pop.

Passwords Suck: Learn about and use multi-factor authentication

BryanHalfpap February 13, 2012

Passwords suck.

They are long, hard to remember (even if you have easier-to-remember phrases), moreso when new, and are largely a difficulty for users to user properly. Combined with the fact that many users choose easy-to-guess or easy-to-ascertain passwords based off of commonly-known facts about themselves and that they will try all of their passwords when told one isn’t working…the list goes on.

Passwords are essential to our daily lives because they provide a facet of authentication security — the ability to confirm that you are who you say you are, because you know something someone else does not. In security circles, this type of password authentication is known as “something you know.” In authentication security experts have long pointed out that just having “something you know” is not good enough. Three key facets are required for good authentication:

Something you know
Somewhere you are
Something you have

Something you know is a password or passphrase, pin, code, or pattern that you transmit to a party in order to identify yourself. Somewhere you are is a location which is restricted or confirmed through GPS. Something you have is more commonly used, and includes tokens, RFID chips, badges, USB keys, CAC cards, and much more. These facets of authentication security, when blended together, form multi-factor authentication. The most common form of multi-factor authentication is Something You Have and Something you Know. This is what is commonly referred to when “Dual-factor authentication” is mentioned.

Why would you use multi-factor authentications? For the same reasons you might want to have multiple core routers in your datacenter, multiple employees performing the same job, or tape backups — redundancy.

Consider the following scenario:

Acme company is deploying a new web application for their business partners to better access manufacturing information. Unfortunately a SQL injection bug made it’s way into the production version, and a hacker managed to steal the database of usernames and passwords. In a single-factor password-based authentication scheme, the hacker would be able to crack hashed passwords in the database, login, and steal property from Acme business partners. Fortunately, Acme issues each customer a token which securely and pseudo-randomly generates a secure PIN every 10 seconds. Partners using the site use this as a secondary confirmation mechanism and will be unable to perform any actions without first providing a valid PIN.

Hacker halted.

Properly encrypted passwords would have also stopped this attacker from gaining access to user accounts, but what if the users or user accounts themselves were targeted? Even if someone guesses, cracks, or steals their way into a user password or access, without the token they will be unable to access systems with just a password.

Deploying dual-factor authentication can be as easy as turning it on in Google, deploying turnkey Active Directory solutions, purchasing solutions for websites, or adjusting Linux Pluggable Authentication Module (PAM) and authentication settings.

A Final Note:

Password security is important as a barrier to entry, but proper password procedures ensure that post-breach data theft including password databases are less severe. Proper password storage is essential to provide time post-breach for an organization to reassess and refactor security policy and reset affected user accounts.

Usually when a breach occurs involving passwords, we are talking about a Windows SAM credential dump or a SQL database (Unix/Linux passwords are automatically stored correctly with proper security measures). Ensure that Windows machines are NOT useing Lanman hashes, and consult your web developers to be sure that passwords are not being stored as simple hashes.

A proper password hash algorithm should look like this:

hash(hash(password) + salt)

Where the hash() function is a non-trivial (read: not MD5 or Base64) hash and the salt is a random string of data added to the password in order to slow password cracking by defeating rainbow tables and pre-generated brute-force dictionaries.

For more context see:

Cyberwar Is Now A 3-Way Cage Fight

Good Writing By Krebs: Half of all Phishing Sites Now Have the SSL Padlock

In Space, No One Can Hear You Scream

Hadoop Quickstart: Build a Cluster In The Cloud In 20 Minutes Or Less

BryanHalfpap January 10, 2012

Editor’s note: The tips Bryan Halfpap provides below really work. I stood up a working Hadoop Cluster in under 20 minutes, from cold iron to production ready, using just his guidance and a Rackspace account. bg

I’ve been working with Apache Hadoop in my lab, spending much of that with CDH3 (the Cloudera Distribution including Apache Hadoop). As part of my examination of the best way to move from test/evaluation/prototyping to production systems I’ve been examining cloud-based cluster capabilities. CTO Bob Gourley and I recently set up clusters in Amazon EC2 and Rackspace and were both impressed with the ability to get powerful systems up and running fast for very low cost.

With this post I’ll share some of the tips and techniques used to get these clusters up and running. These tips flow from the guide provided by Cloudera (see link at end of this post) but include a few other important tips that will help get you operational fast. I also provide more explanatory context to help you understand more what is going on as you execute these instructions. My only assumption here is that you have an understanding of the basic Linux command-line environment and have access to systems (perhaps in your environment or, like we did, at Amazon or Rackspace). Please note that this particular guide is written for 64-bit Redhat-flavored systems (CentOS/Fedora/ect…), and will need to be tailored for other OSs. Also note that we did things this way because we want to learn and to teach and thought many of you would like to see what is going on here as well. But there are actually better/faster ways than this. You can download configuration management systems that let you plan, start, manage and operate clusters very smartly using great GUIs and then manage them over the lifecycle of a production environment (see for example, the free Cloudera Manager). We will provide more on tools like that in the future. But now we want to show you how to stand up your environment using CDH3 by use of the command line. This is the fun way. We will walk you through the enterprise way in the near term.

Our first step flows from an understanding that Hadoop utilizes a lot of Java, and for that, you want to be sure that you are using the Sun/Oracle version. This ensures that any problems you come across aren’t in the Java implementation itself.

Bring up the Linux system you wish to install CDH/Hadoop on and run the commands below inside of the command prompt. This will return a list of installed packages called “OpenJDK“. OpenJDK is an open-source implementation of Java that we will need to remove.

rpm -qa | grep openjdk

If that returns a list of openjdk packages, you need to remove them before heading further. This command below will remove anything in yum which starts with “java”, and will remove openJDK from the system. We want to do this because the Sun/Oracle Java is much more stable and will ensure that all required features are present in Java.

yum remove java*

Once the packages have been removed, run the following commands in order to download and install the Sun/Oracle Java environs.

For 64-bit versions:

wget http://download.oracle.com/otn-pub/java/jdk/7u2-b13/jdk-7u2-linux-x64.rpm

Now we download and install the Java Runtime Environment.

For 64-bit versions:

wget http://download.oracle.com/otn-pub/java/jdk/7u2-b13/jre-7u2-linux-x64.rpm

Now that we have the packages downloaded, it is time to install them. Use the Redhat package manager (rpm) to install these files with the commands below.

rpm -i jre-7u2-linux-x64.rpm
rpm -i jdk-7u2-linux-x64.rpm

At this point we are ready to start downloading the Cloudera Distribution including Apache Hadoop (CDH). Redhat-flavor linux users of later distributions (Fedora 12+, latest versions of CentOS) can safely use the Redhat/CentOS 6 repositories from cloudera to install hadoop. If you are running later versions of those flavors, use the CentOS 5 repo.

For Latest Versions: http://archive.cloudera.com/redhat/6/x86_64/cdh/cloudera-cdh3.repo

For Legacy Versions: http://archive.cloudera.com/redhat/cdh/cloudera-cdh3.repo

You may notice that this is a repo file — it’ll add a repository to your package manager and help you keep your CDH software up to date. Simply add it to the yum repository like this (as root):

cd /etc/yum.repos.d/
wget http://archive.cloudera.com/redhat/6/x86_64/cdh/cloudera-cdh3.repo

Then update your package information from the repo with yum.

yum update

We use the repositories because it’s easier to keep CDH up-to-date with the package manager rather than building from source every time there is an update. Maintenance and setup are easier, and it’s easier to reinstall should we need to do that. Adding the repositories also allows you to install additional packages with yum with a few simple commands, rather than having to search for, then download and install any additional software add-ons for your Hadoop ecosystem.

Now you should be ready for the next step — downloading and installing Cloudera’s Hadoop. Do this with yum by using the following commands:

yum install hadoop-0.20

This will install the base of Hadoop and allow you to install the other packages that Hadoop requires to operate — namely the jobtracker, tasktracker, namenode, and datanode. All of those trackers and nodes are required in order to operate a Hadoop cluster. Choose which ones this machine will run, or install all of them at once if you are running in “pseudo-distributed mode”. Pseudo-distributed mode treats one computer as if it is an entire cluster, and is great for testing out prototype jobs or performing development on.

to install the packages, simply follow this syntax:

yum install hadoop-0.20-[NAME OF THE PACKAGE]

If you want to run hadoop just on one node, install all of the packages, then use the following configuration for Hadoop in the /etc/hadoop-0.20/conf folder: http://hadoop.apache.org/common/docs/r0.20.2/quickstart.html#PseudoDistributed

Below is a simple bash script for pseudo-distributed installs of CDH. This script will start the Hadoop services for you:

#!/bin/bash
service hadoop-0.20-datanode start
service hadoop-0.20-jobtracker start
service hadoop-0.20-namenode start
service hadoop-0.20-tasktracker start

Copy and paste that code into a file named “start-hadoop”, save it, and mark the file executable with the chmod command:

chmod +x start-hadoop

Then run it to see if everything starts up fine with:

./start-hadoop

If you did everything well, then you should be able to perform the following actions:

hadoop fs -copyFromLocal [filename] [filename in hdfs]
hadoop fs -cat [filename in hdfs]

Where the names in brackets are the names of a file you wish to import and then display in HDFS (Hadoop distributed file system). If you weren’t able to get that started fine, then you may need to repeat one of the steps or make sure you installed all the components you needed.

If you have any questions or if you’ve found an error in this Tutorial, please let me know on Twitter (@Crypt0s).

A full documentation including steps to install on Debian-based systems can be found at Cloudera’s website.

Note: Keep checking CTOvision.com for more Hadoop updates on software for Hadoop like Whirr. Whirr is another great capability, enabling the fast standup of cloud clusters by use of a simple properties file you update with your cloud provider info. We also have posts coming on the use of Cloudera’s Manager. Also, as previously mentioned, if you would like fast tips to standing up cloud-based servers to start these Hadoop Clusters on, see our Quickstart guides to Amazon and Rackspace servers.

Choosing Your First Programming Language

5 Webapps to Add to Your Security Tool Arsenal

The Problem With PEAP

The Wireless Attack Toolkit

What’s Trending For Hackers?

Bug Bounty Programs : Encourage Responsible Disclosure

Passwords Suck: Learn about and use multi-factor authentication

Hadoop Quickstart: Build a Cluster In The Cloud In 20 Minutes Or Less

Cyber War

Optimizing Corporate Intelligence with OODA Loop

The Short Story From Arthur C. Clarke Every Officer in The Military Should Read and Heed

This One Little Configuration Change Will Make It Harder For People To Steal Your Information

SciFi

Science Fiction Gadgets You Can Get Today

Super Bowl Ads Indicate Big Businesses Think You Are Afraid of AI

HBO’s Westworld: The Man In Black is a special kind of Twitter user

BryanHalfpap

About BryanHalfpap

More Content

Cyber War

SciFi