Do you have a program or project that is evaluating “big data” approaches to the solution. You are not alone. With the explosion of data hitting almost every facet of IT, project leaders and program managers everywhere are downloading and evaluating the open source Apache Hadoop.

Since the word Hadoop is so frequently used to refer to the entire suite of open source tools around the core capability, one thing to keep in mind is that you will also need to download several other packages. You don’t really want to do that individually do you? If you download every package separately then you are responsible for making sure it is all integrated, and you will be responsible for making sure that changes to the underlying components are maintained and updated.

That means the best way to start with Hadoop is to download a distribution. Start with the always free to use CDH from Cloudera.  CDH stands for Cloudera Distribution including Apache Hadoop. CDH includes components that have been tested together and are known to work. So starting your activity with CDH is the smart way to go.

To download CDH see:  https://ccp.cloudera.com/display/SUPPORT/Downloads

You can also download the free version of a a configuration manager called the Service and Configuration Manager (SCM) Express Edition which will get you up and running fast.

More from the download page:

Find out why Cloudera’s Distribution Including Apache Hadoop (CDH) is downloaded more than all the other distributions combined.

Say you’re a Crip gangbanger and want to declare a “Crip Holiday” for you and your homies. You put it out on Twitter, bring a 9mm pistol along with you, and congregate at a local park at 7:30. You get there only to find out that the police are waiting for you. Sounds stupid? Well, it actually happened.

Quoth Mashable’s Radhika Marya:

The New York Daily News reports that police learned the alleged Crips members were using Twitter to call for a gathering at Amersfort Park in East Flatbush, Brooklyn Wednesday. Members of the NYPD’s Gang Unit teamed up with officers from the 63rd Precinct, the borough task force and the Emergency Service Unit, arriving at the park at about 7:30 p.m, police told the Daily News.

While commendable, the bust illustrates a problem with the emerging discipline of law enforcement social media exploitation–we can’t expect everyone to be that stupid. As law enforcement agencies continue to datamine Facebook, Twitter, and other social media to find evidence of crimes, track down fugitives, identify suspects, and gain intelligence on criminal groups, their targets will inevitably learn to cover their tracks. Gang members, violent “black bloc” activists, and other related groups are not supermen and fall prey to the very same privacy mistakes that law-abiding citizens regularly make. But any group hoping to be successful will place a premium on social media operational security (OPSEC). And many already have. For example,  many activist groups wary of police surveillance publish extensive guidelines on practicing OPSEC, which can be accessed for free on the Internet.

Police and federal law enforcement and security units seeking to leverage social media have become, in other words, victims of their own success. This isn’t to say that objects of surveillance will go completely analog. Many have grown so accustomed to the use of social media that they are loath to give it up–and simply will better camouflage their usage. But crime and riots originated thousands of years before the first tweet or Flickr photo, and criminals are perfectly capable of organizing themselves without social media. That’s why law enforcement social media exploitation will be one part of a larger data puzzle.

As Alex Olesker argues, the predominant intelligence challenge is micro-targeting with an increasingly overwhelming amount of data:

For example, at the end of 2010, General Cartwright noted that it took 19 analysts to process the information gathered by a Predator drone, and that, with new sensor technology being developed, “dense data sensors” meshing together video feeds to cover a city while simultaneously intercepting cell phone calls and e-mails, it would take 2,000 analysts to manually process the data gathered from a single drone.

On the law enforcement/homeland security level, police will be working with far smaller quantities of data than the military and the strategic intelligence community, but also far less resources. Even when criminals decide to hatch their plots the old-fashioned way, social media can still be used effectively in combination with other sources of information:

Surveillance footage from the 7-Eleven at 13001 Wisteria Drive shows two dozen suspects entering the convenience store en masse, grabbing snacks before leaving. Police posted the footage on their YouTube channel two days later, which has since tallied more than 140,000 views. …So far, police have identified 16 of the 25 suspects and plan to make arrests this week, Starks said. Most are county residents, according to police. The bulk of those positive ID’s came after investigators turned to the in-school police officer for Germantown and pored through Facebook profiles and high school yearbooks.

This sort of Social Network Analysis is a basic part of policing, and social media acts as another element in this time-honored process. However, Alex correctly points out that Hadoop can replicate and automate the more data-intensive aspects of this process (especially when it involves large and complex groups), as well as carry out textual analysis of large data repositories such as chat forums or email inboxes. Even without explicit instructions about capabilities and intentions, Hadoop can identify relationships and point police and federal personnel in the right direction.

As Katherine Maher pointed out on Twitter, no one gets credit when contingency plans work. And it is truly amazing how much government-citizen information collaboration has evolved–not to mention the growth of data journalism even in the most traditional news outlets. The average citizen had a wealth of accurate (and useful) hurricane information to choose from. There was also plenty of data to be used for those seeking to tinker, and very useful Twitter/Flickr/YouTube overlays on Google Earth interfaces. Anyone unlucky enough to turn on their television was assaulted by an overwhelming wave of hysteria and action shots of news anchors bravely intoning into the camera that it was windy and raining–an astonishing revelation, undoubtedly, to the millions told to evacuate.

Alex Howard of O’Reilly Radar has a very helpful list of Hurricane Irene tracking links from both government,media, and crowdsourced sources. Among the highlights:

• New York City government datamine with lots of Irene-related geographic data.
• Google Earth kmz files for New York City hurricane evacuation zones.
• Maryland state government Google Earth-based iReport for hurricane damage reporting.
• New York Times hurricane tracking map feeding off National Weather Service data.
• Hurricane Irene cleanup map using Ushahidi interface.
• Geographic information system (GIS) and tweet mashup Florida GATOR with lots of downloadable files.
• Google’s hurricane GIS/weather data aggregator.

My personal favorite was ESRI’s full screen mega-aggregator, built on ArcGIS API for Flex and running on an ArcGIS server. It combined data from the National Hurricane Center, Open Street Map, Telvent, and allowed you to look at layered Flickr pictures, Tweets, and YouTube videos on top of the usual amount of hurricane data. Incidentally, ESRI has entered into a strategic alliance with MetaCarta, which Alex Olesker wrote about in May after attending the Department of Defense Intelligence Information Systems (DoDIISS) conference. MetaCarta’s large database of locations, Olesker notes, is seven times the size of the National Geospatial-Intelligence Agency (NGIA)’s gazateer, making it a natural tool for geospatial analysis. It goes without saying that government Twitter accounts have also been a source of timely and accurate information. A wealth of government agencies, large and small, used Twitter and Facebook to spread the word before, during, and after the high point of the hurricane.

The Federal Emergency Management Agency (FEMA) also developed a host of mobile applications for hurricane preparedness and evacuation. Granted, the usual amount of Twitter rumor and innuendo prevailed, but the strong presence of “curated” information from the government, helped counteract information overload. I myself had to completely ignore my usual Twitter and Facebook timelines and focus exclusively on the applications listed above in order to avoid drowning in data. While government resources at times strained under the weight of unprecedented traffic, performance on the whole was very strong and reflected an internalization of what big data and social media pioneers have been urging for a long time.

With this post we provide some info on a capability we believe most of our readers will want to know more about, the Plug and Play Tech Center. For an overview see the video clip below:

The Plug and Play Tech Center is a Silicon Valley facility with a very apropos name. If you have a business idea or a hot technology and you want to accelerate it into the world, you want to go somewhere for support and when you do you want to plug in and play.  Several of us on the East Coast (including our Editor Bob Gourley) make periodic trips to the valley to stay up on what’s hot and the Plug and Play Tech Center is a great place to visit to scan the technology horizon. Their business model is to serve startups with all the needs for a growing firm, so a company can go from big ideas to big sales. The tech center offers everything from funding to office space to mentorship. They also have tremendous infrastructure, including, of course, Internet connections. They have a huge SAS 70 Type II certified data center on location, providing on-site data center access. They offer hiring help, executive mentoring, legal, accounting, offshoring and even PR help.

The Plug and Play Tech Center has great connections to the high tech financial community (their site indicates partnerships with 170 investors who participate in regular screening sessions with Plug and Play members). They also have their own internal funding arm that can help startups achieve their goals. Their hiring center can help startups staff and fill needed slots, while the legal, accounting, offshoring and PR firms all provide invaluable knowledge to help craft business strategy and gain competitive edge.

The Plug and Play Tech Center offers valuable services that accelerate a startup’s viability in the market, and in doing so they are providing great help to us the technology ecosystem and well as to the country’s economy. Visit their site to learn more. We plan on writing more about firms of interest that have either launched from their capability or are currently hosted there. If you have favorites you know of who have launched from the Plug and Play Tech Center, please let us know.

You can find them at:  http://plugandplaytechcenter.com

Photo credit: Kiyoshi Takahase Segundo

Do you have a need to index and search pictures and video? If you are working for any of the intelligence agencies, a police department, or any other security force that involves video and photo, you may. You may even have a need for this outside of the intelligence/security realm. How do you do it now? Does a human need to look at each video and picture and index what objects are visible in the video? That could take ages if so, and may be nearly impossible to get through the large amounts of data.

Right now there are several companies that can index and search video in different ways, but most do not offer a general Google-like search of video and pictures. Additionally, most of these products are not scalable for enterprise class purposes.  Some of the companies out there include Numenta, Pixsy, IDEE TinEye, and KEVA, which is only available to government currently.

California based piXlogic has brought an interesting product to the table that is changing the way enterprises search video and photos. piXserve is an enterprise class 64bit Java application for Windows. It indexes all objects in video and photos and attempts to classify the objects.  It does this by breaking a picture down into smaller pieces and creating a specific mathematical description of that image to leave metadata that can be searched.  piXserve extracts small parts of a picture or video (in any format) and creates a 3D object rotation in space, filling in the blanks about what the oject would look like from other angles.  piXserve works for many use cases, but pays specific attention to facial recognition and text recognition. It also works very well with geospatial imagery.

In one demo, piXserve took a picture of from a freeze frame of video. It extracted information from the face of a passerby and searched other videos in the database for the same person. It actually picked up many other videos of the same person, including videos looking at the same person from different perspectives. I could see this product having great application to law enforcement and intelligence agencies specifically, especially in cities with a large amount of cameras around the city. It could be highly beneficial for investigative and forensic purposes. What other areas do you see this product being useful?

To watch the Carahsoft webinar about this product, click here.

Image via Wikipedia

I carry my smartphone with me everywhere. It does everything from Facebook and email to keeping me in the loop with SMS, instant messaging, and real, live phone calls. There’s just one problem — what happens if my cell phone is stolen and I can’t remote wipe it? What happens if I’m stopped at the border of a foreign country and my mobile devices are searched, or what if I’m on a GSM network and am kept under surveillance that includes the decryption of my normally encrypted GSM phone calls and text messages?

You may think that these scenarios are some sort of Orwellian science fiction but they are all very possible and in some cases easy to perform. If you find yourself in a position of power or trust, you have a real need to encrypt your data from prying eyes and sticky fingers (especially with laptops and cell phones — prime targets for identity thieves and social engineers).

The following is a list of free or nearly-free utilities that you can use in your day-to-day life to secure everything from your mobile phone* to the data on your thumb drive without too much fuss.

*You may notice that blackberry is missing from this list — my research showed them to be incapable of making convenient user-friendly SMS encryption, and their connection is controlled and encrypted from the Blackberry Enterprise Server and the client device (usually BES is for business users).  I also wanted to focus on more consumer-oriented smartphones.  If you’re looking for a enterprise level view, check out this chart from Infoworld that contrasts control between the major smartphone OSes.

:::ANDROID:::

Encrypted Emails:

APG + K-9 Mail (free): APG acts as a keyring manager, generator, file encrypter/decrypter and email encrypter/decrypter all in one, and it interfaces with the K-9 mail program to allow you to easily use your PGP keys to send and receive emails. In my tests it was incredibly simple to set up and use keys through K-9 Mail and APG. You don’t need to use K-9 Mail with APG if you don’t want to, however. APG also allows you to select from many different algorithms and protects your keys from prying persons.

Encrypted Phone Services:

RedPhone (free):This free android program by whispersys.com will use VOIP technology to encrypt the phone calls that you make to other RedPhone users. It keeps the connection secure and your call log and contacts encrypted and private and integrates well with the android system, where you can select phone numbers to be called using the regular phone or RedPhone at your whim.

TextSecure: Another gem created by whisper systems this program encrypts the text messages using known secure algorithms (unlike many other applications in the android app store using weak encryption). Not only does this application encrypt the text messages, it can import and encrypt your entire SMS archive and password-lock it so that nobody else can access your text message history or contacts but you. I highly recommend this to anyone seeking a modicum of privacy in their communications.

LokPixPro (Paid): The only paid application on the list for Android, this application allows you to encrypt photos in your photo gallery, and will even allow you to encrypt pictures directly from the camera device itself before it’s stored on the phone according to their http://closecrowd.com/. The program even uses AES encryption algorithms (Though it doesn’t disclose what key strength is being used.)

Additional Security Features:

Lookout Mobile (free): I personally use Lookout Mobile on my Android device. It allows me to perform remote wipes, locate my phone, even make it yell so I can find it if I lose it. It also protects me from potentially malicious applications with a built-in antivirus and remote backups to the Lookout Mobile website automatically! This is a must-have for any android user. The only gripe with this program is the feature it has for enumerating all of the access features each app asks for. This feature is for paying members only (and is the only feature one gets for paying) but you can easily do this yourself by going through the android settings. Other than this, it’s one of the best and most useful applications in the android market.

:::iPhone:::

Disappointingly I could not find any (trustworthy) applications on the Apple app store that were able to securely encrypt phone calls (for free and not as a paid monthly service) or SMS messages, or even email. If you use an iPhone application that provides encryption capabilities, please contact me (@crypt0s on twitter or Bryan@crucialpointllc.com). The lack of secure device encryption on the iPhone is something of a well-known problem, illustrated here in this wired.com article. Also, unless the database of text messages and the phone call history can be encrypted, then the messages and your privacy are at risk. Even a search through several jailbreak software repositories didn’t turn up any promising applications.

Other Security Features:

The iPhone does support remote wipe and a few other interesting security features, but to get these you need to connect the phone to a Microsoft Exchange server and be or get an administrator-level user account. If you are connecting to your corporations network and planning on keeping secret data on the phone, be sure to understand your companies usage rules, some may stipulate that they can seize your phone for security investigations.

:::Windows Mobile/Phone 7:::

SMS:

WMKits (paid): WMKits has a very comprehensive SMS encryption application for Windows Mobile as well as an application to encrypt photos on Windows Mobile phones, however it’s not compatible with the new Windows Phone 7 operating system, which unfortunately does not have any encryption built into the phone at this time (including email). You can check out the WMKits software at their website, http://www.wmkits.com

Other Security Features:

Windows Mobile versions under version 7 have great interaction with Microsoft Exchange server and you can set all sorts of security policies for these phones from there. For Windows Phone 7, this is not currently the case, as Windows Phone 7 is not integrated with Microsoft Exchange Server. It’s the opportunity cost of owning a next-generation Windows Phone (for now).

:::Browsers:::

Firefox:

Firefox has a few extensions that will attempt to use HTTPS connections instead of HTTP connections:
Force TLS (for versions of firefox below 4)
STS-UI for Firefox 4.0 and above (Still in beta at the time of writing.)
HTTPS-Everywhere — HTTPS-Everywhere was created by the EFF to force websites to serve HTTPS websites.

Chrome:

Chrome currently has several extensions that force HTTPS connections, however they currently leak information (unless running in incognito mode) when first determining if a target website has HTTPS, which can put your personal information at risk if the website has stored cookies (like for auto-login or “remember me” features).

Two extensions for Chrome that provide the Force-TLS and HTTPS Everywhere functionality are Use HTTPS and KB SSL Enforcer

Opera

There is a HTTPS-redirect extension in Opera, however it does load the insecure page first, which can potentially leak sensitive information to everyone on your network (esp. wireless). If you have to use it, be aware of this.

Internet Explorer

As of the time of writing, there is no way to consistently load HTTPS pages in IE through redirects or otherwise.

:::Chat and Instant Messaging:::

Generally the best way to secure your instant messaging is to use Off The Record messaging plugins inside your chat client. Perhaps the easiest client to set up (and the one I reccomend to people) is a combination of Pidgin and Off The Record.

Pidgin can be downloaded from pidgin.im

OTR Plugins can be downloaded from cypherpunks.ca/otr/
Pidgin should support your instant messaging service of choice, be it AIM, MSN, or ICQ or more obscure services.

:::Operating Systems and Data Storage:::

Windows (all versions):

Truecrypt is perhaps the best single solution you can have on Windows because it allows for free system-level encryption on every version of Windows. You can also create encrypted volumes, use stenography, or encrypt entire drives like USB thumb drives. http://www.truecrypt.org/downloads

Mac:

Whole disk encryption isn’t available for Mac from TrueCrypt, but it is available from pgp.com. The rest of the Truecrypt features are available to Mac OSX users, including stenography.

Linux:

You can easily set up encryption at install time on Linux, selecting either whole disk encryption or just the home directory during the install process. To encrypt your main drive even after you have installed linux, look into using LUKS and dm-crypt.  That combination of software is currently the preferred encryption method of many Linux flavors, including Fedora and Ubuntu.  If you don’t want to go down that road for a removable or secondary drive, Truecrypt is available for Linux as well.

If Truecrypt doesn’t do what you want on Windows, Mac, or Linux, you can find a large list of disk encryption software and their features at this Wikipedia article.

Have something to say or add?  Contact me on twitter: @crypt0s and I’ll add it to this article!

What is Hadoop?

Hadoop is a collection of software originally spawned from the Apache Nutch project (Read a little more of its history HERE that is now its own project within the Apache Foundation. Its goal is to provide a highly redundant, self-repairing cloud of computers that can fail out and still be robust, fast, and efficient. To accomplish this goal, it leverages several pieces of software at its most basic installation (there are other distributions of Hadoop which can contain modified or extra software). The entire purpose of the software is to perform tasks that would normally crush computing clusters by breaking apart a question, distributing it, solving the, and then returning an answer — all while using throwaway hardware that is allowed to fail out, be replaced, or simply drop from the node network at any time. How does it do this? It uses Mapreduce, Zookeeper, and the Hadoop Distributed File System. Lets find out a little about how they work together to make some of the worlds best distributed computing software.

Mapreduce

Mapreduce is a software framework designed and patented by Google which allows applications written in many different languages to pass operations into the framework to be broken down into component operations, distributed across a cluster, and then reassemble and return an answer to the application.

“Mapping” is the act of de-constructing an operation into different parts for consumption by multiple computers in a distributed system.

“Reducing” is when the operation parts are received from the cluster systems and built into a single answer.

HDFS–Hadoop Distributed File System

The HDFS is a highly redundant filesystem that sits on top of the filesystem of the host computer. The HDFS has two different nodes in the filesystem — namenodes and datanodes. The namenode is a control node and the datanodes act as storage nodes on the filesystem. Inside the filesystem, the namenode has “rack-awareness” of the datanodes, meaning that it knows where each datanode is and attempts to use this information to reduce communication over the backbone. In fact, jobs will execute on the same rack first, or, failing that, on the same switch.

Nodes, Nodes, Nodes:

Namenodes

Namenodes are the master nodes that store the metadata of the filesystem and control the behaviour of the filesystem. This control is exerted over replication, deletion, and creation of files on the datanodes. This means that if the namenode detects that one file is being requested over another file, it will instruct the datanodes to replicate the more-needed file more often to increase availability. Namenodes (and mapreduce) also structure the filesystem (and jobs in the case of mapreduce) in an efficient, rack-aware manner to decrease the amount of traffic running on the backbone of the datacenter. Replication first goes within the same rack/network switch. Namenodes also perform access control for users and groups.

Secondary namenodes can also be deployed (usually on a separate computer) to take snapshots of the filesystem and the journal of changes to the filesystem. Like many database systems, hadoop uses a journaling filesystem, which means that changes are logged to a file before being applied to the file system to allow recovery in the event of failure. The secondary namenode decreases the amount of time it takes to recover because it keeps more up-to-date snapshots and a shorter change log to be applied on restart than the primary namenode. In short, its a form backup that makes the system faster in the event that it must be restarted.

Datanodes

Datanodes store the data and perform replication, deletion, and creation of files as instructed by the namenode. They make and read or write operations to existing files without explicit input from the namenode in order to carry out assigned tasks from mapreduce. Each datanode has a rack id in order to route traffic around the filesystem in an efficient manner.

ZooKeeper

Zookeeper is a distributed coordination service for distributed applications. It performs configuration management, provides a naming service, does leader election (selecting nodes to organize a task), Synchronization, and notifications. Basically this service keeps all of the nodes in line, working together, and disseminates this information using a distributed model.

Learn more at CTOlabs.com.

Doug Cutting is the creator of Lucene, Nutch and Hadoop. Doug and these projects are increasingly being mentioned in enterprise environments. So with this post I’ll provide a bit more of context on each.

Doug Cutting has earned a reputation as an innovative, community-supporting developer.  He is on the board of directors of the Apache Software Foundation and is now its Chairman. He is the Architect at Cloudera, the firm famous for delivering an end to end platform built on Hadoop.

Lucene has quite a bit of fame in enterprise circles. It is a free/open source search software library first written in Java (but now ported to many other languages). Although lacking in the discovery capabilities of more advance information retrieval tools, it is very reliable, easy to code to, and has a growing developer community. Lucine is frequently the basic search capability found in large government IT projects. It provides full text indexing and search.

Nutch is a web crawling and HTML parsing capability designed to be highly scalable and feature rich. It can be run on one system or scale to clusters of 100′s of systems. Nutch has a feature enabling “bias” so crawling and fetching of “important” pages can be done first.

Hadoop is a software framework for data-intensive solutions. Most enterprise technologists usually first think of the word “database” when they hear Hadoop.  But it is really more than a filesystem. It provides a new way of storing and interacting with stored data and a software framework designed to support data-intensive distributed applications.  Hadoop was inspired by Google’s MapReduce and Google File System papers.  It was originally written to support the Nutch project, but is now its own top level Apache Software Foundation capability.  Apache provides the kernel of capabilities for Hadoop.  Cloudera provides a Hadoop distribution that provides a full framework of capabilities.

Here is more context on these capabilities from Doug himself:

httpv://www.youtube.com/watch?v=qxC4urJOchs

Related articles

• 11 Apache Technologies for the Enterprise (itexpertvoice.com)
• Webcast: Better Search Results Faster with Apache Solr and LucidWorks Enterprise (lucidimagination.com)
• Sematext – Lucene, Solr, Nutch, Hadoop Experts (sematext.com)
• Pentaho brings BI, integration to Hadoop (newstatesman.com)
• Search Analytics: Hadoop World Presentation (sematext.com)