NSF's guidance on Cloud Computing will inform research dollars in the near future

The National Science Foundation released their report on cloud computing. It can be found here. The intent of this report is to provide information that guides funding programs. The NSF used NIST’s guidance on cloud computing to inform their research and decision making. This report will be instrumental in informing Federal decision makers and cloud investment. It is important to pay attention to such documents as they can help businesses align with Federal government priorities.

The NSF has identified the following areas as research areas worthy of financial support from the federal government:

1. Computer Systems
2. Computer Networks
3. Security and Privacy
4. Algorithms and data management
5. Applications and software engineering
6. Computer science education

The report concludes with the following:

“As recognized by both NIST and the America COMPETES Reauthorization Act, cloud computing is an area vital to the economic growth and competitiveness of the nation. The CISE community has responded vigorously to the challenge, as evidenced by the wide range of research efforts underway that are supported by numerous programs throughout the CISE Directorate.”

The NSF’s Computer Information Science and Engineering (CISE) directorate will continue its research into cloud computing (and hopefully will continue to advance cloud computing knowledge and research).

Related articles

• Federal Researchers Push Limits Of Cloud Computing (fedcyber.com)
• Cloud Security, Costs Concern Federal IT Pros (ctolabs.com)
• Cloud Computing: Why Should I Care? (blogs.cisco.com)

You may also like -

Lessons Learned from MagellanThe Cloud and CybersecurityCTOs: Provide your inputs on how the government will implement cloud computin ...Virtualization, Cloud Computing, Green IT conference 26-27 Oct, DC.Myths and realities of cloud securityCloud Security, Costs Concern Federal IT ProsCrucialPointLLCFederal Researchers Push Limits Of Cloud ComputingCrucialPointLLCFederal CIO says FedRAMP to be mandatoryCrucialPointLLCNASA Plans Cloud Marketplace For ScientistsCrucialPointLLCNIST identifies cloud computing standards gapsCrucialPointLLC

CumuLogic is ready to provide PaaS services for your enterprise

CumuLogic is a cloud computing company founded by Sun Microsystems’ alumni. The company has developed a Java Platform-as-a-Service (PaaS) software that makes it easier and faster to develop and deploy Java applications (apps) in the cloud. By automating the management of the runtime environments for developing and deploying Java apps in the cloud, including automated scaling, CumuLogic’s Java PaaS can reduce the current time these steps take from upwards of multiple weeks to under a day. Cloud computing is not just about Infrastructure-as-a-Service (IaaS). The cloud is about the applications, and getting them to your customers in a timely and effective way.

CumuLogic’s Java PaaS is now in beta with general availability coming soon. Their offering provides a no-lock in PaaS solution for Java apps that works on either public or private clouds. An enterprise can easily move apps between their own private cloud and a public cloud if needed. CumuLogic PaaS supports Amazon EC2, Eucalyptus, Citrix/Cloud.com CloudStack, OpenStack, and VMware private clouds. The architecture is flexible, cloud agnostic, hypervisor agnostic, offers mix-and-match infrastructure components, and has IT integration points. The beta product supports the following open source web servers, app servers, and databases: Ngnix, Apache, Tomcat, JBoss, Jetty, GlassFish, MongoDB and MySQL. The product also supports connections to an Oracle database. CumuLogic PaaS architecture is designed for high availability and failover. Fault tolerance is built in at every tier of the platform to meet the needs of mission critical applications.

Unlike today’s PaaS platforms like Google App Engine or application server-based approaches such as IBM WebSphere or Oracle/BEA WebLogic, Cumulogic recognizes that enterprise applications often exist in a tightly integrated ecosystem of highly varied and version dependent infrastructure components such as databases, web servers, libraries, and the like. Infrastructure component choices and their versions are often dependent on the goals of the executing application. The same is true for different application deployment scenarios where the choice of components is dictated by the specific service goals to be achieved.

CumuLogic is the first and only PaaS offering for application service development and deployment that supports full choice of an application’s infrastructure components regardless of whether they are open source, purchased from ISVs, or built in-house and proprietary. Existing PaaS platforms give limited choice of infrastructure components while application server vendors require significant component knowledge, integration, and testing to guarantee and maintain application functionality and integrity. CumuLogic PaaS provides freedom of choice for a wide selection of infrastructure components automating the otherwise costly integration of those components into a fully functioning application environment. CumuLogic PaaS also allows those applications to function on a CumuLogic platform(s) hosted on one or multiple internal sites and/or external service providers. Additionally, CumuLogic’s PaaS addresses enterprise security and compliance issues. Applications that require isolation can be managed separately to execute in isolation if required. Other applications without this requirement can share infrastructure components.

With the above features, the benefits of CumuLogic are dramatic:

• Enhanced developer productivity,
• A twenty to thirty percent cost savings in application lifecycle management, and
• Increased overall business agility.

Check out CumuLogic’s Java PaaS at www.cumulogic.com for your cloud computing application needs.

Useful Links

• Data sheet
• White paper
• Demos
• Beta registration

Related articles

• What You Need To Know About FedRAMP (ctovision.com)
• CumuLogic PaaS is Verified as Citrix Ready (prweb.com)
• Cloud Security, Costs Concern Federal IT Pros (ctolabs.com)

You may also like -

What's a "Cloud Operating System"?Dr Cloud's Flying Software CircusJava Gets CloudyDr Cloud's Flying Software Circus

Editor’s note: This is a guest post by Kyle Lagunas of Software Advice .

Recent activity – including the acquisitions of SuccessFactors by SAP and Rypple by Salesforce – is pumping new energy into web-based  HR software in the HCM market. Things are clearly heating up in this sector, and investors have taken note. In the last two weeks, VC funds have been flowing into the human resources software market, with large investments in SmartRecruiters and iCIMS. As such, 2012 shows much promise for the continued expansion of HR solutions into the Cloud, as well as further integration of social media functionality into HR applications.

The increasingly diverse needs of organizations big and small demand myriad options in human resources and talent management software solutions. As they continue to fight for the biggest piece of the pie, I believe more large software companies will make strategic acquisitions. I’ve taken an in-depth look at what SAP and Salesforce stand to gain from their HR acquisitions, how it will impact this dynamic software market, and where other contenders in this space might make moves of their own.

The Opportunity for SAP in HR: It’s All About the Data

Over the years, SuccessFactors added analytics and reporting capabilities to their product, in part through the acquisition of Inform Business Impact (previously InfoHRM) in early 2010. “The ability to create some telling analytics will set one vendor ahead, because that’s what it’s going to take to give HR a seat at the table,” says independent HR technology consultant Tiffani Murray. With SAP’s vast resources and experience delivering hard data in the ERP space, there’s a great opportunity here to further develop an HR analytics tool the market craves. If SAP is successful, we might expect to see more acquisitions at the intersection of HR and analytics in 2012.

Why HR Makes Sense for Salesforce’s Social Enterprise

Social tools have seen the most widespread adoption in these two markets, as CRM and HR have historically been the first business sectors to adopt new communications tools. With the evangelist support of Salesforce CEO Marc Benioff and John Wookey (previously of Oracle and SAP) heading their HCM unit, Salesforce is in a strong position to help HR departments extend social media functionality across their various applications. “This new blood drives innovation, which energizes the market,” says Murray. “It will drive the market in a new direction.”

Three HR Software Areas to Watch in 2012

As vendors great and small continue developing Cloud-based solutions, I believe there are three HR applications that offer the greatest opportunity for strategic growth through acquisition in 2012: talent management software, applicant tracking and recruiting software, and analytics and reporting applications. And in these areas, there are a few vendors who are ahead of the pack.

1. Because business leaders increasingly see the value in a system for developing their workforce, Talent Management Software continues to experience rapid growth in the Cloud. This sector has also seen some movement, as large vendors acquire best-of-breeds, particularly in learning management and performance management. Many analysts are have their eyes on Taleo – a leading talent management suite, and serious competitor in this space. I expect they’ll begin flexing some muscles in the marketplace, taking advantage of the opportunity to position themselves as SuccessFactors’s biggest competition.

2. Many of the core functions of Applicant Tracking and Recruiting Software thrive in the social media arena, and it makes sense that you’ll see more social capabilities evolving in this area first. Bullhorn – a leader in recruiting software – is also breaking into social with their new product, Bullhorn Reach. Currently in its infancy, I see a lot of potential in this product.

3. Your ability to identify and track trends in your retention rates, boosts in productivity, and benchmark performance will set your HR department apart from your competitors. And though HR has never had a problem with gathering data, HR software is recently making great strides in delivering technology that allows them to use that data. Analytics and Reporting Tools like eThority’s Workforce Analytics 5.0 (acquired by Equifax and integrated into their TALX business unit last October) will continue to drive innovation in HR software.

As legacy vendors continue their age-old ERP battle, and as innovative companies like Salesforce continue to dazzle, the moves made by these (rather different) business management providers have certainly set the stage for an exciting 2012. Customers and analysts alike would be wise to pay attention to continued movement in the market in the coming months.

For further reading, check out the full analysis and forecast on my HR blog at: http://blog.softwareadvice.com/articles/hr/the-hr-software-round-up-setting-the-stage-for-2012-1011912/. Have your own thoughts? Join the discussion, and leave a comment.

Related articles

• More HR Tech: SuccessFactors, Jobs2Web & SAP (zdnet.com)
• With Rypple Salesforce becomes a bigger social provider (gautamblogs.com)
• Salesforce makes an HR splash, buys Rypple (zdnet.com)
• Should SAP + SuccessFactors Worry Salesforce.com? (informationweek.com)
• Salesforce.com Acquires Rypple for Push Into Cloud HR Software (pcworld.com)
• Why Enterprise Software Will Never Be the Same (pcworld.com)
• The “Big Five” IT trends of the next half decade: Mobile, social, cloud, consumerization, and big data (bobgourley.com)

You may also like -

How Apple is Impacting Enterprise Software DevelopmentBob Gourley

We have previously provided a Quickstart guide to standing up Rackspace cloud servers (and have one for Amazon servers as well). These are very low cost ways of building reliable, production ready capabilities for enterprise use (commercial and government).  And Bryan Halfpap has provided a Quickstart guide which shows you how to build a Hadoop Cluster (leveraging Cloudera’s CDH3).  Using Bryan’s guide you can have a Hadoop Cluster up and running in under 20 minutes.

With this post we would like to provide you with some additional tips that flow from these other posts. We will show you how to build clusters even faster using another common tool in community use, Whirr.

What is Whirr? Apache Whirr is a set of libraries for running cloud services. Here is more from http://whirr.apache.org/ 

Whirr provides:

• A cloud-neutral way to run services. You don’t have to worry about the idiosyncrasies of each provider.
• A common service API. The details of provisioning are particular to the service.
• Smart defaults for services. You can get a properly configured system running quickly, while still being able to override settings as needed.

And the great news is you can use Whirr as a command line tool for deploying clusters.

If you follow the tips below you can use Whirr to quickly standup distributed clusters. Our assumptions in this guide are that you have stood up RedHat severs using our Rackspace tutorial. But if this is not the case you should be able to easily modify the tips below to suit your situation.

SSH into your Rackspace account by terminal window:

sudo ssh root@50.56.237.236

After logging in, it is always a good idea to make sure you have the latest packages. In Red Hat, type:

sudo yum upgrade

Now it is time to install Whirr.  This is easy since you are running RedHat. RedHat uses YUM, a package management application that makes software installation easy. Type:

yum install whirr

Your installation will be complete in under a minute.

You will now need to generate a keypair for use with Whirr. This will let you enable secure communications with the Whirr cluster without needing passwords. To do that, enter the following command:

ssh-keygen -t rsa -P ”

You will see:

Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa):

Just hit “enter”.

You will see something like:

Created directory ‘/root/.ssh’.
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
c6:31:f7:f5:97:e4:8c:b3:2a:f4:0d:a0:93:e4:c1:06 root@RedHat6-1-Hadoop-Test
The key’s randomart image is:
+–[ RSA 2048]—-+
| |
| E |
| o o . .. |
| * = . .=..|
| + S . .o +o|
| * . . o .|
| o . o. |
| . … |
| .. |
+—————–+

Now you must define your Whirr cluster. you do that by creating a properties file. For simplicity, you will name it hadoop.properties. You will need your rackspace username and API to fill out the whirr properties file.  Your API is found in my account page under “API Access”

You can create the properties file many ways. Here is how to do it in nano:

nano hadoop.properties

Now  enter the following info in that file, subsituting your login and API info for what you see below:

whirr.cluster-name=myhadoopcluster
whirr.instance-templates=1 hadoop-jobtracker+hadoop-namenode,1 hadoop-datanode+$
whirr.provider=cloudservers-us
whirr.identity=enteryourloginidfromrackspace
whirr.credential=[youuseyourownapi]
whirr.private-key-file=/root/.ssh/id_rsa
whirr.public-key-file=/root/.ssh/id_rsa.pub
whirr.cluster-user=newusers
whirr.hadoop-install-function=install_cdh_hadoop
whirr.hadoop-configure-function=configure_cdh_hadoop

Now to launch a cluster, type:

$ whirr launch-cluster –config hadoop.properties

This will take a few moments to run. As it runs you should see messages like:

Bootstrapping cluster
Configuring template
Starting 1 node(s) with roles [hadoop-datanode]
Configuring template
Starting 1 node(s) with roles [hadoop-jobtracker, hadoop-namenode]

As things are started up, servers are being automatically built. Keep watching your e-mail, you will be getting notices of server standup. Remember, this is costing you money. When you finish using your clusters you will want to terminate them. You can do that through Whirr or by just nuking the servers using your Rackspace account and control panel.

Note the info being provided in the terminal window. Information is being provided on the instances being stood up.  As you skim this info you will notice a couple URL’s are provided that give you a web UI into the namenode and job tracker. For example, mine are:

Namenode web UI available at http://50.56.211.206:50070

Jobtracker web UI available at http://50.56.211.206:50030

You will also see that a site file was created for you at:

/root/.whirr/myhadoopcluster/hadoop-site.xml

You need to update your your local Hadoop configuration to use this file.  Type the following commands:

cp -r /etc/hadoop-0.20/conf.empty /etc/hadoop-0.20/conf.whirr
rm -f /etc/hadoop-0.20/conf.whirr/*-site.xml
cp ~/.whirr/myhadoopcluster/hadoop-site.xml /etc/hadoop-0.20/conf.whirr
alternatives –install /etc/hadoop-0.20/conf hadoop-0.20-conf /etc/hadoop-0.20/conf.whirr 50
alternatives –display hadoop-0.20-conf

A proxy script was created for you at:

/root/.whirr/myhadoopcluster/hadoop-proxy.sh

You should now start that proxy.  It is there for security reasons.  All traffic from the network where your client is running is proxied through the master node of the cluster using an SSH tunnel.  This script launches the proxy.  Run the following command to launch the script:

~/.whirr/myhadoopcluster/hadoop-proxy.sh

If that doesn’t run make sure you have the right permissions on the file by

chmod +rwx hadoop-proxy.sh

Then try again.

With the above you are now able to use your Hadoop Cluster.

Prove that by browsing HDFS:

hadoop fs -ls /

Now it is time to run a MapReduce job!  We are going to use one of the example programs provided in the Hadoop installation. The program is in the file Hadoop-*examples*.jar  First, a lets review list of options available form the program. See these by entering:

hadoop jar $HADOOP_HOME/hadoop-examples-*.jar

You will see:

An example program must be given as the first argument.
Valid program names are:
aggregatewordcount: An Aggregate based map/reduce program that counts the words in the input files.
aggregatewordhist: An Aggregate based map/reduce program that computes the histogram of the words in the input files.
dbcount: An example job that count the pageview counts from a database.
grep: A map/reduce program that counts the matches of a regex in the input.
join: A job that effects a join over sorted, equally partitioned datasets
multifilewc: A job that counts words from several files.
pentomino: A map/reduce tile laying program to find solutions to pentomino problems.
pi: A map/reduce program that estimates Pi using monte-carlo method.
randomtextwriter: A map/reduce program that writes 10GB of random textual data per node.
randomwriter: A map/reduce program that writes 10GB of random data per node.
secondarysort: An example defining a secondary sort to the reduce.
sleep: A job that sleeps at each map and reduce task.
sort: A map/reduce program that sorts the data written by the random writer.
sudoku: A sudoku solver.
teragen: Generate data for the terasort
terasort: Run the terasort
teravalidate: Checking results of terasort
wordcount: A map/reduce program that counts the words in the input files.

So lets put this info to use. We will make a directory put some info in there, and run the wordcount program:

$ export HADOOP_HOME=/usr/lib/hadoop
$ hadoop fs -mkdir input
$ hadoop fs -put $HADOOP_HOME/CHANGES.txt input
$ hadoop jar $HADOOP_HOME/hadoop-examples-*.jar wordcount input output
$ hadoop fs -cat output/part-* | head

Now you are off and running.

You now have a platform capable of scaling to very large jobs. And it runs CDH3, the most reliable, capable distribution of Hadoop and related technologies. Let the fun begin!

But one final note. Think about the lifecycle of your system. At some time you will need to spin it down and turn it off.  To destroy the cluster gracefully using Whirr, enter this command:

whirr destroy-cluster –config hadoop.properties

Using the information above you can create and better manage Hadoop Clusters on Rackspace very easily. This is how we create our CDH Clusters.  In future posts we will show you how to get data prepared for analysis and how to run some queries.  We will also provide tips on how to use Cloudera’s free management tools and how to upgrade to Cloudera Enterprise when you are ready.

You may also like -

Hadoop Quickstart: Build a Cluster In The Cloud In 20 Minutes Or LessQuickstart Guide: Stand up your cloud-based servers with RackspaceAdmin Note: Another upgrade to the blogEnterprise CTOs: Learn Hadoop and Cloudera's CDH3 on 21 AprilFrom the Apache Software Foundation Blog: The Apache Software Foundation Anno ...Hadoop Quickstart: Use Whirr to automate standup of your distributed cluster ...CrucialPointLLCQuickstart Guide: Stand up your cloud-based servers with Amazon Web Services EC2Bob GourleyCloudera Day in DCCTOvision.comImproving Hadoop Performance with Optimization, CDH3 Update 3, and CDH4CTOvision.comData Wizards Know Hadoop is Powerful: But they want more automationCTOvision.com

While virtualization offers many benefits to enterprise such as lower costs and greater flexibility, it also creates new challenges. One of the greatest concerns with switching over to virtualized infrastructure, espeically in government, is security and compliance in a complex and dynamic environment which legacy software can no longer handle. Catbird offers automated security solutions tailored to virtualized data centers and has recently unveilled the next generation of its vSecurity software, vSecurity 5.0.

By integrating with the hypervisor, which manages all of the virtual machines, vSecurity 5.0 delivers access control, intrusion detection, secure auditing, automated protection, visibility, and efficiency. It can enforce FISMA, NIST, HIPAA, and other standards so that  users can virtualize more assets faster and with better return on investment. As a result, vSecurity provides a level of securityand compliance that exceeds traditional physical security devices.

In vSecurity 5.0, Catbird will incorporate a variety of security features. The Event Viewer displays security data from various controls such as the firewall and intrusion detection system and shows in real time what each is detecting as well as how they interact. The Compliance Radar Graph visually shows data center compliance to a given standard in real time to automate validation. Automated Asset Inventory accurately catalogs for for security monitoring and enforcement, which is integrated with Network Flow Mapping to effectively regulate access controls and mitigate risks from suspicious communication. It also allows for Policy Automation via vSecurity TrustZones, logical groupings of assets with common a pre-defined policy and compliance rules. Version 5.0 also allows data to be analyzed by leading enterprise management systems such as ArcSight and Splunk.

Catbird’s vSecurity has won many awards and garnered recognition for its revolutionary solutions, including receiving four consecutive VMworld Best-of-Show Finalist awards and Gartner’s Cool Vendor 2011 designation. If you’re running a virtualized data center, give Catbird a try for yourself by requesting a free demo here.

Related articles

• What You Need To Know About FedRAMP (ctovision.com)
• Invincea Continued to Gain Momentum (ctovision.com)
• Splunk: Bringing Big Data Analysis to the Rest of Us (ctovision.com)

You may also like -

Splunk: Bringing Big Data Analysis to the Rest of UsIs the Instant-On Enterprise Right for You?Catbird’s vSecurity 5.0CrucialPointLLCWMware: The Leader in Virtulization SolutionsCrucialPointLLCSplunk to Provide Integration with Apache HadoopCrucialPointLLCNicira Launches: Will change the world of networking giants and enterprise ITCTOvision.comSplunk: Make Your Machine Data Accessible, Useful, and ValuableCrucialPointLLC

Protecting and connecting

Here are a few recent links of note on cybersecurity and disruptive technology:

• The Cyber Power Index by Booze Allen Hamilton. The G20 are ranked by their potency in cyber, determined by their legal and regulatory framework, economic and social context, technology infrastructure, and industry application. The United States is ranked second with the United Kingdom surprisingly first and China in 13th place. This tool is interactive and, if you disagree with Booze Allen Hamilton’s assessment, you can change the weighting of the determining factors.

• Federal Researchers Push the Limits of Cloud Computing. The results of the 2 year Magellan project, where the DOE”s national labs compared their traditional High-Powered Computing to cloud models, were released last month. Can the national labs use cloud computing to measure the expansion of the universe? Maybe, but not quite yet. We also did some further analysis on Lessons Learned from Magellan.

• Nearly a Third of Americans Now Own an E-Reader or a Tablet. Almost 20% own each. Increasingly mobile is becoming the norm, which means that mobile applications, services, and security need to keep pace and are some of the most pressing topics in information technology.

• Military’s New Plan to Weed Out Counterfeits: Plant DNA. Counterfeit goods are a tremendous concern in military logistics, but serial numbers can be rubbed out and even holograms can be mimicked. Applied DNA Sciences Inc. has just won a contract with the Defense Logistics Agency to try a novel solution –  coating electronics with plant DNA markers. Matching the plant DNA would be almost impossible for counterfeiters.

Related articles

• Alex’s 2012 Tech Predictions (ctovision.com)
• Adam’s 2012 Tech Predictions (ctovision.com)
• Ryan’s 2012 Tech Prediction (ctovision.com)

You may also like -

CTO Perspectives on Cyber Security BillInterested in Cyber Security? Read (and support) the new Cybersecurity Legisl ...Deputy Secretary of Defense Lynn: Cyber Strategy’s Thrust is DefensiveFedCyber.com Cybersecurity Summit on Wednesday, September 28Announcing FedCyber.com: A resource for firms with cybersecurity solutions fo ...The Cyber Power IndexCTOvision.comGuardians of the Grid: Agencies Unite to Bulk Up Utility CybersecurityCrucialPointLLCDHS’s National Cyber Security Division and Idaho National Laboratory win cybe ...CrucialPointLLCNIST, DHS trying to tame cyber workforce definitionsBob GourleyDHS cyber security operations see leadership changesCrucialPointLLC

Two years ago, the Department of Energy‘s Office of Advanced Scientific Computing Research launched the Magellan project, a research and development effort aimed at harnessing cloud computing for the most demanding information processing of the national labs. A distributed testbed infrastructure was deployed at the Argonne Leadership Computing Facility and the National Energy Research ScienticComputingCenter, then benchmarked for some of the most challenging applications such as a particle physics experiment at the Large Hadron Collider or measuring the expansion of the universe. They also tested Hadoop, MapReduce, and the Hadoop ecosystem on massive scientific problems in the cloud. Their final results, published in December, show both the potential and current limitations of cloud computing for cutting-edge science. The primary appeal of cloud computing for the national labs was flexibility and agility. Through virtualization, researchers could create whatever custom computing environment they need, bring their own software stack, and try out new environments. Resources are also more flexible in the cloud, and researchers enjoyed being able to rapidly scale to a problem and tap into economies of scale for massive data sets and workflows. Another benefit of the cloud for science was that it simplified collaboration, allowing researchers to share software and experiments with their peers. Hadoop and MapReduce showed promise for high-throughput data and very large workloads. Often, the High-Performance Computingsolutions in place at the national labs have scheduling policies that aren’t compatible with this type of analysis. Problems with applying deep science to the cloud, however, currently outweigh the benefits for most applications, so the national labs will not be switching over from HPC just yet. Adapting to the cloud, porting applications, and building up infrastructure took considerable time and skill, raising costs. For most applications, which deal with truly massive workloads, have idiosyncratic needs, and are input/output intensive, traditional HPC currently performs better. Cloud worked best for applications that required minimal communication. The research team also had concerns about meeting the specific security and monitoring requirements of the national labs. Price was perhaps the biggest obstacle to implementing a cloud model, as using a commercial cloud would cost between 3 and 7 times as much as the current computing centers, which already pool resources to cut costs. Even switching over to private clouds would exceed a lab’s budget. Cloud computing for deep research isn’t doomed, however, as almost 40% of scientists would still want a cloud model even if performance suffered. There is also a lot of room for growth in this area, and even during the 2 years of the study, researchers marked dramatic improvements to the open source software powering the cloud such as Hadoop. To move forward, researchers sought improvements to MapReduce to better fit scientific data and workflow as well as ways to bring some of the benefits of the cloud to the traditional HPC platforms the national labs have spent decades perfecting.

Related articles

• Federal Researchers Push Limits Of Cloud Computing (fedcyber.com)
• Big Data Success in Government (ctovision.com)
• Congress Funds Exascale Computing (fedcyber.com)

You may also like -

Congress Funds Exascale ComputingBob GourleyFederal Researchers Push Limits Of Cloud ComputingCrucialPointLLC

The Federal Risk and Authorization Management Program (FedRAMP) is a government-wide program established in December 2011 to speed the adoption of cloud computing. FedRAMP includes a set of requirements for federal cloud computing and universal procedures for approving services and providers to work with the government. When contractors feel that they have met FedRAMP requirements, they must have their security control implementations independently verified and validated by a FedRAMP accredited Third Party Assessment Organization for compliance which then submits  a security assessment package for review by the cross-agency Joint Authorization Board (JAB) . FedRAMP is expected to be operational by June and will be mandatory for all government cloud deployments of low to moderate risk levels except for single agency private clouds. Agencies can also add additional requirements on top of the FedRAMP controls. The goal is to establish standards to ease fears about cloud security while saving time and labor through one federal standard rather than redundant agency standards, allowing organizations to leverage past approvals elsewhere.

After looking at over 1000 comments from government and industry, FedRAMP released its list of security controls earlier this month. The controls are based on the National Institute of Standards and Technology special publication 800-53, Revision 3, which are already in place for each federal agency through the Federal Information Management Security Act (FISMA), with additions relating specifically to security in the cloud. The 800-53 standards are characterized by measures to ensure the consistent application of security practices and continuous monitoring of near-real time data.

Additions include controls to deal with trust on shared resources and to dictate secure practices for Platform-as-a-Service, Software-as-a-Service, and Infrastructure-as-a-Service. PaaS and IaaS are to have session locks and SaaS needs to have cryptography up to federally mandated standards. Service providers must support the capability to produce, control, and distribute asymmetric cryptographic keys. Identity and privilege are to be tightly managed, with means to identify foreign nationals and contractors on government networks and enforce role-based access controls at the file, table, row, column, or even cell level if necessary. There are extensive documentation requirements. Service providers must maintain a list of software programs authorized to execute on the information system and submit it to the JAB for approval, and must also document all outsourced security services as well as conduct a risk assessment of future outsourced security services to be approved by the JAB. To gain authorization, service providers must also submit updated code analysis reports and, in the Continuous Monitoring Plan, how new code will be reviewed. The JAB must also approve a list of security functions that must be routed for DHS monitoring such as authentication and resource provisioning and what internal communications traffic will be routed through authenticated proxy server to which external networks. Service providers are to logically or physically separate administrator information security tools, mechanisms, and support components and set resource allocation priorities for the moderate impact systems. The full list of controls contains more additions and specifics.

FedRAMP has already gotten a mixed response. Government executives say that the program will speed up the adoption of cloud computing by simplifying the authorization process for cloud services. If a Third Party Assessment Organization and the Joint Authorization Board find a service to be compliant, any and all government agencies can adopt it. That is, however, only if individual agencies don’t add too many additional conditions to the controls, which some researchers fear will happen. Also, as noted above, the authorisation process is very documentation intensive with many steps that may create a bureaucratic nightmare as cloud services rush to get authorized. Rather than speed up the adoption of cloud services, FedRAMP could create a bottleneck. To combat this, the JAB intends to view authorization packages in order of priority and grant provisional authorization if necessary.

Related articles

• GSA to hold industry workshop on federal cloud security controls (fedcyber.com)
• FedRAMP includes 168 security controls (fedcyber.com)
• Federal officials launch FedRamp (ctolabs.com)

You may also like -

Live from the Gov 2.0 Expo - Security in the CloudThe Cloud and CybersecurityA few minutes with Justice's CIO, Vance HitchDHS CIO discusses 12 Cloud ServicesA look at VMware's vFabric Cloud Application PlatformGSA to hold industry workshop on federal cloud security controlsCrucialPointLLCFederal officials launch FedRampCrucialPointLLCFederal CIO says FedRAMP to be mandatoryCrucialPointLLCFedRAMP includes 168 security controlsCrucialPointLLCNov 4 NIST Cloud Computing Forum, Day 1: "Cloud Computing is not a fad"Dr Cloud's Flying Software Circus

Big Data solutions company Splunk has just filed to go public and is looking to raise $125 million in IPO. Splunk, which will list under the symbol SPLK, collects, indexes and harnesses all the fast moving machine data generated by enterprise applications and devices.  It makes this data, which can be hundreds of terabytes a day for large enterprises, easy to break down and query so that even non-technical users can solve Big Data problems, reducing the IT bottleneck.  It can also be used to manage and monitor distributed computing, Platform-as-a-Service infrastructure, and Software-as-a-Service. Splunk Enterprise for Hadoop will soon be available, allowing the software to integrate with with Apache Hadoop, the leading open-source Big Data platform.

Splunk has over 3,300 customers, including most of the Fortune 100.  Credit Suisse, Bank Of America, Comcast, Salesforce, Zynga, LinkedIn, T-Mobile, Swisscom, Shutterfly, Heroku and the US Departments of Labor and Energy are all clients. Splunk generates revenue based off the amount of data its clients want to index, and in the 2009, 2010, and 2011 fiscal years generated $18.2 million, $35 million and $66.2 million in revenue. On top of this rapid growth, Splunk has also raised $40 million in venture capital.

Related articles

• Splunk to Provide Integration with Apache Hadoop (ctolabs.com)
• Enterprise Data Software Company Splunk Files for $125 Million IPO (techcrunch.com)
• Splunk: Bringing Big Data Analysis to the Rest of Us (ctovision.com)

You may also like -

Splunk: Bringing Big Data Analysis to the Rest of UsDigital Reasoning's SynthesysCatbird's vSecurity 5.0Technology Firms at the DoDIIS WorldwideThe Insider Threat Worst-Case ScenarioSplunk Is Going PublicCrucialPointLLCSplunk to Provide Integration with Apache HadoopCrucialPointLLCCatbird’s vSecurity 5.0CrucialPointLLCSplunk: Make Your Machine Data Accessible, Useful, and ValuableCrucialPointLLCDigital Reasoning’s SynthesysCrucialPointLLC

I’ve been working with Apache Hadoop in my lab, spending much of that with CDH3 (the Cloudera Distribution including Apache Hadoop). As part of my examination of the best way to move from test/evaluation/prototyping to production systems I’ve been examining cloud-based cluster capabilities.  Crucial Point CTO Bob Gourley and I recently set up clusters in Amazon EC2 and Rackspace and were both impressed with the ability to get powerful systems up and running fast for very low cost.

With this post I’ll share some of the tips and techniques used to get these clusters up and running. These tips flow from the guide provided by Cloudera (see link at end of this post) but include a few other important tips that will help get you operational fast. I also provide more explanatory context to help you understand more what is going on as you execute these instructions. My only assumption here is that you have an understanding of the basic Linux command-line environment and have access to systems (perhaps in your environment or, like we did, at Amazon or Rackspace). Please note that this particular guide is written for 64-bit Redhat-flavored systems (CentOS/Fedora/ect…), and will need to be tailored for other OSs. Also note that we did things this way because we want to learn and to teach and thought many of you would like to see what is going on here as well. But there are actually better/faster ways than this. You can download configuration management systems that let you plan, start, manage and operate clusters very smartly using great GUIs and then manage them over the lifecycle of a production environment (see for example, the free Cloudera Manager). We will provide more on tools like that in the future. But now we want to show you how to stand up your environment using CDH3 by use of the command line. This is the fun way. We will walk you through the enterprise way in the near term.

Our first step flows from an understanding that Hadoop utilizes a lot of Java, and for that, you want to be sure that you are using the Sun/Oracle version.  This ensures that any problems you come across aren’t in the Java implementation itself.

Bring up the Linux system you wish to install CDH/Hadoop on and run the commands below inside of the command prompt.  This will return a list of installed packages called “OpenJDK“.  OpenJDK is an open-source implementation of Java that we will need to remove.

rpm -qa | grep openjdk

If that returns a list of openjdk packages, you need to remove them before heading further.  This command below will remove anything in yum which starts with “java”, and will remove openJDK from the system.  We want to do this because the Sun/Oracle Java is much more stable and will ensure that all required features are present in Java.

yum remove java*

Once the packages have been removed, run the following commands in order to download and install the Sun/Oracle Java environs.

For 64-bit versions:

wget http://download.oracle.com/otn-pub/java/jdk/7u2-b13/jdk-7u2-linux-x64.rpm

Now we download and install the Java Runtime Environment.

For 64-bit versions:

wget http://download.oracle.com/otn-pub/java/jdk/7u2-b13/jre-7u2-linux-x64.rpm

Now that we have the packages downloaded, it is time to install them.  Use the Redhat package manager (rpm) to install these files with the commands below.

rpm -i jre-7u2-linux-x64.rpm
rpm -i jdk-7u2-linux-x64.rpm

At this point we are ready to start downloading the Cloudera Distribution including Apache Hadoop (CDH).  Redhat-flavor linux users of later distributions (Fedora 12+, latest versions of CentOS) can safely use the Redhat/CentOS 6 repositories from cloudera to install hadoop. If you are running later versions of those flavors, use the CentOS 5 repo.

For Latest Versions:  http://archive.cloudera.com/redhat/6/x86_64/cdh/cloudera-cdh3.repo

For Legacy Versions: http://archive.cloudera.com/redhat/cdh/cloudera-cdh3.repo

You may notice that this is a repo file — it’ll add a repository to your package manager and help you keep your CDH software up to date. Simply add it to the yum repository like this (as root):

cd /etc/yum.repos.d/
wget http://archive.cloudera.com/redhat/6/x86_64/cdh/cloudera-cdh3.repo

Then update your package information from the repo with yum.

yum update

We use the repositories because it’s easier to keep CDH up-to-date with the package manager rather than building from source every time there is an update.  Maintenance and setup are easier, and it’s easier to reinstall should we need to do that.  Adding the repositories also allows you to install additional packages with yum with a few simple commands, rather than having to search for, then download and install any additional software add-ons for your Hadoop ecosystem.

Now you should be ready for the next step — downloading and installing Cloudera’s Hadoop. Do this with yum by using the following commands:

yum install hadoop-0.20

This will install the base of Hadoop and allow you to install the other packages that Hadoop requires to operate — namely the jobtracker, tasktracker, namenode, and datanode. All of those trackers and nodes are required in order to operate a Hadoop cluster. Choose which ones this machine will run, or install all of them at once if you are running in “pseudo-distributed mode”. Pseudo-distributed mode treats one computer as if it is an entire cluster, and is great for testing out prototype jobs or performing development on.

to install the packages, simply follow this syntax:

yum install hadoop-0.20-[NAME OF THE PACKAGE]

If you want to run hadoop just on one node, install all of the packages, then use the following configuration for Hadoop in the /etc/hadoop-0.20/conf folder: http://hadoop.apache.org/common/docs/r0.20.2/quickstart.html#PseudoDistributed

Below is a simple bash script for pseudo-distributed installs of CDH.  This script will start the Hadoop services for you:

#!/bin/bash
service hadoop-0.20-datanode start
service hadoop-0.20-jobtracker start
service hadoop-0.20-namenode start
service hadoop-0.20-tasktracker start

Copy and paste that code into a file named “start-hadoop”, save it, and mark the file executable with the chmod command:

chmod +x start-hadoop

Then run it to see if everything starts up fine with:

./start-hadoop

If you did everything well, then you should be able to perform the following actions:

hadoop fs -copyFromLocal [filename] [filename in hdfs]

hadoop fs -cat [filename in hdfs]

Where the names in brackets are the names of a file you wish to import and then display in HDFS (Hadoop distributed file system).  If you weren’t able to get that started fine, then you may need to repeat one of the steps or make sure you installed all the components you needed.

If you have any questions or if you’ve found an error in this Tutorial, please let me know on Twitter (@Crypt0s).

A full documentation including steps to install on Debian-based systems can be found at Cloudera’s website.

Note: Keep checking CTOvision.com for more Hadoop updates on software for Hadoop like Whirr. Whirr is another great capability, enabling the fast standup of cloud clusters by use of a simple properties file you update with your cloud provider info. We also have posts coming on the use of Cloudera’s Manager. Also, as previously mentioned, if you would like fast tips to standing up cloud-based servers to start these Hadoop Clusters on, see our Quickstart guides to Amazon and Rackspace servers.

Related articles

• If You Use Hadoop You Have Been Waiting For This: Cloudera Enterprise 3.7 (ctovision.com)
• Quickstart Guide: Stand up your cloud-based servers with Rackspace (ctolabs.com)
• Join Cloudera and Carahsoft for Big Data Success in Government (ctolabs.com)

You may also like -

Cloudera Day in DCHadoop Quickstart: Use Whirr to automate standup of your distributed cluster ...Data Wizards Know Hadoop is Powerful: But they want more automationImproving Hadoop Performance with Optimization, CDH3 Update 3, and CDH4If You Use Hadoop You Have Been Waiting For This: Cloudera Enterprise 3.7Cloudera and SGI Partner: With new benchmarks and better mission support the ...CTOvision.comEnterprise CTOs: Learn Hadoop and Cloudera's CDH3 on 21 AprilCTOvision.comThe Quickest Way To Deploy A Well Engineered Apache Hadoop Solution To A Prod ...CTOvision.comBig Data Tip For The New Project Manager: Starting With Apache HadoopCrucialPointLLCSGI ESTABLISHES NEW WORLD RECORD APACHE HADOOP BENCHMARKCrucialPointLLC