CTOvision.com

Context for the CTO, CIO, CISO and Data Scientist

You are here: Home / Archives for Artificial Intelligence / Big Data and Analytics

Big Data Is About New Designs and New Ways Of Doing Things

Big Data Can Be A Useful Term If It Motivates Continued Change

big-data In today's world where data is everywhere the term "Big Data" may be losing some of its usefulness. The term came into widespread use in 2007 (see a short history of Big Data here), and was very important in helping people to realize that new approaches were needed to data challenges. Now that large data systems are common and AI/ML is being used to leverage these big data sets, the use of the term is dropping off. We still use it, since when used correctly it can help motivate change. In most every case where Big Data is being usefully applied today, it is associated with analytics. This is frequently the "so what" of Big Data. It is also closely related to the construct of Artificial Intelligence, since for AI concepts like Machine Learning and Deep Learning to work, large quantities of data and well thought out data architectures are required.
And clearly continued change is still needed. There are no simple answers to overwhelming data challenges, but one thing is clear. The answer to our challenges is not “less data.” The reasons we have so much data is the continued drive to improve business outcomes in industry, the need to pursue scientific discovery in academia, and the need to improve government support to citizens and the mission of government. We need new concepts and technologies to deal with these massive amounts of new data. We need new ways of doing more analysis and gaining more understanding, and of course we need smart data solutions.

Stay In The Loop: Sign Up To Receive the CTOvision Daily

When it it time to research the great providers of Big Data Analytics Solutions, kick off  your search in the CTOvision Disruptive Technology Finder. Pay particular attention to our Big Data Companies section. For help putting all this information into context for your business, consider tapping into Crucial Point's CTO-as-a-Service or Technology Due Diligence offerings, both are ways to quickly leverage the technical talents of experienced enterprise technology professionals.

The White House Office of Science and Technology Policy and the National Science Foundation Present: Challenges and Opportunities in Big Data

On Thursday 29 March 2012 from 2:00pm to 3:30pm EDT the White House Office of Science and Technology Policy (OSTP) and the National Science Foundation (NSF) will hold an event focused on “Big Data” Research and Development.  Federal government science heads from OSTP, NSF, NIH, DoE, DoD, DARPA and USGS will outline how their agencies are engaged in Big Data research.  The event will also include a discussion by a panel of thought leaders from academia and industry, moderated by Steve Lohr of the New York Times.

Our view: This event will be important for many reasons. Here is what to expect:

  • Although scientists and technical leaders are the primary participants, all who are participating are also masters at explaining things in plain english. So this will be a GREAT event for those who would like to learn how to explain Big Data concepts.
  • This type of evolution can be very helpful to enhancing cross-government sharing of lessons learned and concepts. I believe this will be very positive.
  • This initiative will discuss R&D activities. Although at a high level, federal R&D focus is usually expressed in terms of frameworks and broad goals, these goals normally translate into budgets so the research community outside of the government should really pay attention.
  • You will be proud of the people speaking at this event. These government servants are smart people doing good things.

Would you like to watch this event?  Thanks to the miracles of the modern science you can watch this event right from your web browser.  It will be webcast live.

We will be watching here and will capture salient points for summarizing here and in the Government Big Data Newsletter.

Hadoop and Business Intelligence

Like my colleague Alex Olesker, I too attended Cloudera Day 2012.  While there were many panels of interest, perhaps one of the most important was Amr Awadallah‘s talk about big data applications to business intelligence. Many CTOVision readers with backgrounds in the intelligence community may think of corporate espionage when the phrase “business intelligence” is uttered, but I assure you that this is definitely not the case. Business intelligence is different from competitive intelligence, which is primarily based on open-source analysis of competitors and markets. Rather, business intelligence is quantitative analysis of internal data using advanced analytics techniques.

As we’ve noted before on CTOLabs, business intelligence is a changing field that is increasingly awash in information. Analysts face three core problems:

  • More information – Critical information now comes from new sources, such as online customer reviews and Facebook updates.
  • More change – The pace of competition, changes in customer preferences and organizational changes are all accelerating.
  • More questions – More people are asking new questions, such as: Why did that happen?How do these trends relate?

Adding to the analytical difficulties is the fact that complex and unstructured data, as opposed to relational data, is exploding. The question increasingly becomes–how do you manage the data and categorize it?

Awadallah talked about the unfavorable use/price dimension involved in archiving data. Once you’ve archived it, you’ve more or less lost it because it becomes too expensive to retrieve on demand. The solution? A combined computer/storage layer with Hadoop applications. Awadallah discussed opportunities to employ HDFS and MapReduce in business analytics. Hadoop offers three “core values”: scalability to grow nodes, flexibility in data storage and analysis, and data longevity. No transformation–implied under the previous model–is necessary, as data can start flowing any time with Hadoop applications. Awadallah contrasted what he called the slow “Data Council” model with a new forward-thinking approach he dubbed “Data Scientist” built on Hadoop products. Hadoop can grow too without requiring developers to re-architect their applications and algorithms. Both structured and raw data can be logged from multiple applications, and Hadoop offers centralized logging across all execution platforms.

All of these approaches, Awadallah argued, offered a much better RoB (Return on Byte) for business intelligence analysts looking to use big data to optimize their enterprise.

Quickstart Guide: Stand up your cloud-based servers with Rackspace

Rackspace is a growing cloud computing capability provider that differentiates themselves by what they call “Fanatical Support.”  We try not to dwell too long on the marketing slogans of firms but have first-hand experience with Rackspace and can tell you their support is really really different. It must be the culture, but every support experience has left me convinced they are a great company to do business with.

We host several sites at Rackspace, including CTOvision.com  We also host data in Rackspace’s clouds and use their Cloud Files as part of a content delivery system. Additionally we use Rackspace cloud servers for standing up testing and prototyping environments and have also stood up Hadoop analytical clusters in the Rackspace cloud.

We have a coming post on quickly standing up an Apache Hadoop Cluster (CDH3-based) on cloud servers and to prepare you for that step by step tutorial we want to make sure you had a familiarity with Rackspace’s capabilities. To that end, this post will walk you through how to establish your own Cloud Server on Rackspace.

First, a word about cost. If you start small you can give yourself enough power to test the Rackspace cloud for about $11.00 per month. If your day job is as an enterprise techie or leader we recommend doing this with RedHat enterprise (which is probably in use in your organization) and there is a slight licensing fee for that too, so plan on another $20.00 per month there).  Your rates are decided based on how many servers you stand up and how powerful you make the servers. Keep in mind that when you are done building servers you should smartly scale them back. If you are only using a small bit of compute power you should only pay the minimum (that is the beauty of cloud computing).

The standup of a server for this tutorial will only cost you a few bucks, but once you stand up the Hadoop Cluster we tell you about in later posts you can easily run into expensive tasks, depending on the size of the data sets you crunch.

Rackspace cloud servers provide hosting via “virtual private instances.” These give you the ability to deploy, utilize and pay for as little or as many resources as you need. You can have one up and running in about 5 minutes (you can time yourself as you work through the tips below).

Now lets get into it, follow along and you can stand up your Rackspace server in no time at all:

Step one: Create your login:

Step two: Stand up your server:

  • You will now have access to Rackspace’s Cloud Server Control Panel. Login there with the credentials you created in the previous step.
  • Click the button that says “Add Server”  (how is that for intuitive?).
  • You will now see an easy to read list of servers you can select from. There are three tabs at the top, one for Linux, one for Windows, and one for your Server Images.  For now, stick with Linux. Select a Server (your choice).  If you select enterprise Red Hat Linux you will pay a support license fee.  We do that for our images because we want that level of quality support. But if you are just testing things out and building prototypes select another, like Ubuntu.

 

  • You now have an option to give your server a name. Pick something unique and easy to understand, like BobsUbuntu or BobsRedHat (you can change the name you pick later).
  • Also select your initial server size. This is the great part about this. You can start small and scale up anytime you please. So start with the smallest option.
  • You will now be given a password.  This is your ROOT password so treat it with respect.  You must have this, but must protect it.
  • You will now see a window that shows the server is being built. In about a minute the server will be up and you can remotely log into it and operate on it like any other server you own.
  • As soon as your server is built you will get an e-mail with the public IP address of your server and the root password.
Now, how do you connect to this new server?
  • Upon successful connect you will get a console command window and can operate on your box as if it is any of your other servers.
  •  How you connect is a matter of personal choice. I use the old fashioned terminal window and SSH. You can bring one up on most any modern computer. You will use the info you received from Rackspace to connect.
  • The command you enter in your terminal window will looks somethting like:

sudo ssh root@50.56.237.236

You may also see:

The authenticity of host ‘50.56.237.236 (50.56.237.236)’ can’t be established.
RSA key fingerprint is ba:86:9b:de:f5:50:4a:3b:07:b2:ab:03:33:ef:c1:1c.
Are you sure you want to continue connecting (yes/no)?

  • Type yes then when prompted enter your password. Now you are in!
  • The first command we recommend you enter is one to change your root password. The one created by Rackspace was sent via a clear e-mail and could have been intercepted by bad programs or government agents. So while logged in over SSH type the command psswd and enter.  Now enter a new password, twice.  Save this new password in a safe location.
  • It is also important to update the software packages on your server. If you built yours with RedHat type this command in and hit enter:
sudo yum upgrade

We hope this quickstart guide was of use. We recommend establishing servers like this with Rackspace so you can continue to track this emerging cloud computing capability. We also recommend working through this example so you will be ready to stand up your own Hadoop Cluster.  We will be spelling out how to do that in a coming article.

And to review the Amazon Web Services (AWS) EC2 version of this guide see:
Quickstart Guide: Stand up your cloud-based servers with Amazon Web Services EC2

McKinsey on Big Data Part 2: Production, Supply, and Logistics

For the second post in the CTOVision summary of the McKinsey report on Big Data, we take a look at the impact of Big Data on production, supply, and logistics. When we discuss Big Data, there’s often an implicit assumption that it has to do entirely with IT functions. However, Big Data actually will have a large impact on manufacturing value chains.

While the manufacturing sector has historically been a driver of GDP and employment for many developed economies, globalization has made it a global activity with extended supply chains made possible by advances in information and communications technology as well as reduced transportation and market entry costs. This has resulted in incredibly complex and fragmented webs of globe-spanning chains, and increasing country specialization in specific changes of the production process. Manufacturers, in turn, have assembled global production and supply chain networks for cost advantage.

Where does Big Data come in? McKinsey argues that manufacturers will need to leverage large datasets in order to continue achieving large levels of productivity. The “raw material” is also very much available:

Manufacturing stores more data than any other sector—close to 2 exabytes of new data stored in 2010. This sector generates data from a multitude of sources, from instrumented production machinery (process control), to supply chain management systems, to systems that monitor the performance of products that have already been sold (e.g., during a single cross-country flight, a Boeing 737 generates 240 terabytes of data).  And the amount of data generated will continue to grow exponentially. The number of RFID tags sold globally is projected to rise from 12 million in 2011 to 209 billion in 2021. IT systems installed along the value chain to monitor the extended enterprise are creating additional stores of increasingly complex data, which currently tends to reside only in the IT system where it is generated. Manufacturers will also begin to combine data from different systems including, for example, computer-aided design, computer-aided engineering, computer-aided manufacturing, collaborative product development management, and digital manufacturing, and across organizational boundaries in, for instance, end-to-end supply chain data.

McKinsey forecasts the biggest applications of big data will be in Research and Development, Supply Chain, and Production functions:

  • The collaborative use of data can enhance product lifecycle management, using customer data to create better design to value, and better enable open innovation through sorting of high-value ideas.
  • Manufacturers can use data to improve demand forecasting and supply planning, but can also use big data to integrate reams of data from various sources in near-real time to adjust production.
  • Big data can also revolutionize the production process by creating “digital factory” simulations to better uncover optimal production layousts, sprinkle sensors throughout the supply chain and production process to reduce waste and cut operations and maintenance costs.
  • Marketing and sales/after sales support can also use sensor data and analytics to make customer service more reliable. As the report notes, “a repair technician can be dispatched before the customer even realizes that a component
    is likely to fail.”

However, the report goes on to note that Big Data may pose some cultural challenges for typical manufacturing organizations. One telling anecdote from the report described oil refineries that rely on managers with spreadsheets instead of algorithms using data collected directly from machinery. In order to reap the benefits from Big Data, manufacturers will have to develop capabilities and human resources up to the task of getting the best use out of the vast amount of information potentially available to them.

Related Reading:

The Insider Threat Worst-Case Scenario

GSA USASearch Wins 2011 Government Big Data Solutions Award

The Government Big Data Solutions Award was established to highlight innovative solutions and facilitate the exchange of best practices, lessons learned and creative ideas for addressing Big Data challenges. The top five nominees and overall winner was announced at Hadoop World in New York City on November 8 2011.

The Government Big Data Solutions Award Program is coordinated by CTOLabs.com.  The 2011 judging panel included: Doug Cutting, creator of Apache Hadoop and architect at Cloudera, Alan Wade, former CIA and IC CIO, Ryan LaSalle, Director of Accenture Cyber R & D, Ed Granstedt, Senior VP Director of the QinetiQ Strategic Solutions Center and Chris Dorobek, founder, editor and publisher of DorobekInsider.com.

The top five honorees of the Government Big Data Solutions Award are:

  • USA Search: Hosted search services over more than 500 government sites.  Provides search and suggestion services plus analytical tool dashboards.
  • GCE Federal: Cloud-based financial management solutions.
  • PNNL Bioinformatics: Advancing understanding of health, biology, genetics and computing.
  • SherpaSurfing: A cybersecurity solution that analyzes trends, finds malware, and writes alerts.
  • US Department of State, Bureau of Consular Affairs: Large data set with critically important applications for citizen service and national security.

The winner of the 2011 Government Big Data Solutions Award is the USA Search Program of the US General Services Administration Office of Citizen Services and Innovative Technologies.

Award judges saw the USASearch Program as a great example of solving Big Data problems to improve government agility and provide better service for less. In line with the GSA’s cost-saving “build once, use many times” paradigm, USASearch has provided hosted search services for USA.gov and, through its Affiliate Program, over 500 other government websites. This is done in an incredibly cost-effective way, especially for the agencies involved (which receive these services from GSA for free).

In 2010, USASearch adopted an open architectural model to better exploit shared solutions and open source technology. This model leverages Cloudera’s Distribution including Apache Hadoop (CDH3). The move brought further cost savings and scalable shared search services, which drove up usage. USASearch adopted Hadoop to improve the search results by aggregating and analyzing  information on what users are searching, their success in finding it, the time of the search, the affiliate, the results, and which results users click on, among other trends. To do so across hundreds of affiliates with growing traffic, USASearch considered scaling up or dividing its database systems, but they knew that these solutions would be costly and temporary.  Instead, it turned to  HDFS, Hadoop, and Apache Hive—a big data system that could grow cost effectively and without downtime, be naturally resilient to failures, and sensibly handle backups.

The overhaul of USASearch’s analytics is a dramatic success story. In the space of a few months, USASearch went from having a brittle and hard-to-scale RDBMS-based analytics platform to an agile, scalable Hadoop-based system.  By using a state-of-the-art open source technology, USASearch has created a radically different search service that transforms the customer experience Hadoop’s uses and effects will continue to expand as more data sources and tools are added. Having a government-owned and controlled search service allows provides insight on the needs and concerns of Americans to drive enhancements to other delivery channels. The public has a much improved experience when interacting with the government due to USASearch.

The GSA is to be congratulated for their mission-focused, citizen-centered, open approach to a big data challenge and a resulting solution that improves the experience of a broad swath of users of federal services. On behalf of our judges and the many citizens who use this capability on a daily basis we say thank you, and congratulations on this well deserved recognition.

The Database Society and You

There is much commentary right now about Facebook, social networking, and the meaning for the enterprise technologist. All of the tumult over Facebook’s new features (frictionless sharing, the timeline, etc) inspired me to dig out an old post I had written for Huffington Post on what I called the “database society:”

The dominant metaphor of the Cold War era was cybernetics — the study of closed, self-regulating systems. At its heart, George Orwell’s 1984 is a depiction of a giant cybernetic system, like the Soviet bloc countries David Ronfeldt describes in his paper “The Prospects for Cyberocracy.” Some critics of current movements in science and technology such as Jaron Lanier denounce the legacy of cybernetics as something that gave rise to “cybernetic totalism” — a system of beliefs that views all of the processes of life as interactions of information. So is Facebook a realization of Orwell on the level of the individual? Not really, but the reality of the database society is hardly comforting either. If there is a “cybernetic totalism” in Facebook, it lies in our own desire to make our own lives closed, self-regulating systems. The emerging database society reflects individualism, a desire for personal control, and hyper-consumerism. Facebook gives people the ability to categorize, organize, and experience the raw data of their social worlds. These qualities are by no means unique to social networking — it is embedded in the lived experience of everyday life. Everything from iPods to online dating have evolved similar search, sorting, tagging, and categorization mechanisms.

As I wrote, Hiroki Azuma’s research popular culture in Japan to some degree reflects the shaping effect of the database society.  Anime fans no longer have a preference for long Star Wars-like sagas distinguished by a tree-like narrative structure. Instead, anime is now largely devoid of originality or system-building narrative ambition. Users prefer a basic database of character designs, plot twists, and traits that are mixed and matched into different commercial products–which in turn are dissected and re-arranged in online fanfiction communities. Hollywood’s preference for “gritty reboots” and formulaic reusing of basic narratives and character archetypes shows that we’ve moving toward to this sort of cultural reality too.

Lanier’s use of the term “cybernetics” bears some explanation and contextualization. The dominant metaphor of Web 2.0 is complexity, self-organization, and self-synchronization. Lots of tomes have been written about the potential for social networks to enable the crowd. But the other side of Web 2.0 is cybernetics, especially as Facebook increasingly “eats” what we consider to be the broader Web:

Facebook offers businesses its own brand of access and indeed, a better branding tool. Your prospective customers are all here in one place, gathered together under the illusion of a social fabric; never mind whether they’re actually talking with one another or not, never mind whether the carrots or cattle they’re raising in Farmville are real.

Matt Devost has also written on CTOVision about how many of the search and categorization mechanisms discussed here could be improved, possibly dealing a deathblow to Twitter. Sounds vaguely totalitarian? Not entirely. One interpretation (suggested by Lanier’s work ) is that Facebook represents one side of an emerging techno-regime of micro-surveillance that increasingly exercise biopower through the transmission of life into data (which can then be sold, seized, or controlled). Critics of cybernetic technologies in the Cold War (particularly hippies, who once regarded computers as tools of The Man) made these arguments before the personal computer turned computing into a populist medium.

But as one of my favorite futurists Jamais Cascio noted, there’s also a very real (and sometimes overhyped) empowerment of the user inherent in the growth of a database society and the proliferation of devices that allows users to contribute and organize it. Cybernetics was once thought of in a highly positive manner that historically-minded enterprise technologists may remember.  Norbert Weiner, Gregory Bateson, Ludwig von Bertalanffy, and other pioneers of cybernetics saw great human potential in the ability of a systems understanding and the notion of feedback and self-regulation to improve business, society, and life as a whole. Cybernetics also moved far beyond its early focus on closed regulatory systems to open–and indeed living–systems, and much of contemporary environmental study and biology is influenced by large-scale systems views of the natural environment.

Enterprise technologists should take note of the growing ability of social networks to transform micro-activities of basic life into data. As Facebook and other networks–as well as the trends of ubiquitous computing–further deepen our dive into the database society, there is a revolutionary opportunity to create analytics for complex activities once thought to be entirely qualitative. as well as use network tools to enhance business productivity.

Our associates, clients, and friends are busy at work on many of these applications. As the database society looms, we have a choice between trying to fight what technology, social structures, and human nature is increasingly creating or we could think a bit more about how to harness it to empower the individual user and the enterprise owner. I vote for the latter.

Other topics of interest:

Artificial Intelligence Companies – A fast overview of Artificial Intelligence companies we believe are poised to cause the most positive disruption in the enterprise.

Big Data Companies – Reference to the greatest, most disruptive Big Data companies in the tech ecosystem.

Business Intelligence Companies – We assess these to be the Business Intelligence Companies most impactful for delivering real decision advantage.

Cybersecurity Companies – We apply our deep expertise in cybersecurity to assessing the best across multiple categories including:

More Content

Cyber War

SciFi

Super Bowl Ads Indicate Big Businesses Think You Are Afraid of AI

Tom Loftus provided an insightful review of some of Super Bowl ads in the 4 Feb 2019 Wall Street Journal. Like AI parody in SNL, these ads really poked fun at computers: Seriously, did anyone catch the commercials? If the Super Bowl is the zeitgeist, then it is safe to say that America is not […]

HBO’s Westworld: The Man In Black is a special kind of Twitter user

HBO’s Westworld is a modern take on the Michael Crichton SciFi tale of a robotic amusement park. The HBO version has a new twist. The creators/writers were all exposed to years of Twitter, which has influenced every episode and the entire arch of two seasons of the show. Now that season two is wrapping up, […]