CTOvision.com

Context for the CTO, CIO, CISO and Data Scientist

You are here: Home / Archives for Artificial Intelligence / Big Data and Analytics

Big Data Is About New Designs and New Ways Of Doing Things

Big Data Can Be A Useful Term If It Motivates Continued Change

big-data In today's world where data is everywhere the term "Big Data" may be losing some of its usefulness. The term came into widespread use in 2007 (see a short history of Big Data here), and was very important in helping people to realize that new approaches were needed to data challenges. Now that large data systems are common and AI/ML is being used to leverage these big data sets, the use of the term is dropping off. We still use it, since when used correctly it can help motivate change. In most every case where Big Data is being usefully applied today, it is associated with analytics. This is frequently the "so what" of Big Data. It is also closely related to the construct of Artificial Intelligence, since for AI concepts like Machine Learning and Deep Learning to work, large quantities of data and well thought out data architectures are required.
And clearly continued change is still needed. There are no simple answers to overwhelming data challenges, but one thing is clear. The answer to our challenges is not “less data.” The reasons we have so much data is the continued drive to improve business outcomes in industry, the need to pursue scientific discovery in academia, and the need to improve government support to citizens and the mission of government. We need new concepts and technologies to deal with these massive amounts of new data. We need new ways of doing more analysis and gaining more understanding, and of course we need smart data solutions.

Stay In The Loop: Sign Up To Receive the CTOvision Daily

When it it time to research the great providers of Big Data Analytics Solutions, kick off  your search in the CTOvision Disruptive Technology Finder. Pay particular attention to our Big Data Companies section. For help putting all this information into context for your business, consider tapping into Crucial Point's CTO-as-a-Service or Technology Due Diligence offerings, both are ways to quickly leverage the technical talents of experienced enterprise technology professionals.

SitScape, Inc. Receives Multi-Million Dollar Award from United States Air Force

One of the companies I have the pleasure of advising and interacting with is SitScape, a firm making incredible contributions to the ability of command centers, fusion centers and those that work with dynamic situations to bring vast quantities of information together for operational decision-making. If you have ever been in a command center you know this is an area in need of dramatic improvement and SitScape is doing just that. Their potential has been recognized by many, resulting in not just accolades but investment from thought leading firms like In-Q-Tel. SitScape provides significant improvements to any environment in need of rapid visualization, visual real-time collaboration and mobility support. They help move disparate legacy system data into web-based collaborative User Defined Operational Pictures on the fly, and enable significantly enhanced information access, sharing and collaboration from any device for real-time on-demand situational awareness and understanding.

With this note I want to share some very exciting news just issued in a press release from Sitscape. The release is copied below:

SitScape, Inc. Receives Multi-Million Dollar Award from United States Air Force 

November 15, 2012 – Vienna, VA SitScape, Inc. — the global leader in Collaborative User-Defined-Operating-Picture (UDOP) software, for shared situational awareness and visual dynamic collaboration — today announced that it has received a Rapid Innovation Fund (RIF) Award from the U.S. Air Force. The award was granted September 2012.

The National Defense Authorization Act (NDAA) for FY2011, Section 1073, provided the Department of Defense (DoD) with the authority and funds to facilitate the rapid insertion of innovative technologies into military systems or programs meeting critical national security needs.

Regarding the award: the Office of the Secretary of Defense (OSD) has established a policy for implementing RIF technologies. The Air Force RIF is managed through the Secretary of the Air Force for Acquisition, Deputy Assistant Secretary for Science, Technology, and Engineering (SAF/AQR).

The winning technology initiatives are to resolve operational challenges characterized by the following four critical national security areas of particular interest to the Air Force or Joint Urgent Operational Needs (JUONs) with an Air Force application:

1) Rapid Fielding to Support Overseas Contingency Operations

2) Cyberspace Superiority and Mission Assurance

3) Improved System Sustainment

4) Power Generation and Energy for Platforms

SitScape’s collaboration and situational awareness software was determined to be an innovative technology that meets Air Force RIF requirements.

“We are excited and honored to receive this prestigious award from the U.S. Air Force,” said Kevin Yin, CEO of SitScape. “The Air Force’s selection and use of SitScape’s innovative technology and capabilities will further enhance our efforts to support the Department of Defense and its national security missions.”

By leveraging SitScape’s award-winning software, Air Force mission users, operators, and analysts will gain greater utility and integration over their current disparate information sources and siloed applications. By simply using a Web browser, and with no programming needed, they can contextually visualize, communicate, share, and collaborate in real-time across globally distributed users and organizations. This can be done via a PC or mobile device for rapid situational awareness, enhanced shared situational understanding, and actionable operational intelligence.

###

About SitScape 

SitScape is the global leader in Collaborative User-Defined-Operating-Picture (UDOP) Situational Awareness software solutions. SitScape is ideally suited for Cyber Security Watch, Command & Control Centers, Operations Centers, Data Centers, Situation Rooms, Traffic Monitoring, Emergency Response Teams, Intelligence Analysts, and C4ISR applications. SitScape has large customers with varied applications in both the Federal and Commercial spaces.

To learn more about SitScape and its software, please visit www.sitscape.com, and contact: Jeff Heubusch, Director of Marketing and Communications, jheubusch@sitscape.com.

 

Congrats to all at Sitscape, especially to my friend and the innovative, mission-focused thought leader bringing all this together, Kevin Yin.  Keep pressing Kevin! The world needs what you are delivering.

The winner of the 2012 Government Big Data Solutions Award is the NCI Frederick National Laboratory

The Government Big Data Solutions Award was established to highlight innovative solutions and facilitate the exchange of best practices, lessons learned and creative ideas for addressing Big Data challenges. The Top Five Nominees of 2012 were chosen for criteria that included:

  • Focus on current solutions: The ability to make a difference in government missions in the very near term was the most important evaluation factor.
  • Focus on government teams: Industry supporting government also considered, but this is about government missions.
  • Consideration of new approaches: New business processes, techniques, tools, models for enhancing analysis are key.
The top five honorees for the 2012 award include:
  • Veterans Health Administration: New Big Data approaches and frameworks provide data and tools for 20,000 clinicians to track medical trends, better anticipate outcomes. The scope of the data set is over 80 billion data files. Focused on service to 25 million veterans. Judges selected Veterans Health Administration because of the impact and best practices in Big Data solutions.
  • NASA: Multiple and extensive activities. One of many exemplars was the NASA Center for Climate Simulation (NCCS). Their work includes scalable Hadoop clusters for large scale climate simulations.
  • Bureau of Engraving and Printing: This government agency is the largest producer of security documents in country. They have fielded an Oracle based Big Data solution enhanced quality and mission support, reduced waste. Judges characterized this as a good match of right business processes to and a modern technical approach.
  • AMSAA: Army Materiel Systems Analysis Activity. Vehicle data analysis program instruments vehicles in theater to collect operational and environmental parameter historical data. Massive data pattern screening and analysis toolsets put in place. Result: rapid identification of issues before mission impact.
  • National Cancer Institute Funded Frederick National Laboratory: Extensive research and working prototypes of cutting edge systems based on Cloudera Distribution of Apache Hadoop (CDH) and the Oracle Big Data Appliance. Judges noted the significant potential impact of this solution as well as the strength of the technical approach.

The winner of the 2012 Government Big Data Solutions Award is the National Cancer Institute Funded Frederick National Laboratory.

The National Cancer Institute (NCI) has spent years focusing thought and technical research on issues of concern to all of humanity. The are the part of the National Institutes of Health (NIH) responsible for coordinating the US National Cancer Program. They conduct  and support research, training, health information dissemination and other related activities related to the causes, prevention, diagnosis and treatment of cancer; the supportive care of cancer patients and their families, and cancer survivorship. NCI funds the Frederick National Laboratory.

The NCI Funded Frederick National Laboratory has been using Big Data solutions in pioneering ways to support researchers working on complex challenges around the relationship between genes and cancers. In a  recent example, they have built infrastructure capable of cross-referencing the relationships between 17000 genes and five major cancer subtypes across 20 million biomedical publication abstracts.  By cross referencing TCGA gene expression data from simulated 60 million patients and miRNA expression for a simulated 900 million patients. The result: understanding additional layers of the pathways these genes operate in and the drugs that target them. This will help researchers accelerate their work in areas of importance for all humanity.  This solution, based on the Oracle Big Data Appliance with the Cloudera Distribution of Apache Hadoop (CDH), leverages capabilities available from the Big Data community today in pioneering ways that can serve a broad range of researchers. The promising approach of this solution is repeatable across many other Big Data challenges for bioinfomatics, making this approach worthy of its selection as the 2012 Government Big Data Solution Award.

On behalf of our judges and all those who may one day benefit from research into the prevention and treatment of cancer, we congratulate the Frederick National Laboratory for this pioneering work, and add our appreciation to the teams of industry technologists developing well engineered solutions that make this work possible (with a special thanks to Oracle and Cloudera).

Top Five Nominees For The 2012 Government Big Data Solutions Award

The amount of data being created in governments is growing faster than humans can analyze. But analysis can solve tough challenges. Those two facts are driving the continual pursuit of new Big Data solutions. Big Data solutions are of particular importance in government. The government has special abilities to focus research in areas like Health Sciences, Economics, Law Enforcement, Defense, Geographic Studies, Environmental Studies, Bioinformatics, and Computer Security. Each of those area can be well served by Big Data approaches, and each has exemplars of solutions worthy of highlighting to the community.

The Government Big Data Solutions Award was established to help highlight some of the best innovation in the federal space. The 2012 award process solicited nominations from across federal, state and local governments. Nominations were evaluated based on how well submissions addressed three key factors:

  • Focus on current solutions: The ability to make a difference in government missions in the very near term was the most important evaluation factor.
  • Focus on government teams: Industry supporting government also considered, but this is about government missions.
  • Consideration of new approaches: New business processes, techniques, tools, models for enhancing analysis are key.
Nominees were evaluated by a panel of judges that included: Doug Cutting of Cloudera (creator of Hadoop), Alan Wade (former government CIO), Chris Dorobek (leading government journalist and publisher of the DorobekInsider.com, and Ed Granstedt (industry leading federal sector technologist).
The judges selected five exemplars for recognition as the top five nominees for the 2012 award. They include:
  • Veterans Health Administration: New Big Data approaches and frameworks provide data and tools for 20,000 clinicians to track medical trends, better anticipate outcomes. The scope of the data set is over 80 billion data files. Focused on service to 25 million veterans. Judges selected Veterans Health Administration because of the impact and best practices in Big Data solutions.
  • NASA: Multiple and extensive activities. One of many exemplars was the NASA Center for Climate Simulation (NCCS). Their work includes scalable Hadoop clusters for large scale climate simulations.
  • Bureau of Engraving and Printing: This government agency is the largest producer of security documents in country. They have fielded an Oracle based Big Data solution enhanced quality and mission support, reduced waste. Judges characterized this as a good match of right business processes to and a modern technical approach.
  • AMSAA: Army Materiel Systems Analysis Activity. Vehicle data analysis program instruments vehicles in theater to collect operational and environmental parameter historical data. Massive data pattern screening and analysis toolsets put in place. Result: rapid identification of issues before mission impact.
  • National Cancer Institute: Extensive research and working prototypes of cutting edge systems based on Cloudera Distribution of Apache Hadoop (CDH) and the Oracle Big Data Appliance. Judges noted the significant potential impact of this solution as well as the strength of the technical approach.

Details on the judges top pick will be announced on 20 Nov 2012

Perhaps The Most Important Big Data Presentation From Hadoop World

“Somewhere, something incredible is waiting to be known” said Carl Sagan. This inspiring quote is a tremendous mantra for those in the Big Data movement and it played a central organizing theme to Mike Olson‘s keynote at the 2012 Hadoop World/Strata Conference in NYC.  Mike highlighted this quote in his motivational keynote because of its perpetual truth. No matter what we learn we are always on the edge of new knowledge. There is always something else. Which makes this a great time to be in this industry. We are poised on the edge of great discoveries for industry and society.

Mike then hit on some examples of how Big Data analysis is transforming society and industry today:

  • The Large Hadron Collider at CERN, which generates 27 terabytes of useful data per day, is giving tremendous insights into how the universe works, including the recent discovery of the Higgs Boson, the particle which gives mass to matter. Researchers globally have access to this data for their own research. For example, the Unviversity of Nebraska Lincoln has captured all this data in a Hadoop Cluster and make this available to researchers globally. Discoveries will continue.
  • Healthcare is full of incredible Big Data use cases and success stories. Many of those come from activities around the Human Genome. New research has indicated that our genetic makeup is not the only biological determinate that makes us human. Human’s have about 1 trillon cells all including the same genome and this is incredibly important to continue to research but new research shows over 10 trillion other cells (mostly bacteria) ride on the human and have impacts on how our body function and even which genes are triggered. This world of epigenetics forces us to ask continually bigger questions in our search for more knowledge, and it is helping us better understand the onset and progress of disease and new treatments.  This includes breakthrough research by the National Cancer Institute, for example. All the breakthroughs in these fields are connected to Hadoop-based projects.
  • New predictive models on social matters and policy issues are also making a difference in the world including analytical models and model enabled analysis.  Much of this is based on the Hadoop family of capabilities.
  • What if how we generate and consume electricity could be better measured and action taken to understand and act upon those measurements?

The above are a few of the many use cases in place now using these capabilities.

Far more is being done with the current big data capabilities, and new more powerful capabilities are also coming, fast. Community-focused companies like Cloudera and the open source community itself are moving towards more simplified, unified, efficient Big Data platforms with a vision towards a single repository and comprehensive analytics that can explore it. And capabilities are now being announced to enhance the ability to work with speed over big data.

The Hadoop framework includes many tools centered around Hadoop, but at its core is an ability to run mapreduce jobs. Although info from those mapreduce jobs can be put in HDFS and queried in real time, it actually takes time to produce the results.  Current mapreduce jobs are not real time by any means. It is hard to ask and rapidly iterate new questions over Big Data. In the opinion of many, it is just too slow and in need of a serious, significant improvement.

Which leads to Mike Olson and Cloudera’s big announcement in this video, a capability known as Impala. Cloudera Impala is designed from the ground up with a powerful goal in mind: enabling you to work over big data at the speed of thought.  Cloudera has been working on this for 2 years.  It is a 100% open source real time query engine that lets you work in real time query engine fully integrated with Hadoop.  It allows you to take advantage of map reduce and ask speed of thought queries of your data. This is a new way to get at your data. And the data doesn’t move. It runs on your data in your Hadoop cluster.

Implala has been in a private beta with several customers. One, for example, is a Fortune 500 company specializing in agriculture and genomics which needed this capability to better automate their data-driven R&D decisions to reduce time to market from years to months. Result: better ways to feed humanity, now and in the future. This is seriously good. Cloudera, Hadoop and Impala have knocked down silos in these crucial area.

Mike ends his note by understanding that we need to keep asking big (and bigger) questions.

The rest of the conference included many detailed presentations on ways the community is asking and answering these big questions. We will report on more of them here.

[Watch this at if embed does not show: YouTube]

Related Reading:

Cloudera: Extract Benefit From All Your Data And Work Across Clouds

Hadoop’s Impact on the Future of Data Management: Insights from Mike Olson at Strata and Hadoop World 2013

Presenting Tech To Decision Makers: Be Bold, Be Brief, Be Gone….

What Is Hadoop? Here is a 101 with Mike Olson

Gain Decision Advantage With Innovative Enterprise Software

Rap Genius: Coming to your enterprise

With this post I want to lay out a way rap music is going to forever change the world of enterprise IT. Rap will have this impact on IT in many enterprises, but one of the biggest impacts, I predict, will be on the US Intelligence Community and in DoD.

First some background so you can see why I’m so convinced: Enterprises exist because the our biggest challenges require people to come together to collaborate on outcomes. This is true in and out of government. It is how humanity really moves things forward. That is one of the reasons enterprise IT is so cool. The enterprise CTO gets to impact some huge missions.

One of the hardest challenges in enterprise IT today is sensemaking over data. This is a driver of the current “Big Data” movement sweeping the community. But sensemaking is more than just machine analysis and algorithms. Sensemaking requires an ability to tap into institutional knowledge of your workforce. It can also require information collected by distant parts of your enterprise or perhaps collected long ago.

Some of the biggest challenges I’ve seen in sensemaking are in the intelligence community. If you are trying to understand something as complex as adversary intentions you need to tap into every source possible and need to understand the strength and weakness of every source. In the intelligence world, there are no perfect sources. Every source has flaws and must be put into context.  Every source, whether that is a satellite picture or a document translation or a communications intercept or a defector debrief or a report from an agent in a position to observe your enemy, must be put into context if you want to know what is going on.

The need for context over intelligence reporting is not a new concept. It has been part of our tradecraft for years.  Reports from satellite imagery include numbers that tell the image quality. Reports from communications intercept can contain information that provides the error associated with the location or perhaps analyst comments on whether it seemed the speakers were lying. Reports from agents can contain text that indicate if the ultimate source can be trusted or was in a position to observe. There are many other means of providing context. There are related means of capturing context and providing feedback to other elements of the intelligence process, including mechanisms to give feedback to those that plan, collect, process and analyze intelligence. But every means I know of for providing this context is the product of industrial age processes.  Most all of those are in need of modernization.

But how can rap help with this? It is not the music itself, but a capability developed by three fast-thinking rap fans designed to help in sensemaking over rap. The result, a new way to capture the thoughts of people via annotations and assessments. Their company and the sensemaking tool they provide is known as:  RapGenius.

RapGenius is a site that lets users upload lyrics. Then it lets other users annotate and explain the meaning of the words for each song. It has rocketed to the top of lyrics sites and is quickly becoming the top destination for lyrics on the net. Part of the reason why is the incredibly smart way they built their annotation capabilities. It is powerful and easy. Users can upload lyrics (or any other text) and others can very easily comment on that. And then people can evaluate the comments and suggested continued improvements. In building a platform for lyrics annotation they have built a great means to add context to text.

This is exciting for many reasons. They have built something that humans can easily use (and the market is proving that), they have built something that can easily track who made the comments, and they have built something that has been shown to be scalable to millions of users a day.

So you may already be wondering, will they be building an enterprise version?

RapGenius has attracted the attention of VC’s including one of the great investors in enterprise IT, AndreessenHorowitz. And along with the investment is coming advice from a team with a world class record in fielding enterprise grade software.

I had a chance to talk with RapGenius co-founder Ilan Zechory and focused in on this topic of the potential of RapGenius in the enterprise. Here is how the conversation went:

GOURLEY: Thanks for the time. What a great capability you have built.

ZECHORY: Thanks. You into rap?

GOURLEY: Not so much, but you guys have some of my favorite songs in Rap Genius (even American Pie, which I help explain here) and I’ve tried your capability out. I like the Rap Genius interface and ease of use.

ZECHORY:  Thanks. We are excited about our ability to add context and depth to rap and know this same framework can add context and depth over text from many other fields.

GOURLEY: Seems like some clear potentials for this framework as an enterprise IT capability. And with this recent a16z news lots of us are wondering about your future. What can you tell us about that?

ZECHORY: We know enterprises are full of people seeking truth about information. Collaboration and communication over text is traditionally the way most of that truth seeking and sensemaking gets done. But the technology to help do that has not improved much over the years. And once you make sense over data how do you record it for other’s use?  And how do you give feedback to others who have previously provided context? These are challenges faced by enterprises everywhere and they are things we have addressed in our platform.

GOURLEY: How can this help?

ZECHORY: One non-rap example is the Apple iTunes Terms of Service.  Clay Shirky uploaded that to Rap Genius after he saw another document (the Mayflower Compact) had been uploaded.  Apple’s terms of service, like so many others, is long and hard to understand. Shortly after Clay tweeted about his upload people started coming in and annotating it and clarifying that is is you are agreeing to if you accept these terms. Other than just explaining, people are also pointing out places that make these terms potentially voidable and other issues of note. This is turning into a great example of how our platform can help people communicate and clarify meaning and form assessments on action the meaning might compel.

GOURLEY: What sort of things will you add in to make this more enterprise ready?

ZECHORY: We are ready for enterprises now and we are continually adding more of the features that will make us even more attractive, like an ability to tap into the power of enterprise directories and to leverage enterprise PKI. We already, by design, build in significant audit and authorization mechanisms like the enterprise CTO will want out of the box. We also built this in a very open way that will let us work with other enterprise systems out of the box, including your behind the firewall enterprise social media systems.

GOURLEY: Great, we look forward to hearing more. Can we come back with more questions from our readers?

ZECHORY: Absolutely.

Closing comments: We will keep tracking Rap Genius and their progress, and hope to meet with Ilam and his co-founders in person in the near term. Please let us know what is on your mind regarding their capability, and please check out Rap Genius yourself to get a feel for their ability to annotate. Find them at http://rapgenius.com

 

 

Breaking News: Enterprise IT Now Sexy

OK I’ve always thought enterprise IT was cool. But now the hottest IT professionals from Silicon Valley, guys like Kirk Dunn of Cloudera, Aaron Levie of Box, Todd McKinnon of Okta and Justin Rosenstein of Asana are articulating why it is especially hot right now. These great innovators helped highlight some of the big reasons why IT for enterprises is so important during a panel at TechCrunch’s Disrupt.

My view: Kirk Dunn did a great job spelling out the importance of the datacenter.  Levie did a great job of articulating how new capabilities available to enterprises can be very disruptive, empowering new organizations in ways that make them more agile. And Justine Rosenstein provided a very compelling articulation about the importance of empowering people to get big missions done. All these were topics that really underscored well the importance of the enterprise and IT support to enterprise missions. This is well worth a watch.

For more see:

 

Related Posts:

Your Enterprise has too many Cyber-ish People(Opens in a new browser tab)

Cloud Computing(Opens in a new browser tab)

The Future of Enterprise IT: Insights that can drive your decisions today(Opens in a new browser tab)

 

 

What Does It Mean to Disconnect?

Are we too connected to our gadgets? Many journalists and social thinkers seem to think so. Are we all too fixated on emailing, tweeting, just plain staring at our screens to participate in the world around us? The answer may be more complex than that, but you won’t find it in posts like this recent effort from TechCrunch:

[O]ur obsession with technology, the right and wrong of it, the speed with which we have to create, consume, and engage with it, is not sustainable. People aren’t meant for this, not forever. It’s an imperfect state of being. We humans are always looking for balance. Today’s parents are realizing that they can’t achieve work/life balance with their children, for example. The same goes for technology. Tech/life balance is a similar myth. There’s no such thing, not really. You’ve just gotten used to it in the same way a coffee drinker gets used to caffeine. The effects are still there, but you don’t feel them until the drug is gone.

These sentiments will likely resonate with many tired of seeing iPhones whipped out during dates and family events, but we should recognize that they are based on emotion rather than careful analysis. As Nathan Jurgenon observed on Twitter, the post equates disconnecting from technology with not looking at a screen. Online and offline are synergistic—Facebook is, after all, populated by real beings rather than machines. Our entire motivation for going online is the real-life social and professional networks embedded into our technologies. I post a cat picture, for example, in part because I know that certain people on my FB field will find it enjoyable. I’m also motivated to log on because I care about the people in my network and use Facebook as a means of following their lives. I may see Alex Olesker and CTOVision alum Dillon Behr in person, but social networks are also an important way for me to keep in contact with them. It doesn’t replace in-person contact, but as Jurgenseon says, “[I]f we can fix this false separation and view the digital and physical as enmeshed, we will understand that what we do while connected is inseparable from what we do when disconnected.”

This leads us to the more troubling implication of the recent trend of “disconnect” pieces: they confuse disconnecting from technology with disconnecting from social and professional networks. The TechCrunch post bemoans the way that the demands of modern work life and social life motivate people to constantly plug in. But is the problem technology, or modernity itself? There is nothing said in the post that sociologists like Georg Simmel or Zygmunt Bauman have not said better over a hundred year period: modernity is fast-paced and constantly demands stimulation and attention. And what is the solution? Unmaking the structure of modernity is not really a reasonable option for most workers and consumers. The TechCrunch post at least looks at some technological solutions for making automated agents take on the burden of connection, but that’s not without significant problems either.

Having Internet-of-Things automated agents handling one’s life does not constitute disconnection. People will still monitor their agents, as they cannot isolate themselves from the social and professional consequences of what those agents do. There is something also highly bizarre about the idea of handing progressively larger and larger slices of one’s social and professional lifeworlds to robots in order to combat technological alienation from everyday life? It may be more efficient, and less stressful, but let’s not pretend that it somehow means a decoupling of information technology from our social life. Wouldn’t that also alienate us more? To take an example from the online dating industry, it’s one thing to use OKCupid’s sophisticated matching systems to help us find a mate, another to entrust those systems with handling replies to messages sent from prospective suitors.

If people highly value their work life and social networks, why should they be ashamed of using technology to interact with them? Calls to unplug and the fetish-making of being “offline” should be seen for what they really are: a fantasy of solitude that privileges the individual’s absolute desire to be free from the demands of the outside world. Is selective disconnection good or helpful? Sure, everyone could use a break. But the assumption that taking a break is somehow revolutionary has to go, as it inevitably privileges solipsism. Taking a break does not mean that you escape from the consequences of larger networks you are personally embedded in. The IRL fetish is at heart an urge to have the control necessary to be an island whenever you want to be. Sherry Turkle , for example, can run off to Cape Cod to completely withdraw from family, friends, colleagues, and the rest of the world. Not everyone has that luxury, but not everyone needs or wants it.

Defining Big Data for the Public CIO

The following is a piece I wrote for Public CIO. It is reposted here with their permission. I would appreciate your thoughts on this topic. How do you think public CIO’s should define Big Data? -bg

Enterprise IT professionals, including public CIOs, have long recognized the power of data, and the exciting new sense-making capabilities around big data approaches have generated a great deal of buzz and excitement. If history is a guide, however, we are about to see that term lose much of its meaning. Here is what I mean:

Do you remember service-oriented architecture (SOA)? This concept led to tremendous new capabilities and efficient, mission-focused designs. Enterprises established architectures in which application interfaces, logic and data were separated and smartly reusable. After the term went mainstream, every company in the IT ecosystem grabbed onto it and began to use the acronym SOA to mean anything they wanted it to. Although it’s still a useful construct for IT professionals, when it comes to interacting with industry, the term has now lost much of its meaning.

Then there’s cloud computing. When enterprise IT professionals use that term among themselves, there is huge value in the concept. It conveys a great deal of meaning regarding a need to change business processes to take maximum advantage of modern IT and new offerings. Now, however, most IT vendors describe what they do as cloud computing. When it comes to interacting with industry, that term, like SOA, has lost much of its meaning.

Now what about big data? Today it remains a very helpful term. Practitioners, including IT architects, systems engineers, CIOs, CTOs and data scientists, all use this term in dialog over ways to improve sense-making over data. The term remains a useful way of introducing others, including non-technologists, to new approaches like the Apache Hadoop framework. We have a continuing need to discuss these topics, and the term “big data” will likely be with us for quite a while.

But just like SOA and cloud computing, big data is now a hot topic among the vendor community. All indications show that most IT vendors are aware of the exciting dialog under way on this term. All have either already shifted their marketing strategy to include this topic — or they soon will. Odds are that most every firm in the IT industry will soon be proclaiming itself to be a big data company.

I’ve already seen plenty of evidence that this rebranding is under way. I’ve heard makers of network switches and routers assert that they are big data companies because they move large amounts of data. I have met with mapping companies that want to be called big data companies because they plot data. I know of an old-school storage company that wants to be known as a big data company because it stores lots of information. A great information integration company I know and love has told me it’s the big data solution of choice since it integrates data. The leading chip-maker is about to kick off a big data campaign, because it takes processors to process big data.

And in every case, the firms are creating their own definitions of what big data is. History is going to repeat itself here. Very soon, every vendor you deal with will want to get you to use its definition of big data.

So what should public-sector technologists do in an environment like this?

I recommend doing what enterprise technologists do best: Focus on your mission needs; don’t let anyone convince you to conform to their concepts of how those needs should be met.

And when it comes to definitions, you should be prepared to articulate one that best meets your organization’s needs. As a starting point, I recommend the definition at Wikipedia.org, since this community-edited site captures the input of many. Wikipedia’s definition is this: “Big Data implies the need for a strategy for dealing with large quantities of data. The term is also used to describe the new platform of tools required to successfully handle sense-making over large quantities of data, as in the Apache Hadoop Big Data Platform.”

I like this definition because it focuses on sense-making over data, which is why we have the data to begin with. I also like the reference to Apache Hadoop, since every big data solution I know of uses this framework. Hadoop is usually key to big data, but other important capabilities in this framework include HDFS, HBase, Hive, Cassandra and Mahout.

If you select a definition that doesn’t key in on sense-making over data, then you automatically open yourself up to letting every maker of any IT capability say it is a big data company. And if you don’t mention the Apache Hadoop framework in your definition, you open yourself up to allowing every maker of legacy software to say it is a big data company even though it has the same old approach. There’s something new about big data designs, and that is the distributed processing of large data sets over clusters of computers enabled by the Hadoop framework.

Whatever definition you decide to use, I would recommend you dive deep into learning the capabilities of the Apache Hadoop software library. This framework enables distributed parallel processing of huge amounts of data across inexpensive, commodity servers — and no vendor should bring you a big data solution unless it has leveraged the powerful capabilities of this framework.

Big data and how the community uses the term is a topic in need of more discussion, and my hope is that technologists from across the public sector, at local, state and federal levels, have a greater dialog on what that term means to public-sector missions. Discussing this topic could prove to be very positive for organizational missions and will help the IT vendor community better understand public-sector needs.

More Content

Cyber War

SciFi

Super Bowl Ads Indicate Big Businesses Think You Are Afraid of AI

Tom Loftus provided an insightful review of some of Super Bowl ads in the 4 Feb 2019 Wall Street Journal. Like AI parody in SNL, these ads really poked fun at computers: Seriously, did anyone catch the commercials? If the Super Bowl is the zeitgeist, then it is safe to say that America is not […]

HBO’s Westworld: The Man In Black is a special kind of Twitter user

HBO’s Westworld is a modern take on the Michael Crichton SciFi tale of a robotic amusement park. The HBO version has a new twist. The creators/writers were all exposed to years of Twitter, which has influenced every episode and the entire arch of two seasons of the show. Now that season two is wrapping up, […]