CTOvision.com

Context for the CTO, CIO, CISO and Data Scientist

You are here: Home / Archives for Cloudera

Cloudera

Full reporting on Cloudera see the CTOvision Cloudera Overview

Technologies To See At DoDIIS

The DoDIIS Conference is almost upon us. August 12-15, 2018, members of the defense intelligence community will gather in Omaha for several days to listen to the latest in mission needs and requirements from key government leaders and to witness demonstrations of cutting edge IT on a very well attended expo floor.

If you have not registered to attend yet do so now here: 2018 DoDIIS Worldwide Conference

One of the challenges we have mentioned before is the overwhelming amount of innovation that will be on the expo floor. We recommend everyone put thought into who to spend time with before the event. We recommend reviewing the full list of exhibitors to get a gist of who is going to be there, then refine your intent on who to spend time with by considering your requirements and how these firms might potentially address them. It is also important to build in time for serendipity of course, but a bit of planning in advance can make this a much more productive conference.

As another input to your planning we reviewed every company with an eye towards potential positive/disruptive innovation in national security missions. idea of where you should spend your time. The list of who you see should be shaped by your own requirements of course, but we have a list below that may help. The list below are the tech firms we want to bring your attention to. In my view they are coming with innovations that could be very positive for the many important missions of defense intelligence.

 

The CTOvision List Of Must See Tech At DoDIIS

  • Alteryx: We like Alteryx for their focus on ease of use for business intelligence use cases, especially around self-service data analytics.
  • Amazon Web Services: We all know them but with their continuous innovation it is important to track the latest. Ask them about Kinesis for data and video.
  • Appian: They enable any company to be a software company. Would love to see more of this in government agencies.
  • Atlassian: Every developer seems to love them, largely because they remove friction in fielding real capabilities. Great collaboration for techies.
  • Attivo Networks: Advanced deception technology.
  • Anomali: Threat intelligence identifying threats to networks.
  • Basis Technology: Continues to innovate around language.
  • Blue Prism: Pioneers in robotic process automation and digititizing
  • Bricata: Innovative next generation intrusion prevention, detection, response
  • Carahsoft: Removing friction for government decision-makers. See at DoDIIS for access to 100’s of great technologies
  • Cloudera: The absolute leader in fielding enterprise data solutions that scale and operate at speed. Modern platform for machine learning and analytics optimized for the cloud.
  • Data Robot: Automating tasks for data scientists. Great capability.
  • Fidelis Cybersecurity: Providing a new platform to detect, investigate and stop advanced cyber attacks.
  • Palantir: Transforming the way organizations use data.
  • Percipient.ai: Artificial Intelligence for national security missions.
  • PFP Cybersecurity: If you want to make it hard on nation states trying to hack you, use these guys.
  • Recorded Future: Fast insights into cyber threats put in context of your enterprise needs.
  • VMware: Accelerating digital transformation for the enterprise.

By the way, we track these and many other hot technology firms in the CTOvision Disruptive Technology Finder . Use that to kickstart your market research.

Also see:

Seems Like We Should All Thank Cloudera, Tamr, Zoomdata, Trifacta, Kinetica For Helping GSK Help Humanity

Jessica Davis just published a piece at Information Week reviewing an architecture Glaxo Smith Kline (GSK) put in place to help make drug development and fielding faster. Creating new drugs always takes years, and after they are created it takes years more to get them fielded.

Mark Ramsey is the chief data officer at GSK. He and his team have some incredible goals regarding speeding drug development times. They intend on helping GSK reduce what is normally an 8 to 20 year timeline to 2 years. The GSK approach has been to design in ways that break down data-flow barriers.

GSK has not achieved their goal yet, but are well on their way. The architecture they put in place to do it is based on great technologies, most of which CTOvision has reported on in the past. The foundation of the GSK platform is based on Cloudera. Others include Tamr, Zoomdata, Trifacta, Kinetica.

From Information Week:

Ramsey’s data and analytics stack includes multiple technologies, with the foundation based on Cloudera’s Hadoop

“That’s our primary data and information platform — the source where we store our curated data and our analytics processes.” The stack also includes Kafka and Spark. Other technologies include StreamSets for data ingestion (which has been completely automated with bots), Tamr for machine learning data curation, Trifacta for data wrangling, and AtScale for virtualization across environments. AtScale lets users leverage familiar BI tools for insights from the Hadoop environment. GSK also uses Zoomdata for data visualization, Docker for some of its containerization, Kinetica for GPU-based analytics, and Waterline Data for storage and search. The total solution amounts to more than 5 petabytes of data, all on-premises.

The community should keep watching this one for lots of reasons. One is that GSK is in business to help humanity out. So if their architecture helps them help us it is something we should all have an interest in helping improve. And to the extent GSK will share lessons on their architecture it is something others can replicate.

It is also good seeing which technologies were put in place by GSK. This public recognition of great firms with enterprise grade technology is something that all these firms should be proud of and all enterprise technologies can consider an endorsement by the market.

Stay in the loop and our other enterprise IT reporting with our Enterprise IT Newsletters. 

We track Cloudera in our list of Big Data disrupters.

 

Examining New Mission-Focused Capabilities Coming From The Cloudera And Intel Engineering Partnership

The video at this link and embedded below captures the content and dialog of a webinar which examined new capabilities of Cloudera and Intel, with a focus on capabilities that provide a full stack solution to many key enterprise mission needs. The webinar included insights by Cloudera’s Senior Director of Technology Webster Mudge and Intel’s Enterprise Technology Specialist Ed Herold, plus questions from an informed audience. Results of the recent CTOlabs.com white paper on this topic were also presented.

 

https://www.youtube.com/watch?v=0KgxCb2X_Bo

This information can inform your design trades and is relevant to enterprise architects, engineers and line of business leaders.

We would love to get you feedback on the webinar video and/or white paper. Please contact us with your thoughts.

We track Cloudera in our list of Big Data disrupters.

We track Intel as a Tech Titan

Big Data in the Public Sector

In the video at this link and embedded below I provide some context on new approaches to data can enhance outcomes for public sector organizations, with a focus on real world use cases. I also mention key requirements which apply at most government organizations for their data and how organizations are addressing their unique requirements with technology provided by Cloudera:

Intel CEO Brian Krzanich and Cloudera CSO Mike Olson on Intel and Cloudera’s Technology Collaboration

Two of my favorite technology companies, the great American firms Intel Corporation and Cloudera, are now working very strategically together in a way that will have a positive impact on the future of enterprise IT.

For context, see the video at this link and embedded below:

In the video, Intel CEO Brian Krzanich talks about the equity investment and importance of the technology collaboration between Intel and Cloudera. The bottom line here: The leader in data center hardware and high performance computing (Intel) is now strategically teamed with the leader in the big data software platform (Cloudera) to produce a great innovation for developers, users and solution providers across every realm. This is bringing solutions to the mainstream in a way that leverages a standard platform.

What do I look for next? As a CTO I would like to see new documentation, including use cases, reference architectures and repeatable guidance that enterprises and developers can use for this new platform. Both Cloudera and Intel have many reference architectures and use cases available now, but new ones that combine the power of both these firms will be an awesome read. I really can’t wait.

Mike Olson of Cloudera provided some context on what this means strategically in a blog post at Cloudera. He summarized the essence of the news as:

  • Intel will replace its own Apache Hadoop-based software distribution with CDH. We’ll transition current customers and partners over a reasonable time period.
  • Intel and Cloudera will cooperate to enhance the open source projects that make up CDH so that they take advantage of the power and features of Intel’s chipset. Intel silicon powers 94% of the world’s servers today, so we expect to make life better for an enormous number of users.
  • We’ll work together to promote and distribute the software worldwide, directly and through the channels and partnerships we each bring to the relationship.
  • Intel Capital has acquired a stake in Cloudera, adding to the funds we announced last week. It’s the largest data center investment in Intel Capital’s history, and the investment makes Intel our largest strategic shareholder.

Mike concludes his post with some important relevance on this news, saying:

Our collaboration means that customers will be able to rationalize their IT spending. They’ll be able to do existing work better, faster and cheaper. More importantly, they’ll be able to attack new workloads, analyzing more data than ever before in ways previously impossible. They’ll get value from data they couldn’t capture before.

Mike’s full blog is here: The Next Ten Years

Also see more from the Intel press release below:

Cloudera, Intel Commit to Accelerate and Transform How Enterprises Use Big Data; Intel Makes Significant Equity Investment in Cloudera

PALO ALTO, Calif., and SANTA CLARA, Calif., March 27, 2014 – Intel Corporation and Cloudera today announced a broad strategic technology and business collaboration, as well as a significant equity investment from Intel making it Cloudera’s largest strategic shareholder and a member of its board of directors. This is Intel’s single largest data center technology investment in its history. The deal will join Cloudera’s leading enterprise analytic data management software powered byApache Hadoop™ with the leading data center architecture based on Intel® Xeon® technology. The goal is acceleration of customer adoption of big data solutions, making it easier for companies of all sizes to obtain increased business value from data by deploying open source Apache Hadoop solutions. Both the strategic collaboration and the equity investment are subject to standard closing conditions, including customary regulatory approvals.

Cloudera will develop and optimize Cloudera’s Distribution including Apache Hadoop (CDH) for Intel architecture as its preferred platform and support a range of next-generation technologies including Intel fabrics, flash memory and security. In turn, Intel will market and promote CDH and Cloudera Enterprise to its customers as its preferred Hadoop platform. Intel will focus its engineering and marketing resources on the joint roadmap. The optimizations from Intel’s Distribution for Apache Hadoop/Intel Data Platform (IDH/IDP) will be integrated into CDH and IDH/IDP and will be transitioned after v3.1 release at the end of March. To ensure a seamless customer transition to CDH, Intel and Cloudera will work together on a migration path from IDH/IDP. Cloudera will also ensure that all enhancements will be contributed to their respective open source projects and CDH.

The collaboration with Cloudera promises to enhance both Intel data center and Internet of Things (IoT) technology initiatives by enabling customers to manage and analyze machine-generated data from a variety of sources, spanning sensors to gateways and a range of devices.

“By aligning the Cloudera and Intel roadmaps, we are creating the platform of choice for big data analytics,” said Diane Bryant, senior vice president and general manager of Intel’s Data Center Group. “We expect to accelerate industry adoption of the Hadoop data platform and enable companies to mine their data for insights that inform the business. This collaboration spans our data center technology from compute to network, security and storage, and extends to our initiatives for the Internet of Things.”

Both companies remain committed to the open source community that has been the driving force behind Hadoop. The companies will also execute joint go-to-market strategies that include coordinated product, channel and sales programs.

“Hadoop has changed the entire conversation around data,” said Tom Reilly, chief executive officer, Cloudera. “Based on our ability to store, process and analyze all kinds of data, increasingly in new ways, the potential for advances in business, social and economic environments are vast. Teaming with Intel, the industry leader in data center technology innovation, presents an opportunity for us to layer Hadoop into a platform’s core framework. The result is rapid performance improvements on workloads such as those extracting intelligent insight. Intel’s vision for delivering open, performance optimized solutions for big data is synergistic to our vision to help companies accelerate time to achieving success.”

About Intel
Intel (NASDAQ: INTC) is a world leader in computing innovation. The company designs and builds the essential technologies that serve as the foundation for the world’s computing devices. Additional information about Intel is available at newsroom.intel.com andblogs.intel.com.

Intel is a trademark of Intel Corporation in the United States and other countries.

  • Other names and brands may be claimed as the property of others.

About Cloudera
Cloudera is revolutionizing enterprise data management by offering the first unified Platform for Big Data: The Enterprise Data Hub. Cloudera offers enterprises one place to store, process and analyze all their data, empowering them to extend the value of existing investments while enabling fundamental new ways to derive value from their data. Founded in 2008, Cloudera was the first and is still today the leading provider and supporter of Hadoop for the enterprise. Cloudera also offers software for business critical data challenges including storage, access, management, analysis, security and search. With over 20,000 individuals trained, Cloudera is a leading educator of data professionals, offering the industry’s broadest array of Hadoop training and certification programs. Cloudera works with over 900 hardware, software and services partners to meet customers’ big data goals. Leading organizations in every industry run Cloudera in production, including finance, telecommunications, retail, internet, utilities, oil and gas, healthcare, biopharmaceuticals, networking and media, plus top public sector organizations globally.http://www.cloudera.com/ www.cloudera.com

Connect with Cloudera
Read our blog: www.cloudera.com/blog/

Follow us on Twitter: twitter.com/cloudera

Visit us on Facebook: www.facebook.com/cloudera

Cloudera, Cloudera’s Platform for Big Data, CLoudera Enterprise Basic Edition, Cloudera Enterprise Flex Edition, Cloudera Enterprise Data Hub Edition and CDH are trademarks or registered trademarks of Cloudera Inc. in the United States, and in jurisdictions throughout the world. All other company and product names may be trade names or trademarks of their respective owners.

The Guide To Professional Hadoop Solutions For The Enterprise

hadoopsolutions I’ve known Kevin Smith of Novetta Solutions for years and have been a huge fan of both his ability to write about technology and his ability to field enterprise grade solutions that really work. He knows real enterprise solutions require focused planning, solid designs and an attitude of enterprise-focused teamwork. He is the kind of author that writes well because he writes frequently and writes relevant because he fields real solutions.

This makes anything Kevin collaborates on to write a good project. His latest book is written with Boris Lublinsky (principle architect at Nokia and author of 70 other publications) and Alexey Yakubovich (a Hadoop architect). Together they have collaborated on “Professional Hadoop Solutions“, a great “how-to” book and guide for the systems architect.

I am particularly impressed with the security chapter of this book. There are many challenges in the security environment related to Hadoop deployments and it is important for architects to know both the history and status of core Hadoop in this regard. There are configuration steps that can be taken that greatly reduce security risks and this book can guide you through ways to optimize your clusters in this regard. For those requiring real industrial strength security you will want solutions like Sentry on the latest distribution from Cloudera as well as their enterprise management tools and of course commercial support. But if you are just looking to scope out the kinds of issues you will need to deal with you will find a great introduction to security in this book.

My recommendation: Add Professional Hadoop Solutions to your library. It is a great read that will give you lots of food for thought and help you speed your deployment.

Related Reading:

CTO Guide to the Business of Artificial Intelligence

CTOvision Guide To The Cybersecurity Technology Marketplace

Why Cybersecurity is Looking to AI to Mitigate Business Risks

Working in Cybersecurity: Life on the front lines, in the C-suite, and everywhere in-between

Deep Insights Into The Future of Hadoop By Doug Cutting of Cloudera

Cloudera’s Doug Cutting delivered a presentation at Hadoop World that outlined key forces driving the data world forward which shed some important insights on where enterprise technology is going. This presentation is summarized and embedded below:

Doug starts with a review of his models for predicting the future and this is important, since every model has strengths and weaknesses. His model for predicting the future of data is to take a linear approach. We know what has happened in the past and can know what is happening now so can make linear predictions on the near term future.

 

For example, with hardware the past has always been that hardware gets cheaper and transistor counts go up so this cheaper hardware gets more powerful. This power and drop in price applies not just to processors but storage. The prediction this leads us to is a simple one, we will be able to store and process more data in the future and will do so more economically. Why do we care? Turns out that in parallel with this growth in capabilities in hardware is the adoption of technology. We have seen technology invade every element of industry and our personal lives and all this generates data. They leave tracks of what we are doing, what our institutions are succeeding and failing at and give us an image of ourselves and our businesses and if we look at the growth in industries they are greatest when they are automated.

Regarding software, in the past decade we have seen remarkable transition towards open source software as platform technologies. Linux and Android are obvious examples of this. The benefits and drivers for individuals and business are becoming very clear. Consider, for example, the fact that business are now built around technology. It is bad to have your business overly dependent on others in ways that increase risk and this translates to strong pushes for open source technologies. You don’t want your business to be hostage to someone else’s business. Open source avoids this lock-in and does so economically. You pay for the value you receive. So what prediction can we make from this? In the future, platforms for data must be open source. It is a basic requirement.

A key prediction can be made about open source in general and Hadoop specifically. A decade ago Doug Cutting began work on the project that became Hadoop. The community came together and made it reliable. The community enhanced its security. High availability was introduced. Improvements continued and continued. So we can generalize and predict from this. The Hadoop platform, already very capable, will continue to improve.

What has been happening around Hadoop? We have seen innovation around this platform. It started as a background component and with frameworks around it became easier to use, then new capabilities like HBase, Impala and Search. It is now very capable. The prediction to make: more and more types of workloads will be supported.

What does this mean for us all? The fact is that Hadoop now dominates in the Big Data space. It is no longer the place you come to to process data, it is the locus of your enterprise data. It lets data be shared and hardware resources be shared. People are taking workloads out of silos and bringing them into Hadoop.  We are seeing this used as an enterprise data hub.

On the topic of transactions on data: Transactions have long been thought to be out of scope for Hadoop. Doug believes this is clearly an important class of workload that is not served by the current Hadoop platform. A year ago Google published another paper describing their internal system they built for internal use that does this, demonstrating this is possible. In the past when we have seen this is possible, in two years it happens. The prediction: it is inevitable. It will come to this platform.

In Conclusion: Hadoop is playing its role as the center of an enterprise data hub. We are in the middle of a revolution in data. Revolutions can be scary times. But Hadoop provides a clear path that will endure, supporting a wide variety of workloads. Have no fear. Be comfortable adopting Hadoop.

Hadoop’s Impact on the Future of Data Management: Insights from Mike Olson at Strata and Hadoop World 2013

big_data_5Below and at this link is a video of Mike Olson at the opening of the 2013 Strata Conference and Hadoop World. The context in this discussion is important for technologists, strategists, analysts and executives alike, it provides insights in easy to understand ways and succinctly articulates the impact of some key trends in the environment.

The ecosystem around Apache Hadoop has continued to mature so updates like this from one of its key leaders are critically important.

Mike opened with a great overview of where the community has come in the last five years:

  • In 2008 the Big Data meme had not happened yet. And you probably hadn’t heard of Hadoop yet.
  • In 2009 the first Hadoop world was held and over 700 people showed up. This year, 3000 people attended in a sold out venue.
  • Now consider the many vendors in the ecosystem.
  • And consider the big trends.
  • When Hadoop was born was a compliment to traditional data processing. It was off on the side. Good for batch and for storage but could not handle real time. Much of the market did not pay much attention. But real time was always desired. Just a year ago Cloudera announced a real time platform, Impala, an open source real time SQL engine.

For context on where were are today and where we are going Mike reviewed that:

  • Other real time capabilities have been added including Cloudera search. In the single year since they have been announced over 5000 enterprises have added Impala and Search. Real time has always mattered.
  • Now that real time and search are both available, more work can be done on the real platform. Now other applications and uses can be supported on Hadoop. This platform is attracting work and attracting data. And it is attracting more and more users.
  • At enterprise deployments are showing a strong trend. Hadoop is emerging as an enterprise data hub. This meme is big in the industry now.

What is a data hub? A scale-out, affordable, reliable platform. Can hold any data in any format for as long as you want it. It is a storage layer with security built in that can do access control, auditing, logging, providence of data. And a secure storage substrate would also require a rich collection of engines for working with data. You want query, search, machine learning and analytics in place without moving it out. That collection of capabilities is hugely valuable and lets you work with the data where it  lives. But still this is not a hub. A hub needs to connect to the infrastructure you already rely on. That makes a hub and makes this concept very virtuous. This is something new. It is an enterprise data hub. This is a very big deal.

Cloudera announced, via Mike, the release of Cloudera 5, the industry’s first Enterprise Data Hub.

Bottom line of this new Enterprise Data Hub capability: Scale out storage, security, good data governance capability, a rich collection of engines for working on the data in place and delivering results to your systems and people.

For more see Mike expand on this concept here

 

 

More Content

Cyber War

SciFi

Super Bowl Ads Indicate Big Businesses Think You Are Afraid of AI

Tom Loftus provided an insightful review of some of Super Bowl ads in the 4 Feb 2019 Wall Street Journal. Like AI parody in SNL, these ads really poked fun at computers: Seriously, did anyone catch the commercials? If the Super Bowl is the zeitgeist, then it is safe to say that America is not […]

HBO’s Westworld: The Man In Black is a special kind of Twitter user

HBO’s Westworld is a modern take on the Michael Crichton SciFi tale of a robotic amusement park. The HBO version has a new twist. The creators/writers were all exposed to years of Twitter, which has influenced every episode and the entire arch of two seasons of the show. Now that season two is wrapping up, […]