An interview with Robert Fink, Architect of Foundry, Palantir’s open data platform Part Three: Open Development Environments

Bob Gourley October 1, 2018

Editor’s note: This post is the third of a series of three capturing the result of recent interview/discussions I had with Robert Fink of Palantir. The conversation was wide ranging, hitting on topics of design, development environments and a bit on the philosophy of enterprise tech. Several common themes emerged in those topic areas, including ways that Palantir has been leveraging open approaches to data architectures, system design and even developer environments. The first post focused on open data architectures. The second post continues the discussion of open approaches including open source software. This third post gets into the meat of modern, open development environments .– bg

BG: The third broad topic I want to ask you about is open-source for development environments. How does this relate to open-source software and open data architectures?

RF: Once again these are in theory separate concerns, but related by mindset. Hypothetically, you could imagine a development environment based on closed, proprietary tools that produces an open data system, but in reality I have never seen that done. The most powerful and flexible development environments today are based on collections of open-source tools that have been designed from the ground up to produce highly functional, well documented, sharable code. Modern languages like Typescript are entirely open-source, including the compiler, the build system, and the IDE. Contrast that to the early days of C++, where the best compilers and IDEs were closed-source and very, very expensive.

Palantir’s developers are addicted to a large amount of open-source developer tooling. For example, all of our Java developers build their projects with Gradle, and all of our frontend developers code in Typescript. We have made a number of our internal productivity tools available as open-source. For Gradle, examples include plugins for coding standards, container-based integration testing, or for building Docker images. For the frontend community, we maintain the standard linter for Typescript.

BG: On one hand, Palantir is a commercial software company, on the other hand, you give away some of your code as open-source. How do you balance these concerns, how do you decide which part of your software you publish?

RF: In the case of developer-tooling the answer is straightforward: I feel morally obliged to maintain our tools in the open-source domain because it incurs no overhead for us and because we would gain no competitive advantage from keeping it closed-source. A second key advantage of open-source tooling: it makes it much easier for our customers to interact programmatically with our open data platform.

BG: Can you give me an example of that?

RF: Since our data platform is focused on serving customer needs for data integration and analysis, we support standard formats and APIs with built-in interoperability, for example SQL, Hadoop, Parquet, as well as downstream analysis applications like Tableau or PowerBI. But we also provide ways for customers to extend the platform with bespoke functionality in the form of plugins and applications. We try to make this easy by providing building blocks for application development. In many cases we have turned these building blocks into open-source projects themselves so anyone can contribute to and use these capabilities. For example, Blueprint and Plottable are Javascript libraries that both we and our customers employ to develop rich Web-based frontend applications with consistent look and feel. We are in the process of open-sourcing our RPC framework, Conjure. Conjure is built on industry standards like HTTP and JSON and makes is very easy to interact with our platform APIs from a variety of programming languages.

BG: I noticed from your GitHub page that you remain personally involved in the open-source community. Can you tell us some of your favorite Palantir projects?

RF: On the top of my list is probably AtlasDB. AtlasDB is a distributed transactional key-value store built on top of Cassandra. Inspired by research papers on MVCC databases, we began internal development of AtlasDB around 2012 and decided to contribute the codebase to the open-source community in 2014. Today, AtlasDB is the database backend for all of our products. Specifically for Palantir Gotham, AtlasDB has enabled our customers to migrate from Oracle to open-source databases.

As a second example, we partnered with Pepperdata, Red Hat, Bloomberg, and Google to design and implement a Kubernetes backend for Spark. After a number of iterations, the main components developed as part of this collaboration were merged into mainline Spark. Both Spark and Kubernetes are industry-changing software ecosystems and I am really happy to see that their “marriage” is now available to the wider Spark community. Palantir continues to be an active participant in the vision and implementation of the project.

BG: Can you elaborate a little on legal concerns with open-source software, and maybe also on security concerns?

RF: Both are non-trivial subjects, in particular when organizations or individuals “go rogue” and blindly download and deploy open-source components. A big advantage of commercially supported open-source software is that the providers take care of the issues of indemnification and compliance which organizations would have otherwise have to solve themselves.

Open-source components, including code libraries, graphics and fonts, and entire applications are subject to a license grant. Different license grants impose different constraints on how the licensed component and the resulting software can be used. For example, the MIT license allows for permissive use of the covered components, including modifications and redistribution under commercial terms, whereas the GPL license is far more restrictive in how one can package and distribute a combination of the open-source component and proprietary code. We monitor our external dependencies to ensure that our use of open-source components is compatible with their licenses.

The same dependency monitoring is useful in the security context. Like any software, open-source software can be susceptible to malicious and unintentional bugs and security problems. Fortunately, the developer community and security experts around the world continually scan source code for vulnerabilities and maintain lists of known “bad” libraries. We use our dependency monitoring tools to identify and remove such code from our systems. A small amount of common sense also helps in this context: we try to avoid dependencies on code produced by small or obscure development teams, as well as code that is no longer actively maintained.

BG: Thanks, Robert!

RF: My pleasure!

This concludes our series based on our interviews of Robert Fink. Find all three of the series at:

Track the most disruptive technologies by diving into our categorized index:

Artificial Intelligence Companies – A fast overview of Artificial Intelligence companies we believe are poised to cause the most positive disruption in the enterprise.

Big Data Companies – Reference to the greatest, most disruptive Big Data companies in the tech ecosystem.

Business Intelligence Companies – We assess these to be the Business Intelligence Companies most impactful for delivering real decision advantage.

Cybersecurity Companies – We apply our deep expertise in cybersecurity to assessing the best across multiple categories including:

Cloud Computing Companies – We include both platform and software as a service providers, capturing only the most innovative and disruptive.

Collaborative Tool Companies – These are the firms that help humans connect to humans to create, manage and lead.

Infrastructure Companies – Critical enterprise foundations for business agility.

IoT Companies – Internet of Things and Industrial Internet of Things are here. How do you manage them?

Mobile Companies – Help manage, configure, secure and optimize these very powerful capabilities.

Robotics Companies – Including innovations in Robotic Process Automation, Drones, and industrial robotics.

Services Companies – We only track a few, the ones we really know well.

Tech Titans – These are the big players. We track the tech titans closely since their capabilities change continuously.

VC, PE and Finance Companies – Keeping an eye on the investors can give indications of coming developments.

You can also use our topical pages to get up to speed quickly on the current status of the major megatrends. See our pages on Cloud Computing, Artificial Intelligence, Mobility, Big Data, Robotics, Internet of Things, Cybersecurity and Blockchain and Cryptocurrencies.

We also provide special pages focused on high interest topics, including Science Fiction, Entertainment, Cyber War, Tech Careers, Training and Education and Tech Tips.

White House Takes Action Against Chinese Intellectual Property Theft

Junaid Islam March 22, 2018

On March 21, 2018 the New York Times reported that the White House was going to take aggressive action against China for illegal trade practices as well as intellectual property theft. The administration is right to do so.

Drive thru middle America and you’ll see closed factories whose products are now manufactured in China. For the past few decades China has implemented an aggressive policy of stealing manufacturing technology and then providing zero interest loans to Chinese companies to re-create American products. The transfer of wealth to China from the United States directly due to stolen products is in the order of $Trillions. More recently Chinese companies have moved into Silicon Valley to steal technology from leading edge start-ups before they even go to market.

The Chinese government utilizes three techniques to identify and steal technology in Silicon Valley. First, Silicon Valley is famous for meet ups to discuss new technologies. Every night there’s one or more events, often at a bar or coffee shop, to discuss new ideas such as machine learning or cybersecurity. Chinese operatives are regularly seen at all these events listening to ideas as well as identifying the people behind them. Second, once “smart” people have been identified, Chinese operatives utilize email password phishing attacks to access all of their correspondence. Third, if the company is viewed as valuable, Chinese operatives will launch a direct attack on server infrastructure to take source code, product designs and marketing information including customer lists.

Irrespective of how the next few months play out with China, here are some countermeasures you should implement now to protect your intellectual property:

Multi-factor Authentication: Email phishing attacks is the most common technique used by Chinese intelligence because it works. Thus you should implement some form of multi-factor authentication to make sure email isn’t stolen.

Server Isolation: After email, direct attacks on server infrastructure is the most common technique. Subsequently all Internet-facing servers should be locked down as well as connectivity between frontend and backend servers.

Certificate-based Mutual TLS VPN: Hacking WiFi hot spots is a popular technique. Given that R&D personnel love to go to coffee shops and check email or do some coding this is a big attack vector. Thus implement a certificate-based mutual TLS VPN to ensure the connections to corporate assets cannot be hijacked.

Application Layer Access: Only allow application layer access to authorized personnel and their devices. This way if attackers are able to steal an employees credentials or install malware on their devices the ability to install code on servers or laterally move within the data center is limited.

Do not let your company’s intellectual property be used against America.

The Cyber Threat Provides New Insights Into Bad Actors: Book updated with latest on threat actors and the tech ecosystem

Bob Gourley January 9, 2018

How can I learn about cyber crime? Good question.

The Cyber Threat was written to help executives, especially those without a deep background in cybersecurity, understand the nature of adversaries in cyberspace. The book includes a new section on the technological environment that can help decision-makers get their heads around the new tech enabled world arising around us. The book also captures key lessons from the most important cyber attacks in history, providing insights any modern executive can benefit from knowing.

Now more than ever, organizations need their executives and workforce to have a better grasp of the threats to business outcomes outlined in this book.

The book is available in paperback and electronically via Kindle.

For more info and to order see: The Cyber Threat.

What They Are Saying

“The Cyber Threat captures insights into dynamic adversaries that businesses and governments everywhere should be working to defeat. Knowing the threat and one’s own defenses are the first steps in winning this battle.”
Mike McConnell, Admiral, USN (Ret), Former Director of National Intelligence and Director, NSA

“There are no excuses anymore. Trying to run a business without awareness of the cyber threat is asking to be fired. The Cyber Threat succinctly articulates insights you need to know right now.”
Scott McNealy, Co-founder and Former CEO, Sun Microsystems and Chairman Wayin

“When I’m researching my own books, I always turn to Bob Gourley. I make diasasters up. He’s seen them for real. And most important, he knows how to stop them. Read this. It’ll scare you, but also protect you.”
Brad Meltzer, #1 bestselling author of The Inner Circle

“The insights Bob provides in The Cyber Threat are an essential first step in developing your cyber defense solution.”
Keith Alexander, General, USA (Ret), Former Director, NSA, and Commander, US Cyber Command

“Vaguely uneasy about your cyber security but stumped about what to do? Easy. READ THIS BOOK! “The Cyber Threat” will open your mind to a new domain and how you can make yourself safer in it.”
Michael Hayden, General, USAF (Ret), Former Director, NSA and Director, CIA

“Bob Gourley was one of the first intelligence specialists to understand the complex threats and frightening scope, and importance of the cyber threat. His book can give you the edge in what has emerged as one of the most compelling, mind-bending and fast moving issues of our time.”
Bill Studeman, Admiral, USN (Ret), Former Director, NSA and Deputy Director, CIA

For more see: TheCyberThreat.com

WTF?: What’s the Future and Why It’s Up to Us

Bob Gourley October 17, 2017

Tim O’Reilly’s latest book is out! Everything he writes is worth reading. He has always brought insights into technology, business and society, and this time he is no different. WTF?: What’s the Future and Why It’s Up to Us is a fantastic guide and overview of the tech and business megatrends sweeping over society.

Topics covered include artificial intelligence, cloud computing, on-demand services, the Internet-of-Things, autonomous vehicles and so many other related technology enabled constructs. He also makes clear arguments that income inequality, declining upward mobility and job loss due to technology are all topics that society must now address.

This book gets our strongest possible endorsement for anyone who wants to play a role in shaping the future of our tech enabled world (and that should be all of us). For more and to order see: WTF?: What’s the Future and Why It’s Up to Us

Additional Reading:

The Air Traffic Control System: We all assumed it was vulnerable, but now we know

WTF?: What’s the Future and Why It’s Up to Us

Pulitzer Prize-Winning Fred Kaplan Explores The Secret History of Cyber War

What You Need To Know About The Administration’s Cybersecurity National Action Plan

Like Kip in Napoleon Dynamite, We love Technology

Breakthrough In Energy We Should All Track: New green technology generates electricity ‘out of thin air’

Future Crimes: Everything Is Connected, Everyone Is Vulnerable and What We Can Do About It

The Security Yearbook by Richard Stiennon

I Thought I Had Seen Some Crazy Things In The Intelligence Community, But Nothing Like Nuking The Moon

CTO Bob Gourley in People Magazine: Security tips for the general public

Bob Gourley September 16, 2016

One of the key lines of OODA LLC’s business is our cybersecurity practice. We help enterprises build action plans to mitigate risks and improve their security posture. Increasingly we are also asked to provide tips and techniques for employees to consider in their personal lives. Every person with a computer or phone has information at risk and should take personal responsibility for improving their security posture. The problem is reaching people with the right information on what they should do.

My personal belief is that local law enforcement should be enlisted to help get the word out to 100% of our citizens on cyber threats and mitigation strategies. That type of approach has scale and police all need to be cyber experts these days anyway.

Media can and should also play a role, which is one reason I was very glad to have a chance to discuss cyber security with People Magazine in a story titled “Keep Your Email Secure: Experts on How to Avoid the Hacks That Impacted Colin Powell and Hillary Clinton.” In it I provide a few tips, simply put, for those that are not computer experts.

The following captures tips from that article plus a few others that can help citizens become more secure:

Stay aware of the threat and be suspicious of people trying to trick you out of personal information.
Pick passwords that are impossible to guess but easy to remember (a favorite method is creating an easy to remember sentence and taking the first letter of each word and adding symbols in the middle, if you can remember the sentence you can remember the password).
Since you have many passwords across multiple systems and can’t remember them all, use a password manager like Dashlane. Dashlane can help you see when passwords are not secure and can help you replace them with more secure ones.
Don’t use free email from your ISP. Use Google mail.
Use two factor authentication whenever you can (you can do this in Google mail and many other online services).
Look for spoofed emails and links, and don’t click on links or attachments from spoofed sources.
Know what https is and how to spot it in your browser.
Make sure you use the most recent edition of your computer’s operating system and all applications.
Contact all three of the major credit bureaus (Equifax, Experian and TransUnion) and ask for a copy of your credit report and also ask for a credit lock to make it a little harder on some of the bad guys.
Use a modern anti-virus system on your PC or Mac (Norton Security).

The challenge with building a list like that is keeping it simple but at the same time relevant and actionable. I’m not sure if I have hit the mark exactly right yet and would appreciate your feedback. Do you think that is too general/generic? Does it overlook anything else major that the average user should know about?

Changing Government Requirement For Market Research to Continuous Market Assessment

Bob Gourley July 26, 2016

During interview today with GovMatters.tv I’ll be discussing an observation many of us in the enterprise technology community have been discussing lately.

There is a growing consensus in the enterprise tech community (CIO, CTO, CISO) regarding the Cambrian Explosion of great capabilities in the tech sector. Although this is an incredibly exciting time to be an enterprise technologist, this explosion of capabilities presents new challenges. New approaches are needed in enterprise evaluation of new technologies. The process of studying capabilities on the front end is getting harder and harder.

There is room for improvement in many areas of this process. Vendors, for example, should consider better ways to market and sell. The old ways of seeking time on a technologist’s schedule are becoming less effective due to issues of time. There just isn’t enough time to meet with everyone.

Consider also the old approach of buying a booth at a massive conference. The biggest example of the mis-match here is at the RSA tech expo floor, where 500 cybersecurity vendors are fighting for time. If you tried to spend an hour with each of those you would need 10 weeks of dedicated time. Not going to happen.

On the vendor side, there is much more room for smart interactions with technologists via production of white papers that articulate user-focused use cases. That can show why a technologists should pay attention.

On the buy side there is a huge need for new methods of evaluation as well. No where is this needed more than in the federal sector, where all enterprises are compelled to follow a massive codex of hard to update regulations. The Federal Acquisition Regulations (FAR) are designed to serve government missions and do so while being fair to all in industry while preventing anyone in government or the contracting world from making decisions out of self interest. One of the many sections of the FAR deals with how to learn what industry has to offer. This is FAR Part 10 on Market Research.

The concepts and requirements in this section are great, for a previous age. Not so long ago, inserting brief spurts of technology assessment into a procurement was just fine. But today the government needs something else. The Cambrian Explosion of incredible capabilities in the tech sector means we need continuous market knowledge and a persistent ability to know what is available. The government’s acquisition system needs to be fueled by more accurate and up to date knowledge and it needs access to it on an ongoing basis. We need we don’t need periodic market research. We need Continuous Market Awareness.

We plan on doing our part to help with Continuous Market Research. You can track our views on the market and the future it is bringing us by signing up for the CTOvision Daily. And look for our online assessments categorized with the helpful mnemonic acronym CAMBRIC, which stands for Cloud Computing, Artificial Intelligence, Mobility, Big Data, Robotics, Internet of Things, CyberSecurity.

The Wisdom Of Carl Sagan On Science, Government, and Even Enterprise IT and Digital Risk

Bob Gourley July 1, 2015

Of course you know Carl Sagan the distinguished astronomer and great explainer of science via best selling books and the TV series Cosmos.

One of his last interviews was conducted by Charlie Rose in May 1996. The clip below captures a very important message from this discussion, one I think he really nailed for its relevance to the technological world we live in.

His message was about the full spectrum of science and technology and the fact that we people need to be smart enough to drive the right decisions in society vice abdicating to leaders who themselves are ignorant.

Key points:

“if the general public doesn’t understand science and technology, then who is making all of the decisions about science and technology that are going to determine what kind of future our children live in, some members of congress? There are only a handful who have any background in science at all, and some of them don’t even want to know about it.”
and
“We’ve arranged a society on science and technology in which nobody understands anything about science and technology, and this combustible mixture of ignorance and power sooner or later is going to blow up in our faces. I mean, who is running the science and technology in a democracy if the people don’t know anything about it.”

I believe this is important to revisit and consider on its own merit. However, there are special considerations in light of enterprise technology and digital risk. In large enterprises we are at the point where business leaders know too little about the technology that empowers their organizations. And in government the same is true. Too frequently, the leaders of organizations are too ignorant, but still very powerful. And we the people who must watch over them, as well as the government oversight processes from the executive branch, congress and the courts, are also too frequently too ignorant to really improve things.

The OPM breach, for example, is clearly a situation exactly like the one Carl Sagan described: a combustible mixture of ignorance and power that has blown up in our faces.

But this is a story best told by the great explainer himself. Watch the clip below and at this link and ask yourself, did Carl Sagan nail it or not?

More importantly, I believe the tech community should try to put thought into how we can help inform policy both in our own enterprises and broadly in society. It is not a foregone conclusion that we must always have things blow up in our face. We can structure in ways to reduce risk and improve enterprise functionality and the world overall.