OODAloop Network Benefit: Direct participation with other members in monthly research meetings

Bob Gourley November 24, 2020

OODA Network members are invited to participate in a monthly video meetings to discuss items of common concern to our membership. These highly collaborative sessions are great ways for our members to meet and interact with each other while talking about items of common interest. We also use these sessions to help better focus our research and reporting on member needs.

To encourage openness of discussion, these sessions take place with Chatham House rules, where participants are free to use the information in the meeting but are asked not to directly quote or identify other participants.

For our members who could not make our last monthly session, here is an overview of the topics we reviewed:

Content Of Interest:

In September our analyst team published insights on a variety of topics (all available to members here) including:

Member Interview Series:

Chris Ward continues her interview series focused on introducing our members to each other. Recent subjects include Masha Sedova of Elevate, Joel Wallenstrom of Wickr and journalist Marc Ambinder. All in this series are available at: OODA Network Expert Series.

The OODAcast:

We continued to interview thought leaders with insights for our members and publish the results to both our audio and video channels (to subscribe see OODAcast). Recent subjects include:

Topics Of Potential Impact:

Each month we provide lists of topics and ask members to provide feedback by adding new topics or letting us know what is really not relevant. This helps inform decisions by our members by bringing focus to what is important but also helps drive our research agenda. Some discussion topics and comments included:

AI in national security: Much broader than DARPA AlphaDogFight and support to logistics etc. Point was made that if the US Air Force is in a dogfight something has already gone wrong, so perhaps the nation should be applying AI to other parts of national security.
AI in every part of our lives: Healthcare noted example, including AI for AI for cancer detection/diagnostics
CRISPR Developments: Including gene editing to produce “super-dad” livestock. Another interesting development is CRISPR created potential treatment for muscular dystrophy
Watch for continued CRISPR breakthroughs including from nation’s that do not share our ethical framework and from researchers that could be violating laws of their own countries. Potential for research by proxy using criminal organizations.
Brain-controlled devices. Biggest/latest news is Neuralink. More clearly coming
Watch For Continuous disruption via social media: including disruption of democracy, quality deep fakes and engineered disinformation/misinformation adding fuel to the fire. Disrupting all democracies.
Track space developments closely: including Artemis. But also Hayabusu2, which will return to earth with sample from asteroid (6 Dec)
Quantum Advancements: Including Quantum Key Distribution and Quantum Networking, Quantum Supremacy including modeling of atoms,
Communication revolution: 5G, Wifi 6, Spacebased communications.
Cyber Conflict: This is not just about hacking, but also about using cyberspace for unfair advantage. Example given of nation state funding of exploit development to sell for bug bounty and gain major contracts, which then leads to deeper insights for IP theft.
Embedded Computing: In this case we are talking about embedded in the body. Visceral reaction by many, but clearly this is a trend.
Infodemic: Concern by many members on this issue of trust and knowing who to trust.

The meeting and this input is already shaping our focus, insights on most of the above is available at our member resources page.

Approach To Our Next Meeting:

Next month we will continue our examination of topics of interest and will facilitate a dialog between members. In response to suggestions from members we will also be asking some our our members to provide 3 minute overviews on topics of interest designed to share knowledge on priority topics. The next meeting will include a 3 minute discussion from one of our members who is a AI practitioner who will discuss transparency in AI. We have also asked the CEO of a major threat intelligence company to provide his insights on AI since in a recent discussion we found his views to be very helpful in gauging what is possible and what is improbable in near term AI capabilities.

We Need Your Feedback:

If you are a member we need you feedback on the above to keep our reporting focused on your needs. Please reply to our weekly member newsletter with any feedback, guidance or questions you have for us.

If you are not a member yet, please take a moment to join and deepen your engagement with your peers on these and other topics. Do so at: Join OODAloop.

The Titan Release of Palantir Gotham: An Interview with Ryan Beiermeister

Bob Gourley August 1, 2019

At this point most everyone in the technology world has heard of Palantir. Their solutions are in use across multiple industries and government agencies. But Palantir continues to evolve in capability and with the current release of their platform they are bringing together data, analytics and visualizations in ways that can add significant value to end users and the enterprise missions they support.

To get an update on the latest capabilities we sought out an interview with Ryan Beiermeister, the lead product manager of Palantir’s Gotham platform. This post captures insights from that interview.

In Ryan’s seven years at Palantir she has succeeded at multiple functions in the firm, including key roles like customer facing deployment strategist, which required her to bring knowledge of all Palantir capabilities to customer challenges. She also helped ensure product developments were rapidly and accurately communicated to customers via product marketing. For the last 4 years she has been the product manager of Palantir Gotham, the company’s flagship platform. In this role she leads engineers, developers, designers, and planners in continuous development of new capabilities focused on customer success. In the interview I was struck by Ryan’s leadership style and vision and will be making that a topic of a future post. For now we will focus on Palantir Gotham and the new Titan release.

Gourley: Ryan can you tell us what Palantir Gotham is and how it originated?

Gotham is Palantir’s core Operations and integrated Intelligence platform — honed over a decade of partnership with military, civil, and intelligence communities. We built Gotham because we deeply believe in empowering institutions to make data-driven decisions to counter threats and abuse. The challenge, however, is that as our world accelerates — the data our organizations depend on is ever-increasing in volume and complexity, while response times and human resources narrow.

Palantir’s goal is to provide clarity amongst this storm — to build technology so that organizations are supercharged, not hindered, by Big Data. Gotham turns this data into knowledge: integrating an organization’s disparate datasets, mapping them onto a common ontology, and layering analytical tools for users to generate and share insights. This is all backed by powerful security frameworks and fine-grained access controls — creating an expressivity that no other data platform has — so that users can securely collaborate, discover, and evolve knowledge across time. Gotham thus gives organizations the advantage against a changing landscape — so their experts can act on the most informed, comprehensive intelligence to achieve their mission.

One of the things that makes Gotham so unique is the level of partnership we have with our customers. We are not technologists in a lab in Silicon Valley — we are engineers on the ground, in the battlefield, at the operations centers. I was just talking to colleagues yesterday who have been on base abroad for 4 straight weeks, working side-by-side with the customer and channeling feedback so our software supports key operational missions. Real collaboration with our partners is what pushes the Gotham product forward.

Gourley: How are your customers using the Gotham platform?

For our partners, Gotham represents the integrated picture of what they know about the world. Customers range from military or intelligence organizations who might be conducting counterterrorism work, to regulatory agencies investigating financial or securities fraud, to nonprofits helping law enforcement find missing and exploited children.

More and more, we’re also supporting operational workflows. In a high-risk scenario, you need everyone coordinated on the same page, acting on the same information — and you can’t afford any disjoint between intelligence and operations teams. That’s why we’re deploying thousands of ruggedized laptops with Gotham to the US Army to assist soldiers in the field. Users can live collaborate geospatially and quickly answer questions about threats, equipment, and terrain. Operations that previously required weeks of preparation now take hours and each mission creates intelligence that is stored in Gotham to inform preparation for the next. That knowledge capture is so important — we once had a Commander in Afghanistan tell us that the 17-year war in Afghanistan more resembled 17 “one-year” wars — because they had to start over between deployments and rotations. Instead, Gotham facilitates an integrated “common operating picture” that lends efficiency and accuracy to soldiers operating under the highest stakes.

One of the other use cases close to my heart is disaster response. I’m a native Houstonian (who happens to be named after MLB-great Nolan Ryan…), so it was very moving to see our 2017 deployment with Team Rubicon in the aftermath of Hurricane Harvey. Within 24 hours of the storm, we had teams of engineers on the ground with Team Rubicon volunteers — fusing geographical, weather, and vulnerability data to facilitate rescue operations for over 70 people. That’s another type of impact I’m really proud of.

Gourley: We also hear about Palantir Foundry. What is Foundry, and how are Gotham and Foundry different?

Great question! Foundry is another key offering in the Palantir platform ecosystem. Whereas Gotham traditionally sings in operations and investigations — taking large data sources, unifying them, and enabling users to mission-plan and find insights from the “ground up” — Foundry was built to answer the computationally challenging questions enterprises have at scale. This means integrating petabytes of information into a common data foundation — empowering discovery and cross-functional collaboration that is truly changing whole industries. For example, via our partnership with Airbus, we are transforming aviation by deploying Skywise, a common data environment powered by Foundry, for almost 100 airlines and their suppliers. This enables airlines to share and operationalize massive-scale data around production, inventory, sensors, and maintenance systems — so that end-to-end, the industry is more efficient. The scale of data collaboration is really incredible. Another example is the World Food Programme, which serves 90 million people in 80 countries and uses Foundry to scale and optimize its global supply chain.

We’re seeing more and more government customers use Gotham in the traditional sense combined with Foundry. This might look like Gotham as the backbone intelligence platform for an agency — where analysts, agents, and operators build knowledge — with Foundry as the key system used for massive-scale analytics and management of data pipelines. And of course, both Gotham and Foundry share the same Palantir DNA: open APIs, fine-grained security and access controls, audit capabilities, interoperability with legacy and 3rd party systems, and cross-organization information sharing to name a few.

Gourley: The latest release of Gotham is called Titan. What was it like to build this release – and what are customers excited about?

To build Titan, we really had to go back to Gotham’s roots — and understand what has made the platform so valuable and enduring over the last decade. There’s a lot of dimension to the product — but at its core, what Gotham delivers is very simple and profound. It’s enabling customers to build a data asset — their most holistic, integrated picture of the world. It’s a phenomenally obvious concept — but one that requires an exorbitant amount of technical innovation and orchestration to get right. So we wanted Titan to extend that value — making that data asset more robust, open, and connective to the external (or “federated”) data stores to which the customers have access. The data asset is also powerful due to Gotham’s ontology — or the common model we use to translate raw data into meaningful representations. So instead of rows and columns, you have people, places, things. With Titan, we aimed to take the value of an ontology even further — by enabling custom, domain-specific views of this data that integrate various pieces of intelligence and add rich visualizations. We call these “Custom Object Views.” So instead of wading through thousands of unordered things we know about an entity, analysts can look at COVs and answer their first 10 questions within seconds — such as the last flagged financial transaction, or any suspicious travel abroad.

Finally, Titan introduces four new Modules into the Gotham platform: Ava, Dossier, Gaia, and Table. These Modules derive their power from seamless integration with the Gotham ecosystem: the underlying data asset, ontology, search, collaboration, and security frameworks. They build on what has been tried, true, and differentiating about Gotham all along — while injecting new methods of finding insight.

Gourley: Is a module like an add on?

Correct — Modules are additional investments on top of the Gotham platform. We think about them as accelerants to Gotham’s value, uniting the data asset with elements like real-time knowledge management (Dossier), geospatial collaboration (Gaia), high scale search and triage (Table), and Artificial Intelligence (Ava). Each Module was driven by real customer need — such as coordinated response to a crisis in the aftermath of an attack — and was prototyped on the ground with deep partnership. The Titan site is an excellent resource to learn more.

Gourley: Seems like openness is a key focus for Gotham.

That’s absolutely right. We know that it’s essential for governments to have extensible, interoperable technologies — and that they are the ultimate owners of their data. Period. All data in Gotham can be exported in a variety of open formats, on the front and backends, so that data is never trapped.

As I mentioned, one of the other core components of Titan is “federated” search of external systems to which the customer has access — so users can explore that data without the need for backend ingesting. Gotham makes it possible, from a single search interface, to search every data source at an enterprise and present users with views of data that include insight into where the results actually came from. Records of interest can be imported on the fly while respecting data provenance and security, with Titan supporting additional frameworks to write back to those systems.

We are also finding more and more opportunities to collaborate with developers on the customer side. Custom Object Views are one such opportunity (we ship Titan with an Object View editor for easy-to-configure views of your data). Titan also ships industry-standard RESTful HTTP/Json protocols for all APIs — so that experts can develop against the nuances of their institution.

Gourley: You mentioned the Ava Module. How is Gotham investing in AI/ML technologies?

It’s a major part of our roadmap. So many organizations are bogged down by data complexity, with their analysts and operators spending 80% of their time just searching and manually crunching data. That overhead is untenable moving forward — so we spend a lot of time thinking about how Gotham can use AI to flip that paradigm. How can we use technology so that experts spend less time on repetitive tasks, and more time interpreting and making decisions?

We also want to unite AI/ML innovations with what makes Gotham so differentiated in value. Fundamentally, these algorithms are only as good as the data that backs them — if you don’t have a holistic data asset from which to train and drive connections, you’re not unlocking true potential. You’re just building another piecemeal dataset. For AI to be transformative, it needs to be operationalized within a data and collaboration platform — and that’s where we’re focused with the future of Gotham.

You can see this throughout the platform — where we’re using AI to drive insights for operators, for mission planners, for intelligence analysts. For example, our Ava module automates the discovery of connections across the data asset for humans to review. This might look like alerts for complex patterns of behavior across millions of data points — all sent directly to users at a scale that would be impossible to track by human capacity alone.

Of course, all of this must be done transparently — and there is no replacement for human judgement. But we can and should enable analysts to use AI to offload specific complex and routine tasks – and are deeply investing in it. Gotham leverages the best analysts and operators and scales them — so that they can focus on building intelligence and enacting missions to keep us safe.

Gourley: What excites you most about the future of Gotham product?

I’m most excited by Gotham’s rate and speed of innovation. There’s an electricity within our product organization — where features are shipping faster than ever and we’re working on problems that most thought impossible to solve. This is all driven by deep engagement with our customers — a testament to the scale and impact Gotham has had over the past 10 years. I once met an Army veteran at a conference — who approached me to say that Gotham had saved his friends’ lives down range by alerting them to placed IEDs. It’s easy to get lost in pixels and data — but truthfully, we’re all working towards real issues that matter to real people in the world. And we know we need to do more. The challenges our institutions face are exorbitant – we have to keep pushing to ensure they have the technological advantages they need. That mission motivates us every day — and though it’s been a decade, Gotham feels like it’s just getting started.

Gourley: I’ve heard many customers are amazed by how different Gotham looks today and the new capabilities in Titan. Do you have more information online?

Yes! For more information on the Titan release, you can check out our website which includes visuals for the core upgrade features and new Modules: https://www.palantir.com/palantir-gotham/titan/

Additionally, you can contact Gotham-Titan-Requests@palantir.com for any questions or requests related to the release. And as always, we encourage our partners to engage with Palantir engineers in the field so that Gotham innovations can be as transformative as possible.

Track the most disruptive technologies by diving into our categorized index:

Artificial Intelligence Companies – A fast overview of Artificial Intelligence companies we believe are poised to cause the most positive disruption in the enterprise.

Big Data Companies – Reference to the greatest, most disruptive Big Data companies in the tech ecosystem.

Business Intelligence Companies – We assess these to be the Business Intelligence Companies most impactful for delivering real decision advantage.

Cybersecurity Companies – We apply our deep expertise in cybersecurity to assessing the best across multiple categories including:

Cloud Computing Companies – We include both platform and software as a service providers, capturing only the most innovative and disruptive.

Collaborative Tool Companies – These are the firms that help humans connect to humans to create, manage and lead.

Infrastructure Companies – Critical enterprise foundations for business agility.

IoT Companies – Internet of Things and Industrial Internet of Things are here. How do you manage them?

Mobile Companies – Help manage, configure, secure and optimize these very powerful capabilities.

Robotics Companies – Including innovations in Robotic Process Automation, Drones, and industrial robotics.

Services Companies – We only track a few, the ones we really know well.

Tech Titans – These are the big players. We track the tech titans closely since their capabilities change continuously.

VC, PE and Finance Companies – Keeping an eye on the investors can give indications of coming developments.

You can also use our topical pages to get up to speed quickly on the current status of the major megatrends. See our pages on Cloud Computing, Artificial Intelligence, Mobility, Big Data, Robotics, Internet of Things, Cybersecurity and Blockchain and Cryptocurrencies.

We also provide special pages focused on high interest topics, including Science Fiction, Entertainment, Cyber War, Tech Careers, Training and Education and Tech Tips.

CTO Guide to the Business of Artificial Intelligence

Bob Gourley June 3, 2019

artificial-intelligence-head This is the CTOvision guide on the megatrend of Artificial Intelligence. This report gives a high level overview of the most important factors of the trend, gives updated insights into the activities of the major AI companies, and succinct descriptions of AI tech. It also points to key CTOvision reporting to help readers dive deeper into the topic.

We start with a definition of AI:

Artificial Intelligence (AI) is the application of thinking machines to real world problems.

This definition is unique. We like it because it focused on practitioners. Once you take a practitioners view you see AI is really about far more than algorithms. There are a wide range of technical and non-technical factors that must come together to deliver results.

Examples of tech components and non-tech components of real world AI solutions follow:

Tech Components:

Analytic Algorithms (including Machine Learning, Deep Learning)
Natural Language Processing
Robotics
Computer Vision
Data Management
Sensors
Hardware architectures
Technical security measures

Non Tech Components:

New business strategies
Cybersecurity policies
Business risk policies
Ethics
Legal and regulatory regimes
Training and testing
Operation and maintenance
Hiring, promotion, career management

Business Impact of AI:

Here are the key points we recommend any enterprise tech professional consider regarding AI:

With business models returning profit now, all indications are AI will continue to improve.
The evolution of AI has been accelerating due to its coupling with incredibly low cost cloud computing.
Creators use a “generate and test” approach to creating functionality, with no accepted protocol for security or testing in AI. This is a huge negative.
There are four major problems with AI today: 1) Some of the most capable AI is not scrutable (you can’t see how it works), 2) AI can be easy to deceive, trick or hack, 3) AI can be unfair, unethical and unwanted, and 4) AI can be leveraged by competitors and even criminals to your detriment.
There are ways to balance the risks and opportunities around AI.
AI, especially Machine Learning, is playing a huge role in modernizing the cybersecurity industry.
AI is also being used by cyber criminals, with many in the security community predicting AI enabled malware coming soon.
AI can be easier to deceive than current computer software (see Generative Adversary Networks: A very exciting development in Artificial Intelligence).
There are many lessons that can be learned from others on ways to improve your corporate governance over AI including ethics around AI.
More on optimizing AI for business can be found at OODAloop.com

Open AI questions decision-makers should track:

Will job displacement caused by AI be a crisis? Will government put regulations on companies because of this?
Will companies use AI in ways their customers regard as ethical?
Will there ever be a widely-accepted security framework for AI?
Can behavioral analytics enhance security?
How can machine learning improve cybersecurity?

The field is growing dramatically with the proliferation of high powered computers into homes and businesses and especially with the growing power of smartphones and other mobile devices. AI requires lots of data to be effective and with the proliferation of mobile devices there is more data now than ever.

Due Diligence Assessments and Artificial Intelligence

The trend of Artificial Intelligence is an increasingly important element of corporate Due Diligence since it is so disruptive business models.

On the sell side: Firms should ensure their use of AI is done securely and ethically (see our special report at OODAloop.com on “When AI goes Wrong” for insight into issues and mitigation strategies). This applies to any firm that uses any AI enabled capability. However, firms that produce AI (vendors) should pay particular attention to this, it will make a big difference in how well a firm will be valued.
On the buy side: Buyers should pay particular attention to the use of AI in the target to ensure a well thought out architecture that mitigates risks. External and independent verification and validation of AI ethics and security policies and practices are key, as well as the degree that the target is complying with appropriate compliance regimes.

Strategically, the acquisition of technology firms is an art requiring assessment of how unique the capability is and how much in demand it will be in the market. We provide a special focus on due diligence for artificial intelligence companies via our parent company, OODA LLC.

The Technologies of Artificial Intelligence

There are many key technologies used in fielding AI. These are the components of AI technologies we recommend tracking:

Machine Learning: Machine Learning is a subset of AI that focused on giving computers the ability to learn without being explicitly programmed. Machine Learning involves the automated training and fitting models to data. ML is the most widely used AI related technology and is frequently the front end of more complex solutions. This is a broad technique with many methods. Methods commonly taught and applied in ML solutions all have different strengths and weaknesses and part of the art of ML is knowing which applies to the need at hand.

Neural Networks: Considered a more complex form of Machine Learning, this approach uses data flow mappings similar to artificial “neurons” to weigh inputs and related them to outputs. This approach views problems in terms of inputs, outputs and variables that associated inputs with outputs.

Deep Learning: Highly evolved neural networks with many layers of variables and features. Important to most modern image and voice recognitions and for extracting meaning from text. Deep learning models use a technique called “back propagation“ to optimize the models that predict or classify outputs, which adds to complexity of the end model. The end model may have so many 1000’s of variables that no human can really understand how the model functions or how a conclusion was arrived at.

Natural Language processing: This class of technology analyzes and understands human speech and text. Used in modern applications of speech recognition including chatbots and intelligent agents. NLP also requires training data, in this case the output is knowledge about how language relates, often referred to as a “knowledge graph” for a particular domain.

Rule-based expert systems: This is an older approach to AI solutions. It involves establishing sets of logical rules derived from the way people actually work. Used in many processes where sets can be clearly defined. This was the dominant form of AI in the past and is still around today, but is really just complex programming. Imagine a large number of “if-then” statements in a program, but in this case the rules were built by domain experts.

Robots and Robotics: This is the automation of physical tasks. Primarily used in factory and warehouse tasks but growing use in heath care, small businesses and homes. Training data for robots is critically important, but in this case the training data may include location for movement or a wide variety of expected changes in the environment.

Robotic Process Automation: This is the automation of structured digital tasks in the enterprise or factory. This is a highly evolved form of scripting actions. It is a combination of software and workflows built to help automate business processes. RPA is at its best when it provides users with the benefits of other AI capabilities like Machine Learning.

Other Related AI Terms and Concepts:

Supervised Learning: The most common type of training for AI models. Data is labeled by humans so the algorithm can be taught based on what was established by humans. This is very similar to older techniques of statistics like regression analysis. Once a model has been developed using supervised learning, it can be used with new data to provide predictions. This is called “scoring”. Training models on labeled data generally takes large quantities of data that have known outcomes, and in many use cases the outcome that is being sought is actually a rare occurrence (this is called a “class imbalance”).

Unsupervised Learning:This is the development of AI models in ways that detect patterns in data that are not labeled and results are not known.

Training Data:The data used for the development of the model. This is often validated using another subset of data for which the outcome to be predicted is known.

The methods and concepts above are almost always combined in any real world AI solution.

The AI Vendor Community

Most AI capabilities today have their roots in academia, but real implementations are being driven by the corporate world. There are many reasons for this. One is the profit motive. Another is the large collections of data available to the big firms. We provide more focused reporting on the firms driving AI forward in our Disruptive IT Directory in the categories of Tech Titans and Artificial Intelligence Companies. It is especially important to track the AI developments of Google, Microsoft, IBM, Amazon, Apple,

You can see our reference to Truly Useful AI You Can Use Right Today.

For Further Study

Artificial intelligence software is assisting people in most every discipline. The many functions of AI are considered to be threatening jobs across multiple industries, but others consider it a great producer of jobs since it will help create entirely new industries and free more humans to innovate and create. This impact on jobs is best considered in conjunction with the megatrend of Robotics, since together those two trends are going to impact some of the largest sectors of jobs in the U.S. (consider, for example, the impact on retail and shipping).

For alerts on future posts on this topic see CTOvision Newsletters.

Some of the AI companies we are tracking include:

[wp_show_posts id=”108847″]

For more on these topics see:

There are seven key megatrends driving the future of enterprise IT. You can remember them all with the mnemonic acronym CAMBRIC, which stands for Cloud Computing, Artificial Intelligence, Mobility, Big Data, Robotics, Internet of Things, CyberSecurity.

Insights for your enterprise approach to AI and ML ethics: Advice for the C-Suite

Bob Gourley April 9, 2019

This post is the third in a series of three based on interviews of Courtney Bowman and Anthony Bak of Palantir. The first post laid a foundation which can help any organization seeking to establish an ethics policy and program around AI and ML. The second post picked up from there and dove into the role of humans in AI systems for the enterprise. This third post dives deeper into actionable advice.

Gourley: What advice would you have for the C-Suite executive in an enterprise thinking through their approach to AI?

Bowman: Like Anthony just mentioned, testing is key, but for any enterprise anywhere I would also say be sure to keep humans in the loop in any system. And help those people understand and appreciate the potential implications of the systems they work with as they relate to civil liberties and privacy, compliance, security, etc. Your developers will help ensure your systems work as well as can be designed, but human oversight is key.

Bak: This is going to sound strange, but the first piece of advice I have is to not do AI at all. What I mean by that is that when approaching a problem, AI is often the last area where you should spend capital (in terms of both personnel and technical resources). First you need to have your data story in order – this is frankly where you need to spend most of your time and you’ll get the most ROI on your efforts and this is also where we spend most of our product energy.

It’s worth thinking ahead here about how you want to use the data. One common way to integrate ML into a product is rather than automatically having the ML make a decision, we make use ML to makea recommendation to a subject matter expert and let them decide if they want to use the recommendation or not. When you start trying to improve the decision process – and this can be either through improving the ML recommendation or other aspects such as SME education, user interface design, policies etc. you need to be able to reconstruct what the options were at the time the decision was made, what extra context may the SME have used that is was not captured by the workflow – not what options and context are now. Whether you’re taking an algorithmic-driven or more human-centric analytical path, you still want to be able to figure out what is working? where are we making mistakes? how can we improve? Doing this kind of retrospective analytics involves sophisticated data management software so that you know exactly what data was used, when, where, how and by whom. Trying to leverage ML without having these foundations in place is a recipe for failure.

Alongside the data foundation, you are of course building out the operational workflows driving your decision making. Here too, you don’t want to build out the ML as the first step. You need to understand the decision first and make sure the decision is being made with the right informational context for the decision maker.

Bowman: In a similar vain with respect to bias and ethical issues, I would say that in our experience, machine ethics and machine issues with bias are not separate from human ethics or bias. Both are important to address in the organization. In both cases the corporate leadership needs to think through what is right and wrong at the institutional and systemic levels and seek to set the standards and policies that reflect well upon your organization, your employees, and all stakeholders.

Many of the ethical issues exist independently of any particular technological solution. While there is some evidence that AI can make certain issues more severe (e.g., by amplifying existing bias in the data), one of the benefits we see emerge from customers preparing to deploy AI responsibly is that the commitment to rigorously evaluating data bias and imposing algorithmic scrutiny may open their eyes to examining — and remedying — more fundamental structural maladies.

Gourley: Both of you bring and interesting mix to issues of ethics. I wonder if that is serendipitous or if positions like the ones you are in require that type of broad experience/education base. I’m thinking about this from the perspective of an organization that might need to fill positions like yours and also about people early in their careers who may want to think about experiences they should get if they aspire to positions like you two have. Any thoughts on that?

Bowman: To some significant degree, I think my perspective benefits from having both a hard sciences and humanities background. I can appreciate the power of what technology can do, but I also temper those expectations with a richer understanding of the irreducible qualities of life. All of that helps me maintain a balanced perspective that resists the urge to denounce AI/ML as inherently unethical or irremediably flawed, while also carrying an appreciation of when and how to put the technology in its place and let human elements flourish.

Bak: Coming at this topic from a technical perspective I have to constantly remind myself to have humility about the limits of technical solutions. I have a natural inclination to look to technical ways to use AI to solve some of societies biggest problems – and to some degree you could say that is Palantir’s mission – but many of the complications that arise, such as bias in AI models having at their root complicated social phenomena and furthermore and technical solution that I build becomes part of feedback cycles in this much larger system that includes legal, social, political and technical components. So while it’s important to have broad perspective, even more important is humility around ones own area of expertise and how it fits into a larger whole.

Insights for your enterprise approach to AI and ML ethics: The role of humans in AI systems

Bob Gourley April 2, 2019

This post is the second in a series of three based on interviews of Courtney Bowman and Anthony Bak of Palantir. The first post laid a foundation which can help any organization seeking to establish an ethics policy and program around AI and ML. This one picks up from there and dives into the role of humans in AI systems for the enterprise.

Gourley: The approach to ethics issues around AI and ML you introduced us to has a clear respect for the roles of humans in the equation. How do you articulate the role of humans and computers in an enterprise AL system? What it is the humans are adding to the equation?

Bowman: Humans provide the understanding. In part, this involves the deeper semantics that comes from living in the world, having shared lives with other people, being entrenched in linguistic, social, and cultural practices that are often ambiguous and very context-dependent. Understanding allows us to do much more than just tell apart subtly distorted images of school buses from ostriches. It also forms the basis of other, much more delicate, forms of cognitive reasoning, like what’s just, fair, virtuous, or beautiful.

Gourley: That’s really interesting. It’s also kind of complicated. In your view, how do you get an institution to internalize these subtle but important distinctions? In other words, is it possible to reduce ethics and ethical AI to compliance frameworks and policies that can be embodied in process or even checklists?

Bowman: Policies are critical for establishing a baseline for what constitutes acceptable decisions. But I can tell from your question that you understand where policies fall short. You can’t enshrine every nuance of ethics in policy, nor can you rely on process to keep you in the clear. It’s just as critical to create a culture with shared principles, vocabulary, and frameworks for understanding and discussing ethical considerations. For a tech company like Palantir, we recommend starting with principles to ground your technical and business decisions (you can read ours here), and provide clear escalation paths and an open environment to ensure that developers see and raise flags when something doesn’t align with those principles. But being able to identify and raise those flags requires more than just operational policies. It’s a matter of culture. Building a culture of ethical engineering is like learning to speak a language: you have to practice using the language all the time, and you improve through repetition and lived experience. You need training and workshops, and you need to engage in transparent dialogue about the decisions you make and how they align with your principles.

Bak: So, I think what Courtney’s trying to say is, no, ethical engineering is not a compliance task. You don’t get to morally defensible technology applications just by ticking a lot of boxes because those boxes alone won’t always instruct you what to do when a truly novel situation presents itself or when two seemingly countervailing considerations demand a decision one way or another, in the same way the rote memorization of vocabulary won’t instruct you how to articulate a unique thought in a new language you’re studying.

Gourley: I see what you’re saying. But even skilled language instructors employ tools to help students learn. How do you help instruct computer scientists to become grounded in a culture of ethical engineering?

Bowman: One thing you do is focus design practices on methodologies and frameworks for architecting systems that have a vision of morally good outcomes, or at least certain critical features of those outcomes. What does that mean in practice? Well it means you’re reminding yourself to grapple with potentially difficult questions at each stage of development, asking yourself questions like, “What is this technology intended to do in a defensible way, but also what could it be re-purposed to do in ways that reasonable people would find objectionable?,” “How do I minimize the risk of unintended consequences and harms?,” “How do I keep my technology out of the hands of irresponsible or unsavory users?,” Etc.

Having a framework for asking these sorts of questions at critical stages in AI/ML development is the most important task. But also it is important to give engineers the resources, vocabulary, and sense of responsibility to act and build appropriately.

Gourley: I think I see what you’re suggesting in a broad sense, but can you make this more concrete in terms of frameworks that might be applied in AI/ML applications?

Bak: Sure. Applying AI ethics is multi-layered. The first layer is evaluating whether you should be using AI to solve a problem at all. We work in domains where decisions are “high consequence.” We see poor problem selection all the time. For instance there was a recent paper where researchers were trying to predict criminality using facial recognition software. Now, without knowing more, I could’ve told you at the outset that this was a problematic study because my moral intuition tells me that there’s no such thing as a criminal face (I also hear my mother’s voice in my head reminding me, “never judge a book by its cover!”).

As it turns out the flaws in this study were simpler than you might even suspect. In the images used to train the model, the researchers used mugshots for the criminals and actor “headshots” for the non-criminals. The software was able to be highly accurate just due to picking up on queues such as “is the person smiling.” Of course, not many people are at their best in their mugshots and are therefore not inclined to smile. Nevertheless, I would argue this isn’t a problem worth solving in the first place, even as an academic exercise, and no matter how you try to control for training data bias (say, by using glamour shots of convicted felons and/or having actors stage mugshots), the dataset is still going to lead to reinforcement of one kind of entrenched systemic bias or another – be it race, ethnicity, or something else.

So if you’re dealing with a problem type where the cost of making false predictions (either True or False) is high and the nature of the data is such that a certain level of false predictions are inevitable (for systemic or other reasons), AI/ML is not the approach you should be taking. We call this “problem selection” and it’s a threshold consideration.

Gourley: Okay, but let’s assume you’ve selected a problem that meets your threshold criteria. What next?

Bak: Assuming you’ve found a legitimate AI problem, you then need to ensure that the training data you’re building upon is appropriately representative. This means not just looking at distributions of variables within the data, but considering the system that generated the data and what the implications are for inferences you’re trying to make from the data. The data you use to train a model is the most important factor determining the models performance characteristics – both in terms of overall accuracy, and in terms of fairness or other normative concerns.

This issue of where the data comes from is really subtle. One example is from image recognition where you want to identify objects in images. A fairly common way to get image data is to use images posted on the web and social media as training data for AI – so here the task for the machine is to say what’s in an image – here’s a person, here’s a car, etc. People then try to transfer these models to a robot – so it can identify objects. What you find is that even with similar photo sensors being used, the accuracy of the model drops significantly. This can be traced back to the fact that images posted online are purposely framed for aesthetic reasons, while the robot is more haphazardly scanning the environment – this small difference in generating method can make a big difference in the performance.

Many ethical issues have similar roots in that the generating process for the data is different than the application domain. For example, if you use the same dataset to find people in photographs – you may find, and in fact people do find, that it doesn’t do as well at finding minorities due to them being under-represented in the training data. Google famously released a model that identified many people of African descent as gorillas.

Gourley: It seems like there’s a whole cluster of challenges that arise in working with training datasets.

Bak: Right. As the robot example shows, it’s not just about over/under representation of objects of interest in the data and it’s also not obvious how to “fix” the training data as in the second example (gather more photos with minority representation). This becomes particularly pernicious when you create a feedback loop between the data your model uses and the decisions it enables. Consider, for example, using records of stop-and-frisk encounters to build a model suggesting who to stop and frisk — the data is subject to human biases around race, gender, and class, and when you feed it into a system that doesn’t understand that you end up with a model that says those are the type of people who should be stopped. If people follow the model then new training data is created that further exacerbates the initial problem. Studies have even shown that there’s a tendency for ML to make biases even more pronounced than what might exist in the training data. Understanding what it means for the data to be biased is not just a matter of looking at the data itself – you have to consider the larger system in which it sits – both in terms of the training data used and how a model built on that training data will feed back into the larger system.

Gourley: So then what do you — as a tech company — do to address these AI training data problems?

Bak: For Palantir, one of the questions we ask when faced with these sorts of complicated phenomena is – Are there ways in which we can build technology to help solve these issues? Up front, it’s clear that we won’t be able to completely obviate the need for careful thought, but we do look for ways to help people or nudge people along in their thought processes.

There are two areas where we see technology as helping avoid some of these ethical pitfalls (1) Model testing, validation, and monitoring and (2) data management.

Our goal is to have Palantir provide industry leading monitoring and validation for AI models, this means tracking performance for the model – what it is getting right or wrong as well as how it performs on different subgroups of the data, visualizations of the data being used by the model to understand representation bias and other issues, and where possible aid the end-user in interpreting how the model is making decisions by examining the relative importance of different characteristics for a data point or even other more sophisticated methods. Part of this involves good, out-of-the-box defaults so that, at a minimum, you are encouraged to track a wide range of performance issues and metrics. But we also understand that that alone will not be sufficient because each model application must be contextually tuned. So we provide flexible, configurable tools to make it as easy as possible to evaluate ML bias – either coming from the data used to train the model or in model outcomes.

For data management our system is built around being able to easily and transparently track where data comes from, who and how it’s been used, including being used as training, testing, or live data for AI models, as well as any decisions made using the model.

Model tracking and data management provide entry points to start to discover and address bias issues. We understand that by themselves they are necessary but not sufficient for addressing all ethical issues.

Gourley: Does Palantir put in place other capabilities to make that easier?

Bak: We make it easier both on our developers creating solutions and customers operating them by ensuring methods for algorithmic explainability and interpretability. The same is true for more standard data protection features and capabilities, like granular access control, data provenance and lineage tracking, data retention and deletion management, and audit logging.

Bowman: Notice that what Anthony’s describing are practices and implementations of AI support tools that are intended to enforce both good data science hygiene and morally defensible outcomes. I can’t stress this enough: It’s our belief that most of the time, there isn’t really a trade-off to be made between either doing high quality AI/ML or doing ethical AI/ML. In other words, it turns out the methods and practices of of qualitatively good AI/ML tend to be wholly consistent with the demands of ethical AI/ML.

Gourley: That sounds like a principle. Is that what you mean when you say you established principles?

Bowman: Exactly. It’s one of our guiding precepts in working on AI and ML.

Gourley: What would you say are some of your other principles?

Bak: Whether a solution has fully automated decision-making or machine assisted decision-making, we humans have to think through the inherent limitations of the solution. AI systems are frequently encoding human behavior but without the understanding that an actual human provides. Our default position and guiding principle here is, generally speaking and especially when the stakes are high, a human should be in the loop in the systems we field, exactly so that their understanding is retained. A further corollary is that the higher consequence the system has, the more we should be rigorous around evaluating if AI is appropriate.

Bowman: Another related principle is that, in addition to carefully think through the inherent limitations of the solution, we also have a responsibility to evaluate how the solution could be re-purposed or extended in abusive ways. Potential application of AI must include not just an assessment of what the technology can and cannot do, but also (and perhaps more importantly), what it should and should not be permitted to do.

The third post in our series based on the interview with Bowman and Bak will bring more focus to advice for the C-Suite.

An interview with Robert Fink, Architect of Foundry, Palantir’s open data platform Part Three: Open Development Environments

Bob Gourley October 1, 2018

Editor’s note: This post is the third of a series of three capturing the result of recent interview/discussions I had with Robert Fink of Palantir. The conversation was wide ranging, hitting on topics of design, development environments and a bit on the philosophy of enterprise tech. Several common themes emerged in those topic areas, including ways that Palantir has been leveraging open approaches to data architectures, system design and even developer environments. The first post focused on open data architectures. The second post continues the discussion of open approaches including open source software. This third post gets into the meat of modern, open development environments .– bg

BG: The third broad topic I want to ask you about is open-source for development environments. How does this relate to open-source software and open data architectures?

RF: Once again these are in theory separate concerns, but related by mindset. Hypothetically, you could imagine a development environment based on closed, proprietary tools that produces an open data system, but in reality I have never seen that done. The most powerful and flexible development environments today are based on collections of open-source tools that have been designed from the ground up to produce highly functional, well documented, sharable code. Modern languages like Typescript are entirely open-source, including the compiler, the build system, and the IDE. Contrast that to the early days of C++, where the best compilers and IDEs were closed-source and very, very expensive.

Palantir’s developers are addicted to a large amount of open-source developer tooling. For example, all of our Java developers build their projects with Gradle, and all of our frontend developers code in Typescript. We have made a number of our internal productivity tools available as open-source. For Gradle, examples include plugins for coding standards, container-based integration testing, or for building Docker images. For the frontend community, we maintain the standard linter for Typescript.

BG: On one hand, Palantir is a commercial software company, on the other hand, you give away some of your code as open-source. How do you balance these concerns, how do you decide which part of your software you publish?

RF: In the case of developer-tooling the answer is straightforward: I feel morally obliged to maintain our tools in the open-source domain because it incurs no overhead for us and because we would gain no competitive advantage from keeping it closed-source. A second key advantage of open-source tooling: it makes it much easier for our customers to interact programmatically with our open data platform.

BG: Can you give me an example of that?

RF: Since our data platform is focused on serving customer needs for data integration and analysis, we support standard formats and APIs with built-in interoperability, for example SQL, Hadoop, Parquet, as well as downstream analysis applications like Tableau or PowerBI. But we also provide ways for customers to extend the platform with bespoke functionality in the form of plugins and applications. We try to make this easy by providing building blocks for application development. In many cases we have turned these building blocks into open-source projects themselves so anyone can contribute to and use these capabilities. For example, Blueprint and Plottable are Javascript libraries that both we and our customers employ to develop rich Web-based frontend applications with consistent look and feel. We are in the process of open-sourcing our RPC framework, Conjure. Conjure is built on industry standards like HTTP and JSON and makes is very easy to interact with our platform APIs from a variety of programming languages.

BG: I noticed from your GitHub page that you remain personally involved in the open-source community. Can you tell us some of your favorite Palantir projects?

RF: On the top of my list is probably AtlasDB. AtlasDB is a distributed transactional key-value store built on top of Cassandra. Inspired by research papers on MVCC databases, we began internal development of AtlasDB around 2012 and decided to contribute the codebase to the open-source community in 2014. Today, AtlasDB is the database backend for all of our products. Specifically for Palantir Gotham, AtlasDB has enabled our customers to migrate from Oracle to open-source databases.

As a second example, we partnered with Pepperdata, Red Hat, Bloomberg, and Google to design and implement a Kubernetes backend for Spark. After a number of iterations, the main components developed as part of this collaboration were merged into mainline Spark. Both Spark and Kubernetes are industry-changing software ecosystems and I am really happy to see that their “marriage” is now available to the wider Spark community. Palantir continues to be an active participant in the vision and implementation of the project.

BG: Can you elaborate a little on legal concerns with open-source software, and maybe also on security concerns?

RF: Both are non-trivial subjects, in particular when organizations or individuals “go rogue” and blindly download and deploy open-source components. A big advantage of commercially supported open-source software is that the providers take care of the issues of indemnification and compliance which organizations would have otherwise have to solve themselves.

Open-source components, including code libraries, graphics and fonts, and entire applications are subject to a license grant. Different license grants impose different constraints on how the licensed component and the resulting software can be used. For example, the MIT license allows for permissive use of the covered components, including modifications and redistribution under commercial terms, whereas the GPL license is far more restrictive in how one can package and distribute a combination of the open-source component and proprietary code. We monitor our external dependencies to ensure that our use of open-source components is compatible with their licenses.

The same dependency monitoring is useful in the security context. Like any software, open-source software can be susceptible to malicious and unintentional bugs and security problems. Fortunately, the developer community and security experts around the world continually scan source code for vulnerabilities and maintain lists of known “bad” libraries. We use our dependency monitoring tools to identify and remove such code from our systems. A small amount of common sense also helps in this context: we try to avoid dependencies on code produced by small or obscure development teams, as well as code that is no longer actively maintained.

BG: Thanks, Robert!

RF: My pleasure!

This concludes our series based on our interviews of Robert Fink. Find all three of the series at: