Vinod Khosla once said it takes seven years and $30m to train a venture capitalist (VC).Failure may be the best teacher, but failure in early-stage investing comes at a high cost. When I started my career in private equity I knew I would have my share of experiential education through failed investments, but I also believed I could shorten my learning curve and, by finding a way to make smarter decisions, possibly limit the tuition costs of becoming a venture investor.
The framework I adopted to improve my investment judgment was decision analysis, a rigorous and sophisticated set of tools that have been adopted as best practice
in industries analogous to venture, such as pharmaceutical research and development and upstream oil and gas exploration. All three industries require large amounts of initial capital, face significant uncertainty, and achieve success, if ever, years after the original investment. I found the similarities compelling, and with my academic background
in decision analysis, including a PhD from Stanford, and 20 years of experience applying it to a wide range of problems and industries, I was well prepared to develop a decision analysis framework for venture investing.
This article opens by examining the challenges of venture decision-making and describing how a decision analytic approach can overcome these challenges to produce more informed decisions. The main discussion illustrates how we at Ulu Ventures, where I am a co-founder and partner, applied this approach to an early-stage investment in Inkling, an interactive textbook platform for the iPad. I conclude with a consideration of the implications of decision analysis for the venture industry.
The inherent difficulty in venture decisions
Venture is a tough business that requires meaningful commitments of resources in the face of extreme uncertainty and dynamic markets. Most start-ups involve some combination of unproven technologies, inexperienced teams, undeveloped markets and untested business models. Information tends to be incomplete and unreliable. The most promising markets tend to change quickly and domain expertise becomes obsolete just as quickly.
To make matters worse, venture is a terrible learning environment. The average venture capitalist makes only a few decisions a year, and it takes years to see if these decisions result in good or bad outcomes. Few data points and long feedback loops make for slow learning, while the extreme uncertainty makes it difficult to learn the right lessons.
In this business, it is too easy to confuse luck with skill.
Cognitive psychologists Daniel Kahneman and Amos Tversky have shown that humans are prone to thinking errors – cognitive biases – when assessing uncertain events or making decisions in the face of uncertainty.
Most people rely on heuristics to simplify the complex task of assessing probabilities and forecasting value when faced with uncertainties, but while "in general, these heuristics are quite useful … sometimes they lead to severe and systematic errors."
Relying solely on heuristic analysis creates a number of potential dangers for VCs and the companies they work with. For example, representativeness is a pattern-matching
bias in which a person judges risk based on how closely one situation resembles another, while not being sensitive enough to relevant statistical facts – such
facts could include the underlying base rates, the validity of results from small samples, the likelihood of "black swans",4 and other limits to predictive accuracy.
When a VC falls in love with a start-up, it is susceptible to confirmation bias – a tendency to search for or interpret information in a way that confirms one’s preconceptions.
To combat this bias some venture firms assign a devil’s advocate or require a unanimous partner vote for an investment.
Anchoring is the tendency to rely too heavily on one factor or piece of information when making decisions. Anchoring often comes disguised as hard-won experience, and painful experiences tend to create strong anchors. For example, if a promising portfolio company fails because of the chief executive’s (CEO’s) technical failings, I am likely
to be extra sensitive to any CEO’s technical expertise in the future, regardless of its importance for that particular venture.
Thinking you are smart – a common trait among VCs I know – deepens the effects of these biases. According to Chris Argyris, a learning expert from Harvard, smart people tend to make the worst learners.
When smart people think they have figured something out – for example, successful start-up CEOs must have deep technical expertise – they become closed to other points of view. As one becomes immune to contradictory evidence, one’s bias becomes self-sealing and learning slows. Being aware of these potential decision traps, I knew I wanted a more reliable system than just heuristic analysis.
Decision analysis as best practice
Decision analysis is a framework – principles, processes and tools – for making high-quality decisions that marries the art and science of decision-making through a disciplined
process. The definition I find most evocative is "high-quality conversation leading to clarity of action".
Decision analysis was developed at the Stanford School of Engineering and the Harvard Business School. Because of the academic orientation and historic focus on quantitative analysis, users of decision analysis have typically been leading companies in large industries, such as pharmaceutical, oil and gas, utilities, automotive, and, to a lesser extent, financial services. These organisations have generally used decision analysis to make resource allocation decisions, capital decisions, portfolio decisions – any decision that could be mathematically modelled from a quantifiable set of values.
After finishing my PhD in decision analysis at Stanford, I spent 14 years building companies that delivered decision analysis software, training and consulting to these large companies. For the past three years I have been applying these same principles at Ulu Ventures, where we have created – and are continuously modifying and improving – an investment framework that has informed dozens of investment decisions.
This framework does not ignore intuition, pattern matching or judgment – these are the foundations on which venture decisions are made. Decision analysis can and does,
however, provide a structure that guides our investment intuition, much as navigation instrumentation in a plane helps pilots find an airport in darkness or bad weather.
Done well, this framework simplifies the complex problem of early-stage investing, focusing attention on what matters most and allowing investors to ignore immaterial factors with confidence. The magic is in the simplification. Every early-stage investment has a myriad considerations, risks and dynamics to keep straight – but only a small number of them matter. The best investors have a knack for intuitively identifying those important few. However, as returns over the past decade have demonstrated, this knack does not always adjust to changes in the market. At Ulu, we think our framework of instruments can help.
Decision analysis applied: Inkling Miriam Rivera, my partner at Ulu Ventures, and I first met Inkling in October 2009 when it was raising a seed round to build an interactive textbook platform for the iPad. While this seems like an obvious idea now, at the time the iPad was still just a rumour. This section presents Inkling’s journey through Ulu Ventures’ investment decision-making process, highlighting the role of decision analysis along the way.
Ulu Ventures investment criteria
We look at three primary decision criteria for all of Ulu’s investments – a visionary entrepreneur, a home-run market opportunity and attractive risk/reward. The first two are
fairly common and straightforward, but our approach to assessing risk/reward is quite different and is where the decision analysis framework shines. When we applied our first two criteria to Inkling, the
results were encouraging. Matt MacInnis, Inkling’s founder and CEO, had worked in Apple education for many years and told a compelling story about how coming changes in education would be driven by tablet computers. We were impressed by Matt’s deep domain expertise and vision for how Inkling would transform the textbook market using this new platform. We also admired Matt’s integrity – he did not disclose any confidential, non-public information about Apple’s iPad development programme.Visionary entrepreneur – check.
The textbook market is large, broken, and ripe for disruption. Annual new US textbook sales are about $5bn, and used sales are around $2bn.7 It is often called a "broken market" because those choosing the textbooks – faculty – are not the ones buying the textbooks – students – a disconnect that reduces pricing pressure as faculty prefer up-to-date content and are less concerned about costs.
Publishers take advantage of this disconnect by releasing new editions with only modest changes in content or by bundling in other materials, tactics that drive new book sales and minimise cannibalisation from used books.
Even so, long production times mean material in new textbooks is often one or two years old. High prices, dated information and perceived gimmicks have created a large, underserved
base of customers hungry for better options. Home run market opportunity – check.
Matt made a real impression on us during that first meeting and we immediately moved to the next step in our process-evaluating the risk/reward. Our target metric is a 10-times probability-weighted multiple on our investment. Many VCs target a 10-times return, but a probability-weighted 10-times is an entirely different matter, as shown
below.
Risk assessment: early stage: We start our risk assessment by looking at the market, product, team, and financial risks for each company (figure 1). While this is standard operating procedure for venture investors, at Ulu we go a step further by assigning probabilities to each factor, representing the likelihood the entrepreneur will be able to overcome each particular risk. By using probabilities, we discipline ourselves to think clearly and communicate precisely about our intuition.
Market risk: We gave Inkling an 80% chance of achieving market success – namely, finding paying visionary customers and developing compelling reasons for other customers
to buy. This probability was based on our experience with other early-stage start-ups similar to Inkling and our knowledge of the education space. The higher education industry is hungry for innovation, and we knew of a number of schools looking to experiment with new technologies.
Since Inkling needed to find only a few schools willing to try a pilot project or a few professors willing to test an iPad textbook in their class, we believed they had a high but not
certain chance of overcoming the early-stage market risks, hence our 80% assessment (figure 1, row 1).
Product risk: Inkling did not have a working product yet – they did not even have an iPad to program. But they did have a working proof of concept on the iPod, and their
engineers had built successful products on this platform in the past. Like the market area, there was risk, but it seemed manageable: another 80% assessment.
Team risk: Matt MacInnis worked in Apple education for eight years prior to starting Inkling. He had impressive domain expertise, knew many of the important players in
this market and had experience managing teams. On a personal level, he was passionate, driven and a compelling salesman. He seemed ideally suited to lead Inkling
through the early stages – a 95% assessment.
Financial risk: Inkling’s other co-investors were a savvy group from Google with deep pockets and domain expertise. We knew many of them and had confidence they would continue funding Inkling if they were making good progress: another 95 percent assessment (figure 1, row 4).
Overall early-stage risk assessment: For Inkling to achieve early-stage success, it would have to overcome all four of these risks – any one of them could kill the company. Therefore, to calculate the chance of Inkling achieving early-stage success, we take the product of the small risks. In other words, 0.80 x 0.80 x 0.95 x 0.95 = 58%
(figure 1). This is only slightly better than a 50:50 chance.
As this example illustrates, small risks add up quickly in early-stage investing.
Risk assessment: crossing the chasm
Early-stage success is just the first step in a long journey to building a meaningful company. The other life stages, adapted from Geoffrey Moore’s work,10 include "crossing
the chasm" and achieving mass-market success.
For Inkling to cross the chasm, it would have to transition from visionary customers willing to put up with hassles to
pragmatic customers who buy only products that are easy to find and use. Inkling faced an additional obstacle – a pragmatic customer would first have to buy an iPad or a
tablet computer of some sort.
At the time we were considering this investment, the iPad had not launched, so iPad adoption in education was unknown and one of Inkling’s biggest risks. As a result,
we assessed only a 40% probability that Inkling would be able to overcome the market risks required for crossing the chasm (figure 2, row 1).
We also assessed the specific product, team and financial risks inherent in moving from early-stage development to market success, and taking the product of all four risks, we calculated Inkling had a 24% chance of crossing the chasm (figure 2). For many investors, this assessmentwould be the kiss of death. However, this evaluation is only one piece of a larger puzzle. In early-stage investing, large
risks can be outweighed by even larger returns.
Risk assessment: mass market
If Inkling successfully crosses the chasm, it becomes a going concern serving a niche market such as medical schools or California state universities. To achieve mass
market success, Inkling must overcome another set of operational and financial risks (figure 3).
Educational institutions tend to share best practices, so it was plausible that Inkling’s success in one niche market would naturally lead it to opportunities in other niches and eventually the mass market. We assessed a 60% probability of overcoming the market risks. Product, team, and financial challenges to mass-market success also existed – scaling production, scaling a sales organisation,
establishing appropriate revenue sharing relationships – but we felt all were manageable. Taken together, we saw these combined risks translating into a 37% chance Inkling
would be able to overcome the hurdles and achieve success in the broader market for educational textbooks.
Mass market share
If Inkling achieved mass market success, what share of this market could Inkling capture and sustain? In The Gorilla Game, Geoffrey Moore describes a framework for understanding
the competitive dynamics in high-technology markets. Based on this framework, we assessed Inkling’s ability for sustainable differentiation at this level through switching costs, network effects, and relationship and
technology barriers to entry – how these come together as a quantifiable probability is material for another article.
Our conclusion was that if Inkling became a mass market player, it would have a 25% chance of becoming the market leader, a 50% chance of being a challenger and a 25% chance of being an also-ran.
Representing risks in simplified form All these risk assessments from early stage to mass market share can be summarised in a decision tree
Each node in the tree represents the product of all risks for each life stage. The first node shows the 58% chance of early-stage success that we computed, as well as the
corresponding 42% chance of failure. If Inkling achieves early-stage success, the next branch is the 24% versus 76% likelihood we determined for the company’s chances of crossing the chasm, and so on.
The numbers at the right side of the branches show the ultimate probability of each scenario. For example, Inkling has a 1% chance of achieving the topmost branch of market
leadership by being on the topside of each of these branches. The chance of this scenario can be calculated
by multiplying the chance of early-stage success, the chance of crossing the chasm, the chance of mass-market
success and the chance of becoming a market leader – 0.58 x 0.24 x 0.37 x 0.25 = 1%.
The bottom two branches represent no money back to the investors. In this sense, Inkling has an 86% chance of failure. This sounds terrible, of course, but this is the
nature of early-stage investing. It is a high-risk undertaking, but if you get it right, the rewards can be tremendous.
The top four branches show that Inkling has a 14% chance of becoming a going concern and returning some
money to investors. It looked bad for Inkling, but remember – Inkling had a visionary entrepreneur and a largeripe for disruption. The key now was to determine how big an Inkling success could be.
Mapping Inkling’s potential
To assess Inkling’s potential value, we locked ourselves in a room with Matt, tore his business model apart, and then built it back up graphically (figure 5). This decision diagram represents a map of our conversation, showing the key drivers of risk and value in Inkling’s business.
We started by examining Inkling’s target market – higher education. Our goal was to understand Inkling’s total addressable market (TAM) in higher education, as shown by the bubble in the upper left labelled "HiEd TAM."
We identified four key drivers – tablet adoption among higher education students (tablet adoption), average digital textbook unit sales per student with a tablet (unit sales), average retail price per digital textbook (price) and the percentage of this price Inkling would keep after Apple’s cut (net price). We then moved on to the grade school (K-12) market, which Matt thought would require a different business model. Where selling individual textbooks to individual students suited the college market, Matt believed selling a suite of textbooks and charging on a per-student basis would be a better fit for K-12 customers.
Our conversation continued through revenue, earnings, exit multiples, dilution and so on. Every important concern
or potential driver of value had a spot on the map.
On the surface this may look like just another block diagram, but it precisely specifies a quantitative decision model. Each oval represents an uncertainty with an underlying
probability distribution. Each rectangle represents a decision with an underlying set of choices. The eight-sided node represents the value we were using to base
our choices. The arrows represent influences, showing the mathematical relationships between the nodes. Our next step was to quantify all the uncertainties.
Ranges/structure/intuition
The conventional wisdom governing financial projections is "garbage in, garbage out". Inkling’s business case showed a 25% tablet-adoption rate in higher education in five years, but this adoption rate was informed guesswork at best. No single number forecast for adoption of a new technology is credible. Frankly, no single number forecast for anything five years in the future is likely to be credible
even for the most mature businesses. This is why most venture investors have only limited use for models in their decision-making.
Ranges are a more useful approach to forecasting, as they solve the garbage-in-garbage-out problem by recasting the projection question from "Where will we be?" to "How wrong could we be?" In the case of Inkling we were assessing how wrong we could be about tablet adoption in higher education. Matt’s range of possibilities for tablet adoption spread from a low of 10%, to a base of 25%, to a high of 40%. This format of low-base-high captures Matt’s knowledge about tablet adoption, both what he knows and what he does not know. The width of the range represents Matt’s level of uncertainty.
To debias these assessments and ensure assessmentsare consistent across all uncertainties, we use specific techniques such as back casting. Examples of back casting questions are "Suppose we were sitting here five years from now and I show you a report detailing tablet adoption in higher education. What would be a surprisingly low percentage of adoption? What happened?"
As an expert in the field, Matt should be able to generate thoughtful and compelling answers, and his rationale provided an added layer of insight and richness to our understanding of the risks. This same exercise was repeated to capture the high side of the range.
Ranges effectively move the discussion from "precisely wrong" to "approximately right". When done consistently across all uncertainties, we are able systematically to test
our intuition.
Sensitivity analysis focuses due diligence
We used the ranges to test our decision model, producing a sensitivity analysis showing which factors mattered the
most.The horizontal scale at the top is the probability-weighted multiple on investment (PWMOI) – the metric we care most about. Along the left side of the chart we listed all th variables from the model. Along the right side of the chart we listed each variable’s base value from our low-base-high range discussion. For
example, the base case for tablet adoption is 25%.
With each variable set to its base case, the value of the Inkling investment is a 9-times PWMOI, represented by the vertical line running through the middle of the diagram
labelled Base value = 9. Now we asked What if we are wrong about one of our assessments?
Suppose Inkling’s mass market share was only 5%, the low side of our range. Looking at the left end of the mass market share bar, if mass market share is at 5% then the PWMOI multiple drops below 5 times. If on the other hand,
Mass market share is 50%, represented by the right end of the bar, the PWMOI is close to 30 times.
The size of this bar represents how uncertainty about mass market share affects Inkling’s probability-weighted
multiple, holding all other uncertainties constant at their base values.
We then repeated this exercise for each uncertainty, swinging its value from low to high while holding all other uncertainties at their base value. We recorded the swings
in the form of bars and organised them in a Pareto sense from the largest to smallest, as in figure 6.
We were now able to address the garbage-in-garbageout issue. Or, to put it more usefully, we could answer the question If we were wrong about one of these assessments,
does it matter? Sometimes it did, as in the case of mass market share, exit earning multiple and the other top bars shown in figure 6. The value of an Inkling investment
varied significantly based on the outcome of these uncertainties, hence we focused our due diligence on these issues.
More often than not, however, the precise value within a range did not change the attractiveness of an Inklinginvestment. For example, pre-money valuation is at the
bottom of figure 6. We, of course, preferred $3m whileMatt wanted $5m. However, this range did not affect our PWMOI nearly as much as tablet adoption, exit multiples
and the other bars above.
In addition, this sensitivity analysis shows only 12 of the 29 uncertainties in Inkling’s decision model. In other words, Inkling could have been on the low side or the high
side of most of these ranges and the attractiveness of the investment would not materially change.
More precisely, these uncertainties were not material to our decision in that a more accurate assessment would not change our decision to invest or not. There was no
decision value in collecting more information, allowing us confidently to deprioritise these factors in our diligence. In effect, we took what most VCs would consider "garbage"
inputs and through decision analysis transformed them into credible, actionable information.
The bottom line
We now had all the components we needed to build a complete picture of Inkling’s risk and reward. Returning to
our decision tree, we added the values of each scenario from our value analysis in figure 6 to produce a scenario analysis (figure 7). Recall that Inkling had a 1% chance of
achieving the market leader scenario. If Inkling achieved this, it would have $750m in revenue, an enterprise value of $6bn and the MOI would be 420. In other words, for
every $1 we invest, Ulu would get $420 back. However, the important number is this multiple discounted by the chance that it would happen. PWMOI is the product of
the value of the scenario (420) and the chance of achieving that scenario (1%). Therefore this market leader scenario contributes 4.2 to our overall assessment of Inkling’s PWMOI.
We repeated this analysis for each scenario and their sum generated a 9-times PWMOI. While this was short of our 10-times goal, we do not blindly follow the numbers.
The purpose of this exercise is to clarify our thinking. We make decisions based on our judgment, incorporating both quantitative and qualitative considerations. In Inkling’s case the founder was a visionary entrepreneur, the textbook market was large and ripe for disruption, and the risk-return analysis was close to our target. As we analysed Inkling from many different perspectives we found a coherent, compelling story. We invested.
What happened
The iPad is a runaway hit. Apple sold more than 11 million iPads during its September 2011 quarter and more than 40 million units since its launch 18 months ago.19 Hundreds of universities are experimenting with the iPad. Inkling is riding the wave.
Sequoia Capital led Inkling’s Series A. Pearson & McGraw-Hill, the two largest textbook publishers in the world, led Inkling’s Series B. Inkling currently has more
than 80 titles available and partnerships with mostmajor textbook publishers in the US, giving it access to the majority of currently used textbooks. By autumn 2012, it should have hundreds of titles available. Inkling seems to be a home run in the making.
As I look back at this investment two years later, I think we called it about right. We made a smart risk-return tradeoff
and we got lucky. That combination is the heart of most successful venture investments.
Conclusion
Decision analysis helps people make faster, more informed decisions. At Ulu Ventures, it helps us structure judgment and quantify intuition in forms that can be easily
discussed and tested with logic and evidence, driving clarity of thought through a disciplined process. The goal is not "the right answer". The goal is to apply intuition intelligently
to an investment and to guide useful, decision-focused conversations.
Decision analysis helps us learn faster – about the risk and return of individual investments and also about how to practice the craft of venture capital. Inkling’s case is typical
of early-stage investing in that out of the many possible risks and considerations, only a small handful matter.
This framework focuses our attention on what matters most. Anchoring and other biases are reflected in model results that are out of line with our intuition. Our first step is to review the model for errors – if the model checks out, we adjust our intuition.
Our framework also naturally improves over time. Each time we evaluate a company, we refine our framework and add to our set of benchmark data. We believe this sets us on a path toward competence in this industry, more quickly and cost-effectively than the "seven year and $30m plan" Vinod Khosla referenced.
Decision analysis also helps Ulu’s entrepreneurs build their business. Whether we invest or not, we share our analysis with our entrepreneurs. About 20% of the time,
they change their business model or target market based on our analysis. Most of the time, entrepreneurs walk away with a deeper understanding of their business model.
These conversations also help transition us from a diligence role focused on our needs as an investor to a supporting
role focused on the needs of our entrepreneurs.
While intuition and judgment will always underlie successful venture capital investing, I believe venture firms will produce better and more consistent results by using
disciplined frameworks to assist the investment process.
Frameworks act like maps, helping the venture investor see all relevant aspects of a decision in context. Like a well-drawn map, a disciplined framework can help investors
plan where to go next and how to get there, which is particularly important when the terrain is difficult or confusing.
I hope other VCs will adopt decision analysis or a similar framework as part of their investment processes. In addition to improving results, decision frameworks make it
easier for us to work together as an industry in developing and sharing best practices. They codify the art and scienceof capital formation, allowing us to communicate what we
do and how we do it to a wide audience.
Frameworks allow us, as a Society of Kauffman Fellows, to make contributions beyond our immediate circle of concern. They offer the potential for our society and our industry
to reach beyond start-ups to help unlock the creativity and innovation potential of people in other industries and
countries.
This article originally appeared in Volume 3 of the Kauffman Fellows Report, Spring/Summer 2012, www.kauffmanfellows.
org/vision-journal.aspx. Reprinted with permission from the Society of Kauffman Fellows.