External Views on Internal Truth

The way to maintain a truth-seeking culture on your data science team while respecting the constraints of sales and marketing and presenting to the board and simplicity and etc is to separate messy, inexplicable truth-oriented analysis from useful views on top of the truth.

This is an example of a more general heuristic: any time you're optimizing for "two objectives at once", take a step back and try to figure out how you can optimize for just one objective. A common solution is to split The Thing into multiple parts, each of which can optimize for just one objective, and then combine the results. That's what we'll do here. (Another common solution is to combine your objectives via some sort of parameter, expose the parameter as a "business" lever, refactor until the parameter you're exposing is a good business lever, then optimize for the explicit combination of objectives.)

Suppose for example you're a ~Kiva making micro-loans to under-served individuals. You've got a pretty great repayment rate, like 95%. But also you reject a number of possible loans. What can you say about the rejected population?

The most blatant problem is if the external-facing repayment rate of 95% is taken as The Important Metric internally, and then you completely ignore the rejected population, and talk internally as if 95% is a fundamental number rather than one point on a tradeoff curve. Maybe you reject 10% of possible loans, and actually 90% of them would have repaid, so you could just stop rejecting and still have a 94.5% external-facing repayment rate. But maybe you're really good at discrimination and actually only 20% of them would have repaid and you'd have a 87.5% external-facing repayment rate. Internally, it's super-important to figure out which is true. Externally, maybe you straight-up can't talk about this, because it's too fraught with PR peril. Don't let the PR peril dictate what you study internally.

But maybe you do need to talk about it. Maybe you need to say "we estimate that only 10% of those we reject would actually have been fruitful partners" or something. I'm not good at PR. Basically, external-facing false-positives. Here it becomes crucial to do good internal analysis and then sanitize it for external consumption. You probably don't actually simply have an accept/reject binary internally. Maybe you have bins, like "absolutely loan", "probably loan", "on the fence", "pretty bad", and "definitely do not". And what you really do is cherry pick the "pretty bad", accept all the better, and reject the "definitely do not", except also you accept 5% of what you think you should reject to get an idea of ground truth. Internally you want to know the actual counts of repayment in each bucket, your estimates and confidence intervals of extrapolating ground truth data to unseen in various ways, etc. Make sure this happens, even if the external formula has to be literally just "# of that 5% which repayed / total # in that 5%". That's just one piece of interesting data internally. Keep digging. Stratify and recombine, smooth small data, etc etc. But don't just say "this is what we present externally, so this is the number we will inscribe into our intuitions internally".

Comments

Post a Comment

Popular posts from this blog

"Liberty" Tasks

A "Liberty task" is a small, self-contained task, ready to pick up and just start. No dependencies or blockers. It must be useful rather than make-work, and very often it is one of many similar tasks. It should not require deep thought; it is a thing which can be done, not a thing which surely could be done after figuring out how. In a codebase this might look like a list of tech debt noted over the years, or like the dozen refactorings needed before an unwieldy component can be broken up into three clean components. At home this might look like the ten yearly maintenance jobs needed to avoid early breakage, or like writing out the ingredients needed for each recipe you intend to try soon so that a legitimate shopping list can be created.  Some years ago one of the cofounders of the company to which I sell my labor found out he had a brain tumor. He took a step back from being the CTO while he dealt with the immediate mental and physical issues, including brain surgery, but d...
Read more

Make A Bad Plan

One of my more lucrative techniques at work I call Make A Bad Plan. This is where you know  that something  has to happen, but no one really has a good idea of exactly what or exactly how to do it, and also it's a lot easier to go work on less ambiguous projects, so how do you make it happen anyway, when no one, including you, really knows what "it" is? Two common ways this comes up are when you can see that there's an important problem in a system even if you don't know how to solve it, and when you can see that there's an important piece of information that humans know that isn't baked into automated systems and if only it was then something would be better, who knows what, but clearly humans know and act on this kind of thing so we should be able to use it, right? Make A Bad Plan calls for you to make a bad plan. Think of a few " something "s, pick one or synthesize one from your (bad) options, write it up as a draft proposal with enough detai...
Read more

Creating an extra character

For Halloween, I'm running a light horror-ish RPG for some friends. We play other RPGs regularly. I had a great little game in mind called ViewScream, designed for online play. Everyone plays a character isolated on a spaceship that has Problems (TM). Each character has two life-threatening emergencies which must be solved lest they die in the end. Each character also has three technobabble solutions to help solve others' emergencies. These may or may not work; the player knows, but the other players don't know. Everyone takes turns dialoguing, telling about their emergencies, offering up solutions, seeing how they worked (or didn't). But there are 6 of us, and I didn't have a prebuilt scenario that went past 5. Characters are pretty minimal. They have a tiny personality description and a little bit of relationship info, a couple of barebones sentences explaining how they feel or think about another character. In addition they have one secret or twist or scripted mi...
Read more