Reflexive Consistency About The Right Thing

 Today at work I proposed a sampling scheme for our ML training. Our data is highly imbalanced and more so because we don't know ground truth on most of the examples which are (roughly) at least 2% likely to be the minority class. Our architecture does not allow us to use all the data in training, so we downsample the target=0 examples and to a lesser extent the no-ground-truth examples, giving them weights inversely proportional to their sample rate.

The proposal: What if we sample our data in proportion to its probability of target=1? And then calibrate to get probabilities from the new model, then resample and train, calibrate, repeat? That would definitely converge.

The rebuttal: "I agree that [it] should get convergence, but can you explain why you think it would converge to the correct result?"

No, no I cannot, and I'm pretty sure it would not. I didn't suggest re-labeling, just adjusting the sampling scheme. Why would I expect that sending the training algorithm a huge variety of the most extreme examples would result in good discrimination? Shouldn't I rather send over a good variety of every type of example?

My modified proposal: Okay, right, this is about the sampling scheme. That mostly affects the target=0 examples, and to some extent the no-ground-truth examples. Maybe the actual win here is in sampling approximately equal amounts from each "type" of example, and maybe we can approximate that by binning by equal-in-some-space score bands and sampling approximately equal amounts from each. And then, sure, if that works well, we can make it reflexively consistent and try that experiment too.

What mistake did I make? I started by thinking about reflexive consistency. I thought to make the scores output by the system consistent with the scores input to the system. But that sounded scary (still maybe it's a good idea!), so I backed off to "hey, we downsample low scores quite a lot, [cue key mistake] so since we downsample high scores less than low maybe we should rescore so that we know which examples are now thought of as high score and can be downsampled less!" This conflated sample rate with score; we weren't actually downsampling low scores, we were downsampling overrepresented examples, and as it happens, the overrepresented examples mostly have low scores. So the correct observation should have been:

"We downsample overrepresented examples for performance reasons. Let's try (a) being more deliberate about checking for overrepresentation even within each class of example, for example we know we end up with tons of really low scoring examples, and (b) if that works, then our overrepresentation checks use the output of past models as an input, so iterate on new models until we hit an approximate fixed point such that we're using the most up-to-date guess about scores, i.e. from the new model, to get good information to the new model's training."

Comments

Post a Comment

Popular posts from this blog

"Liberty" Tasks

A "Liberty task" is a small, self-contained task, ready to pick up and just start. No dependencies or blockers. It must be useful rather than make-work, and very often it is one of many similar tasks. It should not require deep thought; it is a thing which can be done, not a thing which surely could be done after figuring out how. In a codebase this might look like a list of tech debt noted over the years, or like the dozen refactorings needed before an unwieldy component can be broken up into three clean components. At home this might look like the ten yearly maintenance jobs needed to avoid early breakage, or like writing out the ingredients needed for each recipe you intend to try soon so that a legitimate shopping list can be created.  Some years ago one of the cofounders of the company to which I sell my labor found out he had a brain tumor. He took a step back from being the CTO while he dealt with the immediate mental and physical issues, including brain surgery, but d...
Read more

Make A Bad Plan

One of my more lucrative techniques at work I call Make A Bad Plan. This is where you know  that something  has to happen, but no one really has a good idea of exactly what or exactly how to do it, and also it's a lot easier to go work on less ambiguous projects, so how do you make it happen anyway, when no one, including you, really knows what "it" is? Two common ways this comes up are when you can see that there's an important problem in a system even if you don't know how to solve it, and when you can see that there's an important piece of information that humans know that isn't baked into automated systems and if only it was then something would be better, who knows what, but clearly humans know and act on this kind of thing so we should be able to use it, right? Make A Bad Plan calls for you to make a bad plan. Think of a few " something "s, pick one or synthesize one from your (bad) options, write it up as a draft proposal with enough detai...
Read more

Creating an extra character

For Halloween, I'm running a light horror-ish RPG for some friends. We play other RPGs regularly. I had a great little game in mind called ViewScream, designed for online play. Everyone plays a character isolated on a spaceship that has Problems (TM). Each character has two life-threatening emergencies which must be solved lest they die in the end. Each character also has three technobabble solutions to help solve others' emergencies. These may or may not work; the player knows, but the other players don't know. Everyone takes turns dialoguing, telling about their emergencies, offering up solutions, seeing how they worked (or didn't). But there are 6 of us, and I didn't have a prebuilt scenario that went past 5. Characters are pretty minimal. They have a tiny personality description and a little bit of relationship info, a couple of barebones sentences explaining how they feel or think about another character. In addition they have one secret or twist or scripted mi...
Read more