Mystery Analysis

So what I see in organisations is there is no proper rigour in evaluating mysteries and hence organisations don’t learn a lot. Suppose the sales goes down in a territory and you talk to salespeople, you will hear various different kinds of storylines and storylines that contradict each other. So what we are going to discuss today is the rigour of trying to do a mystery analysis in a very systematic way. And hence, develop an ability to learn further as an organisation.

what we are going to discuss today, mystery analysis is something that has already happened. And now we are trying to analyse what has happened, and why has it happened? Is that right? Would that be right to say?

Now, the problem is the word when I use the word motivated dealers now, how do you how do you find out so the randomization is very, very important. Yeah. So randomization is very important. So your people should should be randomly assigned to the control group and or the treatment group, right? The work treatment here is where the scheme marketing scheme is being applied. So there has to be equal chances of all the participants to be part of this group versus that group, then you account for all kinds of hidden variables. So the golden rule of all factors remaining the same is kind of applied. That’s the real RCT. Right, okay. So the problem is, see the real problem is in organisations, right, many times, you cannot do this so called randomization, you cannot even run a test, for example, you want to reduce the prices, right? Can you do a pilot that can reduce the prices?

Yeah. Right. So let’s understand if he can’t do RCT, what is a good enough method? Right? And, and thankfully, in social sciences over the last decade or so, people have come out with lot of good enough methods to establish causality from you know, from from observations, so you can’t do RCT for example, this this, this problem, as you can guess, is also there in economics. Right. Yeah, You, you can’t do this kind of changes in our in our country or not improve the rate decrease the rate for half the country? Yes. Yeah. Correct. So many things can’t be done. So, so. So people have invented very interesting methods and Okay, so one of the things that people do and we will  talk about these methods, which is called the interrupted time series method or regression, discontinuity analysis or natural experimentation. These are these are what I call as good enough methods to establish causality in absence of a proper RCT.

We have people who have equal chances of being on the either side, we can say that they are nearly the same. Okay, so what they’re saying our control group is here only. So at the edge between 89 to 90, let’s say there are 50 students, and between 90 to 91. There are 50 students, right? Okay. So let’s take, take the data of the 50 students who are the on the other side of the border, who got the scholarship and the 50 students, or let’s say 60, or students who didn’t get the scholarship and they are equal in all other aspects. you know, those 50 I run the data, after five years, I find it out of 50, you know, 49 pursued higher studies. And in the other case, where people were between 89 to 90, you know, only two of them, let’s say pursued higher studies, then I can say, Okay, guys, that’s a very clever idea of choosing your control groups, I think very, very, very, very, very, very clever idea. And, but that is only used when there is there is this cutoff. So any solution which which says, you know, there is this cutoff beyond that, you know, you get it and you don’t get it. So any anything, any any motivation schemes that you have for salespeople, so, in fact, we did an analysis for a company and prove it to them that the so called, they had something called Super achievers club, and they used to, you know, go out for foreign countries and have good fun and so we prove it to them that it’s not working, right and by by doing a regression discontinuity analysis and telling them because their hypothesis was that, you know, the guys who get it, they get motivated and, and and year on year, the next year, at least they’ll perform well. So we, we ran that and we just to show them that it’s it’s it’s not helping okay. So, so the other one so, so, as I said regression discontinuity can be used only when you have a case which is capable, some kind of a cutoff mark, right, right. But vast majority we don’t implement like this right. So we use the other method, which is called the interrupted time series. Now, what is interrupted time series, it is nothing but the word interrupted time series, right? And it is nothing but trying to find out a before and after.

Yeah, yeah. So the the point here is, how do I how do I establish that, you know, how that whether the, the smoking caused the cancer, so they fought really hard to establish that smoking has no causal link to cancer, you’re just seeing a fake correlation.
you know, when was this conflict got resolved, when was it unrefuted really established when they they built a causality a ladder of causality, which is more incremental, for example, they said smoking leads to deposition of tar, okay. And the tar causes cancer and they all these causation, which is an incremental intermediate effect by step and then then to the step by step and so, when they define this incremental steps, and they were unique, that means this tar was caused only by smoking and the tar also caused cancer they established that tar causes cancer by doing experiments with with rat and they could also establish that smoking led to the position of tar now because of this unique intermediate effect, which is coming only from smoking, right, and and the tar is also causing the, you know, the,
the, the cancer itself. So, this was then established as an unrepeatable cause that smoking lead to cancer because it created a unique effect. Okay.

So, so picking up this knowledge, how do we apply it in in case of mystery analysis, so what we try to do in mystery analysis is try to build those incremental steps. Okay, off what we call as intermediate effects, unique observer observable intermediate effects, leading to the final effect. So let me give an example. Yeah, I have I have.
Okay, so I think the audience would have lost me as well. So let’s say I say, this sales scheme will lead to
higher sales, let’s say there is a dealer network, and I give a fantastic scheme. And the dealers love the scheme and the sales should go up, right? Okay. Now, the sales didn’t go up. And even though you have set up all this control groups, and everything people can really argue a lot, right? So Buca, you are the proponent might say, you know, this factor that factor and even if you saw all this data with the control group, you might have an issue. So what’s very important is to is to establish the incremental steps of outcomes, leading to the final outcome. Okay, for example, let’s think the first step that happened to to it.

Yeah, the first thing, so first of all, first of all, we see a mystery that the sales is actually, you know, didn’t go up, right. Let’s say, sales didn’t go up. But that does not tell us anything at all. Right, from the mystery analysis from the data that we get by the, you know, discontinuity, the interrupted time series analysis, we found the sales didn’t go up with the along with the control group find out that, you know, it’s almost the same trend. Okay. But that does not give us any insight. Yeah. We got to know what exactly happened in so what the only way we can find out what exactly happened is by building up things incrementally, right, right.
Yeah, so incrementally what do we expect to happen before it reaches the sales jump? So for example, here, let’s say it’s a very interesting scheme. When you announced this interesting scheme in the market, what should happen? As the first step?
People should buy more?

No, before buy more, because that is led to sales. So So let’s imagine this is a dealer network, and you have to enrol into that programme. Let’s let’s make that assumption. Right. The first thing that you should see is the enrollment going up. Yeah. Okay. The enrollment first, okay. Yeah. It makes Yeah. And even before that, the place that you are announcing that the enrollment has to go up there has to be attendance as well. So, just think that I try to see whether there was attendance or not. Right. Now the attendance can work.
enrolled. Yeah. Secondary.

But also and you find out that you’re you find out that you have a flop show in the first step itself here it Yeah, something went wrong. So now you you solve your mystery right. Now suppose you say no, no, people came in the right numbers and everything and people enrolled also bad, but you want to find out did the enrollment means who enrolled? If the existing guys enrolled anyways, they would have got the sales right. So if you want the real job bump up in enrollment, you want to know whether the New Dealers enrol Yeah, whether the dormant dealers enrolled. So I expect a big jump of New Dealers into the network, I expect a big jump of dormant dealers to enrol right. So, I want to see that happening. And then what do I expect to see that the first few orders from the dormant and the New Dealers I want to know that right right because the normal dealers anyway would have come in right right. So then you find out okay, these dormant dealers are they repeatedly placing orders or not? So when you try to create these incremental steps of outcomes, right, you can find out at which step things went wrong.

Okay, and if you understand and and that is where you got to have a game check where did things go wrong and and you do a proper game buy in that area for example, we find out that you know, guy, the doorman dealers didn’t even come New Dealers didn’t even come right. Okay. So the what happened is the old dealers who anyway would have you know, join this programme with or without their anyway giving you orders. So, they took over, they came onto the programme and the sales that is why the sales bump didn’t happen, right. Right. So now the the big mystery is why did the dominant dealers dealers and the dormant dealers they didn’t come on board, you find out that there’s something wrong with the computer scheme itself which is not attractive to pull those people Yeah or maybe or maybe there was a fault in communication. So you check all that right. And again, here also you have to keep on falsifying all your hypothesis right? You start thinking what is hypothesis using your communication was wrong. So you say okay, but in that case, the same like communicated a lot of people came in. So maybe something is wrong in my

in my scheme itself, right. So use

Developing this hypothesis, again, taking the past data and invalidating it and then you finding out what is the real reason, okay, then you make the modification, and, and relaunch it. That is how the learning happens. That is how mystery analysis happen. It’s it’s a very

what I say creative process process, it’s a very iterative process and you, you you go through something, then to missteps and go forward. So, this happens only when the solution is being is being tried out in a new area. Or, for that matter, you are innovating, right, you are doing things for the first time. So that’s, that’s where it happens. Lovely, I think. So that is extremely, extremely helpful. I loved

understanding the whole mystery analysis process. And I would I don’t know, for the listeners or not, but I would my say we’ll definitely use it more often.

Okay, so one thing I want to put forward here is that if you if you companies which want to be innovative, right, and want to do things for the first time, and to use what is known as what Elon Musk comes the thinking through first principles, this is the method, this is the method of how do you think and build up a new practice, do something that nobody has done before, right? Or you try to let’s even if you try to implement something from let’s say, somebody else in the industry, you still have to apply this process because the CRT and the party ensures that the solution has been thought through and the problem has been well defined in your environment. Okay, you implement and then you find out, okay, didn’t get the intended results? Did you have all these setups to understand what went wrong and such an organisation? If the develop this entire scientific process of, you know, defining a problem, defining the solution, doing the practical implementation and then learning from it? There is no way anybody can beat this organisation, because this is what the real learning organisation is all about. But, I mean, I know this might sound like a very,

To quickly summarize for our listeners. If you want to solve a chronic problem and develop a rebost solution, there are four important steps
First Step is usage of CRT and FRT to built strong logical foundation of solution that comes up in the mind of people. This steps helps in killing bad solutions
The second Step is to clearly define how causality of your solution leading to the positive effect will be established with proper measurement. Establish the control groups using different methods. At this stage it is also important to establish the ladder of intermediate effects. Think of an intermediate effect which is only unique to your solution.
Then third step is to check for mystery by analysing the results, Look for where the deviation happened in the of proposed intermediate effect and their original estimates, Develop likely hypothesis and go back to step 1 to again kill ideas which lack logical consistency.