Mystery Analysis Part II
In the last episode on Mystery Analysis, we set the groundwork for why and how to do an analysis that would help us reveal the causality behind our post-implementation results. We also touched upon two concepts or approaches that could be used for detecting the mystery. This episode goes into details of these methods and subsequent steps to be followed. A useful article to read while trying to understand the nature of 'cause and effect' can be found here: https://www.vectorconsulting.in/blog/systems-thinking-innovation/thinking-clearly/
| Shubham Agarwal : | So, let us start with regression discontinuity analysis. |
| Satyashri Mohanty : | Yeah, so we use the interrupted time series a lot regression discontinuity, not as much, but the okay, we’ll cover both of them.
So, let me take a case here, to establish how powerful this methodology is. So for example, you know, people get scholarships, right. And I believe that, let’s say good scholarships lead to people pursuing higher studies. So that is, let’s say, the intended effect, x is scholarship and y is let’s a higher studies, just for example, right now, and I said, this is what we want to encourage the good students, right. Now, you might come back and say, I, if I saw you data that hey guys, all the people that I gave scholarships, okay. You know, they most of them went for higher studies. Now, that’s a question for you. Does it establish causality. |
| Shubham Agarwal : | Well maybe good students are studious in nature and they would have pursued higher studies without even the scholarships |
| Satyashri Mohanty : | Yeah. The very difficult to establish causality and very difficult to do a randomised control trial. Just imagine it is unethical to do it, by the way. It’s like giving a group of people good guys scholarship, another set of good guys don’t give them scholarship and for next five, six years, see what is happening with them. Right. And the guys who have not given the scholarship, they you have messed up with their life permanently. Can’t do this RCT, right. So very interesting is how do we then establish causality? Is the scholarship helpful or not? |
| Shubham Agarwal : | Is it really making an impact or not? A parallel to this example would be various schemes organizations have for high performing dealers or employees. They usually have a cut-off points around some business outcomes. Important causality question would be – is the scheme motivating people to be high performers or the high performers are achieving it because of some other factors , nothing to do with the scheme |
| Satyashri Mohanty : | Yes. It’s very important to understand whether it’s working or not otherwise, you try to create all these kind of schemes. And you’re very scared to even withdraw it because you don’t know the knowledge whether it’s working or not. |
| Subham Agarwal : | So let us understand how regression discontinuity help in establishing causality in these cases ? |
| Satyashri Mohanty : | So. So there are two people, social scientists who came up with a very interesting methodology.there is a cutoff Mark. |
| Subham Agarwal : | Yes scholarships have a cut-off point where the students above the number get it and the ones below don’t |
| Satyashri Mohanty : | Yeah. Okay. So there is a cutoff mark. And what is the analysis, they are saying that at the cutoff mark, very near to the cutoff Mark, let’s say the cutoff mark is, let’s say 90 percentage, okay. So all people above 90% will get the scholarship. Now, at the point of cutoff mark, which is, let’s say 89, and 91. Now, the difference between 89 and the student that is getting 91, the assumption is there is hardly any difference, because the same guy over the next day, if he gives a test, there is a good chance that the 89 guy will keep 91 and the 91 guy will keep 89, right. So at the edge where the cutoff or you know, you have two sets of people who are equally likely to be on the either side. |
| Shubham Agarwal : | Right |
| Satyashri Mohanty : | Now just see the, the brilliant idea behind it.
at the edge, right? We have people who have equal chances of being on the either side, we can say that they are nearly the same. Okay, so what they’re saying our control group is here only. So at the edge between 89 to 90, let’s say there are 50 students, and between 90 to 91. There are 50 students, okay. So let’s take, take the data of the 50 students who are the on the other side of the border, who got the scholarship and the 50 students, or let’s say 60, odd students who didn’t get the scholarship and they are equal in all other aspects. Yeah. And it’s they are handled in a way it is, yes, it is totally random. In a way it is randomised, right? Yes. So they say that, if you compare that at the border, I can, I can easily find out whether, you know, those 50 I run the data, after five years, I find it out of 50, you know, 49 pursued higher studies. And in the other case, where people were between 89 to 90, you know, only two of them, let’s say pursued higher studies, then I can say, Okay, guys |
| Shubham Agarwal : | That’s a very clever idea of choosing your control groups, I think
But I guess this methodology can be used only when there is cut-off point for applying solution or giving treatment to a group, like the schemes for high achievers |
| Satyashri Mohanty : | in fact, we did an analysis for a company and prove it to them that the so called, they had something called Super achievers club, so we prove it to them that it’s not working, right and by by doing a regression discontinuity analysis |
| Subham Agarwal : | What about other cases where we cannot apply this regression discontinuity analysis |
| Satyashri Mohanty : | we use the other method, which is called the interrupted time series. Now, what is interrupted time series, it is nothing but the word interrupted time series, right? And it is nothing but trying to find out a before and after. Okay, it’s as good as the on the x axis, you have a timeline. Yeah. And then in between there is a intervention. So you see a before and an after data, okay. So for example, I, I implement, let’s say drum buffer rope, and the before data that WIP is hovering around, you know, 5% plus minus 5%. And just after the implementation of the drum buffer rope, within one week, I see it dropping by 40%. Okay, I can now reasonably say that, you know, the control group was the past data. And I’m comparing with the, the, the, the time is used as a demarcation between the treatment group and the control group. So the past is the control group, and the treated group is the is the future. |
| Shubham Agarwal : | So the assumption is that when I’ve implemented something, it would still take some time to show it results, is it? |
| Satyashri Mohanty : | Yeah, you got Yes, you got to know what is the time that we’ll get to show results. |
| Subham Agarwal : | So the demarcation line should capture this lag in results. But I have doubt, just because the WIP has dropped by 40%, does this establish causality? |
| Satyashri Mohanty : | you got to be very careful.
you find out that there is some other intervention which has also happened in this time. |
| Shubham Agarwal : | Right, |
| Satyashri Mohanty : | So you got to account for that. Okay, if there is something else which has happened, then you can’t claim your solution led to that 40% drop, maybe the other solution also had a contributing factor? |
| Subham Agarwal : | There can be another case, where WIP drop is in the realm of noise. |
| Satyashri Mohanty : | Correct. |
| Shubham Agarwal : | Let us take sales example, which is always the difficult area to establish causality |
| Satyashri Mohanty : | Yeah. So, so let’s say I have thought of a very interesting idea on the sales side, okay. And I expect the sales to go up. Okay. And so now, what I try to do is I, my control group, is I try to look at, let’s say, last five quarters, okay, month on month sales, apply the intervention, okay? And I expect that after one month, post the intervention, right, I see a sudden spike in sales, okay, which is let’s say a 30% bump and it is staying at that level. So, if I draw that trendline I clearly see what is known as an interruption in my trend line, but the trend line was you know, going a particular way and then I see a bump up or a bump down, then I can claim that the intervention you know, really helped okay. |
| Shubham Agarwal : | But there could be a lot of scenarios, you know, say it was the year end and you know, you see a bump at the end. |
| Satyashri Mohanty : | Correct, correct. So, so, it’s very, very important people can say many things here for example, hey guys you know, the before and the after that you are trying to establish here, the way you are measuring things have changed. Yeah, number one, number two, you know, there is another intervention there for example, as you said year end effect okay.
There could be things like you know, the the the trend of the history, right. So, almost the same trend, if I try to extrapolate it, it would have anyway followed that trend line that you’re trying to show as a gain if I if I draw the trendline of the past into the future, and I can extrapolate it and it exactly falls on that trendline right and right or or I can say boss it is in the realm of noise what you’re trying to show the result Yeah, actual fluctuations is almost at that level plus minus 10% aise hi to mera sales upar niche hota hai, what are you telling me it is 10% more right. So, these checks are very, very important right. |
| Subham Agarwal : | When we are only using past as comparision, we cannot always rule out all the 3 counter points – The growth trend was always there – The measurement method has changed – Some other external factor or intervention is working In case of WIP numbers, if the number is nearly static in the past, and we establish that no other intervention was planned at same time, a sudden drop in WIP and a new level line is enough to establish causality from data. But in case of variable like sales, there are many other things which are at play. The numbers are growing year on year in specific trend, many interventions are planned in parallel and number is also noisy. How do we know that a specific intervention helped ? |
| Satyashri Mohanty : | And to do that, when we try to do before and after many times, we have to establish another control group where we have to do before and after understood, so, suppose if I take a territory, okay, and I say in this territory, I apply a before after result, I take another territory, which I claim is equal to the first territory in all aspects. And I do a before after there, right? Because what we’re trying to build is what would have happened if this you know the sale solution was not implemented right, it is only with respect to that comparison we can establish our causality or establish the power of our solution okay. So, that comparison of what would have happened if we had not done is either with the past right, but many times the past and the future you know, there are other factors in the play. |
| Subham Agarwal : | Yes , like the year end effect caused the results to improve. |
| Satyashri Mohanty : | So, I take another territory as for example, you said year end effect, year end effect would have happened on all the other territories right. So, I see that there is a bump up because of the year end effect, but this bump up is much higher than the bump of in the control group right, okay. |
| Subham Agarwal : | The bump also has to be beyond the natural fluctuation of data. It should show a new trend. |
| Satyashri Mohanty : | So, all other factors are remaining the same and I can claim that you know, the, you know, the solution, the solution gave the intended result, right, right. Now starts the point when the results didn’t come right, because all this mystery analysis is all about that. So till now what we discussed how to set it up here, okay? |
| Shubham Agarwal : | I love the fact that, you know, there is so much, you know, so much thought that has gone to just set it up.
This looks like a good enough method, if not a perfect method for establishing causality. Now that we have set up a method to detect outcomes of interventions. So the mystery can get detected. What to do next? |
| Satyashri Mohanty : | So for mystery analysis, I want to take an example of how difficult it was to establish, does smoking lead to cancer or not, you won’t believe it took, it was almost like almost a decade of conflict, that smoking lead to cancer, people should all kinds of data saying that, you know, higher incidence of smoking has led to higher incidence of lung cancer, but the tobacco lobby, they came out with an interesting counter, you know, what was the counter? They said, that boss, this is all correlations, these are just assertions. So, what are the real reason, they said that the real reason is that there is a gene, right, which is a, the gene, if you have the gene, you, you tend to have a risky |
| Shubham Agarwal : | You’re susceptible to being found cancer. |
| Satyashri Mohanty : | You’re susceptible to cancer, and you also are susceptible to smoke. So, they said that smoking does not lead to cancer, it is that gene which leads to smoking and also leads to cancer. Okay, it’s like, it is like a common cause. |
| Satyashri Mohanty : | You know, when was this conflict got resolved, when was it unrefutably established when they they built a causality a ladder of causality, which is more incremental, for example,
they said smoking leads to deposition of tar, okay. And the tar causes cancer and they all these causation, which is a incremental intermediate effect and then, yeah step by step and so, when they define this incremental steps, and they were unique, that means this tar was caused only by smoking and the tar also caused cancer, so they established that tar causes cancer by doing experiments with with rat and they could also establish that smoking led to the position of tar now because of this unique intermediate effect, which is coming only from smoking, right, and and the tar is also causing the, you know, the cancer itself. So, this was then established as an unrefutable cause that smoking leads to cancer because it created a unique effect. Okay. So, so picking up this knowledge, how do we apply it in in case of mystery analysis, so what we try to do in mystery analysis is try to build those incremental steps. Okay, of what we call as intermediate effects, unique observable intermediate effects, leading to the final effect. So let me give an example. |
| Shubham Agarwal : | Yeah, I was about to ask you |
| Satyashri Mohanty : | So let’s say I say, this sales scheme will lead to higher sales, let’s say there is a dealer network, and I give a fantastic scheme. And the dealers love the scheme and the sales should go up, right. Okay?. Now, the sales didn’t go up. |
| Subham Agarwal : | In this case, we can call it a mystery as the outcome did not match expectations
So what’s very important is to is to establish the incremental steps of outcomes, leading to the final outcome. |
| Satyashri Mohanty : | We got to know what exactly happened, so the only way we can find out what exactly happened is by building up things incrementally, right, right. Yeah, so incrementally what do we expect to happen before it reaches the sales jump? So for example, here, let’s say it’s a very interesting scheme. When you announced this interesting scheme in the market, what should happen? As the first step? |
| Shubham Agarwal : | People should buy more |
| Satyashri Mohanty : | No, before buy more, because that is led to sales. So So let’s imagine this is a dealer network, and you have to enrol into that programme. Let’s let’s make that assumption. Right. The first thing that you should see is the enrollment going up. |
| Shubham Agarwal : | |
| Shubham Agarwal : | Yeah. Okay. The enrollment first, okay. |
| Satyashri Mohanty : | Yeah. And even before that, the place that you are announcing that the enrollment has to go up there has to be attendance as well. So, just think that I try to see whether there was attendance or not. Right. Now the attendance can work. |
| Shubham Agarwal : | Percentage of people who enrolled, the second territory, let’s say, would also matter. |
| Satyashri Mohanty : | Correct, and you also find out that you’re, you find out that you have a flop show in the first step itself. |
| Shubham Agarwal : | Correct, yeah. |
| Satyashri Mohanty : | Something went wrong. So now you you solve your mystery right. Now suppose you say no, no, people came in the right numbers and everything and people enrolled, but you want to find out did the enrollment means who enrolled? If the existing guys enrolled anyways, they would have got the sales right. So if you want the real bump up in enrollment, you want to know whether the New Dealers enrol Yeah, whether the dormant dealers enrolled. So I expect a big jump of New Dealers into the network, I expect a big jump of dormant dealers to enrol right. So, I want to see that happening. And then what do I expect to see that the first few orders from the dormant and the New Dealers I want to see that right right because the normal dealers anyway would have come in right. So then you find out okay, these dormant dealers are they repeatedly placing orders or not? So when you try to create these incremental steps of outcomes, right, you can find out at which step things went wrong. Okay, and if you understand and and that is where you got to have again check where did things go wrong and and you do a proper game buy in that area for example, we find out that you know, guy, the dormant dealers didn’t even come New Dealers didn’t even come right. Okay. So the what happened is the old dealers who anyway would have you know, joined this programme with or without they are anyway giving you orders. So, they took over, they came onto the programme and the sales that is why the sales bump didn’t happen, right. Right. So now the the big mystery is why did the new dealers and the dormant dealers they didn’t come on board, you find out that there’s something wrong with the scheme itself which is not attractive to pull those people Yeah or maybe or maybe there was a fault in communication. So you check all that right. And again, here also you have to keep on falsifying all your hypothesis right? You start thinking what is hypothesis, you are saying your communication was wrong. So you say okay, but in that case, the same guy communicated and a lot of people came in. So maybe something is wrong in my, in my scheme itself, right. So you start developing this hypothesis, again, taking the past data and invalidating it and then you finding out what is the real reason, okay, then you make the modification, and, and relaunch it. That is how the learning happens. That is how mystery analysis happen. It’s it’s a very, what I say, it’s a very iterative process and you, you go through something, then to missteps and go forward. So, this happens only when the solution is being is being tried out in a new area. Or, for that matter, you are innovating, right, you are doing things for the first time. So that’s, that’s where it happens. |
| Shubham Agarwal : | Lovely, I think Satya this is extremely, extremely helpful. I loved understanding the whole mystery analysis process. And I would I don’t know, for the listeners or not, but I would my say we’ll definitely use it more often now. |
| Satyashri Mohanty : | Great, so one thing I want to put forward here if the develop this entire scientific process of, you know, defining a problem, defining the solution, doing the practical implementation and then learning from it? This is what real learning organisation is all about. |
| Shubham Agarwal : | Right. So Satya, we’re going to break here. And I think this is a very, very interesting discussion that we’ve had. Once again, apologise for having it after so long. Whenever we discuss something like this. It’s extremely helpful. So thank you so much. And for all the listeners. If you have any questions or concerns, you can shoot us the questions on our social media handles, or you can write to us on our website as well. The link is in the details. Until next time, this is Shubham signing off. Thank you Satya once again. |
Related Information
Get in touch
Vector Management Consulting Pvt. Ltd.
10th floor, Thane One, DIL Complex,
Ghodbunder Road, Majiwada,
Thane (West), Maharashtra - 400610, India.