So before I try to answer let’s try to do a quick recap on what is CRT all about? CRT is all about trying to come out with a very good explanation about what is the real problem given the organisation that we are trying to analyse or let’s say a system that we are trying to analyse, that why this is the problem and this is not a symptom. So, we it is a unrefutable explanation about what is the core problem, right. So that is what a CRT is used for

Shubham Agarwal :	Hello, and welcome to counterpoint Podcast. I’m Shubham Agarwal, and we are joined by Satya here today. To complete our series on thinking processes, which we had started long back, we had promised that we would come up with the discussion on mystery analysis quite some time back, but we took time to record and release the episode. Because COVID hit us, there were some other concerns around recording the episode. And we are finally here to, you know, share the knowledge on mystery analysis. So let’s welcome Satya Hi, Satya, welcome. How are you?
Satyashri Mohanty :	Hi, Hi, Shubham. Nice to be here again
Shubham Agarwal :	So Satya, we talked about mystery analysis in brief in one of the last episodes, but what is what is really a mystery and what is this analysis all about?
Satyashri Mohanty :	Okay, so let’s try to define the word mystery. So, many times what happens is, we think through a solution and we expect the solution to give us some result, but when we start implementing, the results either are much more than what we expected or it is actually the other way around, the results are not seen or worse, the result goes downwards. So this is what we call as mystery which is getting results beyond anticipation So what I see in organisations is there is no proper rigour in evaluating mysteries and hence organisations don’t learn a lot. Suppose the sales goes down in a territory and you talk to salespeople, you will hear various different kinds of storylines and storylines that contradict each other. So what we are going to discuss today is the rigour of trying to do a mystery analysis in a very systematic way. And hence, develop an ability to learn further as an organisation.
Shubham Agarwal :	What I found interesting was that you said not just the negative results, but also the positive results. Because you know, there are a lot of times that happens that, you know, your results are much more than you had expected. Even that is a mystery, which which I kind of you know, a bulb lit up in my mind. Because whenever that happens, we just take the laurels to our name and we move on. But I think there is a lot to learn from the positive results as well, which were beyond our expectations.
Satyashri Mohanty :	Correct, correct. So maybe there are other factors that are playing in so we need to be very careful on how we draw conclusions about about whether the theory worked or didn’t work. So everything that is beyond the intended result is a mystery. That’s the way to look at it.
Shubham Agarwal :	Lovely. Great. So, so we had spoken about, you know, some of the tools like some logical tools like, you know, CRT and some others, why can that not be used for mystery analysis?
Shubham Agarwal :	The as is condition
Satyashri Mohanty :	Yeah, the as is condition. So come out with a unrefutable logic of your entire analysis and synthesis of the situation and and you say that this is the main problem that we need to attack, right. And FRT is about trying to say that the solution that you have thought through, you come out with another unrefutable explanation as to why the solution will deliver the intended results. Yeah, it is it is trying to build this explanations before you start implementing. Okay, so the tools are very useful to kill badly thought solutions, or, or, you know, top of the mind solutions, which have not been well thought out. Or, or let’s say you know, the boss wants to do it. So let’s do it, those kinds of solutions, right. So so CRT and FRT is what you try to do before the implementation, so that what you put onto the, onto the ground is something that is well thought out.
Shubham Agarwal :	Right. what we are going to discuss today, mystery analysis is something that has already happened. And now we are trying to analyse what has happened, and why has it happened? Is that right? Would that be right to say?
Shubham Agarwal :	Right. So it is basically trying to find out what went wrong in the implementation phase. So to say,
Satyashri Mohanty :	Yes, was was was your implementation at fault the solution was incomplete. Or the solution was put in an environment which had a hidden, you know, condition that we all missed out
Shubham Agarwal :	Okay, I think now that the context is very set for why are we doing the mystery analysis? Let us start by understanding first to in order to do this analysis, what do we require? What do we how do we start where do we how do we collect all the requirements to be able to do a good mystery analysis?
Satyashri Mohanty :	Yeah. So, very, very important that we think through the entire, you know, the causal structure before the start of the implementation to understand the, the mystery, right, if you if you don’t set up the implementation, then then we can have big problems, right. So, I just want to you know, take us back to Sir Francis Bacon, who’s kind of considered as the father of the scientific method. So, he said causality can be established by three checks, okay. So, he says that how do you know that x led to y right, he said that you cannot do it by just associations that whenever I see x, I always see y does not mean x caused y, right. So, correlation is not causation that we you know, discussed a lot in the previous podcast. So, what he said is, there are three rules, the first rule is you got to check whenever x was there, y was also there. So that’s the first check. The second check is when the x was not there, you also expect y not to be there, because if x is causing y that means presence of x will lead to presence of y and absence of x will also lead to an absence of y
Shubham Agarwal :	Satya I’m sorry to interrupt you here. But someone might ask how many times? How many places should I check? So, you know, let’s say X causes Y. How many cases of such should I check to be sure that yes, it is a reality?
Satyashri Mohanty :	Yeah, so, so know that that’s why, you know, we got to use representative samples, and we go to got to use those rules that yes, whatever from the samples that we have seen enough samples so that we can conclude about the population from the sample. So that’s, that’s very important. I think the best thing that the industry that does it very often is the pharma industry, which whenever they try to test out a new drug, so they have this gold standard of causality called the randomised control trial, where, and I think we discussed this in one of the episode that they will have a treatment group or the group that is given the medicine, right, and it is compared with the another group that gets the placebo or you know a fake medicine, okay. And and then they check, you know, whether, whether the treatment was there, whether the treated group actually showed the intended result, and the non treated group should not have any results, and that is how the efficacy of the medicine is established. Now, the golden rule in all this, and which is the most difficult part is to ensure that both the groups are equal in all other respects.
Shubham Agarwal :	Correct. Everything else is same.
Satyashri Mohanty :	Yeah, everything else is same, for example, you know, you want to test out a COVID medicine, right. And you give it to a set of very young volunteers. Right. And, and, and they get cured very fast. can you conclude that the medicine cured?
Shubham Agarwal :	No, obviously not. I don’t know.
Satyashri Mohanty :	So you, you might you might say that those young volunteers would have got cured on their own. Right. So you need a control group of again, another set of young volunteers and then compare the two.
Shubham Agarwal :	Correct, Make sense.
Satyashri Mohanty :	Now, one of the things that what RCT does and why that’s why it is considered the gold standard is the word R. Okay, the word R is randomised. So what people try to do is one check that I said is that both groups have to be similar in all aspects, right? Yeah. But there are many hidden things as well, that can cause a havoc for example, you know, maybe the guy guy feels very happy so, their body chemicals behave in a different manner
Shubham Agarwal :	So the hidden variables can lead to one group being mis-represented,
Satyashri Mohanty :	Yes, but not everything can be defined see, because these are many hidden things that you cannot define right.
Shubham Agarwal :	Yeah, how do you ensure the groups are same in all aspects
Satyashri Mohanty :	So the good enough answer to this case is what they call as randomised. Now what is randomised? It means that a guy’s chances of being selected to be in the control group or the treatment group is one and the same Okay. So then so so if you walk in it is not that you are selected to be here you can be randomly picked up and be part of any group. Because of this randomization, they say that all the hidden variables are also kind of, you know, accounted in both the group. So, that’s why the randomization is very, very important and to avoid what is known as a selection bias, right? So, selection bias can creep in without you knowing about it.
Shubham Agarwal :	Yeah, actually. Let us take an example in context of management. How would RCT look like
Satyashri Mohanty :	Yeah, if you’re doing RCT, right, for example, let’s say take a case here that I want to implement a let’s say, interesting marketing scheme or a dealer scheme, right. Yeah. And you can say that I have selected this let’s say five or six dealers at random and there is another five or six dealers who are not given this right?. So, that is how you use the bacons rule right you apply the treatment, you get the effect. So, you get this marketing scheme, you expect the sales to go up wherever you have not given you expect the sales to stay flat, okay. So, by that you, you apply, you establish the causality. So when x was there, intended y happened when x was not there intended y it didn’t happen. Now, the golden rule is all other factors will remain the same. That means if I have if I have big dealers here, there has to be big dealers there if they’re a small dealers here are equal and small dealers has to be there there are motivated dealers here then you have to have motivated dealers. Now, the problem is the word when I use the word motivated dealers now, how do you how do you find out so the randomization is very, very important. Yeah. So randomization is very important. So your people should should be randomly assigned to the control group and or the treatment group, right? The work treatment here is where the scheme marketing scheme is being applied. So there has to be equal chances of all the participants to be part of this group versus that group, then you account for all kinds of hidden variables. So the golden rule of all factors remaining the same is kind of applied. That’s the real RCT. Right, okay. So the problem is, see the real problem is in organisations, right, many times, you cannot do this so called randomization, you cannot even run a test, for example, you want to reduce the prices, right? Can you do a pilot that can reduce the prices?
Shubham Agarwal :	And if you cannot set a pilot and compare with a non-implementation area. We cannot establish causality as per bacon rule. And if you cannot even establish the primary causality of your solution pre-facto – there is no way mystery analysis can be done
Satyashri Mohanty :	Yeah. Yeah. So and always remember the, you know, we need a counter scenario, what is known as a counterfactual to establish causality that means what would have happened if I had not done that? Right. And that’s the difficult part that the counterfactual is not known in an environment you have already made the change. So what would have happened if i had not done this? And that is why you need a control group, right? Where you try to see that, you know, this is the group where I’m not applying anything and let me see what happens. So but I’m giving you a case suppose you want to change the prices right? You cannot say I change the prices for this 10 customers and the other 10 customers, I don’t change the price. But you know, the price is something that if you change you are you are gone like you? Yeah, you know, you change the prices for you know, 10 customers and the other 10 are so pissed off that they don’t give you order. So while running the pilot, you, you screw up the company’s total financial, so you can’t run at times pilots like this. Right? So and the other case, yeah, and the other case, even after you have, let’s say, thought out, let’s say a very interesting sales programme, and, and you want to, you know, you want to apply it in one area, and let’s say it is possible to apply and isolate in one area. But when you ask people in which area should I apply, should we select randomly? People say no, why should we do it randomly? Let’s select the worst case scenario or let’s select where it is easy to implement, right? So, the criteria for selection is never a never a random
Shubham Agarwal :	So two issues At times using principles of randomization to completely eliminate selection bias is difficult And second, we may not be able to do an isolated small scale controlled intervention. Have to go the entire way.
Satyashri Mohanty :	Correct we cannot do RCT in context of an organisation in in the purity of the word RCT, which is a randomised control trial right people coming in if you wish to see the way trials happen. So we we consult a lot of these clinical research organisations, pharma organisations, you, you will find out that the person who is getting the medicine does not even know is he getting the medicine? Or is he getting the placebo? Even the researcher does not know who’s getting what it is so mass and and they say it is that it is it is double blind. That means neither the patient knows whether he’s getting the medicine or is getting a sweetener.
Shubham Agarwal :	Like you said this cannot be used for other organisations,
Satyashri Mohanty :	For example, you come out with the interesting. Let’s say HR policy, right? Would you want to do it for five people and not do it for another five people? And, and it’s so strange, you can’t do it. Right? And it was five people that you don’t give this they leave the organisation and go,
Satyashri Mohanty :	Correct. So what what we just said, you know, you do your CRT, you do your FRT now comes the moment of truth, the moment of truth is you want to implement and you suddenly find out that the way you are selecting your pilots, it is it has got a selection bias. Right? Or you are not even doing a pilot you are actually doing a big bang. Yeah. Right? In which case, how do you set up for testing the causality? So if you can’t even set it up, myster analysis comes much later. Yeah. Right. So let’s understand if he can’t do RCT, what is a good enough method? Right? And, and thankfully, in social sciences over the last decade or so, people have come out with lot of good enough methods to establish causality from you know, from from observations, so you can’t do RCT for example, this this, this problem, as you can guess, is also there in economics. Right. Yeah, You, you can’t do this kind of changes in our in our country or not improve the rate decrease the rate for half the country? Yes. Yeah. Correct. So many things can’t be done. So, so. So people have invented very interesting methods and Okay, so one of the things that people do and we will talk about these methods, which is called the interrupted time series method or regression, discontinuity analysis or natural experimentation. These are these are what I call as good enough methods to establish causality in absence of a proper RCT.
Shubham Agarwal :	Yeah, so we have a we have a parallel or a good enough good enough solution we have a good enough
Satyashri Mohanty :	Yeah, we have a good enough. We don’t have the gold standard called the RCT, but I have good enough,
Shubham Agarwal :	Which is okay. So it was good to have something. Then you said regression discontinuity analysis and interrupted time series, two things, which sounds really complex to me right now on the face of it. But I’m sure you’ll help us understand them better. Okay, yeah.
Shubham Agarwal :	Yeah, I was about to ask you
Satyashri Mohanty :	So regression discontinuity analysis, you know, this is very interesting. So, let me take a case here, to establish how powerful this methodology is. So for example, you know, people get scholarships, right. I believe that, let’s say good scholarships lead to people pursuing higher studies. if I saw you data that hey guys, all the people that I gave scholarships, okay. You know, they most of them went for higher studies. Does it establish causalitySo you will find out you know, these guys, they are already good. They anyway, they would have, you know, done good in life, or, or let’s even pursued higher studies. I just, I just happen to you know, give them give them the scholarship, right. So, the very difficult to establish causality and very difficult to do a randomised control trial. Just imagine it is unethical to do it, by the way. Yeah, it’s like giving a group of people good guys scholarship, another set of good guys don’t give them scholarship and for next five, six years, see what is happening with them. Right. And the guys who have not given the scholarship, they you have messed up with their life permanently. Can’t do this RCT, right. So very interesting is how do we then establish causality? Is the scholarship helpful or not? How do we discount the argument that he guys these were good students to start with, they never would have pursued higher studies, how do you counter that? And if you don’t counter that your entire scholarship budget is going down the drain and you’re not being aware about it. So it’s so important to understand like for example,You don’t know the knowledge whether it’s working or not. So. So there are two people, social scientists who came up with a very interesting methodology. They said that most of the time when these scholarships are given, there is a cutoff Mark. Yeah. Okay. So there is a cutoff mark. And what is the analysis, they are saying that as the cutoff mark, very near to the cutoff Mark, let’s say the cutoff mark is, let’s say 90 percentage, okay. So all people above 90% is will get the scholarship. Now, at the point of cutoff mark, which is, let’s say 89, and 91. Now, the difference between 89 and the student that is getting 91 is hardly the assumption is there is hardly any difference, because the same guy over the next day, if he gives a test, there is a good chance that the 89 guy will keep 91 and the 91 guy will keep 89 Yeah, right. So add the add the edge where the cutoff or you know, you have two sets of people who are equally likely to be on the either side, right?
Shubham Agarwal :	Near the cut-off mark, we can assume, students on either side are same but not beyond the cut off mark We have people who have equal chances of being on the either side, we can say that they are nearly the same. Okay, so what they’re saying our control group is here only. So at the edge between 89 to 90, let’s say there are 50 students, and between 90 to 91. There are 50 students, right? Okay. So let’s take, take the data of the 50 students who are the on the other side of the border, who got the scholarship and the 50 students, or let’s say 60, or students who didn’t get the scholarship and they are equal in all other aspects. you know, those 50 I run the data, after five years, I find it out of 50, you know, 49 pursued higher studies. And in the other case, where people were between 89 to 90, you know, only two of them, let’s say pursued higher studies, then I can say, Okay, guys, that’s a very clever idea of choosing your control groups, I think very, very, very, very, very, very clever idea. And, but that is only used when there is there is this cutoff. So any solution which which says, you know, there is this cutoff beyond that, you know, you get it and you don’t get it. So any anything, any any motivation schemes that you have for salespeople, so, in fact, we did an analysis for a company and prove it to them that the so called, they had something called Super achievers club, and they used to, you know, go out for foreign countries and have good fun and so we prove it to them that it’s not working, right and by by doing a regression discontinuity analysis and telling them because their hypothesis was that, you know, the guys who get it, they get motivated and, and and year on year, the next year, at least they’ll perform well. So we, we ran that and we just to show them that it’s it’s it’s not helping okay. So, so the other one so, so, as I said regression discontinuity can be used only when you have a case which is capable, some kind of a cutoff mark, right, right. But vast majority we don’t implement like this right. So we use the other method, which is called the interrupted time series. Now, what is interrupted time series, it is nothing but the word interrupted time series, right? And it is nothing but trying to find out a before and after.
Satyashri Mohanty :	Okay, it’s as good as the on the x axis, you have a timeline. Yeah. And then in between there is a intervention. So you see a before and an after data, okay. So for example, I, I implement, let’s say drum buffer rope, and the before data that WAP is hovering around, you know, 5% plus minus 5%. And just after the implementation of the drum buffer rope, within one week, I see dropping by 40%. Okay, I can now reasonably say that, you know, the control group was the past data. And I’m comparing with the, the, the, the time is used as a demarcation between the treatment group and the control group. So the past is the control group, and the treated group is the is the future. So the assumption is that when I’ve implemented something, it would still take some time to show it results, is it? Yeah, you got Yes, you got to know what is the time that we’ll get to show results. And and and you go to market out and you say that you know, this is dropped by 40% Right. So, so, if that drop happens, then you can say reasonably sure that it has happened, but you got to be very careful. The careful is that you find out that there is some other intervention which has also happened in this time. Right, correct. So you got to account for that. Okay, if there is something else which has happened, then you can’t claim your solution led to that 40% Drop, can it be the other solution also had a contributing factor? Right? Correct. Or it could be the case that the Wi Fi data of the past was always down coming down by 10%. Okay. And anyways, it is showing the same trend.Correct. And, and, or, or it could be the fact that, you know, the web data fluctuated by plus minus 5%. And when you are seeing the data, the immediate two data points, you know, is showing a 5% drop, then you can claim a claim that it has led tothis, this, this demarcation of time, is still slightly complex for me, personally, so, could we take some other example here as well? Is that possible? Yeah. So, so let’s say I have thought offer a very interesting idea on the sales side, okay. And I expect the sales to go up. Okay. And so now, what I try to do is I, my control group, is I try to look at, let’s say, last five quarters, okay, month on month sales, apply the intervention, okay? And I expect that after one month, post the intervention, right, I see a sudden spike in sales, okay, which is let’s say a 30% Bump and it is staying at that level. So, if I draw that trendline I clearly see what is known as an interruption in my trend line, but the trend line was you know, going a particular way and then I see a bump up or a bump down, then I can claim that the intervention you know, really helpedOkay, now, but but there could be there could be a lot of scenarios, you know, say it was the end, the year end and you know, you see a bump at the end. So, a lot of Yeah, correct, correct. So, so, it’s very, very important people can say many things here for example, you guys you know, the before and the after that you are trying to establish here the way you are measuring things have changed. Yeah, number one, number two, you know, the circumstances another intervention, there is another intervention correct there for example, as you said here and effect Yeah, okay. There could be things like you know, the the the trend of the history, right. So, almost the same trend, if I try to extrapolate it, it would have anyway followed that trend line that you’re trying to show as again if I if I draw the trendline of the past into the future, and I can extrapolate it and it exactly falls on the trendline right and right or or I can say bosses it is in the realm of noise what you’re trying to show the result Yeah, actual fluctuations is almost at that level plus minus 10% I say tomorrow sales, Hooper, new Jota right, what are you telling me it is 10% more right. So, these checks are very, very important right. And to do that, when we try to do before and after many times, we have to establish another control group where we have to do before and afterunderstood, so, suppose if I take a territory, okay, and I stay in this territory, I apply a before after result, I take another territory, which I claim is equal to the first entry in all aspects. And I do a before after there, right? Because what we’re trying to build is what would have happened right if this you know the sale solution was not implemented right, it is only with respect to that comparison we can establish our causality or establish the power of a solution okay. So, that comparison of what would have happened if he had not done is either with the past right, but many times the past and the future you know, there are other factors in the play. So, I take another territory as for example, you said urine urine defect would have happened all the other territories right. So, I see that there is a bump up because of the urine defect, but this bump up is much higher than the bump of in the control group right, okay. So, then I can and that is beyond the realm of noise, all the measurements of before and after is the same it is beyond the So, all the all other factors are remaining the same and I can claim that you know, the, the the, you know, the solution does give the, the solution gave the intended result, right, right. So, once I set it up, like, I used the regression discontinuity analysis or interrupted time series, I first identify the mystery or what is the mystery that this solution is compared to the control group, which is either past or let’s say, another physical group, another geography that I’m comparing it with? Or another plant that I am comparing with? I, once I do that, I can now show that yes, the results have come or it didn’t come. Right. Okay. Now starts the point when the results are good. didn’t come right, because all this mystery analysis is all about battery. So till now what we discussed how to set it up here, okay? How to set it up how to how to define all this before the implementation now comes what happens after the implementation is so much, you know, so much thought that has gone to just set it up. So, correct. Correct So much of our life for example, in in sales pilot if you’re if your control would be to be very careful, somebody might say, you know, my, my treatment group has the best of salespeople or a control group that you have are lousy salespeople, they’re rookies right. So that cannot be the proper control group. So all you have to be very careful about that fact. Right. So, so otherwise, you know, the, so these kinds of comparisons are very, very important. But as I said, you try to find out that even let’s say, you know, getting all the factors, exactly the same, like RCT is not possible. Okay. So when you try to compare with a control group, and you say all other factors remaining the same, let’s say the, the general economic trend of sales, right is following a pattern of let’s say, 15% Bump, with even the rookie sales will have a lower base, and the export sale will have a higher base. Yeah, so even the lower base, I expect, let’s say, 15% Bump, but here with the Expert, yeah, I’m getting a higher bump than then 15%. I’m getting a bump of 30%. Right, right. So can I can I claim that so all that has to be agreed upon set it up? Right? The real mystery comes is when you see the results, not there, as compared to the control, or you see a very big jump, or you see a very, or the entire opposite direction to the and that’s where the analysis starts. Because that’s where the learning is very, very important. Okay. Okay. This This sounds a very interesting process. Yeah. Yeah. Yeah. So let me let me tell you how, how important it is to again, again, do an analysis to really check the mystery, right?So for mystery analysis, I want to take an example of how difficult it was to establish how does smoking lead to cancer or not, you won’t believe it took, it was almost like almost a decade of conflict, that smoking lead to cancer, people should all kinds of data saying that, you know, higher incidence of smoking has led to higher incidence of lung cancer, but the tobacco lobby, they came out with an interesting counter, you know, what was the counter? They said, that boss, this is all correlations, these are just assertions. So, what are the real reason, they said that the real reason is that there is a gene, right, which is a, the gene, if you have the gene, you, you tend to have a risky, you’re susceptible, anything you are you’re susceptible to cancer, and you also are susceptible to smoke. So, they said that smoking does not lead to cancer, it is that gene which leads to smoking and also leads to cancer.Okay, it’s like, it is like a corpsman did not exist. Yeah, yeah. So the the point here is, how do I how do I establish that, you know, how that whether the, the smoking caused the cancer, so they fought really hard to establish that smoking has no causal link to cancer, you’re just seeing a fake correlation. you know, when was this conflict got resolved, when was it unrefuted really established when they they built a causality a ladder of causality, which is more incremental, for example, they said smoking leads to deposition of tar, okay. And the tar causes cancer and they all these causation, which is an incremental intermediate effect by step and then then to the step by step and so, when they define this incremental steps, and they were unique, that means this tar was caused only by smoking and the tar also caused cancer they established that tar causes cancer by doing experiments with with rat and they could also establish that smoking led to the position of tar now because of this unique intermediate effect, which is coming only from smoking, right, and and the tar is also causing the, you know, the, the, the cancer itself. So, this was then established as an unrepeatable cause that smoking lead to cancer because it created a unique effect. Okay. So, so picking up this knowledge, how do we apply it in in case of mystery analysis, so what we try to do in mystery analysis is try to build those incremental steps. Okay, off what we call as intermediate effects, unique observer observable intermediate effects, leading to the final effect. So let me give an example. Yeah, I have I have. Okay, so I think the audience would have lost me as well. So let’s say I say, this sales scheme will lead to higher sales, let’s say there is a dealer network, and I give a fantastic scheme. And the dealers love the scheme and the sales should go up, right? Okay. Now, the sales didn’t go up. And even though you have set up all this control groups, and everything people can really argue a lot, right? So Buca, you are the proponent might say, you know, this factor that factor and even if you saw all this data with the control group, you might have an issue. So what’s very important is to is to establish the incremental steps of outcomes, leading to the final outcome. Okay, for example, let’s think the first step that happened to to it. Yeah, the first thing, so first of all, first of all, we see a mystery that the sales is actually, you know, didn’t go up, right. Let’s say, sales didn’t go up. But that does not tell us anything at all. Right, from the mystery analysis from the data that we get by the, you know, discontinuity, the interrupted time series analysis, we found the sales didn’t go up with the along with the control group find out that, you know, it’s almost the same trend. Okay. But that does not give us any insight. Yeah. We got to know what exactly happened in so what the only way we can find out what exactly happened is by building up things incrementally, right, right. Yeah, so incrementally what do we expect to happen before it reaches the sales jump? So for example, here, let’s say it’s a very interesting scheme. When you announced this interesting scheme in the market, what should happen? As the first step? People should buy more? No, before buy more, because that is led to sales. So So let’s imagine this is a dealer network, and you have to enrol into that programme. Let’s let’s make that assumption. Right. The first thing that you should see is the enrollment going up. Yeah. Okay. The enrollment first, okay. Yeah. It makes Yeah. And even before that, the place that you are announcing that the enrollment has to go up there has to be attendance as well. So, just think that I try to see whether there was attendance or not. Right. Now the attendance can work. enrolled. Yeah. Secondary. But also and you find out that you’re you find out that you have a flop show in the first step itself here it Yeah, something went wrong. So now you you solve your mystery right. Now suppose you say no, no, people came in the right numbers and everything and people enrolled also bad, but you want to find out did the enrollment means who enrolled? If the existing guys enrolled anyways, they would have got the sales right. So if you want the real job bump up in enrollment, you want to know whether the New Dealers enrol Yeah, whether the dormant dealers enrolled. So I expect a big jump of New Dealers into the network, I expect a big jump of dormant dealers to enrol right. So, I want to see that happening. And then what do I expect to see that the first few orders from the dormant and the New Dealers I want to know that right right because the normal dealers anyway would have come in right right. So then you find out okay, these dormant dealers are they repeatedly placing orders or not? So when you try to create these incremental steps of outcomes, right, you can find out at which step things went wrong. Okay, and if you understand and and that is where you got to have a game check where did things go wrong and and you do a proper game buy in that area for example, we find out that you know, guy, the doorman dealers didn’t even come New Dealers didn’t even come right. Okay. So the what happened is the old dealers who anyway would have you know, join this programme with or without their anyway giving you orders. So, they took over, they came onto the programme and the sales that is why the sales bump didn’t happen, right. Right. So now the the big mystery is why did the dominant dealers dealers and the dormant dealers they didn’t come on board, you find out that there’s something wrong with the computer scheme itself which is not attractive to pull those people Yeah or maybe or maybe there was a fault in communication. So you check all that right. And again, here also you have to keep on falsifying all your hypothesis right? You start thinking what is hypothesis using your communication was wrong. So you say okay, but in that case, the same like communicated a lot of people came in. So maybe something is wrong in my in my scheme itself, right. So use Developing this hypothesis, again, taking the past data and invalidating it and then you finding out what is the real reason, okay, then you make the modification, and, and relaunch it. That is how the learning happens. That is how mystery analysis happen. It’s it’s a very what I say creative process process, it’s a very iterative process and you, you you go through something, then to missteps and go forward. So, this happens only when the solution is being is being tried out in a new area. Or, for that matter, you are innovating, right, you are doing things for the first time. So that’s, that’s where it happens. Lovely, I think. So that is extremely, extremely helpful. I loved understanding the whole mystery analysis process. And I would I don’t know, for the listeners or not, but I would my say we’ll definitely use it more often. Okay, so one thing I want to put forward here is that if you if you companies which want to be innovative, right, and want to do things for the first time, and to use what is known as what Elon Musk comes the thinking through first principles, this is the method, this is the method of how do you think and build up a new practice, do something that nobody has done before, right? Or you try to let’s even if you try to implement something from let’s say, somebody else in the industry, you still have to apply this process because the CRT and the party ensures that the solution has been thought through and the problem has been well defined in your environment. Okay, you implement and then you find out, okay, didn’t get the intended results? Did you have all these setups to understand what went wrong and such an organisation? If the develop this entire scientific process of, you know, defining a problem, defining the solution, doing the practical implementation and then learning from it? There is no way anybody can beat this organisation, because this is what the real learning organisation is all about. But, I mean, I know this might sound like a very, To quickly summarize for our listeners. If you want to solve a chronic problem and develop a rebost solution, there are four important steps First Step is usage of CRT and FRT to built strong logical foundation of solution that comes up in the mind of people. This steps helps in killing bad solutions The second Step is to clearly define how causality of your solution leading to the positive effect will be established with proper measurement. Establish the control groups using different methods. At this stage it is also important to establish the ladder of intermediate effects. Think of an intermediate effect which is only unique to your solution. Then third step is to check for mystery by analysing the results, Look for where the deviation happened in the of proposed intermediate effect and their original estimates, Develop likely hypothesis and go back to step 1 to again kill ideas which lack logical consistency.