We use cookies to ensure you have the best browsing experience on our website. Please read our cookie policy for more information about how we use cookies.
Can't seem to direct this at AllisonP but if you see this, please respond.
Your definition of quartiles is certainly a valid definition. I am also certain it can be found in any undergraduate text book on stats, just as you claim. That said, this particular definition for the quartile and IQR was derived from descriptions of continuous distributions and does not actually apply to these discrete cases. There is a similar issue with the standard deviation tutorial, in which you define the standard deviation of a discrete data set without Bessel's correction (which is used in the NIST standard definition). If the goal of these tutorials are, as claimed, to prepare the user for positions in data science, it behooves you to adhere to the standards of the field.
Consider the following
Suppose you were drawing numbers from a hat. Quartile boundaries SHOULD be defined such that there will be about a 25% probability that a value you draw falls between Q1 and Q2, or Q2 and Q3, etc... (hence the name quartile)
Now, suppose that the numbers you are drawing from this hat are decimals with infinite precision (meaning that the numbers drawn from the hat are from a continuous distribution). The quartile range is naturally exclusive because the probability of drawing a number that exactly equals a quartile boundary is precisely zero. Your definition of quartile boundaries applies perfectly well in this situation because you can guarantee that every value drawn is unique.
Now, what if the hat were filled with the values from test case 4.
So there are 55 numbers in the hat, seven 10s, 15 20's and so on.
We charge ahead naively using our understanding of continuous distributions. We find the median and exclude its VALUE. We then find the median of the remaining data which gives us the following
The median Q2 is obviously 30.
Q1 is the median of all values < 30, which is 20
Q3 is the median of all values > 30, which is 50
so the IQR is reported as 50 – 20 = 30.
this means that the quartiles are [,20) , (20, 30) , (30, 50) , (50,] ...
But there is no data on our interval of (20,30) or (50,] ! suppose I were a data scientist in the real world and I defined my quartiles like this and I asked “what values do we see in quartile 4?”, I would find NONE! I have completely lost the fact that almost a quarter of my data is the number 50 ! This alone is a serious problem.
But it’s even worse than that! Suppose there was just one extra number in our hat, 80. Based on our definition so far all the boundaries will remain unchanged Now suppose I ask, “are there any outliers in my data?”. I look in quartile 4 and I find the number 80. I then ask, is this an outlier?
The common upper boundary for outliers is known as the "upper fence" and is defined as Q3 + 1.5*IQR. This is the common choice because it bounds the upper 25% of data beyond Q3. For normally distributed data you want the upper fence to fall just below 3 standard deviations from the mean (meaning that the probability of an event falling outside the upper fence is less than 1%).
The mean of test case 4 with a 80 tacked on the end it is about 31.9 with a standard deviation of about 15.0. This means that the upper fence is going to be 50 + 1.5 * 30 = 95. The 3 standard deviation mark falls at 77.1! four standard deviations is about 92 (and remember the upper fence is supposed to fall just before three standard deviations).
So using our initial IQR definition this 80 would NOT be an outlier even though it falls well above 3 standard deviations beyond the mean. Clearly this quartile definition fails in its purpose.
What if we define the quartiles such that the number of values in each quartile is about 25%?
The boundaries will fall on every 14th number (55 = 13+1+13+1+13+1+13), and so Q1 is 20, Q2 is 30 (the median) and Q3 is 40. This makes the IQR 20
The quartile ranges are defined to be inclusive, so that the intervals are [,20] , [20, 30] , [30, 50] , [40,], and none of these intervals have zero entries.
If we tack on an 80 as before the boundaries can now change slightly to compensate! Q3 becomes 42.5 and the IQR increases to 22.5, which sets the upper fence at 76.25. three standard deviations beyond the mean becomes 77.1.
Under this definition 80 is correctly identified as an outlier as it is not only beyond the upper fence but beyond 3 standard deviations!
This is the difference between defining exclusive boundaries on the value vs inclusive boundaries on the index. The former is fine for continuous distributions but it is NOT applicable to discrete cases!
Again if the purpose of this tutorial is to prepare people for data science you need to adheer to the standards and practices of the field. I do not want to refer my colleagues to this site if that isn't the case. Check the software that's available. I think you will be hard pressed to find any open source libraries that calculate quartiles and IQRs in the fashion you have described.
Cookie support is required to access HackerRank
Seems like cookies are disabled on this browser, please enable them to open this website
Day 1: Interquartile Range
You are viewing a single comment's thread. Return to all comments →
Can't seem to direct this at AllisonP but if you see this, please respond.
Your definition of quartiles is certainly a valid definition. I am also certain it can be found in any undergraduate text book on stats, just as you claim. That said, this particular definition for the quartile and IQR was derived from descriptions of continuous distributions and does not actually apply to these discrete cases. There is a similar issue with the standard deviation tutorial, in which you define the standard deviation of a discrete data set without Bessel's correction (which is used in the NIST standard definition). If the goal of these tutorials are, as claimed, to prepare the user for positions in data science, it behooves you to adhere to the standards of the field.
Consider the following
Suppose you were drawing numbers from a hat. Quartile boundaries SHOULD be defined such that there will be about a 25% probability that a value you draw falls between Q1 and Q2, or Q2 and Q3, etc... (hence the name quartile)
Now, suppose that the numbers you are drawing from this hat are decimals with infinite precision (meaning that the numbers drawn from the hat are from a continuous distribution). The quartile range is naturally exclusive because the probability of drawing a number that exactly equals a quartile boundary is precisely zero. Your definition of quartile boundaries applies perfectly well in this situation because you can guarantee that every value drawn is unique.
Now, what if the hat were filled with the values from test case 4.
10 40 30 50 20 10 40 30 50 20 1 2 3 4 5 6 7 8 9 10
So there are 55 numbers in the hat, seven 10s, 15 20's and so on. We charge ahead naively using our understanding of continuous distributions. We find the median and exclude its VALUE. We then find the median of the remaining data which gives us the following
The median Q2 is obviously 30. Q1 is the median of all values < 30, which is 20 Q3 is the median of all values > 30, which is 50
so the IQR is reported as 50 – 20 = 30.
this means that the quartiles are [,20) , (20, 30) , (30, 50) , (50,] ... But there is no data on our interval of (20,30) or (50,] ! suppose I were a data scientist in the real world and I defined my quartiles like this and I asked “what values do we see in quartile 4?”, I would find NONE! I have completely lost the fact that almost a quarter of my data is the number 50 ! This alone is a serious problem.
But it’s even worse than that! Suppose there was just one extra number in our hat, 80. Based on our definition so far all the boundaries will remain unchanged Now suppose I ask, “are there any outliers in my data?”. I look in quartile 4 and I find the number 80. I then ask, is this an outlier?
The common upper boundary for outliers is known as the "upper fence" and is defined as Q3 + 1.5*IQR. This is the common choice because it bounds the upper 25% of data beyond Q3. For normally distributed data you want the upper fence to fall just below 3 standard deviations from the mean (meaning that the probability of an event falling outside the upper fence is less than 1%).
The mean of test case 4 with a 80 tacked on the end it is about 31.9 with a standard deviation of about 15.0. This means that the upper fence is going to be 50 + 1.5 * 30 = 95. The 3 standard deviation mark falls at 77.1! four standard deviations is about 92 (and remember the upper fence is supposed to fall just before three standard deviations).
So using our initial IQR definition this 80 would NOT be an outlier even though it falls well above 3 standard deviations beyond the mean. Clearly this quartile definition fails in its purpose.
What if we define the quartiles such that the number of values in each quartile is about 25%?
The boundaries will fall on every 14th number (55 = 13+1+13+1+13+1+13), and so Q1 is 20, Q2 is 30 (the median) and Q3 is 40. This makes the IQR 20
The quartile ranges are defined to be inclusive, so that the intervals are [,20] , [20, 30] , [30, 50] , [40,], and none of these intervals have zero entries.
If we tack on an 80 as before the boundaries can now change slightly to compensate! Q3 becomes 42.5 and the IQR increases to 22.5, which sets the upper fence at 76.25. three standard deviations beyond the mean becomes 77.1.
Under this definition 80 is correctly identified as an outlier as it is not only beyond the upper fence but beyond 3 standard deviations!
This is the difference between defining exclusive boundaries on the value vs inclusive boundaries on the index. The former is fine for continuous distributions but it is NOT applicable to discrete cases!
Again if the purpose of this tutorial is to prepare people for data science you need to adheer to the standards and practices of the field. I do not want to refer my colleagues to this site if that isn't the case. Check the software that's available. I think you will be hard pressed to find any open source libraries that calculate quartiles and IQRs in the fashion you have described.