Archive

Posts Tagged ‘Statistics’

Statistical Analysis of Password Strength via Gawker’s Leaked Database

December 15th, 2010

This past weekend, Gawker Media was hacked and its user account database was leaked online. The database contained about 1.3 million rows of information containing usernames, e-mail addresses, and passwords (encrypted via DES). This security breach is unfortunate for people whose information is contained within that database, but the silver lining is that it provides a rare opportunity for statistics nerds like me to analyze some otherwise completely unobtainable data.

Because the passwords were encrypted using such an out-of-date scheme (tsk, tsk, Gawker), about 200,000 of the passwords contained in the database have been decrypted. Of course, the passwords that were cracked were relatively weak. For example, all 2641 accounts that used some trivial modification of “password” or “querty” as their password were of course decrypted. In this post I will look at trends in which users’ passwords were cracked to gain insight into which users do and do not create strong passwords.

It should of course be made clear that, because this data comes from a single database, the results that follow may not be representative of the population as a whole, but rather may be skewed by the fact that people with Gawker accounts are generally more “techy” than the average internet user.

Preliminaries: Cleaning Up the Database

The database of course had to be significantly cleaned before it could be of too much use statistically, so some of the numbers here may differ slightly from the raw numbers you see from news outlets or if you download the raw database yourself. The numbers here are the result of removing any incomplete rows from the database (i.e., rows missing a password, e-mail address, or both) and removing any accounts that were clearly created by SPAMbots (I’m only interested in the password strength of real users).

Also, I will only look at accounts that contain an e-mail address with a domain that was registered in the database at least 50 times. This restriction is in place partly because it is extremely difficult to compute any sort of meaningful statistics on something with a sample size that is much smaller than 50, and it is partly due to the fact that Gawker doesn’t require verified e-mail addresses (so 46993 of the 52593 domain names listed in the database were used by exactly one person, many of which are clearly fake and/or for SPAM).

After making the aforementioned “fixes” to the database, there are 412670 accounts, 157794 (38.2%) of which had their password decrypted.

Password Strength by Domain Name

The following table displays the 10 most frequently-occurring domain names used for e-mail addresses in the database along with how many users of the domain had their password cracked.

Domain Total Accounts Decrypted Passwords Decryption %
gmail.com 158031 50530 32.0%
yahoo.com 94147 40964 43.5%
hotmail.com 66752 27332 40.9%
aol.com 17534 8151 46.5%
comcast.net 7222 2801 38.8%
msn.com 5544 2250 40.6%
mac.com 4951 1750 35.3%
sbcglobal.net 3896 1667 42.8%
hotmail.co.uk 3204 1476 46.1%
verizon.net 2211 860 38.9%

The following table shows the z-values associated with the statistical test that the two given domains have the same proportion of users with strong passwords. Differences that are statistically significant at the α = 0.01 level are in bold. Click on a z-value to see a normal distribution showing the associated p-value. Notice in particular that gmail.com users have stronger passwords than users of any of the other top-10 domain names, while aol.com and hotmail.co.uk users have the weakest passwords.

Yahoo Hotmail AOL Comcast MSN Mac SBC HotmailUK Verizon
GMail 58.28 40.84 38.65 12.10 13.48 5.00 14.27 16.89 6.92
Yahoo -10.26 7.29 -7.81 -4.27 -11.31 -0.89 2.87 -4.33
Hotmail 13.23 -3.55 -0.53 -7.74 2.27 5.75 -1.93
AOL -11.09 -7.70 -13.94 -4.19 -0.44 -6.75
Comcast 2.06 -3.85 4.11 6.98 0.09
MSN -5.52 2.14 5.00 -1.37
Mac 7.14 9.67 2.88
SBC 2.77 -2.97
HotmailUK -5.24

Educational Institutions

Not surprisingly, users who entered an e-mail address from an educational institution typically had stronger passwords than the general population. Of the 2092 users who provided a college or university-based e-mail address, only 697 (33.3%) were decrypted. This proportion is significantly lower than the corresponding proportion for the general population (z = 4.64, p < 0.001).

However, two universities stood out as having particularly weak passwords: of the 56 users who used a University of Texas e-mail address, 27 (48.2%) had their password decrypted, and similarly 101 (45.1%) of 224 New York University passwords were decrypted.

ISP-Provided E-Mail Users

Users who used an e-mail address provided to them by their ISP (such as something@comcast.net) typically had weaker passwords than the general population, a fact that can perhaps be explained by the fact that tech-unsavvy folks are less likely to go out and get a new e-mail address for themselves at a place like GMail. Of the 31667 users who provided an ISP-based e-mail address, 13053 (41.2%) of them had their password decrypted. This proportion is significantly higher than the corresponding proportion for the general population (z = -11.36, p < 0.001).

E-Mail Addresses with Typos

Also unsurprisingly, users who entered an obvious typo in their e-mail address were much more likely to have a weak password than people who entered their e-mail address correctly (by “obvious typo” I basically mean an e-mail address containing a typo of a common domain name, such as “fred@yahoo,com” or “fred@hotmail”). Of the 530 users with a typo in their e-mail address, 280 (52.8%) had passwords that were decrypted. This proportion is significantly higher than the average (z = -6.87, p < 0.001).

Password Strength by Country

The following table shows the strength of user passwords based on the country associated with their e-mail address. Of course some e-mail addresses provide no information about the user’s country, so domains that serve a largely international market (such as gmail.com, mac.com and aim.com) are excluded from this analysis.

Country Total Accounts Decrypted Passwords Decryption %
India 3129 1448 46.3%
United Kingdom 6874 3057 44.5%
China 1411 600 42.5%
Canada 2825 1160 41.1%
United States 30891 12507 40.5%
Germany 1378 484 35.1%
Russia 2223 533 24.0%

So Russia and Germany are the big winners when it comes to password strength, while India and the United Kingdom seem to have the weakest passwords. The following table shows the z-values associated with the statistical test that the two given countries have the same proportion of users with strong passwords. Differences that are statistically significant at the α = 0.01 level are in bold. Click on a z-value to see a normal distribution showing the associated p-value.

UK China Canada US Germany Russia
India -1.67 -2.32 -4.03 -6.26 -6.94 -16.62
UK -1.31 -3.06 -6.05 -6.37 -17.16
China -0.88 -1.49 -3.97 -11.72
Canada -0.57 -3.67 -12.73
United States -3.95 -15.37
Germany -7.18

Attached below is an Excel Spreadsheet containing significantly more detailed information than the snippets contained in this post (though of course all passwords, e-mail addresses and personally-identifiable information has been removed).

Download: Gawker Database Statistics [Excel spreadsheet]

P-Value Calculators and Graphers in Javascript

September 5th, 2010

There are a lot of online tools out there for computing p-values and test statistics associated with common statistical distributions such as the normal or Student’s t-distributions. Unfortunately, most of them are either ad-ridden or powered by Java (and hence slow to initially load and finicky when it comes to which browsers they work with). So one of my summertime projects this year was to create a website that solves both of those problems:

The website computes p-values and test statistics in real-time via javascript (and thus does not need Java or any other plug-in). The computations themselves are fairly straightforward and are performed via the trapezoid rule. The graphic on the right is composed of a static PNG that displays the appropriate distribution. The distribution’s image is transparent under the graph and opaque above the graph, which makes it easy to display the p-value graphically – the light blue area is actually just a blue rectangle that is drawn beneath the distribution’s image.

Additionally, through the magic of PHP the tool automatically creates a URL that links to the current computation (and thus makes it much more citable). So, for example, if you want to know the T-value that corresponds to a right-tailed test with 12 degrees of freedom and a p-value of 0.1, you could simply click here.

Anyway, if you’re a nerd like me then enjoy it and of course feel free to leave any feedback/suggestions that you might have.

Keep the "Info" Before the "Graphic"

November 13th, 2009

The term “infographic” is a ridiculous little buzzword that really took off on the internet sometime last year. It used to refer to genuinely useful things like subway maps and blueprints. Recently, however, the term has come to mean “an obnoxiously oversized image that has numbers on it”. My problem isn’t with infographics like these ones that just display some fun, meaningless information is a visual way, or this one that displays a phenomenon that is inherently visual. My beef is with infographics that reduce a variety of related statistics to an oversized mess of overlapping graphs and charts that are (purposely or otherwise) misleading.

This post will present four rules that infographic designers, if they decide that they absolutely must make an infographic, should always follow (but often don’t). To get the ball rolling, let’s consider an example that made its way around the internet just a couple of weeks ago (source):

american-2009-season-ratings

American 2009 Season Premieres and Averages to Date (click to enlarge)

We are told that the above infographic depicts the US viewership for a variety of shows during their premiere (light red) and on average since they began their 2009 season (dark red). However, I have two main problems with the image, and they’re both problems that are prevalent throughout many infographics and can easily be solved by just using a simple bar graph.

1. Infographics should not require horizontal scrolling. The above infographic is 3133 pixels wide, which means there is no consumer-available monitor in the world capable of displaying the entire image on one screen without scrunching it down. This is apparently exactly what infographic makers want, since they all seem to subscribe to the school of thought that dictates their image deserves 45 inches of horizontal viewing space. This would be fine if infographics were readable when zoomed out, but by their very nature they almost never are.

Computer monitors were not meant to view posters. If you want to make the image high-resolution enough that it can be printed out as a poster then it should be created as a vector graphic, not a raster graphic. If you still insist that your infographic should be a monstrously large bitmap, make it readable from a zoom level that will fit on standard monitor resolutions.

Some other popular infographics that suffer from this problem are the new auto industry breakdown, weight of the world, and the first 100 days.

2. Two-dimensional figures should never be used to compare linear data. The above infographic compares the number of people watching different shows, so why are circles being used to represent the data? What represents the number of viewers — the radius of the circle or the area of the circle? The source doesn’t tell us, so we have no way of appropriately assessing how many more people are viewing NCIS: Los Angeles than The Good Wife. If it’s the radius of the circle, NCIS appears to have about 5% more viewers. If it’s the area of the circle then it’s probably over 10% (and the discrepancy gets much larger if you compare shows that are farther apart).

Furthermore, even if we were told whether it’s the radii of the circles or their areas that we should be looking at, there’s still a problem. If the radii are what are being compared, then the visual is misleading because the differences in areas cause the relative differences to appear larger than they actually are. If the areas are what are being compared, then it should be noted that people just plain suck at visually comparing areas. By looking at the above image (and not getting out a ruler or anything) can you tell which circles have about half as much area as the NCIS: Los Angeles circle? Can you tell how much higher the viewership of The Good Wife is than that of Glee? I certainly can’t, at least not quickly.

InfomationIsBeautiful.net is a particularly notorious violator of this rule, as these three examples show: deadliest drugs, how safe is the HPV vaccine?, reduce your chances of dying in a plane crash (scroll down to the “bad month” and “the odds” sections). What’s worse is they aren’t even consistent with whether it’s the areas of the circles or the radii of the circles they’re comparing.

Problems #1 and #2 can both be rectified by simply turning the data into a bar graph. A plain old-fashioned bar graph. Voila:

American 2009 Season Premieres and Averages to Date (easier to read)

American 2009 Season Premieres and Averages to Date (easier to read)

The above bar graph doesn’t need to be zoomed in to be read, it makes it easier to compare the relative viewership of each show, and it actually contains more data than the previous infographic thanks to the labels on the vertical axis.

The next example (source) supposedly explains how and why low-cost airlines are able to offer flights that are so much cheaper than other airlines. It made its rounds this last spring during recession fever, when anything that had anything to do with something being cheap was instantly popular. While it does not suffer from problem #1 above (since it is readable when zoomed out), it suffers from two instances of problem #2 as well as multiple other problems.

How come airlines are so cheap?

How come cheap airlines are so cheap? (click to enlarge)

3. Infographics (and everything else) should be about substance over style. While there’s no denying that the above infographic is pretty, does it actually tell us anything? Beyond the myriad of small problems such as the average fare of Southwest flights including cents when none of the other numbers do, the misspelling of “Aer Lingus” and “maintenance”, and the mysterious 43% “total advantage” at the bottom that seems to pop out of nowhere, the infographic at its core doesn’t even make sense.

As the infographic itself says, low-cost airlines generally don’t do long-haul flights; they focus on short point-to-point routes. So why are their average fares being compared to the average fares of the likes of British Airways, who regularly do intercontinental flights? Doesn’t it make sense that travel distance makes more of a contribution to the price of the flight than whether or not tickets are sold primarily online? Average fare per kilometer travelled would make more sense to compare, though it would still be misleading because take-off and landing are disproportionately expensive.

Another recent offending infographic that just simply doesn’t say a thing is the $400 million club, which notes that Transformers: Revenge of the Fallen is only the ninth movie in history to gross more than $400 million at the box office in the US during its theatrical run. The infographic then compares the other eight movies, which of course are juggernauts like Star Wars and Titanic. The problem is that none of the figures are adjusted for inflation. If you scale the numbers properly, Transformers: Revenge of the Fallen actually comes in as about the 65th highest-grossing movie. Impressive, sure, but to say that the infographic is misleading is an understatement.

I will finish by presenting a graphic that ran on NewsWeek.com that shows obesity and “life evaluation” trends over the last year or two. It’s debatable whether or not it falls into the category of what most people would consider an “infographic”, but it perfectly illustrates a core problem with them.

Obesity infographic

4. Be careful with your data. Just making your graphic pretty doesn’t give you free reign to ignore basic statistical principles when presenting data. In the above graphic, the left graph shows two lines — one showing how many people have BMI less than or equal to 30 in a given month and one showing how many people have BMI over 30 in a given month. I have a news flash for you, NewsWeek: one of those lines is redundant. Not only that, but the redundant second line manipulates the reader by giving the false impression that the number of obese people is converging toward the number of non-obese people. Nevermind the fact that the vertical scale is completely out of whack and it jumps a vertical distance of 46.4% in the same amount of space that is used to represent about a 2.5% jump elsewhere.

I’m willing to bet that the vertical scale on the right graph is completely out of whack too, but it’s a little difficult to tell since they don’t tell you what percentages any of the intermediate y-values correspond to. On the blue “struggling” line, we are given a value of 48.4% on the left edge of the graph and a value of 49.6% at the right edge of the graph at a nearly identical height. Are we supposed to be able to tell how high and low the peaks in the middle of the graph are based on that? Does the blue line get as low as 40%? 35%? 30%? Would labels along the vertical axis (similar to the bar graph I showed above) really have detracted from the desired aesthetic too much?

So if you have a set of data that you wish to convey graphically, please first consider whether or not it can be presented by a simple bar graph or line graph. If it can, don’t try to make it more complicated than that. If it can’t, at least make sure that the information is the motivating factor in your decisions. If the layout ends up dictating how you present your data, you’ve got your priorities backward.

Tags:

IMDb Movie Ratings Over the Years

October 9th, 2009

It’s time for a random dose of statistics courtesy of The Internet Movie Database. Let’s consider all movies that have been released theatrically over the last 60 years and see whether there is a trend in their perceived quality over time. That is, do new movies generally receive higher or lower scores on IMDb than old movies?

Before looking at the numbers though, we need some rules to clarify what types of movies we are considering:

  • We only consider theatrically-released films — no straight-to-video movies or TV movies.
  • Short films that were released theatrically (such as Pixar’s Presto) are included.
  • We only consider movies that have received 1000 or more votes. This restriction is to prevent movies with only a handful of votes from skewing the results too much.
  • The theatrical release date of the movie must have been at least as recent at 1950.

IMDb contains 10034 movies that satisfy the above criteria. The average score (on a scale of 1 to 10) of those movies is 6.38 and the median score is 6.6. The average score per release year is given by the following graph:

IMDb Ratings

As you can see, older movies (1950 – 1975) have abnormally high scores, as do very recent movies (2000 – 2009). These differences are indeed statistically significant. For example, the p-value associated with the test that the mean score in 1950 is the same as the mean score in 1989 is less than 10-19. The p-value associated with the test that the mean score in 2008 is the same as the mean score in 1989 is about 0.0021. Other nearby years give similar p-values.

So this tells us that, in general, particularly old movies receive the highest scores, followed by newly-released movies, followed by “semi-old” movies from the 1980′s and 1990′s. So why the differences? Were movies from the 1980′s really just that bad? Possibly, but the more likely explanation is that movies from the 1950′s  through 1970′s have artificially higher scores because people don’t generally go back and watch the crummy movies of the last generation, so they get forgotten and do not have 1000 votes on IMDb. Will people be watching Disaster Movie in forty years? I sure hope not.

On the other hand, particularly recent movies tend to draw a fair amount of hype and fanboyism. Remember when The Dark Knight had a score of 9.8 and was at #1 on the IMDb top 250? Now, one year later, it has a score of 8.9 and is located at #9 on the top 250. It will likely dwindle a little further down over the coming years as well.

The Best and Worst of Each Year

While we’re looking at ratings of movies over the years, I suppose I might as well provide a list of the best and worst movie of each year (based on the votes of IMDb users), since such a list is not available on the IMDb website itself to my knowledge. Keep in mind that, as before, only movies with 1000 or more votes are considered. Enjoy!

Year Best Worst
1950 Sunset Blvd. Destination Moon
1951 Strangers on a Train Flying Padre: An RKO-Pathe Screenliner
1952 Singin’ in the Rain Jack and the Beanstalk
1953 Duck Amuck Robot Monster
1954 Rear Window Jail Bait
1955 Nuit et brouillard Bride of the Monster
1956 The Killing The Conqueror
1957 12 Angry Men Beginning of the End
1958 Vertigo The Screaming Skull
1959 North by Northwest Yusei oji
1960 Psycho Ein Toter hing im Netz
1961 Divorzio all’italiana The Beast of Yucca Flats
1962 Lawrence of Arabia Eegah
1963 The Great Escape The Skydivers
1964 Dr. Strangelove or: How I Learned to Stop Worrying and Love the Bomb The Starfighters
1965 Per qualche dollaro in più Monster a-Go Go
1966 Il buono, il brutto, il cattivo. Night Train to Mundo Fine
1967 Cool Hand Luke The Hellcats
1968 C’era una volta il West Girl in Gold Boots
1969 Le chagrin et la pitié Five the Hard Way
1970 Mihai Viteazul Hercules in New York
1971 12 stulyev The Touch of Satan
1972 The Godfather Night of the Lepus
1973 The Sting Gojira tai Megaro
1974 The Godfather: Part II The Bat People
1975 Hababam sinifi Zaat
1976 Tosun Pasa Track of the Moon Beast
1977 Saban Oglu Saban The Incredible Melting Man
1978 Kibar Feyzo Laserblast
1979 Apocalypse Now Angels’ Brigade
1980 Star Wars: Episode V – The Empire Strikes Back L’uomo puma
1981 Raiders of the Lost Ark Le lac des morts vivants
1982 Vincent Megaforce
1983 Jaane Bhi Do Yaaro Los nuevos extraterrestres
1984 Balkanski spijun Ator l’invincibile 2
1985 Esperando la carroza Final Justice
1986 Aliens Zombie Nightmare
1987 L’homme qui plantait des arbres Leonard Part 6
1988 Nuovo cinema Paradiso Hobgoblins
1989 Ilha das Flores R.O.T.O.R.
1990 Goodfellas The Final Sacrifice
1991 The Silence of the Lambs Cool as Ice
1992 Reservoir Dogs Meatballs 4
1993 Schindler’s List Barschel – Mord in Genf?
1994 The Shawshank Redemption Tangents
1995 The Usual Suspects Dis – en historie om kjærlighet
1996 Paradise Lost: The Child Murders at Robin Hood Hills Merlin’s Shop of Mystical Wonders
1997 Masumiyet Pocket Ninjas
1998 American History X Die Hard Dracula
1999 Fight Club The Underground Comedy Movie
2000 Memento The Tony Blair Witch Project
2001 The Lord of the Rings: The Fellowship of the Ring Glitter
2002 Cidade de Deus Ben & Arthur
2003 The Lord of the Rings: The Return of the King From Justin to Kelly
2004 Eternal Sunshine of the Spotless Mind Superbabies: Baby Geniuses 2
2005 Babam Ve Oglum Troppo belli
2006 Kiwi! Pledge This!
2007 Heima Ram Gopal Varma Ki Aag
2008 The Dark Knight Disaster Movie
2009 (so far) Inglourious Basterds Jonas Brothers: The 3D Concert Experience

Downloads: