Advanced Stats: The Non-Conference Scheduling Debate

By most measures, the last two years of Villanova basketball have been wildly successful. After a 2013-2014 campaign in which they emerged as the class of the reborn Big East basketball conference (triumphing in a tough pre-season tournament and then winning 28 games while shredding [most of] the conference competition), they followed it up with a 32 win pre-NCAA tournament campaign in 2014-2015 - the most in program history.

The Wildcats' record over the last two seasons stands at 62-5 (0.925), with some of the finest adjusted overall ratings (per Kenpom) of Jay Wright's tenure.

But said accomplishments mean little when stacked against their far less illustrious post-season campaigns over the same period. With multiple early postseason exits (though I can't complain about a Big East Tournament title), questions about Villanova's ability to break through this unfortunate cycle have been stacking up.

One debate that comes up quite often here in our blog space is the need for strong opponents in the non-conference portion of the schedule. Since the fiery death of the overstuffed, top heavy, but [let's be real] great old Big East conference, the more-consistent-but-lacking-in-programs-perceived-to-be-elite Jesuit Big East that rose from the ashes has borne criticism for not helping to properly prepare Villanova for the more rigorous tests of the postseason. As the reasoning goes, the teams of the new Big East are simply not athletic, tall, or good enough to test Villanova - a team, it must be said, that has stood head and shoulders (during the regular season) above the other teams  in the conference. The only opportunity for loading the schedule with said teams, by the same reasoning, is the non-conference portion.

It's a good theoretical argument - beating up on unworthy teams is more than likely not the best way to prepare for the gauntlet of single elimination competition. Despite the shade such a view throws at the teams currently in the Big East (an ‘insult' I honestly don't think they deserve. It's a collection of very solid teams that have had some truly unfortunate postseason performances so far), I have no real problem with the argument. I also have no problem with the opposite side pointing out that many teams with tough non-conference schedules also bowed out early (which is true), and that the arguably best conference in the country last year (a rating and reputation they had to build in the non-conference portion, by the way) - the Big 12 - had almost all of its top seeds out before the 2^nd weekend.

No, what ultimately bothers me about the argument from both sides is that it continues, with no real effort to back up the arguments with facts from either side, by mostly trading retread opinions and examples. We get it, cupcakes are no good. We also get it, every team plays cupcakes. And yada, yada, yada.

So I endeavored to piece together some statistical ammo for both arguments, taking a big data approach to try and eliminate any biases in the collection. The main questions I sought to answer were - do teams that play tough schedules (conference and non-conference) do any better than teams that don't? Is it a notable effect, or mostly attributable to statistical noise? After all, the data only stretches back 14 seasons or so - it's a good sample, but ultimately not a huge one; only 56 of each seed have played in the last 14 tournaments.

Methodology

I didn't run regressions, linear/non-linear/whatever else on the data, as I honestly don't trust myself to do the math right there. All references to team rank are based on kenpom's adjusted Pythagorean ratings. Much of the analysis was run like this - order data from largest to smallest based on a number of factors (Non-conference NCAA teams faced, overall schedule toughness, etc.), and sample the highest and lowest 10%, 25%, 33%, and then compare. I also ran a made/missed analysis comparing various seeds that ‘made' or ‘missed' a certain round - how the schedules of 1 seeds that made the Sweet Sixteen compared to the schedules of 1 seeds that missed it - and so on. I'll explain the potential problems with this methodology, and what I did, if anything, to combat it later on in the article.

What We Know (Maybe)

I'll start with the clearest results, because I know most people's attention spans barely lasted past that opening. On the biggest possible scale, playing a tough non-conference schedule (drumroll, please...)

Works!

Check out a few quick-ish charts.

So here's the NCAA tournament teams of the last 14 years, organized by number of non-conference NCAA teams played:

Non-Conference NCAA Teams Played

	Top 10%	Bottom 10%
Expected Wins	132.59	48.38
Actual Wins	140	40
Differential	7.41	-8.37
	Top 25%	Bottom 25%
Expected Wins	308.38	144.07
Actual Wins	326	117
Differential	17.63	-27.07
	Top 33%	Bottom 33%
Expected Wins	398.43	207.05
Actual Wins	423	170
Differential	24.57	-37.05

To translate: the expected wins are aggregate wins expected by seed - the average performance of each seed in the tournament was calculated and used for this number. The actual wins are, obviously, the number of games the teams actually won. The top and bottom percentages refer to the slices taken from the top and bottom of each list; the top was the high end of referenced criteria, while the bottom is the low.

So, from a macro standpoint, playing as many strong teams in the non-conference portion of the schedule looks like a solid thing to do! In each case, the teams outperformed their expected wins - though the difference is generally small (each added total is approximately 5% of the total expected wins), while, on the other side, teams at the bottom end severely underperformed their expected win total (21-23%, depending on the slice).

Argument over!

The same effect appears when you take a look at ‘top-25 opponents faced in non-conference play.' While there's going to be some overlap between ‘Teams that played the most NCAA teams in NC play' and ‘Teams that played the most top-25 opponents in NC play,' there are some notably larger effects.

Top 25 Opponents faced NC

	Top 10%	Bottom 10%
Expected Wins	146.73	24.2
Actual Wins	165	19
Differential	18.27	-5.2
	Top 25%	Bottom 25%
Expected Wins	333.14	132.64
Actual Wins	373	120
Differential	39.86	-12.64
	Top 33%	Bottom 33%
Expected Wins	418.34	158.54
Actual Wins	464	141
Differential	45.66	-17.54

And, finally, here's the organization for Unique Top 25 opponents faced (it removes double ups in conference from the equation - playing a top 25 opponent in your conference twice only counts as one top-25 squad).

Unique Top 25 Opponents

	Top 10%	Bottom 10%
Expected Wins	146.79	31.14
Actual Wins	158	28
Differential	11.21	-3.14
	Top 25%	Bottom 25%
Expected Wins	329.11	80.86
Actual Wins	346	62
Differential	16.89	-18.86
	Top 33%	Bottom 33%
Expected Wins	426.41	119.18
Actual Wins	436	97
Differential	9.59	-22.18

Damn! That's noticeable.

It's tough to take these numbers at full face value, though. For starters: check out the disparity between expected wins for the top and bottom portions of these numbers. Especially when we look at the top and bottom 10%, it's enormous.

Most of this difference is obviously related to the ‘seeds' of the teams present on each end of the spectrum. It stands to reason that the 13-16 seeds of the world aren't going to play non-conference schedules as tough as the blue bloods and better programs that get the higher seeds. For example: when you order teams by non-conference NCAA teams played, 53 of the 89 teams at the bottom are seeded between 13 and 16. The average seed of the 89 teams at the top is 5.64; the bottom is 11.53.

Why does this matter? Mainly, it's related to the way the expected win column is calculated, and how rare it is for a low seeded team to win a game. This chart shows the expected wins, broken down by seed, for NCAA teams over the last 14 years (stretching back to the 2001-2002 season).

Seed	Expected Wins
1	3.285714
2	2.375
3	1.982143
4	1.535714
5	1.107143
6	0.892857
7	1.017857
8	0.821429
9	0.482143
10	0.553571
11	0.642857
12	0.607143
13	0.25
14	0.125
15	0.071429
16	0

See the potential problem? While every 8-15 seed will have an ‘expected win' value that will count toward the overall calculation, the ‘actual win' column is an either-or proposition. A fractional value will obviously be greater than 0 (aren't you glad I'm here, guys??), meaning that, in a small sample, it's entirely possible ‘expected wins' will far outstrip the low seeds that actually won a game. This likely contributed at least some to the negative values seen on the charts above for the ‘Bottom' portions - not to take anything away from positive values seen on the ‘Top' side, which are real and spectacular.

So I went through a similar exercise, while restricting the analysis to only top 12 seeds, and then again, while restricting the analysis to only top 4 seeds. The results, mirroring the tables that were shown above, can be seen below.

Top 12 Seeds: Non-Conference NCAA Teams Played

	Top 10%	Bottom 10%
Expected Wins	105.5	60.88
Actual Wins	101	56
Differential	-4.5	-4.87
	Top 25%	Bottom 25%
Expected Wins	261.45	178.61
Actual Wins	284	153
Differential	22.55	-25.61
	Top 33%	Bottom 33%
Expected Wins	338.12	232.25
Actual Wins	362	192
Differential	23.88	-40.25

Top 25 Opponents faced NC

	Top 10%	Bottom 10%
Expected Wins	113.25	57.77
Actual Wins	126	45
Differential	12.75	-12.77
	Top 25%	Bottom 25%
Expected Wins	267.02	162.54
Actual Wins	301	151
Differential	33.98	-11.54
	Top 33%	Bottom 33%
Expected Wins	345.45	222.68
Actual Wins	386	200
Differential	40.55	-22.68

Unique Top 25 Opponents

	Top 10%	Bottom 10%
Expected Wins	107.38	52.95
Actual Wins	110	40
Differential	2.62	-12.95
	Top 25%	Bottom 25%
Expected Wins	251.95	159.89
Actual Wins	276	141
Differential	24.05	-18.89
	Top 33%	Bottom 33%
Expected Wins	327.46	219.84
Actual Wins	339	206
Differential	11.54	-13.84

Top 4 Seeds: Non-Conference NCAA Teams Played

	Top 10%	Bottom 10%
Expected Wins	58.73	51.71
Actual Wins	57	48
Differential	-1.73	-3.71
	Top 25%	Bottom 25%
Expected Wins	137.79	120.32
Actual Wins	142	105
Differential	4.21	-15.32
	Top 33%	Bottom 33%
Expected Wins	179.86	159.18
Actual Wins	184	152
Differential	4.14	-7.18

Top 25 Opponents faced NC

x	Top 10%	Bottom 10%
Expected Wins	54.77	52.3
Actual Wins	60	50
Differential	5.23	-2.3
x	Top 25%	Bottom 25%
Expected Wins	134	126.8
Actual Wins	147	110
Differential	13	-16.8
x	Top 33%	Bottom 33%
Expected Wins	173.39	169.36
Actual Wins	185	150
Differential	11.61	-19.36

Unique Top 25 Opponents

x	Top 10%	Bottom 10%
Expected Wins	47.38	55.75
Actual Wins	56	51
Differential	8.63	-4.75
x	Top 25%	Bottom 25%
Expected Wins	127.13	131.95
Actual Wins	144	119
Differential	16.87	-12.95
x	Top 33%	Bottom 33%
Expected Wins	173.39	170.05
Actual Wins	194	162
Differential	20.61	-8.05

While somewhat muted, the positive showing for teams facing strong non-conference schedules and overall opponents is still there. Simply: teams that played a higher amount of very tough teams have performed better than teams that haven't, when it comes to NCAA tournament time.

Here's Where It Gets Confusing

So, debate over, right? Wrong. We're only 2,000 words in!

So we've seen the ‘effect' that playing strong teams out of conference, and in an overall sense, has on team's NCAA tournament results, right? A logical conclusion to draw from these results would be: the more good teams you play, the better suited you are for success in the NCAAs. Should be true across the board, right?

Not so fast. Check out the tournament success (overall, top 12 seeds, and top 4 seeds) of the top and bottom portions of NCAA teams ordered by number of NCAA teams and top 25 teams played in conference.

Top 4:

Conference Top 25 Opponents
	Top 10%	Bottom 10%
Expected Wins	53.19643	54.89286
Actual Wins	58	50
Differential	4.803571	-4.89286
	Top 25%	Bottom 25%
Expected Wins	124.5893	128.8929
Actual Wins	131	121
Differential	6.410714	-7.89286
	Top 33%	Bottom 33%
Expected Wins	166	170.8929
Actual Wins	169	162
Differential	3	-8.89286

Conference NCAA Teams Played
	Top 10%	Bottom 10%
Expected Wins	48.79	53.16
Actual Wins	44.00	50.00
Differential	-4.79	-3.16
	Top 25%	Bottom 25%
Expected Wins	124.13	129.86
Actual Wins	124.00	124.00
Differential	-0.13	-5.86
	Top 33%	Bottom 33%
Expected Wins	166.95	171.79
Actual Wins	162.00	169.00
Differential	-4.95	-2.79

Top 12:

Conference Top 25 Opponents
	Top 10%	Bottom 10%
Expected Wins	105.84	48.98
Actual Wins	113.00	45.00
Differential	7.16	-3.98
	Top 25%	Bottom 25%
Expected Wins	236.50	156.95
Actual Wins	225.00	154.00
Differential	-11.50	-2.95
	Top 33%	Bottom 33%
Expected Wins	321.61	224.07
Actual Wins	321.00	212.00
Differential	-0.61	-12.07

Conference NCAA Teams Played
	Top 10%	Bottom 10%
Expected Wins	102.68	55.84
Actual Wins	98.00	61.00
Differential	-4.68	5.16
	Top 25%	Bottom 25%
Expected Wins	256.21	153.02
Actual Wins	245.00	152.00
Differential	-11.21	-1.02
	Top 33%	Bottom 33%
Expected Wins	328.84	212.61
Actual Wins	315.00	207.00
Differential	-13.84	-5.61

All:

Conference NCAA Teams Played
	Top 10%	Bottom 10%
Expected Wins	141.64	15.59
Actual Wins	128.00	10.00
Differential	-13.64	-5.59
	Top 25%	Bottom 25%
Expected Wins	330.50	47.39
Actual Wins	317.00	49.00
Differential	-13.50	1.61
	Top 33%	Bottom 33%
Expected Wins	440.20	90.21
Actual Wins	436.00	94.00
Differential	-4.20	3.79

Conference Top 25 Opponents
	Top 10%	Bottom 10%
Expected Wins	136.70	16.95
Actual Wins	137.00	12.00
Differential	0.30	-4.95
	Top 25%	Bottom 25%
Expected Wins	322.66	64.57
Actual Wins	327.00	58.00
Differential	4.34	-6.57
	Top 33%	Bottom 33%
Expected Wins	421.05	96.16
Actual Wins	430.00	91.00
Differential	8.95	-5.16

In almost every case, teams that played more NCAA tournament teams in-conference had worse tournament results than those who played less. And, while it clearly remained somewhat helpful to play more top 25 teams in-conference than less, the effects of same became far more muted than what was seen in terms of both non-conference play and overall top 25 means played.

So what gives? If playing better teams prepares you better for the NCAA tournament, it really shouldn't matter whether they fall in the same conference as a team; good teams are good. Similarly, there are really odd results when checking average NC team rank, average kenpom Pythagorean rating for both non-conference and conference opponents, and various others of the 20 or so categories I ran comparisons for. Did not want to drown the article in tables, but if anyone's interested in taking at a look at the other data, please feel free to contact me.

In essence, the tables I showed you up top were the most conclusive; many of the other factors I checked were contradictory, or so close as to remain inconclusive. This is indicative of the many tricks you can pull with stats - only showing the audience what supports your own previously-arrived-at-conclusion, for example - but the biggest indicator for me is that there's a lot of noise when looking at this data.

Drawing 100% concrete conclusions is kind of dangerous - the margins are truly tiny.

To that end - the final thing I took a look at was a ‘make-miss' analysis that compared seeds that made or missed the various rounds. Again, a ton of noise here. Take a look at the top 4 seeds, in an analysis of ‘made/missed' the round of 32:

									Non-Conference Games				Total Unique Pre-Tournament
			NC NCAA Teams	Conf NCAA Teams	Unique NCAA Teams	Average Non-Tourney Oppo Pythag	Average NC Oppo Pythag (Regular Season)	Average Conference Oppo Pythag (Regular Season)	Top 25 Opponents	Top 50 Opponents	Mini Cupcakes (100-200)	Mega Cupcakes (201-350+)	Top 25 Opponents	Top 50 Opponents
Made Round of 32	1	56	4.45	8.07	8.89	0.672	0.576	0.728	1.95	3.59	3.64	3.54	4.52	8.52
Missed Round of 32	1	0	#DIV/0!	#DIV/0!	#DIV/0!	#DIV/0!	#DIV/0!	#DIV/0!	#DIV/0!	#DIV/0!	#DIV/0!	#DIV/0!	#DIV/0!	#DIV/0!
Made Round of 32	2	53	4.21	8.53	8.60	0.670	0.561	0.732	2.26	3.58	3.19	3.89	4.85	8.57
Missed Round of 32	2	3	4.33	10.00	9.67	0.661	0.540	0.719	1.67	3.33	2.00	4.67	5.00	9.33
Made Round of 32	3	49	3.51	8.73	8.14	0.662	0.536	0.737	1.73	2.96	3.33	4.31	4.67	8.16
Missed Round of 32	3	7	5.00	11.14	10.29	0.702	0.605	0.752	2.00	4.14	2.00	3.71	4.71	10.00
Made Round of 32	4	45	4.04	8.31	8.42	0.660	0.537	0.733	1.91	3.20	3.51	4.27	4.44	8.09
Missed Round of 32	4	11	3.00	9.00	8.00	0.651	0.513	0.735	1.45	2.82	3.27	4.82	4.45	8.09

The two and three seeds that lost in the first round actually had tougher schedules than those who didn't - which pokes some holes in the argument that playing better teams inoculates you against upsets.

A similar general phenomenon can be seen in the make/miss analysis of the Sweet Sixteen:

														Non-Conference Games								Total Unique Pre-Tournament
			NC NCAA Teams	Conf NCAA Teams		Unique NCAA Teams		Average Non-Tourney Oppo Pythag		Average NC Oppo Pythag (Regular Season)		Average Conference Oppo Pythag (Regular Season)		Top 25 Opponents		Top 50 Opponents		Mini Cupcakes (100-200)		Mega Cupcakes (201-350+)		Top 25 Opponents		Top 50 Opponents
Made Sweet Sixteen	1	48	4.58	8.23	9.13		0.678		0.578		0.737		2.02		3.67		3.63		3.56		4.65		8.77
Missed Sweet Sixteen	1	8	3.63	7.13	7.50		0.636		0.562		0.674		1.50		3.13		3.75		3.38		3.75		7.00
Made Sweet Sixteen	2	35	4.34	8.46	8.46		0.665		0.559		0.725		2.34		3.63		3.14		4.00		4.60		8.43
Missed Sweet Sixteen	2	21	4.00	8.86	9.00		0.677		0.562		0.743		2.05		3.48		3.10		3.81		5.29		8.90
Made Sweet Sixteen	3	33	3.42	8.88	8.03		0.661		0.529		0.738		1.82		2.88		3.12		4.55		4.85		8.15
Missed Sweet Sixteen	3	23	4.09	9.26	8.96		0.675		0.566		0.739		1.70		3.43		3.22		3.78		4.43		8.74
Made Sweet Sixteen	4	25	4.40	8.12	8.56		0.659		0.540		0.725		2.00		3.60		3.60		4.20		4.60		8.36
Missed Sweet Sixteen	4	31	3.39	8.71	8.16		0.658		0.526		0.740		1.68		2.74		3.35		4.52		4.32		7.87

While the 1-seeds that bowed out before the Sweet 16 look to have a pretty clearly weaker schedule than those that advanced, the same cannot be said for seeds 2-4. Ultimately, the schedule comparison works out very close to equal for both.

Here's the rest of the table:

														Non-Conference Games								Total Unique Pre-Tournament
			NC NCAA Teams	Conf NCAA Teams		Unique NCAA Teams		Average Non-Tourney Oppo Pythag		Average NC Oppo Pythag (Regular Season)		Average Conference Oppo Pythag (Regular Season)		Top 25 Opponents		Top 50 Opponents		Mini Cupcakes (100-200)		Mega Cupcakes (201-350+)		Top 25 Opponents		Top 50 Opponents
Made Elite Eight	1	37	4.46	8.00	9.03		0.674		0.577		0.732		1.97		3.65		3.81		3.54		4.51		8.76
Missed Elite Eight	1	19	4.42	8.21	8.63		0.668		0.573		0.722		1.89		3.47		3.32		3.53		4.53		8.05
Made Elite Eight	2	27	4.56	8.78	8.89		0.667		0.558		0.729		2.41		3.67		2.93		4.15		4.89		8.67
Missed Elite Eight	2	29	3.90	8.45	8.45		0.671		0.561		0.734		2.07		3.48		3.31		3.72		4.83		8.55
Made Elite Eight	3	16	3.44	9.00	8.31		0.658		0.508		0.747		1.75		2.81		2.88		4.94		5.25		8.25
Missed Elite Eight	3	40	3.80	9.05	8.45		0.671		0.559		0.735		1.78		3.23		3.28		3.95		4.45		8.45
Made Elite Eight	4	9	4.44	8.78	9.56		0.669		0.541		0.742		2.22		3.44		3.78		4.00		5.33		9.33
Missed Elite Eight	4	47	3.72	8.38	8.11		0.656		0.530		0.732		1.74		3.06		3.40		4.45		4.28		7.85
Made Final Four	1	21	4.29	8.00	8.67		0.676		0.569		0.743		2.10		3.76		3.86		3.86		4.71		8.71
Missed Final Four	1	35	4.54	8.11	9.03		0.669		0.580		0.719		1.86		3.49		3.51		3.34		4.40		8.40
Made Final Four	2	12	4.67	9.58	9.58		0.682		0.565		0.748		2.33		3.67		2.83		4.00		5.25		9.33
Missed Final Four	2	44	4.09	8.34	8.41		0.666		0.558		0.727		2.20		3.55		3.20		3.91		4.75		8.41
Made Final Four	3	6	3.17	9.67	8.33		0.674		0.504		0.778		1.83		2.67		2.00		5.17		6.33		9.17
Missed Final Four	3	50	3.76	8.96	8.42		0.666		0.549		0.734		1.76		3.16		3.30		4.12		4.48		8.30
Made Final Four	4	6	4.67	9.50	10.17		0.659		0.528		0.733		2.50		3.50		3.83		4.33		5.67		9.33
Missed Final Four	4	50	3.74	8.32	8.12		0.658		0.533		0.733		1.74		3.08		3.42		4.38		4.30		7.94
Made National Championship	1	13	4.23	7.92	8.62		0.671		0.566		0.740		2.08		3.62		3.69		4.15		4.54		8.38
Missed National Championship	1	43	4.51	8.12	8.98		0.794		0.579		0.725		1.91		3.58		3.63		3.35		4.51		8.56
Made National Championship	2	5	5.60	9.40	10.40		0.684		0.585		0.746		2.80		4.60		2.60		3.80		5.60		9.80
Missed National Championship	2	51	4.08	8.53	8.49		0.767		0.557		0.730		2.18		3.47		3.18		3.94		4.78		8.49
Made National Championship	3	4	3.25	11.00	9.00		0.692		0.509		0.807		1.75		2.75		2.00		5.00		6.75		10.00
Missed National Championship	3	52	3.73	8.88	8.37		0.765		0.547		0.733		1.77		3.13		3.25		4.17		4.52		8.27
Made National Championship	4	1	3.00	11.00	9.00		0.656		0.463		0.790		2.00		3.00		3.00		6.00		7.00		10.00
Missed National Championship	4	55	3.85	8.40	8.33		0.636		0.533		0.732		1.82		3.13		3.47		4.35		4.40		8.05

In Conclusion

There's definitely some solid evidence that beefing up your non-conference schedule with high end opponents is a good indicator for your team's NCAA tournament success - assuming you make it there, of course. There's also some solid-but-not-promising evidence that playing in a conference with a lot of NCAA tournament teams is a BAD indicator of your team's eventual NCAA tournament success - so take both data points as you will.

Mainly, any serious conclusions along these lines must be tempered. Even over the course of 14 years, these data sets are limited. While I'd much rather rely on conclusions drawn from the same than single case instances of confirmation bias among commenters, I ultimately don't feel confident in making too many of them (conclusions, that is). There's so much random variance in the NCAA tournament, and within the results I gathered in general.

A few bullets I feel confident in:

• More top 25 teams is a good thing, in conference or out.
• Scheduling cupcakes has pretty much no effect on a team's NCAA tournament chances, in high numbers or low.
• The teams above the top 25 really don't matter as much; the effect of more top 50, 75, and 100 games is pretty negligible, in terms of actual vs. expected NCAA tournament performance.
• The differences in scheduling between teams that bowed out early and teams that advanced, especially among top seeds, are so miniscule and sometimes contradictory to the general premise so as to be generally ignorable.

So, please find some real, actually data-based fuel for the general debate above. Feel free to continue to pointlessly hash it out in the comments until next year's NCAA championship win.

Also, if you want a look at the larger data set, again, feel free to contact me; can send it via email.