clock menu more-arrow no yes

Filed under:

Advanced Stats: The Non-Conference Scheduling Debate

New, 19 comments

Get your data for the debate here.

Howard Smith-USA TODAY Sports

By most measures, the last two years of Villanova basketball have been wildly successful. After a 2013-2014 campaign in which they emerged as the class of the reborn Big East basketball conference (triumphing in a tough pre-season tournament and then winning 28 games while shredding [most of] the conference competition), they followed it up with a 32 win pre-NCAA tournament campaign in 2014-2015 - the most in program history.

The Wildcats' record over the last two seasons stands at 62-5 (0.925), with some of the finest adjusted overall ratings (per Kenpom) of Jay Wright's tenure.

But said accomplishments mean little when stacked against their far less illustrious post-season campaigns over the same period. With multiple early postseason exits (though I can't complain about a Big East Tournament title), questions about Villanova's ability to break through this unfortunate cycle have been stacking up.

Do teams that play tough schedules do any better than teams that don't?

One debate that comes up quite often here in our blog space is the need for strong opponents in the non-conference portion of the schedule. Since the fiery death of the overstuffed, top heavy, but [let's be real] great old Big East conference, the more-consistent-but-lacking-in-programs-perceived-to-be-elite Jesuit Big East that rose from the ashes has borne criticism for not helping to properly prepare Villanova for the more rigorous tests of the postseason. As the reasoning goes, the teams of the new Big East are simply not athletic, tall, or good enough to test Villanova - a team, it must be said, that has stood head and shoulders (during the regular season) above the other teams  in the conference. The only opportunity for loading the schedule with said teams, by the same reasoning, is the non-conference portion.

It's a good theoretical argument - beating up on unworthy teams is more than likely not the best way to prepare for the gauntlet of single elimination competition. Despite the shade such a view throws at the teams currently in the Big East (an ‘insult' I honestly don't think they deserve. It's a collection of very solid teams that have had some truly unfortunate postseason performances so far), I have no real problem with the argument. I also have no problem with the opposite side pointing out that many teams with tough non-conference schedules also bowed out early (which is true), and that the arguably best conference in the country last year (a rating and reputation they had to build in the non-conference portion, by the way) - the Big 12 - had almost all of its top seeds out before the 2nd weekend.

No, what ultimately bothers me about the argument from both sides is that it continues, with no real effort to back up the arguments with facts from either side, by mostly trading retread opinions and examples. We get it, cupcakes are no good. We also get it, every team plays cupcakes. And yada, yada, yada.

So I endeavored to piece together some statistical ammo for both arguments, taking a big data approach to try and eliminate any biases in the collection. The main questions I sought to answer were - do teams that play tough schedules (conference and non-conference) do any better than teams that don't? Is it a notable effect, or mostly attributable to statistical noise? After all, the data only stretches back 14 seasons or so - it's a good sample, but ultimately not a huge one; only 56 of each seed have played in the last 14 tournaments.

Methodology

I didn't run regressions, linear/non-linear/whatever else on the data, as I honestly don't trust myself to do the math right there. All references to team rank are based on kenpom's adjusted Pythagorean ratings. Much of the analysis was run like this - order data from largest to smallest based on a number of factors (Non-conference NCAA teams faced, overall schedule toughness, etc.), and sample the highest and lowest 10%, 25%, 33%, and then compare. I also ran a made/missed analysis comparing various seeds that ‘made' or ‘missed' a certain round - how the schedules of 1 seeds that made the Sweet Sixteen compared to the schedules of 1 seeds that missed it - and so on. I'll explain the potential problems with this methodology, and what I did, if anything, to combat it later on in the article.

What We Know (Maybe)

I'll start with the clearest results, because I know most people's attention spans barely lasted past that opening. On the biggest possible scale, playing a tough non-conference schedule (drumroll, please...)

Works!

Check out a few quick-ish charts.

So here's the NCAA tournament teams of the last 14 years, organized by number of non-conference NCAA teams played:

Non-Conference NCAA Teams Played


Top 10% Bottom 10%
Expected Wins 132.59 48.38
Actual Wins 140 40
Differential 7.41 -8.37

Top 25% Bottom 25%
Expected Wins 308.38 144.07
Actual Wins 326 117
Differential 17.63 -27.07

Top 33% Bottom 33%
Expected Wins 398.43 207.05
Actual Wins 423 170
Differential 24.57 -37.05

To translate: the expected wins are aggregate wins expected by seed - the average performance of each seed in the tournament was calculated and used for this number. The actual wins are, obviously, the number of games the teams actually won. The top and bottom percentages refer to the slices taken from the top and bottom of each list; the top was the high end of referenced criteria, while the bottom is the low.

So, from a macro standpoint, playing as many strong teams in the non-conference portion of the schedule looks like a solid thing to do! In each case, the teams outperformed their expected wins - though the difference is generally small (each added total is approximately 5% of the total expected wins), while, on the other side, teams at the bottom end severely underperformed their expected win total (21-23%, depending on the slice).

Argument over!

The same effect appears when you take a look at ‘top-25 opponents faced in non-conference play.' While there's going to be some overlap between ‘Teams that played the most NCAA teams in NC play' and ‘Teams that played the most top-25 opponents in NC play,' there are some notably larger effects.

Top 25 Opponents faced NC


Top 10% Bottom 10%
Expected Wins 146.73 24.2
Actual Wins 165 19
Differential 18.27 -5.2

Top 25% Bottom 25%
Expected Wins 333.14 132.64
Actual Wins 373 120
Differential 39.86 -12.64

Top 33% Bottom 33%
Expected Wins 418.34 158.54
Actual Wins 464 141
Differential 45.66 -17.54

And, finally, here's the organization for Unique Top 25 opponents faced (it removes double ups in conference from the equation - playing a top 25 opponent in your conference twice only counts as one top-25 squad).

Unique Top 25 Opponents


Top 10% Bottom 10%
Expected Wins 146.79 31.14
Actual Wins 158 28
Differential 11.21 -3.14

Top 25% Bottom 25%
Expected Wins 329.11 80.86
Actual Wins 346 62
Differential 16.89 -18.86

Top 33% Bottom 33%
Expected Wins 426.41 119.18
Actual Wins 436 97
Differential 9.59 -22.18

Damn! That's noticeable.

It's tough to take these numbers at full face value, though. For starters: check out the disparity between expected wins for the top and bottom portions of these numbers. Especially when we look at the top and bottom 10%, it's enormous.

Most of this difference is obviously related to the ‘seeds' of the teams present on each end of the spectrum. It stands to reason that the 13-16 seeds of the world aren't going to play non-conference schedules as tough as the blue bloods and better programs that get the higher seeds. For example: when you order teams by non-conference NCAA teams played, 53 of the 89 teams at the bottom are seeded between 13 and 16. The average seed of the 89 teams at the top is 5.64; the bottom is 11.53.

Why does this matter? Mainly, it's related to the way the expected win column is calculated, and how rare it is for a low seeded team to win a game. This chart shows the expected wins, broken down by seed, for NCAA teams over the last 14 years (stretching back to the 2001-2002 season).

Seed Expected Wins
1 3.285714
2 2.375
3 1.982143
4 1.535714
5 1.107143
6 0.892857
7 1.017857
8 0.821429
9 0.482143
10 0.553571
11 0.642857
12 0.607143
13 0.25
14 0.125
15 0.071429
16 0

See the potential problem? While every 8-15 seed will have an ‘expected win' value that will count toward the overall calculation, the ‘actual win' column is an either-or proposition. A fractional value will obviously be greater than 0 (aren't you glad I'm here, guys??), meaning that, in a small sample, it's entirely possible ‘expected wins' will far outstrip the low seeds that actually won a game. This likely contributed at least some to the negative values seen on the charts above for the ‘Bottom' portions - not to take anything away from positive values seen on the ‘Top' side, which are real and spectacular.

So I went through a similar exercise, while restricting the analysis to only top 12 seeds, and then again, while restricting the analysis to only top 4 seeds. The results, mirroring the tables that were shown above, can be seen below.

Top 12 Seeds: Non-Conference NCAA Teams Played


Top 10% Bottom 10%
Expected Wins 105.5 60.88
Actual Wins 101 56
Differential -4.5 -4.87

Top 25% Bottom 25%
Expected Wins 261.45 178.61
Actual Wins 284 153
Differential 22.55 -25.61

Top 33% Bottom 33%
Expected Wins 338.12 232.25
Actual Wins 362 192
Differential 23.88 -40.25


Top 25 Opponents faced NC


Top 10% Bottom 10%
Expected Wins 113.25 57.77
Actual Wins 126 45
Differential 12.75 -12.77

Top 25% Bottom 25%
Expected Wins 267.02 162.54
Actual Wins 301 151
Differential 33.98 -11.54

Top 33% Bottom 33%
Expected Wins 345.45 222.68
Actual Wins 386 200
Differential 40.55 -22.68


Unique Top 25 Opponents


Top 10% Bottom 10%
Expected Wins 107.38 52.95
Actual Wins 110 40
Differential 2.62 -12.95

Top 25% Bottom 25%
Expected Wins 251.95 159.89
Actual Wins 276 141
Differential 24.05 -18.89

Top 33% Bottom 33%
Expected Wins 327.46 219.84
Actual Wins 339 206
Differential 11.54 -13.84


Top 4 Seeds: Non-Conference NCAA Teams Played


Top 10% Bottom 10%
Expected Wins 58.73 51.71
Actual Wins 57 48
Differential -1.73 -3.71

Top 25% Bottom 25%
Expected Wins 137.79 120.32
Actual Wins 142 105
Differential 4.21 -15.32

Top 33% Bottom 33%
Expected Wins 179.86 159.18
Actual Wins 184 152
Differential 4.14 -7.18

Top 25 Opponents faced NC

x Top 10% Bottom 10%
Expected Wins 54.77 52.3
Actual Wins 60 50
Differential 5.23 -2.3
x Top 25% Bottom 25%
Expected Wins 134 126.8
Actual Wins 147 110
Differential 13 -16.8
x Top 33% Bottom 33%
Expected Wins 173.39 169.36
Actual Wins 185 150
Differential 11.61 -19.36

Unique Top 25 Opponents

x Top 10% Bottom 10%
Expected Wins 47.38 55.75
Actual Wins 56 51
Differential 8.63 -4.75
x Top 25% Bottom 25%
Expected Wins 127.13 131.95
Actual Wins 144 119
Differential 16.87 -12.95
x Top 33% Bottom 33%
Expected Wins 173.39 170.05
Actual Wins 194 162
Differential 20.61 -8.05

While somewhat muted, the positive showing for teams facing strong non-conference schedules and overall opponents is still there. Simply: teams that played a higher amount of very tough teams have performed better than teams that haven't, when it comes to NCAA tournament time.

Here's Where It Gets Confusing

So, debate over, right? Wrong. We're only 2,000 words in!

So we've seen the ‘effect' that playing strong teams out of conference, and in an overall sense, has on team's NCAA tournament results, right? A logical conclusion to draw from these results would be: the more good teams you play, the better suited you are for success in the NCAAs. Should be true across the board, right?

Not so fast. Check out the tournament success (overall, top 12 seeds, and top 4 seeds) of the top and bottom portions of NCAA teams ordered by number of NCAA teams and top 25 teams played in conference.

Top 4:

Conference Top 25 Opponents

Top 10%

Bottom 10%

Expected Wins

53.19643

54.89286

Actual Wins

58

50

Differential

4.803571

-4.89286

Top 25%

Bottom 25%

Expected Wins

124.5893

128.8929

Actual Wins

131

121

Differential

6.410714

-7.89286

Top 33%

Bottom 33%

Expected Wins

166

170.8929

Actual Wins

169

162

Differential

3

-8.89286

Conference NCAA Teams Played

Top 10%

Bottom 10%

Expected Wins

48.79

53.16

Actual Wins

44.00

50.00

Differential

-4.79

-3.16

Top 25%

Bottom 25%

Expected Wins

124.13

129.86

Actual Wins

124.00

124.00

Differential

-0.13

-5.86

Top 33%

Bottom 33%

Expected Wins

166.95

171.79

Actual Wins

162.00

169.00

Differential

-4.95

-2.79

Top 12:

Conference Top 25 Opponents

Top 10%

Bottom 10%

Expected Wins

105.84

48.98

Actual Wins

113.00

45.00

Differential

7.16

-3.98

Top 25%

Bottom 25%

Expected Wins

236.50

156.95

Actual Wins

225.00

154.00

Differential

-11.50

-2.95

Top 33%

Bottom 33%

Expected Wins

321.61

224.07

Actual Wins

321.00

212.00

Differential

-0.61

-12.07

Conference NCAA Teams Played

Top 10%

Bottom 10%

Expected Wins

102.68

55.84

Actual Wins

98.00

61.00

Differential

-4.68

5.16

Top 25%

Bottom 25%

Expected Wins

256.21

153.02

Actual Wins

245.00

152.00

Differential

-11.21

-1.02

Top 33%

Bottom 33%

Expected Wins

328.84

212.61

Actual Wins

315.00

207.00

Differential

-13.84

-5.61


All:

Conference NCAA Teams Played

Top 10%

Bottom 10%

Expected Wins

141.64

15.59

Actual Wins

128.00

10.00

Differential

-13.64

-5.59

Top 25%

Bottom 25%

Expected Wins

330.50

47.39

Actual Wins

317.00

49.00

Differential

-13.50

1.61

Top 33%

Bottom 33%

Expected Wins

440.20

90.21

Actual Wins

436.00

94.00

Differential

-4.20

3.79

Conference Top 25 Opponents

Top 10%

Bottom 10%

Expected Wins

136.70

16.95

Actual Wins

137.00

12.00

Differential

0.30

-4.95

Top 25%

Bottom 25%

Expected Wins

322.66

64.57

Actual Wins

327.00

58.00

Differential

4.34

-6.57

Top 33%

Bottom 33%

Expected Wins

421.05

96.16

Actual Wins

430.00

91.00

Differential

8.95

-5.16

In almost every case, teams that played more NCAA tournament teams in-conference had worse tournament results than those who played less. And, while it clearly remained somewhat helpful to play more top 25 teams in-conference than less, the effects of same became far more muted than what was seen in terms of both non-conference play and overall top 25 means played.

So what gives? If playing better teams prepares you better for the NCAA tournament, it really shouldn't matter whether they fall in the same conference as a team; good teams are good. Similarly, there are really odd results when checking average NC team rank, average kenpom Pythagorean rating for both non-conference and conference opponents, and various others of the 20 or so categories I ran comparisons for. Did not want to drown the article in tables, but if anyone's interested in taking at a look at the other data, please feel free to contact me.

In essence, the tables I showed you up top were the most conclusive; many of the other factors I checked were contradictory, or so close as to remain inconclusive. This is indicative of the many tricks you can pull with stats - only showing the audience what supports your own previously-arrived-at-conclusion, for example - but the biggest indicator for me is that there's a lot of noise when looking at this data.

Drawing 100% concrete conclusions is kind of dangerous - the margins are truly tiny.

To that end - the final thing I took a look at was a ‘make-miss' analysis that compared seeds that made or missed the various rounds. Again, a ton of noise here. Take a look at the top 4 seeds, in an analysis of ‘made/missed' the round of 32:


Non-Conference Games

Total Unique Pre-Tournament

NC NCAA
Teams

Conf NCAA
Teams

Unique NCAA Teams

Average Non-Tourney Oppo Pythag

Average NC Oppo Pythag (Regular Season)

Average Conference Oppo Pythag (Regular Season)

Top 25 Opponents

Top 50 Opponents

Mini Cupcakes (100-200)

Mega Cupcakes (201-350+)

Top 25 Opponents

Top 50 Opponents

Made Round of 32

1

56

4.45

8.07

8.89

0.672

0.576

0.728

1.95

3.59

3.64

3.54

4.52

8.52

Missed Round of 32

1

0

#DIV/0!

#DIV/0!

#DIV/0!

#DIV/0!

#DIV/0!

#DIV/0!

#DIV/0!

#DIV/0!

#DIV/0!

#DIV/0!

#DIV/0!

#DIV/0!

Made Round of 32

2

53

4.21

8.53

8.60

0.670

0.561

0.732

2.26

3.58

3.19

3.89

4.85

8.57

Missed Round of 32

2

3

4.33

10.00

9.67

0.661

0.540

0.719

1.67

3.33

2.00

4.67

5.00

9.33

Made Round of 32

3

49

3.51

8.73

8.14

0.662

0.536

0.737

1.73

2.96

3.33

4.31

4.67

8.16

Missed Round of 32

3

7

5.00

11.14

10.29

0.702

0.605

0.752

2.00

4.14

2.00

3.71

4.71

10.00

Made Round of 32

4

45

4.04

8.31

8.42

0.660

0.537

0.733

1.91

3.20

3.51

4.27

4.44

8.09

Missed Round of 32

4

11

3.00

9.00

8.00

0.651

0.513

0.735

1.45

2.82

3.27

4.82

4.45

8.09

The two and three seeds that lost in the first round actually had tougher schedules than those who didn't - which pokes some holes in the argument that playing better teams inoculates you against upsets.

A similar general phenomenon can be seen in the make/miss analysis of the Sweet Sixteen:

Non-Conference Games

Total Unique Pre-Tournament

NC NCAA
Teams

Conf NCAA
Teams

Unique NCAA Teams

Average Non-Tourney Oppo Pythag

Average NC Oppo Pythag (Regular Season)

Average Conference Oppo Pythag (Regular Season)

Top 25 Opponents

Top 50 Opponents

Mini Cupcakes (100-200)

Mega Cupcakes (201-350+)

Top 25 Opponents

Top 50 Opponents

Made Sweet Sixteen

1

48

4.58

8.23

9.13

0.678

0.578

0.737

2.02

3.67

3.63

3.56

4.65

8.77

Missed Sweet Sixteen

1

8

3.63

7.13

7.50

0.636

0.562

0.674

1.50

3.13

3.75

3.38

3.75

7.00

Made Sweet Sixteen

2

35

4.34

8.46

8.46

0.665

0.559

0.725

2.34

3.63

3.14

4.00

4.60

8.43

Missed Sweet Sixteen

2

21

4.00

8.86

9.00

0.677

0.562

0.743

2.05

3.48

3.10

3.81

5.29

8.90

Made Sweet Sixteen

3

33

3.42

8.88

8.03

0.661

0.529

0.738

1.82

2.88

3.12

4.55

4.85

8.15

Missed Sweet Sixteen

3

23

4.09

9.26

8.96

0.675

0.566

0.739

1.70

3.43

3.22

3.78

4.43

8.74

Made Sweet Sixteen

4

25

4.40

8.12

8.56

0.659

0.540

0.725

2.00

3.60

3.60

4.20

4.60

8.36

Missed Sweet Sixteen

4

31

3.39

8.71

8.16

0.658

0.526

0.740

1.68

2.74

3.35

4.52

4.32

7.87

While the 1-seeds that bowed out before the Sweet 16 look to have a pretty clearly weaker schedule than those that advanced, the same cannot be said for seeds 2-4. Ultimately, the schedule comparison works out very close to equal for both.

Here's the rest of the table:

Non-Conference Games

Total Unique Pre-Tournament

NC NCAA
Teams

Conf NCAA
Teams

Unique NCAA Teams

Average Non-Tourney Oppo Pythag

Average NC Oppo Pythag (Regular Season)

Average Conference Oppo Pythag (Regular Season)

Top 25 Opponents

Top 50 Opponents

Mini Cupcakes (100-200)

Mega Cupcakes (201-350+)

Top 25 Opponents

Top 50 Opponents

Made Elite Eight

1

37

4.46

8.00

9.03

0.674

0.577

0.732

1.97

3.65

3.81

3.54

4.51

8.76

Missed Elite Eight

1

19

4.42

8.21

8.63

0.668

0.573

0.722

1.89

3.47

3.32

3.53

4.53

8.05

Made Elite Eight

2

27

4.56

8.78

8.89

0.667

0.558

0.729

2.41

3.67

2.93

4.15

4.89

8.67

Missed Elite Eight

2

29

3.90

8.45

8.45

0.671

0.561

0.734

2.07

3.48

3.31

3.72

4.83

8.55

Made Elite Eight

3

16

3.44

9.00

8.31

0.658

0.508

0.747

1.75

2.81

2.88

4.94

5.25

8.25

Missed Elite Eight

3

40

3.80

9.05

8.45

0.671

0.559

0.735

1.78

3.23

3.28

3.95

4.45

8.45

Made Elite Eight

4

9

4.44

8.78

9.56

0.669

0.541

0.742

2.22

3.44

3.78

4.00

5.33

9.33

Missed Elite Eight

4

47

3.72

8.38

8.11

0.656

0.530

0.732

1.74

3.06

3.40

4.45

4.28

7.85

Made Final Four

1

21

4.29

8.00

8.67

0.676

0.569

0.743

2.10

3.76

3.86

3.86

4.71

8.71

Missed Final Four

1

35

4.54

8.11

9.03

0.669

0.580

0.719

1.86

3.49

3.51

3.34

4.40

8.40

Made Final Four

2

12

4.67

9.58

9.58

0.682

0.565

0.748

2.33

3.67

2.83

4.00

5.25

9.33

Missed Final Four

2

44

4.09

8.34

8.41

0.666

0.558

0.727

2.20

3.55

3.20

3.91

4.75

8.41

Made Final Four

3

6

3.17

9.67

8.33

0.674

0.504

0.778

1.83

2.67

2.00

5.17

6.33

9.17

Missed Final Four

3

50

3.76

8.96

8.42

0.666

0.549

0.734

1.76

3.16

3.30

4.12

4.48

8.30

Made Final Four

4

6

4.67

9.50

10.17

0.659

0.528

0.733

2.50

3.50

3.83

4.33

5.67

9.33

Missed Final Four

4

50

3.74

8.32

8.12

0.658

0.533

0.733

1.74

3.08

3.42

4.38

4.30

7.94

Made National Championship

1

13

4.23

7.92

8.62

0.671

0.566

0.740

2.08

3.62

3.69

4.15

4.54

8.38

Missed National Championship

1

43

4.51

8.12

8.98

0.794

0.579

0.725

1.91

3.58

3.63

3.35

4.51

8.56

Made National Championship

2

5

5.60

9.40

10.40

0.684

0.585

0.746

2.80

4.60

2.60

3.80

5.60

9.80

Missed National Championship

2

51

4.08

8.53

8.49

0.767

0.557

0.730

2.18

3.47

3.18

3.94

4.78

8.49

Made National Championship

3

4

3.25

11.00

9.00

0.692

0.509

0.807

1.75

2.75

2.00

5.00

6.75

10.00

Missed National Championship

3

52

3.73

8.88

8.37

0.765

0.547

0.733

1.77

3.13

3.25

4.17

4.52

8.27

Made National Championship

4

1

3.00

11.00

9.00

0.656

0.463

0.790

2.00

3.00

3.00

6.00

7.00

10.00

Missed National Championship

4

55

3.85

8.40

8.33

0.636

0.533

0.732

1.82

3.13

3.47

4.35

4.40

8.05


In Conclusion

There's definitely some solid evidence that beefing up your non-conference schedule with high end opponents is a good indicator for your team's NCAA tournament success - assuming you make it there, of course. There's also some solid-but-not-promising evidence that playing in a conference with a lot of NCAA tournament teams is a BAD indicator of your team's eventual NCAA tournament success - so take both data points as you will.

Mainly, any serious conclusions along these lines must be tempered. Even over the course of 14 years, these data sets are limited. While I'd much rather rely on conclusions drawn from the same than single case instances of confirmation bias among commenters, I ultimately don't feel confident in making too many of them (conclusions, that is). There's so much random variance in the NCAA tournament, and within the results I gathered in general.

A few bullets I feel confident in:

  • More top 25 teams is a good thing, in conference or out.
  • Scheduling cupcakes has pretty much no effect on a team's NCAA tournament chances, in high numbers or low.
  • The teams above the top 25 really don't matter as much; the effect of more top 50, 75, and 100 games is pretty negligible, in terms of actual vs. expected NCAA tournament performance.
  • The differences in scheduling between teams that bowed out early and teams that advanced, especially among top seeds, are so miniscule and sometimes contradictory to the general premise so as to be generally ignorable.

So, please find some real, actually data-based fuel for the general debate above. Feel free to continue to pointlessly hash it out in the comments until next year's NCAA championship win.

Also, if you want a look at the larger data set, again, feel free to contact me; can send it via email.