EX Raid Selection Criteria: What the Data Says About the December 1st Wave

[Dronpes Note: For the sake of timeliness, this preliminary analysis has foregone the Silph Research Group’s standard internal peer review process in the interest of turning this survey around quickly and getting tallies and results back into our travelers’ hands. Please forgive any typo’s or late night errors – I really should have gone to bed many hours ago! ]

Receiving an EX Raid invitation is the key to acquiring the exclusive MewTwo in Pokemon GO.

Niantic recently concluded their multi-month field test of the EX Raid feature and explained three factors that influence the odds of being invited to an EX Raid (plus a note regarding Raid times).

The Silph Road team has conducted a survey of Silph Road travelers who received an invite in the December 1st wave – and for the first time, we’re ready to take a preliminary glance at what the data shows regarding known (and unknown) EX Raid selection criteria.

What We Think We Know

First, let’s review the factors Niantic shared that might be at play:

  • EX Raid Battles will take place most commonly at Gyms found in parks and sponsored locations.
  • Trainers with a high-level Gym Badge are more likely to be invited to EX Raid Battles taking place at that Gym.
  • Trainers who have completed a larger number of Raid Battles are more likely to be invited to EX Raid Battles.
  • EX Raid Battle start times now take into account popular Raid Battle times at that Gym.

This information has prompted many of our travelers to focus their efforts on Gyms in parks and sponsored locations, and has encouraged many to begin working towards Silver and Gold badges at these Gyms.

But what does the data show?

Finding #1: Head to the Parks, Sponsors, & the Next Best Things

Overwhelmingly we found Gyms located in parks to be the most common location for EX Raids, with 53% of all EX Raids happening in park Gyms.  In second place, sponsored locations hosted just under one third (31%) of EX Raids.

Travelers without parks or sponsored locations nearby may look to prioritize universities/schools, statues, and monuments, as these appear to be the most common fallback locations.  Take a look:

Suffice it to say, Niantic’s first factor appears to be in full force.

Finding #2: Gym Medal Levels Suggest They Might Matter

Niantic also claimed that high-level Gym badges will make trainers more likely to be selected for EX Raid passes.

In our view this claim in particular presents an interesting problem. EX Raids currently max out at 40 invitations.  Should Gold and Silver Gym badge holders always be selected according to a “first in line” methodology, Niantic’s selection algorithm would necessarily need to exclude them in certain rounds, or they would consistently prevent lower-level badge holders from receiving EX passes.  Particularly once common EX Raid gyms were made known to local player communities and ‘hardcore’ players all rushed to achieve gold Gym badges at those locations.

It seems likely, therefore, that Niantic may be employing a different approach, possibly either reserving a minority percentage of invitations for lower-level players to ensure they are included when there are ample gold and silver Gym badge holders eligible, or simply selecting all eligible players at a Gym and assigning weighted odds before selection.  Under the first approach, the (far more common) low-level gym badge holders would outnumber the high, so higher-level badge holders would have greater odds in their smaller pool. Under the second, depending on the weights, a dedicated local player group might ‘min-max’ the gym providing “too many” eligible gold and silver badge-holders which might effectively exclude casual players entirely (undesirable from a gamemaker’s perspective).  But enough speculation; let’s examine the data!

In visualizing the Gym badge level breakdown from our sample group, we find the following:

Immediately, the largest proportion being silver jumps out as suggestive of badge level influence – assuming there are (many) more bronze badge holders than silver in the local player ecosystem.

We expected to see bronze badges take the majority by a large margin (remember: completing even one Raid at a Gym will provide a bronze badge, so everyone who ever Raided at that gym has that badge). Instead silver, which requires four completed Raids (or equivalent badge XP) is surprisingly prevalent.  Add to that the gold portion and at this point it would appear that the odds do appear slightly better for silver or gold badge holders.

One caveat to this assumption, however, is that our sample set (of Silph Road travelers) leans heavily towards ‘hard-core’ players.  This may have lead to under-reporting from other bronze badge holders.

Finding #3: Over 30% Have Done 1+ Raid Every Day

The number of Raid battles a player has completed has been announced to be a factor in selection criteria.  But what qualifies as a “larger number of raid battles?”

On December 1st, the Raid system was roughly 160 days old.  If a free-to-play player were to Raid every single day at least 1x, we might expect them to have completed 160 Raids by this date.  In examining the completed Raid counts of our dataset, we find the following:

69% of players selected had completed less than 150 Raids, while 30% had completed more.

Once again, we will point out that the nature of our dataset is skewed towards more active players (though Niantic’s selection of our travelers is not necessarily a representative cross-section of the Silph Road’s most active players).  Assuming the large majority of players have not kept up the pace of 1+ Raid daily for over five months, this nearly 30/70 split along the line of 1x/day Raiding seems rather active and may indicate that those who Raid more are experiencing better odds.

Further research is needed to truly claim this, however.  A baseline survey of how many Raids players have completed (irrespective to EX Raid invites) would provide a valuable reference and enable this comparison.

Finding #4: High Level Trainers Dominate the Dataset

Reports of very low level accounts or very casual/inactive players getting EX Raid invitations spread rapidly when EX waves go out.

Our dataset, however, reflected an entirely different picture:

While 1.5% of accounts reported by our travelers were below level 20, 21% of invites were level 40 players.  In all, 93% of EX Raid invites on the Silph Road went to level 28+ accounts.

Finding #5: Seven Out of Ten Are First-Time Recipients

Throughout the EX Raid field test, we observed largely the same group of players continue to receive EX Raid invitations week after week.

Now, however, we’re observing largely the opposite:

Over 70% of EX Raid recipients on December 1st received their very first invite.  21% were receiving their second, and less than 1 in 10 had received more than two.

Of those who were now receiving more than one invite, all were mid to high level 30s, 81% had a silver Gym badge or above, and 95% had Raided at the Gym in at least the past week.

Finding #6: Raid Invite Times Appear to Reflect Player Raiding Times

This is a particularly notable observation. Niantic mentioned that, contrary to how things operated during the field test period, EX Raids will now function at various times dependent on the Gyms’ “popular Raid battle times.”

Interestingly, we managed to see this reflected in a co-occurrence matrix plotting the time our travelers Raided at this Gym vs. the time of their EX Raid invitation at that gym.  The results are rather striking:

 

It seems unlikely to us that the EX Raid times would align perfectly with all our travelers’ Raiding times, but there is an unusually strong correlation from 10am through 6pm (this wave’s EX Raid window) between travelers who raided in a particular hour and travelers having the EX Raid during the next hour.

This is one we’ll be watching with great interest.

Finding #7: The Gym Eligibility Day Limit May Be One Week

Contrary to the field test (where we saw testers selected for EX Raids who had merely Raided a gym once, many months prior), the selection criteria appears to have tightened down to a shorter period: one week.

95% of travelers selected for an EX Raid on December 1st had Raided at that Gym within the last 7 days.

Notably, 28% had Raided at the selected Gym in the past 24 hours before receiving their EX Raid invitation.  The difference, though, between those who Raided in the past week vs the past two weeks is remarkably more stark than during the field test.  This leads us to hypothesize that 7 days is the cutoff for EX Raid eligibility.

We recommend Raiding at all eligible gyms once a week – while additional frequency has had unclear (and possibly little to no) marginal value in this wave.

Anecdotal Outliers and Questions

All in all, travelers, it appears that much of what Niantic shared regarding EX Raid factors is either confirmed or plausible with the limited data we’ve gathered already.

What, then, of the many anecdotal reports of low-level players receiving invites while more active players have to continue to wait?  It seems undeniable that Niantic has intentionally included some degree of more casual, inactive, or newer players in the EX Raid pool in this December 1st wave.  We may expect this to always be the case with EX Raids for strategic reasons (e.g. inactive player retention).

One apparent irony in Niantic’s selection procedure for these less active players is that it may be (unintentionally) easier for low-level or casual players to net an EX Raid pass if, in fact, very few low-level players Raid at a selected Gym the week before an EX Raid.  The algorithm may, while attempting to include a more casual portion of the playerbase, find itself with few options that qualify (ie, have Raided at that gym in the past week) for a specific EX Raid.

Does that irony automatically mean no casual or less active players should be given the handicap?  The answer is not black and white – and it’s one Niantic will have to make in alignment with their player retention roadmap.  It appears, for the time being, that the selection algorithm may be valuing players who appear to be “coming back” or “testing the waters” greatly and are providing a handicap to these in the weighted EX Raid selection.

The good news is that this does not preclude the many factors identified above from improving your individual selection weight.  (Again, it seems very unlikely to us that Niantic is operating with a crude “first in line” qualification system.  A weighted odds selection process seems a more likely solution – and in fact aligns with their developer notes.)

What We Recommend

For now, we recommend prioritizing the following:

  1. Raid at least weekly at parks, sponsored locations, monuments, and church Gym locations
  2. Try to achieve at least silver gym badge level at these gyms
  3. If possible, perhaps observe these gyms’ most busy Raid times and try Raiding during these hours

We do not recommend:

  1. Spending tons of money trying to Raid 10x/day or rack up a soaring total Raid count
  2. Raiding at every potential EX Gym every day
  3. Using Pinap Berries on a Legendary you need (they really don’t increase your catch chances. No matter what you were told)

Remember: the majority of players have not yet received an EX Raid invite (the Silph Road team included!).  We expect to see more and more EX Raid invite waves in coming weeks, however.  For now, we hope this analysis puts a little more solid information in your hands when puzzling on how to get that EX Raid invite.  Until then, … you can find us at the park!

Travel Safe,

– Executive Dronpes –