Issue turned out to be that in some circumstances, the `group_by` iterator will create multiple groups for a single key - and sometimes those groups can be empty. This lead to situations where an animation track will end up empty until enough changed to re-shuffle things in the query iterator.
The fix is switching to `into_group_map_by` - which will put all grouped items into a hashmap instead, ensuring no duplicate keys.