Performance issues with very large groups
Steps to reproduce
Steps to reproduce:
1. Create 5000 internal users
2. Add all 5000 users to a group (at creation time or later)
3. In the Admin UI, access Organization / Groups
The REST endpoint for groups is extremely slow, as getting a list of groups also loads all of the group members.
Displaying a list of groups should be fast.
In the storage implementation, the group members should be lazy-loaded, i.e. it should be possible to get a list of groups without also loading all members for all groups.
In one case, it took 5.5min for the REST call
to complete. Adding these keys to the database helped slightly (reduced load time to around 2 minutes):
alter table mh_group_member add unique key mh_group_member_i (group_id, member);
alter table mh_group_member add key mh_group_member_member_i ( member);
but the real solution is the groups endpoint should not load the membership unless required to do so.
Workaround (if any):
Don't create groups with large membership.
There may also be some issue with the External API index getting out of sync when you create a very large group, where you get the correct results from:
Note that there is a whole parallel storage and retrieval mechanism for groups in the index service, which is used by the External API: