Performance issues with very large groups

Steps to reproduce

Steps to reproduce:
1. Create 5000 internal users
2. Add all 5000 users to a group (at creation time or later)
3. In the Admin UI, access Organization / Groups

Actual Results:

The REST endpoint for groups is extremely slow, as getting a list of groups also loads all of the group members.

Expected Results:

Displaying a list of groups should be fast.

In the storage implementation, the group members should be lazy-loaded, i.e. it should be possible to get a list of groups without also loading all members for all groups.

In one case, it took 5.5min for the REST call

/admin-ng/groups/groups.json?limit=10&offset=0&sort=name:ASC

to complete. Adding these keys to the database helped slightly (reduced load time to around 2 minutes):

alter table mh_group_member add unique key mh_group_member_i (group_id, member);
alter table mh_group_member add key mh_group_member_member_i ( member);

but the real solution is the groups endpoint should not load the membership unless required to do so.

Workaround (if any):

Don't create groups with large membership.

Activity

Show:
Sven Stauber
April 13, 2017, 9:33 PM

Note that this issue smells similar to https://opencast.jira.com/browse/MH-12157.

Stephen Marquard
January 31, 2017, 10:13 AM

There may also be some issue with the External API index getting out of sync when you create a very large group, where you get the correct results from:

/admin-ng/groups/groups.json

but

/api/groups

stops updating.

Stephen Marquard
January 31, 2017, 8:51 AM

Note that there is a whole parallel storage and retrieval mechanism for groups in the index service, which is used by the External API:

matterhorn-index-service/src/main/java/org/opencastproject/index/service/impl/index/group/GroupSearchQuery.java

Assignee

Unassigned

Reporter

Stephen Marquard

Severity

Performance