Uploaded image for project: 'Opencast'
  1. MH-10995

Gathering workflow statistics for JMX causes extreme performance issues

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed and reviewed
    • Affects versions: 1.6.1
    • Fix versions: 1.6.2, 2.0.1
    • Components: Backend Software
    • Labels:
      None
    • Severity:
      Performance
    • Steps to reproduce:
      Hide
      Steps to reproduce:
      1. Schedule a large number of upcoming recordings (e.g. 3000+)
      2. Schedule a set of new recordings (e.g. 50 in a series) in a group
      3.
       
       Actual Results:
       
      The action to schedule new recordings is extremely slow (e.g. 30+min to complete). Other actions that involve updating workflows are also extremely slow.
       
       Expected Results:
       
      Response time < 60s for group scheduling.
       
      Diagnosis:

      WorkflowServiceImpl.java includes this line for updating statistics that are provided by JMX:

      workflowsStatistics.updateWorkflow(getBeanStatistics(), getHoldWorkflows());

      The method getHoldWorkflows() reads every single upcoming scheduled recording into memory, as these are workflows in PAUSED state.

      This causes extremely high memory usage, frequent GC activity, and slow response times.
      Show
      Steps to reproduce: 1. Schedule a large number of upcoming recordings (e.g. 3000+) 2. Schedule a set of new recordings (e.g. 50 in a series) in a group 3.    Actual Results:   The action to schedule new recordings is extremely slow (e.g. 30+min to complete). Other actions that involve updating workflows are also extremely slow.    Expected Results:   Response time < 60s for group scheduling.   Diagnosis: WorkflowServiceImpl.java includes this line for updating statistics that are provided by JMX: workflowsStatistics.updateWorkflow(getBeanStatistics(), getHoldWorkflows()); The method getHoldWorkflows() reads every single upcoming scheduled recording into memory, as these are workflows in PAUSED state. This causes extremely high memory usage, frequent GC activity, and slow response times.

      TestRail: Results

        Attachments

          Issue links

            Activity

              People

              • Assignee:
                jamesp James Perrin (Inactive)
                Reporter:
                smarquard Stephen Marquard
              • Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  TestRail: Cases