Uploaded image for project: 'Opencast'
  1. MH-12248

Capture Calendar Modification Caching Implementation is very Inefficient

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed and reviewed
    • Affects versions: 2.3.0
    • Fix versions: 3.0
    • Components: Backend Software
    • Labels:
      None
    • Severity:
      Performance
    • Steps to reproduce:
      Hide
      It appears that the CA calendar caching implementation added as part of AdminNG is the same that Entwine created for us in an attempt to implement/improve calendar caching. Unfortunately we found that is caused severe performance issues

      Basically when a CA asks for it's calendar, admin checks for the last modified date in a cache and if nothing has changed returns 304 (etag behaviour) or else the new calendar. However if the CA is not in the list it doesn't just try can calc it's last modified date but gets the *entire* schedule and processes *every* CA ie re-populates the cache. The problem is that the cache entry expiry time has been set to just 60seconds, so all entries are invalid by the time the CA next requests it's calendar.

      Secondly, it fails to take in account that the endpoint is called asynchronously by the CA's hence a large number will trigger the repopulate code at the same time! This caused massive spikes in the application thread counts.

      Unfortunately we didn't realize the code had made it into OC 2.x or would have provided a fix sooner.

      Our solution manages the cache on a per CA basis. No timeout is used and is erroneous anyway since whenever the CA requests it's calendar it wants the lastest version not some out of date one. Schedule operations have therefore been modified update the CA calendar cache. This has worked successfully in production with >300 CAs for over a year.


      Show
      It appears that the CA calendar caching implementation added as part of AdminNG is the same that Entwine created for us in an attempt to implement/improve calendar caching. Unfortunately we found that is caused severe performance issues Basically when a CA asks for it's calendar, admin checks for the last modified date in a cache and if nothing has changed returns 304 (etag behaviour) or else the new calendar. However if the CA is not in the list it doesn't just try can calc it's last modified date but gets the *entire* schedule and processes *every* CA ie re-populates the cache. The problem is that the cache entry expiry time has been set to just 60seconds, so all entries are invalid by the time the CA next requests it's calendar. Secondly, it fails to take in account that the endpoint is called asynchronously by the CA's hence a large number will trigger the repopulate code at the same time! This caused massive spikes in the application thread counts. Unfortunately we didn't realize the code had made it into OC 2.x or would have provided a fix sooner. Our solution manages the cache on a per CA basis. No timeout is used and is erroneous anyway since whenever the CA requests it's calendar it wants the lastest version not some out of date one. Schedule operations have therefore been modified update the CA calendar cache. This has worked successfully in production with >300 CAs for over a year.

      TestRail: Results

        Attachments

          Activity

            People

            • Assignee:
              james.perrin James Perrin
              Reporter:
              james.perrin James Perrin
            • Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                TestRail: Cases