Uploaded image for project: 'Opencast'
  1. MH-8493

Error Distributing to progressinve downloads

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed and reviewed
    • Affects versions: 1.3
    • Fix versions: 1.3
    • Components: Backend Software
    • Labels:
      None
    • Severity:
      Incorrectly Functioning Without Workaround
    • Steps to reproduce:
      Hide
      Seen on our production site:

      a large media zip is placed in the inbox (+/- 1Gb, 1 hour of material)
      Proccessing proceeeds till " Distributing to progressinve downloads" and then just stops.

      On the worker one can sometimes see a nonce time out error and on the admin one can see a an out of memory GC error:

      2012-01-18 12:19:17 ERROR (AbstractFaultChainInitiatorObserver:101) - Error occurred during error handling, give up!
      org.apache.cxf.interceptor.Fault: GC overhead limit exceeded
              at org.apache.cxf.service.invoker.AbstractInvoker.createFault(AbstractInvoker.java:155)
              at org.apache.cxf.service.invoker.AbstractInvoker.invoke(AbstractInvoker.java:121)
              at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:133)
              at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:82)
              at org.apache.cxf.interceptor.ServiceInvokerInterceptor$1.run(ServiceInvokerInterceptor.java:58)
              at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
              at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
              at java.util.concurrent.FutureTask.run(FutureTask.java:138)
              at org.apache.cxf.workqueue.SynchronousExecutor.execute(SynchronousExecutor.java:37)
              at org.apache.cxf.interceptor.ServiceInvokerInterceptor.handleMessage(ServiceInvokerInterceptor.java:106)
              at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:243)
              at org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:110)
              at org.apache.cxf.transport.servlet.ServletDestination.invoke(ServletDestination.java:98)
              at org.apache.cxf.transport.servlet.ServletController.invokeDestination(ServletController.java:423)
              at org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:139)
              at org.apache.cxf.transport.servlet.AbstractCXFServlet.invoke(AbstractCXFServlet.java:142)
              at org.apache.cxf.transport.servlet.AbstractHTTPServlet.handleRequest(AbstractHTTPServlet.java:179)
              at org.apache.cxf.transport.servlet.AbstractHTTPServlet.doPost(AbstractHTTPServlet.java:103)
              at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
              at org.apache.cxf.transport.servlet.AbstractHTTPServlet.service(AbstractHTTPServlet.java:159)
              at org.apache.felix.http.base.internal.handler.ServletHandler.doHandle(ServletHandler.java:91)
              at org.apache.felix.http.base.internal.handler.ServletHandler.handle(ServletHandler.java:77)
              at org.apache.felix.http.base.internal.dispatch.ServletPipeline.handle(ServletPipeline.java:42)
              at org.apache.felix.http.base.internal.dispatch.InvocationFilterChain.doFilter(InvocationFilterChain.java:49)
              at org.apache.felix.http.base.internal.dispatch.HttpFilterChain.doFilter(HttpFilterChain.java:33)
              at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:343)
              at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:109)
              at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:83)
              at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:355)
              at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:97)
              at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:355)
              at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:100)
              at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:355)
              at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:78)
              at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:355)
              at org.springframework.security.web.authentication.rememberme.RememberMeAuthenticationFilter.doFilter(RememberMeAuthenticationFilter.java:119)
              at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:355)
              at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:54)
              at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:355)
              at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:35)
              at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:355)
              at org.springframework.security.oauth.provider.OAuthProviderProcessingFilter.doFilter(OAuthProviderProcessingFilter.java:173)
              at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:355)
              at org.springframework.security.web.authentication.www.DigestAuthenticationFilter.doFilter(DigestAuthenticationFilter.java:319)
              at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:355)
              at org.springframework.security.web.authentication.ui.DefaultLoginPageGeneratingFilter.doFilter(DefaultLoginPageGeneratingFilter.java:90)
              at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:355)
              at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:188)
              at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:355)
              at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:188)
              at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:355)
              at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:105)
              at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:355)
              at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:79)
              at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:355)
              at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:149)
              at org.opencastproject.kernel.security.SecurityFilter.doFilter(SecurityFilter.java:115)
              at org.apache.felix.http.base.internal.handler.FilterHandler.doHandle(FilterHandler.java:80)
              at org.apache.felix.http.base.internal.handler.FilterHandler.handle(FilterHandler.java:68)
              at org.apache.felix.http.base.internal.dispatch.InvocationFilterChain.doFilter(InvocationFilterChain.java:47)
              at org.apache.felix.http.base.internal.dispatch.HttpFilterChain.doFilter(HttpFilterChain.java:33)
              at org.opencastproject.kernel.security.OrganizationFilter.doFilter(OrganizationFilter.java:109)
              at org.apache.felix.http.base.internal.handler.FilterHandler.doHandle(FilterHandler.java:80)
              at org.apache.felix.http.base.internal.handler.FilterHandler.handle(FilterHandler.java:68)
              at org.apache.felix.http.base.internal.dispatch.InvocationFilterChain.doFilter(InvocationFilterChain.java:47)
              at org.apache.felix.http.base.internal.dispatch.HttpFilterChain.doFilter(HttpFilterChain.java:33)
              at org.opencastproject.kernel.rest.JsonpFilter.doFilter(JsonpFilter.java:106)
              at org.apache.felix.http.base.internal.handler.FilterHandler.doHandle(FilterHandler.java:80)
              at org.apache.felix.http.base.internal.handler.FilterHandler.handle(FilterHandler.java:68)
              at org.apache.felix.http.base.internal.dispatch.InvocationFilterChain.doFilter(InvocationFilterChain.java:47)
              at org.apache.felix.http.base.internal.dispatch.HttpFilterChain.doFilter(HttpFilterChain.java:33)
              at org.opencastproject.kernel.rest.CleanSessionsFilter.doFilter(CleanSessionsFilter.java:82)
              at org.apache.felix.http.base.internal.handler.FilterHandler.doHandle(FilterHandler.java:80)
              at org.apache.felix.http.base.internal.handler.FilterHandler.handle(FilterHandler.java:68)
              at org.apache.felix.http.base.internal.dispatch.InvocationFilterChain.doFilter(InvocationFilterChain.java:47)
              at org.apache.felix.http.base.internal.dispatch.HttpFilterChain.doFilter(HttpFilterChain.java:33)
              at org.apache.felix.http.base.internal.dispatch.FilterPipeline.dispatch(FilterPipeline.java:48)
              at org.apache.felix.http.base.internal.dispatch.Dispatcher.dispatch(Dispatcher.java:39)
              at org.apache.felix.http.base.internal.DispatcherServlet.service(DispatcherServlet.java:55)
              at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
              at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
              at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:390)
              at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
              at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:765)
              at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
              at org.mortbay.jetty.Server.handle(Server.java:326)
              at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:536)
              at org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:930)
              at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:747)
              at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:218)
              at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:405)
              at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409)
              at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
      Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded


      I'm not certain at this point if both these errors are consistenly reproducible
      Show
      Seen on our production site: a large media zip is placed in the inbox (+/- 1Gb, 1 hour of material) Proccessing proceeeds till " Distributing to progressinve downloads" and then just stops. On the worker one can sometimes see a nonce time out error and on the admin one can see a an out of memory GC error: 2012-01-18 12:19:17 ERROR (AbstractFaultChainInitiatorObserver:101) - Error occurred during error handling, give up! org.apache.cxf.interceptor.Fault: GC overhead limit exceeded         at org.apache.cxf.service.invoker.AbstractInvoker.createFault(AbstractInvoker.java:155)         at org.apache.cxf.service.invoker.AbstractInvoker.invoke(AbstractInvoker.java:121)         at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:133)         at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:82)         at org.apache.cxf.interceptor.ServiceInvokerInterceptor$1.run(ServiceInvokerInterceptor.java:58)         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)         at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)         at java.util.concurrent.FutureTask.run(FutureTask.java:138)         at org.apache.cxf.workqueue.SynchronousExecutor.execute(SynchronousExecutor.java:37)         at org.apache.cxf.interceptor.ServiceInvokerInterceptor.handleMessage(ServiceInvokerInterceptor.java:106)         at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:243)         at org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:110)         at org.apache.cxf.transport.servlet.ServletDestination.invoke(ServletDestination.java:98)         at org.apache.cxf.transport.servlet.ServletController.invokeDestination(ServletController.java:423)         at org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:139)         at org.apache.cxf.transport.servlet.AbstractCXFServlet.invoke(AbstractCXFServlet.java:142)         at org.apache.cxf.transport.servlet.AbstractHTTPServlet.handleRequest(AbstractHTTPServlet.java:179)         at org.apache.cxf.transport.servlet.AbstractHTTPServlet.doPost(AbstractHTTPServlet.java:103)         at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)         at org.apache.cxf.transport.servlet.AbstractHTTPServlet.service(AbstractHTTPServlet.java:159)         at org.apache.felix.http.base.internal.handler.ServletHandler.doHandle(ServletHandler.java:91)         at org.apache.felix.http.base.internal.handler.ServletHandler.handle(ServletHandler.java:77)         at org.apache.felix.http.base.internal.dispatch.ServletPipeline.handle(ServletPipeline.java:42)         at org.apache.felix.http.base.internal.dispatch.InvocationFilterChain.doFilter(InvocationFilterChain.java:49)         at org.apache.felix.http.base.internal.dispatch.HttpFilterChain.doFilter(HttpFilterChain.java:33)         at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:343)         at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:109)         at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:83)         at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:355)         at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:97)         at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:355)         at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:100)         at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:355)         at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:78)         at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:355)         at org.springframework.security.web.authentication.rememberme.RememberMeAuthenticationFilter.doFilter(RememberMeAuthenticationFilter.java:119)         at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:355)         at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:54)         at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:355)         at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:35)         at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:355)         at org.springframework.security.oauth.provider.OAuthProviderProcessingFilter.doFilter(OAuthProviderProcessingFilter.java:173)         at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:355)         at org.springframework.security.web.authentication. www.DigestAuthenticationFilter.doFilter(DigestAuthenticationFilter.java:319)         at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:355)         at org.springframework.security.web.authentication.ui.DefaultLoginPageGeneratingFilter.doFilter(DefaultLoginPageGeneratingFilter.java:90)         at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:355)         at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:188)         at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:355)         at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:188)         at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:355)         at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:105)         at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:355)         at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:79)         at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:355)         at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:149)         at org.opencastproject.kernel.security.SecurityFilter.doFilter(SecurityFilter.java:115)         at org.apache.felix.http.base.internal.handler.FilterHandler.doHandle(FilterHandler.java:80)         at org.apache.felix.http.base.internal.handler.FilterHandler.handle(FilterHandler.java:68)         at org.apache.felix.http.base.internal.dispatch.InvocationFilterChain.doFilter(InvocationFilterChain.java:47)         at org.apache.felix.http.base.internal.dispatch.HttpFilterChain.doFilter(HttpFilterChain.java:33)         at org.opencastproject.kernel.security.OrganizationFilter.doFilter(OrganizationFilter.java:109)         at org.apache.felix.http.base.internal.handler.FilterHandler.doHandle(FilterHandler.java:80)         at org.apache.felix.http.base.internal.handler.FilterHandler.handle(FilterHandler.java:68)         at org.apache.felix.http.base.internal.dispatch.InvocationFilterChain.doFilter(InvocationFilterChain.java:47)         at org.apache.felix.http.base.internal.dispatch.HttpFilterChain.doFilter(HttpFilterChain.java:33)         at org.opencastproject.kernel.rest.JsonpFilter.doFilter(JsonpFilter.java:106)         at org.apache.felix.http.base.internal.handler.FilterHandler.doHandle(FilterHandler.java:80)         at org.apache.felix.http.base.internal.handler.FilterHandler.handle(FilterHandler.java:68)         at org.apache.felix.http.base.internal.dispatch.InvocationFilterChain.doFilter(InvocationFilterChain.java:47)         at org.apache.felix.http.base.internal.dispatch.HttpFilterChain.doFilter(HttpFilterChain.java:33)         at org.opencastproject.kernel.rest.CleanSessionsFilter.doFilter(CleanSessionsFilter.java:82)         at org.apache.felix.http.base.internal.handler.FilterHandler.doHandle(FilterHandler.java:80)         at org.apache.felix.http.base.internal.handler.FilterHandler.handle(FilterHandler.java:68)         at org.apache.felix.http.base.internal.dispatch.InvocationFilterChain.doFilter(InvocationFilterChain.java:47)         at org.apache.felix.http.base.internal.dispatch.HttpFilterChain.doFilter(HttpFilterChain.java:33)         at org.apache.felix.http.base.internal.dispatch.FilterPipeline.dispatch(FilterPipeline.java:48)         at org.apache.felix.http.base.internal.dispatch.Dispatcher.dispatch(Dispatcher.java:39)         at org.apache.felix.http.base.internal.DispatcherServlet.service(DispatcherServlet.java:55)         at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)         at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)         at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:390)         at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)         at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:765)         at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)         at org.mortbay.jetty.Server.handle(Server.java:326)         at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:536)         at org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:930)         at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:747)         at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:218)         at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:405)         at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409)         at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582) Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded I'm not certain at this point if both these errors are consistenly reproducible

      TestRail: Results

        Attachments

          Activity

            People

            • Assignee:
              twunden Tobias Wunden
              Reporter:
              dhorwitz David Horwitz
            • Watchers:
              0 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                TestRail: Cases