Opencast fails to start

Steps to reproduce

Occasionally Opencast will fail to start with many exceptions that look like

2017-06-21 07:56:31,804 | INFO | (RestPublisher:333) - Registered REST endpoint at /publication/oaipmh
ERROR: Bundle matterhorn-kernel [100] EventDispatcher: Error during dispatch. (org.apache.cxf.service.factory.ServiceConstructionException)
org.apache.cxf.service.factory.ServiceConstructionException
at org.apache.cxf.jaxrs.JAXRSServerFactoryBean.create(JAXRSServerFactoryBean.java:219)
at org.opencastproject.kernel.rest.RestPublisher.createEndpoint(RestPublisher.java:331)
at org.opencastproject.kernel.rest.RestPublisher$JaxRsServiceTracker.addingService(RestPublisher.java:424)
at org.osgi.util.tracker.ServiceTracker$Tracked.customizerAdding(ServiceTracker.java:941)
at org.osgi.util.tracker.ServiceTracker$Tracked.customizerAdding(ServiceTracker.java:870)
at org.osgi.util.tracker.AbstractTracked.trackAdding(AbstractTracked.java:256)
at org.osgi.util.tracker.AbstractTracked.track(AbstractTracked.java:229)
at org.osgi.util.tracker.ServiceTracker$Tracked.serviceChanged(ServiceTracker.java:901)
at org.apache.felix.framework.EventDispatcher.invokeServiceListenerCallback(EventDispatcher.java:990)
at org.apache.felix.framework.EventDispatcher.fireEventImmediately(EventDispatcher.java:838)
at org.apache.felix.framework.EventDispatcher.fireServiceEvent(EventDispatcher.java:545)
at org.apache.felix.framework.Felix.fireServiceEvent(Felix.java:4557)
at org.apache.felix.framework.Felix.registerService(Felix.java:3549)
at org.apache.felix.framework.BundleContextImpl.registerService(BundleContextImpl.java:348)
at org.apache.felix.scr.impl.manager.AbstractComponentManager$3.register(AbstractComponentManager.java:886)
at org.apache.felix.scr.impl.manager.AbstractComponentManager$3.register(AbstractComponentManager.java:873)
at org.apache.felix.scr.impl.manager.RegistrationManager.changeRegistration(RegistrationManager.java:132)
at org.apache.felix.scr.impl.manager.AbstractComponentManager.registerService(AbstractComponentManager.java:940)
at org.apache.felix.scr.impl.manager.AbstractComponentManager.activateInternal(AbstractComponentManager.java:740)
at org.apache.felix.scr.impl.manager.DependencyManager$SingleStaticCustomizer.addedService(DependencyManager.java:1012)
at org.apache.felix.scr.impl.manager.DependencyManager$SingleStaticCustomizer.addedService(DependencyManager.java:968)
at org.apache.felix.scr.impl.manager.ServiceTracker$Tracked.customizerAdded(ServiceTracker.java:1215)
at org.apache.felix.scr.impl.manager.ServiceTracker$Tracked.customizerAdded(ServiceTracker.java:1136)
at org.apache.felix.scr.impl.manager.ServiceTracker$AbstractTracked.trackAdding(ServiceTracker.java:945)
at org.apache.felix.scr.impl.manager.ServiceTracker$AbstractTracked.track(ServiceTracker.java:881)
at org.apache.felix.scr.impl.manager.ServiceTracker$Tracked.serviceChanged(ServiceTracker.java:1167)
at org.apache.felix.scr.impl.BundleComponentActivator$ListenerInfo.serviceChanged(BundleComponentActivator.java:127)
at org.apache.felix.framework.EventDispatcher.invokeServiceListenerCallback(EventDispatcher.java:990)
at org.apache.felix.framework.EventDispatcher.fireEventImmediately(EventDispatcher.java:838)
at org.apache.felix.framework.EventDispatcher.fireServiceEvent(EventDispatcher.java:545)
at org.apache.felix.framework.Felix.fireServiceEvent(Felix.java:4557)
at org.apache.felix.framework.Felix.registerService(Felix.java:3549)
at org.apache.felix.framework.BundleContextImpl.registerService(BundleContextImpl.java:348)
at org.apache.felix.scr.impl.manager.AbstractComponentManager$3.register(AbstractComponentManager.java:886)
at org.apache.felix.scr.impl.manager.AbstractComponentManager$3.register(AbstractComponentManager.java:873)
at org.apache.felix.scr.impl.manager.RegistrationManager.changeRegistration(RegistrationManager.java:132)
at org.apache.felix.scr.impl.manager.AbstractComponentManager.registerService(AbstractComponentManager.java:940)
at org.apache.felix.scr.impl.manager.AbstractComponentManager.activateInternal(AbstractComponentManager.java:740)
at org.apache.felix.scr.impl.manager.DependencyManager$SingleStaticCustomizer.addedService(DependencyManager.java:1012)
at org.apache.felix.scr.impl.manager.DependencyManager$SingleStaticCustomizer.addedService(DependencyManager.java:968)
at org.apache.felix.scr.impl.manager.ServiceTracker$Tracked.customizerAdded(ServiceTracker.java:1215)
at org.apache.felix.scr.impl.manager.ServiceTracker$Tracked.customizerAdded(ServiceTracker.java:1136)
at org.apache.felix.scr.impl.manager.ServiceTracker$AbstractTracked.trackAdding(ServiceTracker.java:945)
at org.apache.felix.scr.impl.manager.ServiceTracker$AbstractTracked.track(ServiceTracker.java:881)
at org.apache.felix.scr.impl.manager.ServiceTracker$Tracked.serviceChanged(ServiceTracker.java:1167)
at org.apache.felix.scr.impl.BundleComponentActivator$ListenerInfo.serviceChanged(BundleComponentActivator.java:127)
at org.apache.felix.framework.EventDispatcher.invokeServiceListenerCallback(EventDispatcher.java:990)
at org.apache.felix.framework.EventDispatcher.fireEventImmediately(EventDispatcher.java:838)
at org.apache.felix.framework.EventDispatcher.fireServiceEvent(EventDispatcher.java:545)
at org.apache.felix.framework.Felix.fireServiceEvent(Felix.java:4557)
at org.apache.felix.framework.Felix.registerService(Felix.java:3549)
at org.apache.felix.framework.BundleContextImpl.registerService(BundleContextImpl.java:348)
at org.apache.felix.framework.BundleContextImpl.registerService(BundleContextImpl.java:355)
at org.apache.aries.jpa.container.impl.ManagedEMF.createAndPublishEMF(ManagedEMF.java:132)
at org.apache.aries.jpa.container.impl.ManagedEMF.updated(ManagedEMF.java:125)
at org.apache.felix.cm.impl.helper.ManagedServiceTracker.updated(ManagedServiceTracker.java:189)
at org.apache.felix.cm.impl.helper.ManagedServiceTracker.updateService(ManagedServiceTracker.java:152)
at org.apache.felix.cm.impl.helper.ManagedServiceTracker.provideConfiguration(ManagedServiceTracker.java:85)
at org.apache.felix.cm.impl.ConfigurationManager$ManagedServiceUpdate.provide(ConfigurationManager.java:1461)
at org.apache.felix.cm.impl.ConfigurationManager$ManagedServiceUpdate.run(ConfigurationManager.java:1417)
at org.apache.felix.cm.impl.UpdateThread.run0(UpdateThread.java:141)
at org.apache.felix.cm.impl.UpdateThread.run(UpdateThread.java:109)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.cxf.service.factory.ServiceConstructionException: There is an endpoint already running on /.
at org.apache.cxf.jaxrs.JAXRSBindingFactory.addListener(JAXRSBindingFactory.java:85)
at org.apache.cxf.endpoint.ServerImpl.start(ServerImpl.java:123)
at org.apache.cxf.jaxrs.JAXRSServerFactoryBean.create(JAXRSServerFactoryBean.java:206)
... 62 more

Note that this exception can be thrown from any bundle (frequently thrown from multiple bundles at once). Restarting seems to fix it sometimes.

Activity

Show:
Stephen Marquard
August 19, 2017, 10:59 AM

Seen on 3.2:

2017-08-19 12:54:10,676 | ERROR | FelixDispatchQueue | (?:?) - FrameworkEvent ERROR - matterhorn-kernel
org.apache.cxf.service.factory.ServiceConstructionException
at org.apache.cxf.jaxrs.JAXRSServerFactoryBean.create(JAXRSServerFactoryBean.java:219)[207:org.apache.cxf.cxf-rt-frontend-jaxrs:3.1.7]
at org.opencastproject.kernel.rest.RestPublisher.createEndpoint(RestPublisher.java:331)[102:matterhorn-kernel:3.0.0.SNAPSHOT]
at org.opencastproject.kernel.rest.RestPublisher$JaxRsServiceTracker.addingService(RestPublisher.java:424)[102:matterhorn-kernel:3.0.0.SNAPSHOT]
at org.osgi.util.tracker.ServiceTracker$Tracked.customizerAdding(ServiceTracker.java:941)[org.osgi.core-6.0.0.jar:]
at org.osgi.util.tracker.ServiceTracker$Tracked.customizerAdding(ServiceTracker.java:870)[org.osgi.core-6.0.0.jar:]
at org.osgi.util.tracker.AbstractTracked.trackAdding(AbstractTracked.java:256)[org.osgi.core-6.0.0.jar:]
at org.osgi.util.tracker.AbstractTracked.track(AbstractTracked.java:229)[org.osgi.core-6.0.0.jar:]
at org.osgi.util.tracker.ServiceTracker$Tracked.serviceChanged(ServiceTracker.java:901)[org.osgi.core-6.0.0.jar:]
at org.apache.felix.framework.EventDispatcher.invokeServiceListenerCallback(EventDispatcher.java:990)[org.apache.felix.framework-5.6.2.jar:]
at org.apache.felix.framework.EventDispatcher.fireEventImmediately(EventDispatcher.java:838)[org.apache.felix.framework-5.6.2.jar:]
at org.apache.felix.framework.EventDispatcher.fireServiceEvent(EventDispatcher.java:545)[org.apache.felix.framework-5.6.2.jar:]
at org.apache.felix.framework.Felix.fireServiceEvent(Felix.java:4578)[org.apache.felix.framework-5.6.2.jar:]
at org.apache.felix.framework.Felix.registerService(Felix.java:3570)[org.apache.felix.framework-5.6.2.jar:]
at org.apache.felix.framework.BundleContextImpl.registerService(BundleContextImpl.java:348)[org.apache.felix.framework-5.6.2.jar:]
at org.apache.felix.scr.impl.manager.AbstractComponentManager$3.register(AbstractComponentManager.java:887)[216:org.apache.felix.scr:2.0.8]
at org.apache.felix.scr.impl.manager.AbstractComponentManager$3.register(AbstractComponentManager.java:874)[216:org.apache.felix.scr:2.0.8]
at org.apache.felix.scr.impl.manager.RegistrationManager.changeRegistration(RegistrationManager.java:132)[216:org.apache.felix.scr:2.0.8]
at org.apache.felix.scr.impl.manager.AbstractComponentManager.registerService(AbstractComponentManager.java:941)[216:org.apache.felix.scr:2.0.8]
at org.apache.felix.scr.impl.manager.AbstractComponentManager.activateInternal(AbstractComponentManager.java:741)[216:org.apache.felix.scr:2.0.8]
at org.apache.felix.scr.impl.manager.DependencyManager$SingleStaticCustomizer.addedService(DependencyManager.java:1012)[216:org.apache.felix.scr:2.0.8]
at org.apache.felix.scr.impl.manager.DependencyManager$SingleStaticCustomizer.addedService(DependencyManager.java:968)[216:org.apache.felix.scr:2.0.8]
at org.apache.felix.scr.impl.manager.ServiceTracker$Tracked.customizerAdded(ServiceTracker.java:1215)[216:org.apache.felix.scr:2.0.8]
at org.apache.felix.scr.impl.manager.ServiceTracker$Tracked.customizerAdded(ServiceTracker.java:1136)[216:org.apache.felix.scr:2.0.8]
at org.apache.felix.scr.impl.manager.ServiceTracker$AbstractTracked.trackAdding(ServiceTracker.java:945)[216:org.apache.felix.scr:2.0.8]
at org.apache.felix.scr.impl.manager.ServiceTracker$AbstractTracked.track(ServiceTracker.java:881)[216:org.apache.felix.scr:2.0.8]
at org.apache.felix.scr.impl.manager.ServiceTracker$Tracked.serviceChanged(ServiceTracker.java:1167)[216:org.apache.felix.scr:2.0.8]
at org.apache.felix.scr.impl.BundleComponentActivator$ListenerInfo.serviceChanged(BundleComponentActivator.java:127)[216:org.apache.felix.scr:2.0.8]
at org.apache.felix.framework.EventDispatcher.invokeServiceListenerCallback(EventDispatcher.java:990)[org.apache.felix.framework-5.6.2.jar:]
at org.apache.felix.framework.EventDispatcher.fireEventImmediately(EventDispatcher.java:838)[org.apache.felix.framework-5.6.2.jar:]
at org.apache.felix.framework.EventDispatcher.fireServiceEvent(EventDispatcher.java:545)[org.apache.felix.framework-5.6.2.jar:]
at org.apache.felix.framework.Felix.fireServiceEvent(Felix.java:4578)[org.apache.felix.framework-5.6.2.jar:]
at org.apache.felix.framework.Felix.registerService(Felix.java:3570)[org.apache.felix.framework-5.6.2.jar:]
at org.apache.felix.framework.BundleContextImpl.registerService(BundleContextImpl.java:348)[org.apache.felix.framework-5.6.2.jar:]
at org.apache.felix.scr.impl.manager.AbstractComponentManager$3.register(AbstractComponentManager.java:887)[216:org.apache.felix.scr:2.0.8]
at org.apache.felix.scr.impl.manager.AbstractComponentManager$3.register(AbstractComponentManager.java:874)[216:org.apache.felix.scr:2.0.8]
at org.apache.felix.scr.impl.manager.RegistrationManager.changeRegistration(RegistrationManager.java:132)[216:org.apache.felix.scr:2.0.8]
at org.apache.felix.scr.impl.manager.AbstractComponentManager.registerService(AbstractComponentManager.java:941)[216:org.apache.felix.scr:2.0.8]
at org.apache.felix.scr.impl.manager.AbstractComponentManager.activateInternal(AbstractComponentManager.java:741)[216:org.apache.felix.scr:2.0.8]
at org.apache.felix.scr.impl.manager.AbstractComponentManager.enableInternal(AbstractComponentManager.java:675)[216:org.apache.felix.scr:2.0.8]
at org.apache.felix.scr.impl.manager.AbstractComponentManager.enable(AbstractComponentManager.java:430)[216:org.apache.felix.scr:2.0.8]
at org.apache.felix.scr.impl.manager.ConfigurableComponentHolder.enableComponents(ConfigurableComponentHolder.java:657)[216:org.apache.felix.scr:2.0.8]
at org.apache.felix.scr.impl.BundleComponentActivator.initialEnable(BundleComponentActivator.java:341)[216:org.apache.felix.scr:2.0.8]
at org.apache.felix.scr.impl.Activator.loadComponents(Activator.java:390)[216:org.apache.felix.scr:2.0.8]
at org.apache.felix.scr.impl.Activator.access$200(Activator.java:54)[216:org.apache.felix.scr:2.0.8]
at org.apache.felix.scr.impl.Activator$ScrExtension.start(Activator.java:265)[216:org.apache.felix.scr:2.0.8]
at org.apache.felix.utils.extender.AbstractExtender.createExtension(AbstractExtender.java:254)[216:org.apache.felix.scr:2.0.8]
at org.apache.felix.utils.extender.AbstractExtender.modifiedBundle(AbstractExtender.java:227)[216:org.apache.felix.scr:2.0.8]
at org.osgi.util.tracker.BundleTracker$Tracked.customizerModified(BundleTracker.java:482)[org.osgi.core-6.0.0.jar:]
at org.osgi.util.tracker.BundleTracker$Tracked.customizerModified(BundleTracker.java:415)[org.osgi.core-6.0.0.jar:]
at org.osgi.util.tracker.AbstractTracked.track(AbstractTracked.java:232)[org.osgi.core-6.0.0.jar:]
at org.osgi.util.tracker.BundleTracker$Tracked.bundleChanged(BundleTracker.java:444)[org.osgi.core-6.0.0.jar:]
at org.apache.felix.framework.EventDispatcher.invokeBundleListenerCallback(EventDispatcher.java:915)[org.apache.felix.framework-5.6.2.jar:]
at org.apache.felix.framework.EventDispatcher.fireEventImmediately(EventDispatcher.java:834)[org.apache.felix.framework-5.6.2.jar:]
at org.apache.felix.framework.EventDispatcher.fireBundleEvent(EventDispatcher.java:516)[org.apache.felix.framework-5.6.2.jar:]
at org.apache.felix.framework.Felix.fireBundleEvent(Felix.java:4562)[org.apache.felix.framework-5.6.2.jar:]
at org.apache.felix.framework.Felix.startBundle(Felix.java:2172)[org.apache.felix.framework-5.6.2.jar:]
at org.apache.felix.framework.BundleImpl.start(BundleImpl.java:998)[org.apache.felix.framework-5.6.2.jar:]
at org.apache.felix.framework.BundleImpl.start(BundleImpl.java:984)[org.apache.felix.framework-5.6.2.jar:]
at org.apache.karaf.features.internal.service.FeaturesServiceImpl.startBundle(FeaturesServiceImpl.java:1287)[6:org.apache.karaf.features.core:4.0.9]
at org.apache.karaf.features.internal.service.Deployer.deploy(Deployer.java:860)[6:org.apache.karaf.features.core:4.0.9]
at org.apache.karaf.features.internal.service.FeaturesServiceImpl.doProvision(FeaturesServiceImpl.java:1176)[6:org.apache.karaf.features.core:4.0.9]
at org.apache.karaf.features.internal.service.FeaturesServiceImpl$1.call(FeaturesServiceImpl.java:1074)[6:org.apache.karaf.features.core:4.0.9]
at java.util.concurrent.FutureTask.run(FutureTask.java:266)[:1.8.0_121]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)[:1.8.0_121]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)[:1.8.0_121]
at java.lang.Thread.run(Thread.java:745)[:1.8.0_121]
Caused by: org.apache.cxf.service.factory.ServiceConstructionException: There is an endpoint already running on /.
at org.apache.cxf.jaxrs.JAXRSBindingFactory.addListener(JAXRSBindingFactory.java:85)[207:org.apache.cxf.cxf-rt-frontend-jaxrs:3.1.7]
at org.apache.cxf.endpoint.ServerImpl.start(ServerImpl.java:123)[206:org.apache.cxf.cxf-core:3.1.7]
at org.apache.cxf.jaxrs.JAXRSServerFactoryBean.create(JAXRSServerFactoryBean.java:206)[207:org.apache.cxf.cxf-rt-frontend-jaxrs:3.1.7]
... 65 more

Stephen Marquard
August 19, 2017, 12:09 PM

Not easy to reproduce this - it disappeared after adding a single log line to RestPublisher and rebuilding kernel and assemblies.

Greg Logan
August 24, 2017, 1:48 AM

I'm thinking this is some kind of race condition, or packaging error. It seems to occur randomly, and in bursts. Sometimes it seems like a repackage of the Karaf assemblies makes it go away, but it's so random that that could just be confirmation bias.

Stephen Marquard
August 24, 2017, 7:43 AM

My bet is on a packaging error. Next time this happens, it would be worth grabbing the assembled package and then comparing it to a subsequent package/deploy that works successfully.

Greg Logan
June 10, 2019, 3:50 PM

Fixed by upstream Karaf changes, unsure when but I have not seen this in a long time.

Assignee

Unassigned

Reporter

Greg Logan

Severity

Incorrectly Functioning With Workaround

Tags (folksonomy)

None

Components

Affects versions

Priority

Critical
Configure