Cannot re-build search Solr index

Steps to reproduce

  • Ingest two videos: 1x presenter 1x presentation

  • Stop Opencast

  • Remove data/solr-index/search

  • Start Opencast

  • Get a lot of errors

  • The indexer cannot access all metadata catalogs (403 error)

  • Trying to add the slidetexts the event adding will completely fail

Stack trace:

2016-12-22 12:50:22,325 | INFO | (SearchServiceImpl:195) - Setting up solr server
2016-12-22 12:50:22,326 | INFO | (SearchServiceImpl:253) - Setting up solr search index at /home/lars/dev/opencast/build/opencast-dist-allinone-2.3-SNAPSHOT/data/solr-indexes/search
2016-12-22 12:50:22,326 | INFO | (SearchServiceImpl:260) - solr config directory doesn't exist. Creating /home/lars/dev/opencast/build/opencast-dist-allinone-2.3-SNAPSHOT/data/solr-indexes/search/conf
2016-12-22 12:50:22,391 | WARN | (SolrCore:389) - Solr index directory '/home/lars/dev/opencast/build/opencast-dist-allinone-2.3-SNAPSHOT/data/solr-indexes/search/data/index' doesn't exist. Creating new index...
2016-12-22 12:50:22,465 | INFO | (SearchServiceImpl:509) - No search index found
2016-12-22 12:50:22,466 | INFO | (SearchServiceImpl:510) - Starting population of search index from database
2016-12-22 12:50:22,609 | WARN | (WorkspaceImpl:330) - Received unexpected response status 403 while trying to download from http://localhost:8080/static/mh_default_org/engage-player/f25f4cd4-6d1d-40be-b6a5-4e14536a61e1/9352000d-cacb-4287-a9b3-1e5ba4525ea9/dublincore.xml
2016-12-22 12:50:22,614 | WARN | (StaticMetadataServiceDublinCoreImpl:317) - Unable to load metadata from catalog 'http://localhost:8080/static/mh_default_org/engage-player/f25f4cd4-6d1d-40be-b6a5-4e14536a61e1/9352000d-cacb-4287-a9b3-1e5ba4525ea9/dublincore.xml'
2016-12-22 12:50:22,741 | WARN | (WorkspaceImpl:330) - Received unexpected response status 403 while trying to download from http://localhost:8080/static/mh_default_org/engage-player/9c13730c-f13a-46a8-b5f0-7c899d202411/42249fe3-3038-4dc0-a7dc-d32dc1e2c412/dublincore.xml
2016-12-22 12:50:22,742 | WARN | (StaticMetadataServiceDublinCoreImpl:317) - Unable to load metadata from catalog 'http://localhost:8080/static/mh_default_org/engage-player/9c13730c-f13a-46a8-b5f0-7c899d202411/42249fe3-3038-4dc0-a7dc-d32dc1e2c412/dublincore.xml'
2016-12-22 12:50:22,747 | WARN | (WorkspaceImpl:330) - Received unexpected response status 403 while trying to download from http://localhost:8080/static/mh_default_org/engage-player/9c13730c-f13a-46a8-b5f0-7c899d202411/f38c7d38-08f4-4b02-b796-1697d92e9ccf/slidetext.xml
2016-12-22 12:50:22,749 | ERROR | (SearchServiceImpl:535) - Unable to index search instances: {}
org.apache.solr.client.solrj.SolrServerException: java.io.IOException: Unable to load metadata from mpeg7 catalog http://localhost:8080/static/mh_default_org/engage-player/9c13730c-f13a-46a8-b5f0-7c899d202411/f38c7d38-08f4-4b02-b796-1697d92e9ccf/slidetext.xml
at org.opencastproject.search.impl.solr.SolrIndexManager.add(SolrIndexManager.java:348)
at org.opencastproject.search.impl.SearchServiceImpl.populateIndex(SearchServiceImpl.java:533)
at org.opencastproject.search.impl.SearchServiceImpl.activate(SearchServiceImpl.java:236)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)[:1.8.0_111]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)[:1.8.0_111]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)[:1.8.0_111]
at java.lang.reflect.Method.invoke(Method.java:498)[:1.8.0_111]
at org.apache.felix.scr.impl.helper.BaseMethod.invokeMethod(BaseMethod.java:231)
at org.apache.felix.scr.impl.helper.BaseMethod.access$500(BaseMethod.java:39)
at org.apache.felix.scr.impl.helper.BaseMethod$Resolved.invoke(BaseMethod.java:624)
at org.apache.felix.scr.impl.helper.BaseMethod.invoke(BaseMethod.java:508)
at org.apache.felix.scr.impl.helper.ActivateMethod.invoke(ActivateMethod.java:149)
at org.apache.felix.scr.impl.manager.SingleComponentManager.createImplementationObject(SingleComponentManager.java:315)
at org.apache.felix.scr.impl.manager.SingleComponentManager.createComponent(SingleComponentManager.java:127)
at org.apache.felix.scr.impl.manager.SingleComponentManager.getService(SingleComponentManager.java:871)
at org.apache.felix.scr.impl.manager.SingleComponentManager.getServiceInternal(SingleComponentManager.java:838)
at org.apache.felix.scr.impl.manager.SingleComponentManager.getService(SingleComponentManager.java:777)
at org.apache.felix.framework.ServiceRegistrationImpl.getFactoryUnchecked(ServiceRegistrationImpl.java:308)
at org.apache.felix.framework.ServiceRegistrationImpl.getService(ServiceRegistrationImpl.java:219)
at org.apache.felix.framework.ServiceRegistry.getService(ServiceRegistry.java:320)
at org.apache.felix.framework.Felix.getService(Felix.java:3568)[org.apache.felix.framework-4.2.1.jar:]
at org.apache.felix.framework.BundleContextImpl.getService(BundleContextImpl.java:468)
at org.apache.felix.scr.impl.helper.BindMethod.getServiceObject(BindMethod.java:572)
at org.apache.felix.scr.impl.manager.DependencyManager.getServiceObject(DependencyManager.java:2012)
at org.apache.felix.scr.impl.manager.DependencyManager$SingleStaticCustomizer.prebind(DependencyManager.java:1005)
at org.apache.felix.scr.impl.manager.DependencyManager.prebind(DependencyManager.java:1439)
at org.apache.felix.scr.impl.manager.AbstractComponentManager.collectDependencies(AbstractComponentManager.java:1119)
at org.apache.felix.scr.impl.manager.SingleComponentManager.getServiceInternal(SingleComponentManager.java:807)
at org.apache.felix.scr.impl.manager.SingleComponentManager.getService(SingleComponentManager.java:777)
at org.apache.felix.framework.ServiceRegistrationImpl.getFactoryUnchecked(ServiceRegistrationImpl.java:308)
at org.apache.felix.framework.ServiceRegistrationImpl.getService(ServiceRegistrationImpl.java:219)
at org.apache.felix.framework.ServiceRegistry.getService(ServiceRegistry.java:320)
at org.apache.felix.framework.Felix.getService(Felix.java:3568)[org.apache.felix.framework-4.2.1.jar:]
at org.apache.felix.framework.BundleContextImpl.getService(BundleContextImpl.java:468)
at org.apache.felix.fileinstall.internal.FileInstall.addingService(FileInstall.java:150)
at org.osgi.util.tracker.ServiceTracker$Tracked.customizerAdding(ServiceTracker.java:932)[karaf-org.osgi.core.jar:]
at org.osgi.util.tracker.ServiceTracker$Tracked.customizerAdding(ServiceTracker.java:864)[karaf-org.osgi.core.jar:]
at org.osgi.util.tracker.AbstractTracked.trackAdding(AbstractTracked.java:256)[karaf-org.osgi.core.jar:]
at org.osgi.util.tracker.AbstractTracked.track(AbstractTracked.java:229)[karaf-org.osgi.core.jar:]
at org.osgi.util.tracker.ServiceTracker$Tracked.serviceChanged(ServiceTracker.java:894)[karaf-org.osgi.core.jar:]
at org.apache.felix.framework.util.EventDispatcher.invokeServiceListenerCallback(EventDispatcher.java:932)[org.apache.felix.framework-4.2.1.jar:]
at org.apache.felix.framework.util.EventDispatcher.fireEventImmediately(EventDispatcher.java:793)[org.apache.felix.framework-4.2.1.jar:]
at org.apache.felix.framework.util.EventDispatcher.fireServiceEvent(EventDispatcher.java:543)[org.apache.felix.framework-4.2.1.jar:]
at org.apache.felix.framework.Felix.fireServiceEvent(Felix.java:4419)[org.apache.felix.framework-4.2.1.jar:]
at org.apache.felix.framework.Felix.registerService(Felix.java:3423)[org.apache.felix.framework-4.2.1.jar:]
at org.apache.felix.framework.BundleContextImpl.registerService(BundleContextImpl.java:346)
at org.apache.felix.scr.impl.manager.AbstractComponentManager$3.register(AbstractComponentManager.java:1003)
at org.apache.felix.scr.impl.manager.AbstractComponentManager$3.register(AbstractComponentManager.java:992)
at org.apache.felix.scr.impl.manager.RegistrationManager.changeRegistration(RegistrationManager.java:134)
at org.apache.felix.scr.impl.manager.AbstractComponentManager.registerService(AbstractComponentManager.java:1044)
at org.apache.felix.scr.impl.manager.AbstractComponentManager.activateInternal(AbstractComponentManager.java:841)
at org.apache.felix.scr.impl.manager.DependencyManager$SingleStaticCustomizer.addedService(DependencyManager.java:931)
at org.apache.felix.scr.impl.manager.DependencyManager$SingleStaticCustomizer.addedService(DependencyManager.java:895)
at org.apache.felix.scr.impl.manager.ServiceTracker$Tracked.customizerAdded(ServiceTracker.java:1480)[73:org.apache.felix.scr:1.8.2]
at org.apache.felix.scr.impl.manager.ServiceTracker$Tracked.customizerAdded(ServiceTracker.java:1401)[73:org.apache.felix.scr:1.8.2]
at org.apache.felix.scr.impl.manager.ServiceTracker$AbstractTracked.trackAdding(ServiceTracker.java:1210)
at org.apache.felix.scr.impl.manager.ServiceTracker$AbstractTracked.track(ServiceTracker.java:1148)
at org.apache.felix.scr.impl.manager.ServiceTracker$Tracked.serviceChanged(ServiceTracker.java:1432)[73:org.apache.felix.scr:1.8.2]
at org.apache.felix.framework.util.EventDispatcher.invokeServiceListenerCallback(EventDispatcher.java:932)[org.apache.felix.framework-4.2.1.jar:]
at org.apache.felix.framework.util.EventDispatcher.fireEventImmediately(EventDispatcher.java:793)[org.apache.felix.framework-4.2.1.jar:]
at org.apache.felix.framework.util.EventDispatcher.fireServiceEvent(EventDispatcher.java:543)[org.apache.felix.framework-4.2.1.jar:]
at org.apache.felix.framework.Felix.fireServiceEvent(Felix.java:4419)[org.apache.felix.framework-4.2.1.jar:]
at org.apache.felix.framework.Felix.registerService(Felix.java:3423)[org.apache.felix.framework-4.2.1.jar:]
at org.apache.felix.framework.BundleContextImpl.registerService(BundleContextImpl.java:346)
at org.apache.felix.scr.impl.manager.AbstractComponentManager$3.register(AbstractComponentManager.java:1003)
at org.apache.felix.scr.impl.manager.AbstractComponentManager$3.register(AbstractComponentManager.java:992)
at org.apache.felix.scr.impl.manager.RegistrationManager.changeRegistration(RegistrationManager.java:134)
at org.apache.felix.scr.impl.manager.AbstractComponentManager.registerService(AbstractComponentManager.java:1044)
at org.apache.felix.scr.impl.manager.AbstractComponentManager.activateInternal(AbstractComponentManager.java:841)
at org.apache.felix.scr.impl.manager.DependencyManager$SingleStaticCustomizer.addedService(DependencyManager.java:931)
at org.apache.felix.scr.impl.manager.DependencyManager$SingleStaticCustomizer.addedService(DependencyManager.java:895)
at org.apache.felix.scr.impl.manager.ServiceTracker$Tracked.customizerAdded(ServiceTracker.java:1480)[73:org.apache.felix.scr:1.8.2]
at org.apache.felix.scr.impl.manager.ServiceTracker$Tracked.customizerAdded(ServiceTracker.java:1401)[73:org.apache.felix.scr:1.8.2]
at org.apache.felix.scr.impl.manager.ServiceTracker$AbstractTracked.trackAdding(ServiceTracker.java:1210)
at org.apache.felix.scr.impl.manager.ServiceTracker$AbstractTracked.track(ServiceTracker.java:1148)
at org.apache.felix.scr.impl.manager.ServiceTracker$Tracked.serviceChanged(ServiceTracker.java:1432)[73:org.apache.felix.scr:1.8.2]
at org.apache.felix.framework.util.EventDispatcher.invokeServiceListenerCallback(EventDispatcher.java:932)[org.apache.felix.framework-4.2.1.jar:]
at org.apache.felix.framework.util.EventDispatcher.fireEventImmediately(EventDispatcher.java:793)[org.apache.felix.framework-4.2.1.jar:]
at org.apache.felix.framework.util.EventDispatcher.fireServiceEvent(EventDispatcher.java:543)[org.apache.felix.framework-4.2.1.jar:]
at org.apache.felix.framework.Felix.fireServiceEvent(Felix.java:4419)[org.apache.felix.framework-4.2.1.jar:]
at org.apache.felix.framework.Felix.registerService(Felix.java:3423)[org.apache.felix.framework-4.2.1.jar:]
at org.apache.felix.framework.BundleContextImpl.registerService(BundleContextImpl.java:346)
at org.apache.felix.scr.impl.manager.AbstractComponentManager$3.register(AbstractComponentManager.java:1003)
at org.apache.felix.scr.impl.manager.AbstractComponentManager$3.register(AbstractComponentManager.java:992)
at org.apache.felix.scr.impl.manager.RegistrationManager.changeRegistration(RegistrationManager.java:134)
at org.apache.felix.scr.impl.manager.AbstractComponentManager.registerService(AbstractComponentManager.java:1044)
at org.apache.felix.scr.impl.manager.AbstractComponentManager.activateInternal(AbstractComponentManager.java:841)
at org.apache.felix.scr.impl.manager.AbstractComponentManager.enable(AbstractComponentManager.java:419)
at org.apache.felix.scr.impl.config.ConfigurableComponentHolder.enableComponents(ConfigurableComponentHolder.java:376)
at org.apache.felix.scr.impl.BundleComponentActivator.initialize(BundleComponentActivator.java:172)
at org.apache.felix.scr.impl.BundleComponentActivator.<init>(BundleComponentActivator.java:120)
at org.apache.felix.scr.impl.Activator.loadComponents(Activator.java:258)
at org.apache.felix.scr.impl.Activator.access$000(Activator.java:45)
at org.apache.felix.scr.impl.Activator$ScrExtension.start(Activator.java:185)
at org.apache.felix.utils.extender.AbstractExtender.createExtension(AbstractExtender.java:259)
at org.apache.felix.utils.extender.AbstractExtender.modifiedBundle(AbstractExtender.java:232)
at org.osgi.util.tracker.BundleTracker$Tracked.customizerModified(BundleTracker.java:479)[karaf-org.osgi.core.jar:]
at org.osgi.util.tracker.BundleTracker$Tracked.customizerModified(BundleTracker.java:414)[karaf-org.osgi.core.jar:]
at org.osgi.util.tracker.AbstractTracked.track(AbstractTracked.java:232)[karaf-org.osgi.core.jar:]
at org.osgi.util.tracker.BundleTracker$Tracked.bundleChanged(BundleTracker.java:443)[karaf-org.osgi.core.jar:]
at org.apache.felix.framework.util.EventDispatcher.invokeBundleListenerCallback(EventDispatcher.java:868)[org.apache.felix.framework-4.2.1.jar:]
at org.apache.felix.framework.util.EventDispatcher.fireEventImmediately(EventDispatcher.java:789)[org.apache.felix.framework-4.2.1.jar:]
at org.apache.felix.framework.util.EventDispatcher.fireBundleEvent(EventDispatcher.java:514)[org.apache.felix.framework-4.2.1.jar:]
at org.apache.felix.framework.Felix.fireBundleEvent(Felix.java:4403)[org.apache.felix.framework-4.2.1.jar:]
at org.apache.felix.framework.Felix.startBundle(Felix.java:2092)[org.apache.felix.framework-4.2.1.jar:]
at org.apache.felix.framework.BundleImpl.start(BundleImpl.java:955)
at org.apache.felix.framework.BundleImpl.start(BundleImpl.java:942)
at org.apache.karaf.features.internal.FeaturesServiceImpl.startBundle(FeaturesServiceImpl.java:516)
at org.apache.karaf.features.internal.FeaturesServiceImpl.installFeatures(FeaturesServiceImpl.java:474)
at org.apache.karaf.features.internal.BootFeaturesInstaller.installBootFeatures(BootFeaturesInstaller.java:93)
at org.apache.karaf.features.internal.BootFeaturesInstaller.start(BootFeaturesInstaller.java:77)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)[:1.8.0_111]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)[:1.8.0_111]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)[:1.8.0_111]
at java.lang.reflect.Method.invoke(Method.java:498)[:1.8.0_111]
at org.apache.aries.blueprint.utils.ReflectionUtils.invoke(ReflectionUtils.java:299)[15:org.apache.aries.blueprint.core:1.6.1]
at org.apache.aries.blueprint.container.BeanRecipe.invoke(BeanRecipe.java:980)[15:org.apache.aries.blueprint.core:1.6.1]
at org.apache.aries.blueprint.container.BeanRecipe.runBeanProcInit(BeanRecipe.java:736)[15:org.apache.aries.blueprint.core:1.6.1]
at org.apache.aries.blueprint.container.BeanRecipe.internalCreate2(BeanRecipe.java:848)[15:org.apache.aries.blueprint.core:1.6.1]
at org.apache.aries.blueprint.container.BeanRecipe.internalCreate(BeanRecipe.java:811)[15:org.apache.aries.blueprint.core:1.6.1]
at org.apache.aries.blueprint.di.AbstractRecipe$1.call(AbstractRecipe.java:79)[15:org.apache.aries.blueprint.core:1.6.1]
at java.util.concurrent.FutureTask.run(FutureTask.java:266)[:1.8.0_111]
at org.apache.aries.blueprint.di.AbstractRecipe.create(AbstractRecipe.java:88)[15:org.apache.aries.blueprint.core:1.6.1]
at org.apache.aries.blueprint.container.BlueprintRepository.createInstances(BlueprintRepository.java:255)[15:org.apache.aries.blueprint.core:1.6.1]
at org.apache.aries.blueprint.container.BlueprintRepository.createAll(BlueprintRepository.java:186)[15:org.apache.aries.blueprint.core:1.6.1]
at org.apache.aries.blueprint.container.BlueprintContainerImpl.instantiateEagerComponents(BlueprintContainerImpl.java:724)[15:org.apache.aries.blueprint.core:1.6.1]
at org.apache.aries.blueprint.container.BlueprintContainerImpl.doRun(BlueprintContainerImpl.java:411)[15:org.apache.aries.blueprint.core:1.6.1]
at org.apache.aries.blueprint.container.BlueprintContainerImpl.run(BlueprintContainerImpl.java:276)[15:org.apache.aries.blueprint.core:1.6.1]
at org.apache.aries.blueprint.container.BlueprintExtender.createContainer(BlueprintExtender.java:300)[15:org.apache.aries.blueprint.core:1.6.1]
at org.apache.aries.blueprint.container.BlueprintExtender.createContainer(BlueprintExtender.java:269)[15:org.apache.aries.blueprint.core:1.6.1]
at org.apache.aries.blueprint.container.BlueprintExtender.createContainer(BlueprintExtender.java:265)[15:org.apache.aries.blueprint.core:1.6.1]
at org.apache.aries.blueprint.container.BlueprintExtender.modifiedBundle(BlueprintExtender.java:255)[15:org.apache.aries.blueprint.core:1.6.1]
at org.apache.aries.util.tracker.hook.BundleHookBundleTracker$Tracked.customizerModified(BundleHookBundleTracker.java:500)[9:org.apache.aries.util:1.1.1]
at org.apache.aries.util.tracker.hook.BundleHookBundleTracker$Tracked.customizerModified(BundleHookBundleTracker.java:433)[9:org.apache.aries.util:1.1.1]
at org.apache.aries.util.tracker.hook.BundleHookBundleTracker$AbstractTracked.track(BundleHookBundleTracker.java:725)[9:org.apache.aries.util:1.1.1]
at org.apache.aries.util.tracker.hook.BundleHookBundleTracker$Tracked.bundleChanged(BundleHookBundleTracker.java:463)[9:org.apache.aries.util:1.1.1]
at org.apache.aries.util.tracker.hook.BundleHookBundleTracker$BundleEventHook.event(BundleHookBundleTracker.java:422)[9:org.apache.aries.util:1.1.1]
at org.apache.felix.framework.util.SecureAction.invokeBundleEventHook(SecureAction.java:1103)[org.apache.felix.framework-4.2.1.jar:]
at org.apache.felix.framework.util.EventDispatcher.createWhitelistFromHooks(EventDispatcher.java:695)[org.apache.felix.framework-4.2.1.jar:]
at org.apache.felix.framework.util.EventDispatcher.fireBundleEvent(EventDispatcher.java:483)[org.apache.felix.framework-4.2.1.jar:]
at org.apache.felix.framework.Felix.fireBundleEvent(Felix.java:4403)[org.apache.felix.framework-4.2.1.jar:]
at org.apache.felix.framework.Felix.startBundle(Felix.java:2092)[org.apache.felix.framework-4.2.1.jar:]
at org.apache.felix.framework.Felix.setActiveStartLevel(Felix.java:1291)[org.apache.felix.framework-4.2.1.jar:]
at org.apache.felix.framework.FrameworkStartLevelImpl.run(FrameworkStartLevelImpl.java:304)[org.apache.felix.framework-4.2.1.jar:]
at java.lang.Thread.run(Thread.java:745)[:1.8.0_111]
Caused by: java.io.IOException: Unable to load metadata from mpeg7 catalog http://localhost:8080/static/mh_default_org/engage-player/9c13730c-f13a-46a8-b5f0-7c899d202411/f38c7d38-08f4-4b02-b796-1697d92e9ccf/slidetext.xml
at org.opencastproject.search.impl.solr.SolrIndexManager.loadMpeg7Catalog(SolrIndexManager.java:697)
at org.opencastproject.search.impl.solr.SolrIndexManager.createEpisodeInputDocument(SolrIndexManager.java:400)
at org.opencastproject.search.impl.solr.SolrIndexManager.add(SolrIndexManager.java:328)
... 144 more
2016-12-22 12:50:22,775 | ERROR | (SearchServiceImpl:547) - Skipped 1 erroneous search entries while populating the search index

Activity

Show:
Greg Logan
May 27, 2017, 6:55 AM

Ugh, ok this is actually some kind of race condition upon further investigation. STR:

1) Process one or more videos, ensure the slide text is present as expected
2) Stop Opencast, clear the search Solr index, start Opencast
3a) Load the video, do not get the slide text even though it is present in the mediapackage
3b) Load the video, get the slide text
3c) Load the video, get the error page

Whether you get a, b, or c appears to be more or less random but whatever you get will persist across all current published recordings until you repeat from step 2.

Greg Logan
June 8, 2017, 10:29 AM

Downgrading to critical since the issue is not that the index can't be rebuilt at all, just that you lose existing slidetext.

Lars Kiesow
June 8, 2017, 7:50 PM

rrolf had the idea that the issue might be that slidetexts are read from the WS/WFR from which they are removed at the end of the workflow by the cleanup operation. I did not test/confirm this, but it sounds reasonable enough to investigate this further.

Dietmar Zenker
September 7, 2017, 7:57 PM

I've just realized the same problem after updating from 2.3.2 to 3.2 and trying to rebuild the SOLR index on the presentation node. After several trials, IMHO the problem is caused when the SOLR indexer starts BEFORE the spring security context is established, and therefore /static is not accessible (that's what the status code 403 indicates).

Here is a part of the log of a trial when the rebuild completely fails:
(lots of error messages here)
...
2017-09-06 12:48:52,290 | ERROR | (SolrIndexManager:404) - Error loading mpeg7 catalog. Skipping catalog: Unable to load metadata from mpeg7 catalog https://my.opencast.domain:8443/static/mh_default_org/engage-player/ff8f5f16-769a-4615-8a9a-b13cda080442/c392825b-54af-4a7b-aa2a-8b1125bc0692/slidetext.xml
2017-09-06 12:48:52,940 | WARN | (WorkspaceImpl:367) - Received unexpected response status 403 while trying to download from https://my.opencast.domain:8443/static/mh_default_org/engage-player/ffaba9b6-7590-4172-9928-05a0981167f5/dc127423-ec2a-46f6-a2de-909c3c736afc/catalog.xml
2017-09-06 12:48:52,950 | WARN | (StaticMetadataServiceDublinCoreImpl:395) - Unable to load metadata from catalog 'https://my.opencast.domain:8443/static/mh_default_org/engage-player/ffaba9b6-7590-4172-9928-05a0981167f5/dc127423-ec2a-46f6-a2de-909c3c736afc/catalog.xml'
2017-09-06 12:48:53,023 | WARN | (WorkspaceImpl:367) - Received unexpected response status 403 while trying to download from https://my.opencast.domain:8443/static/mh_default_org/engage-player/ffaba9b6-7590-4172-9928-05a0981167f5/98af7415-21ea-467e-98ed-c10d8d4471ad/slidetext.xml
2017-09-06 12:48:53,034 | ERROR | (SolrIndexManager:404) - Error loading mpeg7 catalog. Skipping catalog: Unable to load metadata from mpeg7 catalog https://my.opencast.domain:8443/static/mh_default_org/engage-player/ffaba9b6-7590-4172-9928-05a0981167f5/98af7415-21ea-467e-98ed-c10d8d4471ad/slidetext.xml
2017-09-06 12:48:53,647 | INFO | (SearchServiceImpl:554) - Finished populating search index
2017-09-06 12:48:53,724 | INFO | (RestPublisher:333) - Registered REST endpoint at /search
...
2017-09-06 12:49:03,824 | INFO | (SpringSecurityConfigurationArtifactInstaller:96) - registered OsgiBundleXmlApplicationContext(bundle=matterhorn-kernel, config=file:/etc/opencast/security/mh_default_org.xml): startup date [Thu Jan 01 01:00:00 CET 1970]; root of context hierarchy for mh_default_org
...

And here the code of another try when the spring security context is established right at the beginning of the SOLR indexing operation:
...
2017-09-07 09:46:05,389 | INFO | (SearchServiceImpl:195) - Setting up solr server
2017-09-07 09:46:05,393 | INFO | (SearchServiceImpl:253) - Setting up solr search index at /srv/opencast/solr-indexes/search
2017-09-07 09:46:05,394 | INFO | (SearchServiceImpl:260) - solr config directory doesn't exist. Creating /srv/opencast/solr-indexes/search/conf
2017-09-07 09:46:05,889 | WARN | (SolrCore:389) - Solr index directory '/srv/opencast/solr-indexes/search/data/index' doesn't exist. Creating new index...
2017-09-07 09:46:06,188 | INFO | (SearchServiceImpl:515) - No search index found
2017-09-07 09:46:06,189 | INFO | (SearchServiceImpl:516) - Starting population of search index from database
2017-09-07 09:46:07,045 | WARN | (WorkspaceImpl:367) - Received unexpected response status 403 while trying to download from https://my.opencast.domain:8443/static/mh_default_org/engage-player/006c1836-1579-442a-9a82-449666260fbd/de28a2bc-5ef4-4012-850d-c99e0a29a728/catalog.xml
2017-09-07 09:46:07,136 | WARN | (StaticMetadataServiceDublinCoreImpl:395) - Unable to load metadata from catalog 'https://my.opencast.domain:8443/static/mh_default_org/engage-player/006c1836-1579-442a-9a82-449666260fbd/de28a2bc-5ef4-4012-850d-c99e0a29a728/catalog.xml'
2017-09-07 09:46:07,148 | INFO | (SpringSecurityConfigurationArtifactInstaller:96) - registered OsgiBundleXmlApplicationContext(bundle=matterhorn-kernel, config=file:/etc/opencast/security/mh_default_org.xml): startup date [Thu Jan 01 01:00:00 CET 1970]; root of context hierarchy for mh_default_org
2017-09-07 09:46:07,300 | WARN | (WorkspaceImpl:367) - Received unexpected response status 403 while trying to download from https://my.opencast.domain:8443/static/mh_default_org/engage-player/006c1836-1579-442a-9a82-449666260fbd/c6c07819-8f87-4569-bba4-719bd81774e7/slidetext.xml
2017-09-07 09:46:07,441 | ERROR | (SolrIndexManager:404) - Error loading mpeg7 catalog. Skipping catalog: Unable to load metadata from mpeg7 catalog https://my.opencast.domain:8443/static/mh_default_org/engage-player/006c1836-1579-442a-9a82-449666260fbd/c6c07819-8f87-4569-bba4-719bd81774e7/slidetext.xml
2017-09-07 09:46:08,356 | WARN | (StaticMetadataServiceDublinCoreImpl:395) - Unable to load metadata from catalog 'https://my.opencast.domain:8443/static/mh_default_org/engage-player/01b5eaab-9a63-450a-8d11-392564b89df7/35d4dfe3-ad5e-4770-b343-c90ae473b837/dublincore.xml'
2017-09-07 09:46:08,507 | ERROR | (SolrIndexManager:404) - Error loading mpeg7 catalog. Skipping catalog: Unable to load metadata from mpeg7 catalog https://my.opencast.domain:8443/static/mh_default_org/engage-player/01b5eaab-9a63-450a-8d11-392564b89df7/377565ce-a020-45ce-b1da-91f81dfa7a67/slidetext.xml
2017-09-07 09:46:09,082 | INFO | (WorkspaceImpl:364) - Downloading https://my.opencast.domain:8443/static/mh_default_org/engage-player/01d68fea-9cb4-41ca-9610-e7e19602d052/5e31315c-b117-4a0a-8346-bef4958dec66/catalog.xml to /srv/opencast/workspace/https_my.opencast.domain_8443/static/mh_default_org/engage-player/01d68fea-9cb4-41ca-9610-e7e19602d052/5e31315c-b117-4a0a-8346-bef4958dec66/catalog.xml
2017-09-07 09:46:09,233 | INFO | (WorkspaceImpl:364) - Downloading https://my.opencast.domain:8443/static/mh_default_org/engage-player/01d68fea-9cb4-41ca-9610-e7e19602d052/fe09a1d9-6b16-4aa4-98d1-fe94b9507251/slidetext.xml to /srv/opencast/workspace/https_my.opencast.domain_8443/static/mh_default_org/engage-player/01d68fea-9cb4-41ca-9610-e7e19602d052/fe09a1d9-6b16-4aa4-98d1-fe94b9507251/slidetext.xml
2017-09-07 09:46:09,803 | INFO | (WorkspaceImpl:364) - Downloading https://my.opencast.domain:8443/static/mh_default_org/engage-player/025fded5-cc6a-473b-8c2d-ac0a8b7a38f2/a25726a0-62d4-4870-999b-015796c30111/dublincore.xml to /srv/opencast/workspace/https_my.opencast.domain_8443/static/mh_default_org/engage-player/025fded5-cc6a-473b-8c2d-ac0a8b7a38f2/a25726a0-62d4-4870-999b-015796c30111/dublincore.xml
... (from here everything is OK and the index will be nearly completely rebuild when the indexer has finished!)

Could this be the origin of the problem?

Former user
November 17, 2017, 6:13 AM

If this issue is that the search service starting the index rebuild too early, then loading the search service later may resolve this too. Ref: https://bitbucket.org/opencast-community/opencast/commits/28a22d31694938cbf1d261ac2d126671593bad1d

Assignee

Lars Kiesow

Reporter

Lars Kiesow

Severity

Non Functioning

Tags (folksonomy)

None

Components

Fix versions

Affects versions

Priority

Critical