Ingest two videos: 1x presenter 1x presentation
Stop Opencast
Remove data/solr-index/search
Start Opencast
Get a lot of errors
The indexer cannot access all metadata catalogs (403 error)
Trying to add the slidetexts the event adding will completely fail
Stack trace:
2016-12-22 12:50:22,325 | INFO | (SearchServiceImpl:195) - Setting up solr server
2016-12-22 12:50:22,326 | INFO | (SearchServiceImpl:253) - Setting up solr search index at /home/lars/dev/opencast/build/opencast-dist-allinone-2.3-SNAPSHOT/data/solr-indexes/search
2016-12-22 12:50:22,326 | INFO | (SearchServiceImpl:260) - solr config directory doesn't exist. Creating /home/lars/dev/opencast/build/opencast-dist-allinone-2.3-SNAPSHOT/data/solr-indexes/search/conf
2016-12-22 12:50:22,391 | WARN | (SolrCore:389) - Solr index directory '/home/lars/dev/opencast/build/opencast-dist-allinone-2.3-SNAPSHOT/data/solr-indexes/search/data/index' doesn't exist. Creating new index...
2016-12-22 12:50:22,465 | INFO | (SearchServiceImpl:509) - No search index found
2016-12-22 12:50:22,466 | INFO | (SearchServiceImpl:510) - Starting population of search index from database
2016-12-22 12:50:22,609 | WARN | (WorkspaceImpl:330) - Received unexpected response status 403 while trying to download from http://localhost:8080/static/mh_default_org/engage-player/f25f4cd4-6d1d-40be-b6a5-4e14536a61e1/9352000d-cacb-4287-a9b3-1e5ba4525ea9/dublincore.xml
2016-12-22 12:50:22,614 | WARN | (StaticMetadataServiceDublinCoreImpl:317) - Unable to load metadata from catalog 'http://localhost:8080/static/mh_default_org/engage-player/f25f4cd4-6d1d-40be-b6a5-4e14536a61e1/9352000d-cacb-4287-a9b3-1e5ba4525ea9/dublincore.xml'
2016-12-22 12:50:22,741 | WARN | (WorkspaceImpl:330) - Received unexpected response status 403 while trying to download from http://localhost:8080/static/mh_default_org/engage-player/9c13730c-f13a-46a8-b5f0-7c899d202411/42249fe3-3038-4dc0-a7dc-d32dc1e2c412/dublincore.xml
2016-12-22 12:50:22,742 | WARN | (StaticMetadataServiceDublinCoreImpl:317) - Unable to load metadata from catalog 'http://localhost:8080/static/mh_default_org/engage-player/9c13730c-f13a-46a8-b5f0-7c899d202411/42249fe3-3038-4dc0-a7dc-d32dc1e2c412/dublincore.xml'
2016-12-22 12:50:22,747 | WARN | (WorkspaceImpl:330) - Received unexpected response status 403 while trying to download from http://localhost:8080/static/mh_default_org/engage-player/9c13730c-f13a-46a8-b5f0-7c899d202411/f38c7d38-08f4-4b02-b796-1697d92e9ccf/slidetext.xml
2016-12-22 12:50:22,749 | ERROR | (SearchServiceImpl:535) - Unable to index search instances: {}
org.apache.solr.client.solrj.SolrServerException: java.io.IOException: Unable to load metadata from mpeg7 catalog http://localhost:8080/static/mh_default_org/engage-player/9c13730c-f13a-46a8-b5f0-7c899d202411/f38c7d38-08f4-4b02-b796-1697d92e9ccf/slidetext.xml
at org.opencastproject.search.impl.solr.SolrIndexManager.add(SolrIndexManager.java:348)
at org.opencastproject.search.impl.SearchServiceImpl.populateIndex(SearchServiceImpl.java:533)
at org.opencastproject.search.impl.SearchServiceImpl.activate(SearchServiceImpl.java:236)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)[:1.8.0_111]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)[:1.8.0_111]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)[:1.8.0_111]
at java.lang.reflect.Method.invoke(Method.java:498)[:1.8.0_111]
at org.apache.felix.scr.impl.helper.BaseMethod.invokeMethod(BaseMethod.java:231)
at org.apache.felix.scr.impl.helper.BaseMethod.access$500(BaseMethod.java:39)
at org.apache.felix.scr.impl.helper.BaseMethod$Resolved.invoke(BaseMethod.java:624)
at org.apache.felix.scr.impl.helper.BaseMethod.invoke(BaseMethod.java:508)
at org.apache.felix.scr.impl.helper.ActivateMethod.invoke(ActivateMethod.java:149)
at org.apache.felix.scr.impl.manager.SingleComponentManager.createImplementationObject(SingleComponentManager.java:315)
at org.apache.felix.scr.impl.manager.SingleComponentManager.createComponent(SingleComponentManager.java:127)
at org.apache.felix.scr.impl.manager.SingleComponentManager.getService(SingleComponentManager.java:871)
at org.apache.felix.scr.impl.manager.SingleComponentManager.getServiceInternal(SingleComponentManager.java:838)
at org.apache.felix.scr.impl.manager.SingleComponentManager.getService(SingleComponentManager.java:777)
at org.apache.felix.framework.ServiceRegistrationImpl.getFactoryUnchecked(ServiceRegistrationImpl.java:308)
at org.apache.felix.framework.ServiceRegistrationImpl.getService(ServiceRegistrationImpl.java:219)
at org.apache.felix.framework.ServiceRegistry.getService(ServiceRegistry.java:320)
at org.apache.felix.framework.Felix.getService(Felix.java:3568)[org.apache.felix.framework-4.2.1.jar:]
at org.apache.felix.framework.BundleContextImpl.getService(BundleContextImpl.java:468)
at org.apache.felix.scr.impl.helper.BindMethod.getServiceObject(BindMethod.java:572)
at org.apache.felix.scr.impl.manager.DependencyManager.getServiceObject(DependencyManager.java:2012)
at org.apache.felix.scr.impl.manager.DependencyManager$SingleStaticCustomizer.prebind(DependencyManager.java:1005)
at org.apache.felix.scr.impl.manager.DependencyManager.prebind(DependencyManager.java:1439)
at org.apache.felix.scr.impl.manager.AbstractComponentManager.collectDependencies(AbstractComponentManager.java:1119)
at org.apache.felix.scr.impl.manager.SingleComponentManager.getServiceInternal(SingleComponentManager.java:807)
at org.apache.felix.scr.impl.manager.SingleComponentManager.getService(SingleComponentManager.java:777)
at org.apache.felix.framework.ServiceRegistrationImpl.getFactoryUnchecked(ServiceRegistrationImpl.java:308)
at org.apache.felix.framework.ServiceRegistrationImpl.getService(ServiceRegistrationImpl.java:219)
at org.apache.felix.framework.ServiceRegistry.getService(ServiceRegistry.java:320)
at org.apache.felix.framework.Felix.getService(Felix.java:3568)[org.apache.felix.framework-4.2.1.jar:]
at org.apache.felix.framework.BundleContextImpl.getService(BundleContextImpl.java:468)
at org.apache.felix.fileinstall.internal.FileInstall.addingService(FileInstall.java:150)
at org.osgi.util.tracker.ServiceTracker$Tracked.customizerAdding(ServiceTracker.java:932)[karaf-org.osgi.core.jar:]
at org.osgi.util.tracker.ServiceTracker$Tracked.customizerAdding(ServiceTracker.java:864)[karaf-org.osgi.core.jar:]
at org.osgi.util.tracker.AbstractTracked.trackAdding(AbstractTracked.java:256)[karaf-org.osgi.core.jar:]
at org.osgi.util.tracker.AbstractTracked.track(AbstractTracked.java:229)[karaf-org.osgi.core.jar:]
at org.osgi.util.tracker.ServiceTracker$Tracked.serviceChanged(ServiceTracker.java:894)[karaf-org.osgi.core.jar:]
at org.apache.felix.framework.util.EventDispatcher.invokeServiceListenerCallback(EventDispatcher.java:932)[org.apache.felix.framework-4.2.1.jar:]
at org.apache.felix.framework.util.EventDispatcher.fireEventImmediately(EventDispatcher.java:793)[org.apache.felix.framework-4.2.1.jar:]
at org.apache.felix.framework.util.EventDispatcher.fireServiceEvent(EventDispatcher.java:543)[org.apache.felix.framework-4.2.1.jar:]
at org.apache.felix.framework.Felix.fireServiceEvent(Felix.java:4419)[org.apache.felix.framework-4.2.1.jar:]
at org.apache.felix.framework.Felix.registerService(Felix.java:3423)[org.apache.felix.framework-4.2.1.jar:]
at org.apache.felix.framework.BundleContextImpl.registerService(BundleContextImpl.java:346)
at org.apache.felix.scr.impl.manager.AbstractComponentManager$3.register(AbstractComponentManager.java:1003)
at org.apache.felix.scr.impl.manager.AbstractComponentManager$3.register(AbstractComponentManager.java:992)
at org.apache.felix.scr.impl.manager.RegistrationManager.changeRegistration(RegistrationManager.java:134)
at org.apache.felix.scr.impl.manager.AbstractComponentManager.registerService(AbstractComponentManager.java:1044)
at org.apache.felix.scr.impl.manager.AbstractComponentManager.activateInternal(AbstractComponentManager.java:841)
at org.apache.felix.scr.impl.manager.DependencyManager$SingleStaticCustomizer.addedService(DependencyManager.java:931)
at org.apache.felix.scr.impl.manager.DependencyManager$SingleStaticCustomizer.addedService(DependencyManager.java:895)
at org.apache.felix.scr.impl.manager.ServiceTracker$Tracked.customizerAdded(ServiceTracker.java:1480)[73:org.apache.felix.scr:1.8.2]
at org.apache.felix.scr.impl.manager.ServiceTracker$Tracked.customizerAdded(ServiceTracker.java:1401)[73:org.apache.felix.scr:1.8.2]
at org.apache.felix.scr.impl.manager.ServiceTracker$AbstractTracked.trackAdding(ServiceTracker.java:1210)
at org.apache.felix.scr.impl.manager.ServiceTracker$AbstractTracked.track(ServiceTracker.java:1148)
at org.apache.felix.scr.impl.manager.ServiceTracker$Tracked.serviceChanged(ServiceTracker.java:1432)[73:org.apache.felix.scr:1.8.2]
at org.apache.felix.framework.util.EventDispatcher.invokeServiceListenerCallback(EventDispatcher.java:932)[org.apache.felix.framework-4.2.1.jar:]
at org.apache.felix.framework.util.EventDispatcher.fireEventImmediately(EventDispatcher.java:793)[org.apache.felix.framework-4.2.1.jar:]
at org.apache.felix.framework.util.EventDispatcher.fireServiceEvent(EventDispatcher.java:543)[org.apache.felix.framework-4.2.1.jar:]
at org.apache.felix.framework.Felix.fireServiceEvent(Felix.java:4419)[org.apache.felix.framework-4.2.1.jar:]
at org.apache.felix.framework.Felix.registerService(Felix.java:3423)[org.apache.felix.framework-4.2.1.jar:]
at org.apache.felix.framework.BundleContextImpl.registerService(BundleContextImpl.java:346)
at org.apache.felix.scr.impl.manager.AbstractComponentManager$3.register(AbstractComponentManager.java:1003)
at org.apache.felix.scr.impl.manager.AbstractComponentManager$3.register(AbstractComponentManager.java:992)
at org.apache.felix.scr.impl.manager.RegistrationManager.changeRegistration(RegistrationManager.java:134)
at org.apache.felix.scr.impl.manager.AbstractComponentManager.registerService(AbstractComponentManager.java:1044)
at org.apache.felix.scr.impl.manager.AbstractComponentManager.activateInternal(AbstractComponentManager.java:841)
at org.apache.felix.scr.impl.manager.DependencyManager$SingleStaticCustomizer.addedService(DependencyManager.java:931)
at org.apache.felix.scr.impl.manager.DependencyManager$SingleStaticCustomizer.addedService(DependencyManager.java:895)
at org.apache.felix.scr.impl.manager.ServiceTracker$Tracked.customizerAdded(ServiceTracker.java:1480)[73:org.apache.felix.scr:1.8.2]
at org.apache.felix.scr.impl.manager.ServiceTracker$Tracked.customizerAdded(ServiceTracker.java:1401)[73:org.apache.felix.scr:1.8.2]
at org.apache.felix.scr.impl.manager.ServiceTracker$AbstractTracked.trackAdding(ServiceTracker.java:1210)
at org.apache.felix.scr.impl.manager.ServiceTracker$AbstractTracked.track(ServiceTracker.java:1148)
at org.apache.felix.scr.impl.manager.ServiceTracker$Tracked.serviceChanged(ServiceTracker.java:1432)[73:org.apache.felix.scr:1.8.2]
at org.apache.felix.framework.util.EventDispatcher.invokeServiceListenerCallback(EventDispatcher.java:932)[org.apache.felix.framework-4.2.1.jar:]
at org.apache.felix.framework.util.EventDispatcher.fireEventImmediately(EventDispatcher.java:793)[org.apache.felix.framework-4.2.1.jar:]
at org.apache.felix.framework.util.EventDispatcher.fireServiceEvent(EventDispatcher.java:543)[org.apache.felix.framework-4.2.1.jar:]
at org.apache.felix.framework.Felix.fireServiceEvent(Felix.java:4419)[org.apache.felix.framework-4.2.1.jar:]
at org.apache.felix.framework.Felix.registerService(Felix.java:3423)[org.apache.felix.framework-4.2.1.jar:]
at org.apache.felix.framework.BundleContextImpl.registerService(BundleContextImpl.java:346)
at org.apache.felix.scr.impl.manager.AbstractComponentManager$3.register(AbstractComponentManager.java:1003)
at org.apache.felix.scr.impl.manager.AbstractComponentManager$3.register(AbstractComponentManager.java:992)
at org.apache.felix.scr.impl.manager.RegistrationManager.changeRegistration(RegistrationManager.java:134)
at org.apache.felix.scr.impl.manager.AbstractComponentManager.registerService(AbstractComponentManager.java:1044)
at org.apache.felix.scr.impl.manager.AbstractComponentManager.activateInternal(AbstractComponentManager.java:841)
at org.apache.felix.scr.impl.manager.AbstractComponentManager.enable(AbstractComponentManager.java:419)
at org.apache.felix.scr.impl.config.ConfigurableComponentHolder.enableComponents(ConfigurableComponentHolder.java:376)
at org.apache.felix.scr.impl.BundleComponentActivator.initialize(BundleComponentActivator.java:172)
at org.apache.felix.scr.impl.BundleComponentActivator.<init>(BundleComponentActivator.java:120)
at org.apache.felix.scr.impl.Activator.loadComponents(Activator.java:258)
at org.apache.felix.scr.impl.Activator.access$000(Activator.java:45)
at org.apache.felix.scr.impl.Activator$ScrExtension.start(Activator.java:185)
at org.apache.felix.utils.extender.AbstractExtender.createExtension(AbstractExtender.java:259)
at org.apache.felix.utils.extender.AbstractExtender.modifiedBundle(AbstractExtender.java:232)
at org.osgi.util.tracker.BundleTracker$Tracked.customizerModified(BundleTracker.java:479)[karaf-org.osgi.core.jar:]
at org.osgi.util.tracker.BundleTracker$Tracked.customizerModified(BundleTracker.java:414)[karaf-org.osgi.core.jar:]
at org.osgi.util.tracker.AbstractTracked.track(AbstractTracked.java:232)[karaf-org.osgi.core.jar:]
at org.osgi.util.tracker.BundleTracker$Tracked.bundleChanged(BundleTracker.java:443)[karaf-org.osgi.core.jar:]
at org.apache.felix.framework.util.EventDispatcher.invokeBundleListenerCallback(EventDispatcher.java:868)[org.apache.felix.framework-4.2.1.jar:]
at org.apache.felix.framework.util.EventDispatcher.fireEventImmediately(EventDispatcher.java:789)[org.apache.felix.framework-4.2.1.jar:]
at org.apache.felix.framework.util.EventDispatcher.fireBundleEvent(EventDispatcher.java:514)[org.apache.felix.framework-4.2.1.jar:]
at org.apache.felix.framework.Felix.fireBundleEvent(Felix.java:4403)[org.apache.felix.framework-4.2.1.jar:]
at org.apache.felix.framework.Felix.startBundle(Felix.java:2092)[org.apache.felix.framework-4.2.1.jar:]
at org.apache.felix.framework.BundleImpl.start(BundleImpl.java:955)
at org.apache.felix.framework.BundleImpl.start(BundleImpl.java:942)
at org.apache.karaf.features.internal.FeaturesServiceImpl.startBundle(FeaturesServiceImpl.java:516)
at org.apache.karaf.features.internal.FeaturesServiceImpl.installFeatures(FeaturesServiceImpl.java:474)
at org.apache.karaf.features.internal.BootFeaturesInstaller.installBootFeatures(BootFeaturesInstaller.java:93)
at org.apache.karaf.features.internal.BootFeaturesInstaller.start(BootFeaturesInstaller.java:77)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)[:1.8.0_111]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)[:1.8.0_111]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)[:1.8.0_111]
at java.lang.reflect.Method.invoke(Method.java:498)[:1.8.0_111]
at org.apache.aries.blueprint.utils.ReflectionUtils.invoke(ReflectionUtils.java:299)[15:org.apache.aries.blueprint.core:1.6.1]
at org.apache.aries.blueprint.container.BeanRecipe.invoke(BeanRecipe.java:980)[15:org.apache.aries.blueprint.core:1.6.1]
at org.apache.aries.blueprint.container.BeanRecipe.runBeanProcInit(BeanRecipe.java:736)[15:org.apache.aries.blueprint.core:1.6.1]
at org.apache.aries.blueprint.container.BeanRecipe.internalCreate2(BeanRecipe.java:848)[15:org.apache.aries.blueprint.core:1.6.1]
at org.apache.aries.blueprint.container.BeanRecipe.internalCreate(BeanRecipe.java:811)[15:org.apache.aries.blueprint.core:1.6.1]
at org.apache.aries.blueprint.di.AbstractRecipe$1.call(AbstractRecipe.java:79)[15:org.apache.aries.blueprint.core:1.6.1]
at java.util.concurrent.FutureTask.run(FutureTask.java:266)[:1.8.0_111]
at org.apache.aries.blueprint.di.AbstractRecipe.create(AbstractRecipe.java:88)[15:org.apache.aries.blueprint.core:1.6.1]
at org.apache.aries.blueprint.container.BlueprintRepository.createInstances(BlueprintRepository.java:255)[15:org.apache.aries.blueprint.core:1.6.1]
at org.apache.aries.blueprint.container.BlueprintRepository.createAll(BlueprintRepository.java:186)[15:org.apache.aries.blueprint.core:1.6.1]
at org.apache.aries.blueprint.container.BlueprintContainerImpl.instantiateEagerComponents(BlueprintContainerImpl.java:724)[15:org.apache.aries.blueprint.core:1.6.1]
at org.apache.aries.blueprint.container.BlueprintContainerImpl.doRun(BlueprintContainerImpl.java:411)[15:org.apache.aries.blueprint.core:1.6.1]
at org.apache.aries.blueprint.container.BlueprintContainerImpl.run(BlueprintContainerImpl.java:276)[15:org.apache.aries.blueprint.core:1.6.1]
at org.apache.aries.blueprint.container.BlueprintExtender.createContainer(BlueprintExtender.java:300)[15:org.apache.aries.blueprint.core:1.6.1]
at org.apache.aries.blueprint.container.BlueprintExtender.createContainer(BlueprintExtender.java:269)[15:org.apache.aries.blueprint.core:1.6.1]
at org.apache.aries.blueprint.container.BlueprintExtender.createContainer(BlueprintExtender.java:265)[15:org.apache.aries.blueprint.core:1.6.1]
at org.apache.aries.blueprint.container.BlueprintExtender.modifiedBundle(BlueprintExtender.java:255)[15:org.apache.aries.blueprint.core:1.6.1]
at org.apache.aries.util.tracker.hook.BundleHookBundleTracker$Tracked.customizerModified(BundleHookBundleTracker.java:500)[9:org.apache.aries.util:1.1.1]
at org.apache.aries.util.tracker.hook.BundleHookBundleTracker$Tracked.customizerModified(BundleHookBundleTracker.java:433)[9:org.apache.aries.util:1.1.1]
at org.apache.aries.util.tracker.hook.BundleHookBundleTracker$AbstractTracked.track(BundleHookBundleTracker.java:725)[9:org.apache.aries.util:1.1.1]
at org.apache.aries.util.tracker.hook.BundleHookBundleTracker$Tracked.bundleChanged(BundleHookBundleTracker.java:463)[9:org.apache.aries.util:1.1.1]
at org.apache.aries.util.tracker.hook.BundleHookBundleTracker$BundleEventHook.event(BundleHookBundleTracker.java:422)[9:org.apache.aries.util:1.1.1]
at org.apache.felix.framework.util.SecureAction.invokeBundleEventHook(SecureAction.java:1103)[org.apache.felix.framework-4.2.1.jar:]
at org.apache.felix.framework.util.EventDispatcher.createWhitelistFromHooks(EventDispatcher.java:695)[org.apache.felix.framework-4.2.1.jar:]
at org.apache.felix.framework.util.EventDispatcher.fireBundleEvent(EventDispatcher.java:483)[org.apache.felix.framework-4.2.1.jar:]
at org.apache.felix.framework.Felix.fireBundleEvent(Felix.java:4403)[org.apache.felix.framework-4.2.1.jar:]
at org.apache.felix.framework.Felix.startBundle(Felix.java:2092)[org.apache.felix.framework-4.2.1.jar:]
at org.apache.felix.framework.Felix.setActiveStartLevel(Felix.java:1291)[org.apache.felix.framework-4.2.1.jar:]
at org.apache.felix.framework.FrameworkStartLevelImpl.run(FrameworkStartLevelImpl.java:304)[org.apache.felix.framework-4.2.1.jar:]
at java.lang.Thread.run(Thread.java:745)[:1.8.0_111]
Caused by: java.io.IOException: Unable to load metadata from mpeg7 catalog http://localhost:8080/static/mh_default_org/engage-player/9c13730c-f13a-46a8-b5f0-7c899d202411/f38c7d38-08f4-4b02-b796-1697d92e9ccf/slidetext.xml
at org.opencastproject.search.impl.solr.SolrIndexManager.loadMpeg7Catalog(SolrIndexManager.java:697)
at org.opencastproject.search.impl.solr.SolrIndexManager.createEpisodeInputDocument(SolrIndexManager.java:400)
at org.opencastproject.search.impl.solr.SolrIndexManager.add(SolrIndexManager.java:328)
... 144 more
2016-12-22 12:50:22,775 | ERROR | (SearchServiceImpl:547) - Skipped 1 erroneous search entries while populating the search index
Ugh, ok this is actually some kind of race condition upon further investigation. STR:
1) Process one or more videos, ensure the slide text is present as expected
2) Stop Opencast, clear the search Solr index, start Opencast
3a) Load the video, do not get the slide text even though it is present in the mediapackage
3b) Load the video, get the slide text
3c) Load the video, get the error page
Whether you get a, b, or c appears to be more or less random but whatever you get will persist across all current published recordings until you repeat from step 2.
Downgrading to critical since the issue is not that the index can't be rebuilt at all, just that you lose existing slidetext.
rrolf had the idea that the issue might be that slidetexts are read from the WS/WFR from which they are removed at the end of the workflow by the cleanup operation. I did not test/confirm this, but it sounds reasonable enough to investigate this further.
I've just realized the same problem after updating from 2.3.2 to 3.2 and trying to rebuild the SOLR index on the presentation node. After several trials, IMHO the problem is caused when the SOLR indexer starts BEFORE the spring security context is established, and therefore /static is not accessible (that's what the status code 403 indicates).
Here is a part of the log of a trial when the rebuild completely fails:
(lots of error messages here)
...
2017-09-06 12:48:52,290 | ERROR | (SolrIndexManager:404) - Error loading mpeg7 catalog. Skipping catalog: Unable to load metadata from mpeg7 catalog https://my.opencast.domain:8443/static/mh_default_org/engage-player/ff8f5f16-769a-4615-8a9a-b13cda080442/c392825b-54af-4a7b-aa2a-8b1125bc0692/slidetext.xml
2017-09-06 12:48:52,940 | WARN | (WorkspaceImpl:367) - Received unexpected response status 403 while trying to download from https://my.opencast.domain:8443/static/mh_default_org/engage-player/ffaba9b6-7590-4172-9928-05a0981167f5/dc127423-ec2a-46f6-a2de-909c3c736afc/catalog.xml
2017-09-06 12:48:52,950 | WARN | (StaticMetadataServiceDublinCoreImpl:395) - Unable to load metadata from catalog 'https://my.opencast.domain:8443/static/mh_default_org/engage-player/ffaba9b6-7590-4172-9928-05a0981167f5/dc127423-ec2a-46f6-a2de-909c3c736afc/catalog.xml'
2017-09-06 12:48:53,023 | WARN | (WorkspaceImpl:367) - Received unexpected response status 403 while trying to download from https://my.opencast.domain:8443/static/mh_default_org/engage-player/ffaba9b6-7590-4172-9928-05a0981167f5/98af7415-21ea-467e-98ed-c10d8d4471ad/slidetext.xml
2017-09-06 12:48:53,034 | ERROR | (SolrIndexManager:404) - Error loading mpeg7 catalog. Skipping catalog: Unable to load metadata from mpeg7 catalog https://my.opencast.domain:8443/static/mh_default_org/engage-player/ffaba9b6-7590-4172-9928-05a0981167f5/98af7415-21ea-467e-98ed-c10d8d4471ad/slidetext.xml
2017-09-06 12:48:53,647 | INFO | (SearchServiceImpl:554) - Finished populating search index
2017-09-06 12:48:53,724 | INFO | (RestPublisher:333) - Registered REST endpoint at /search
...
2017-09-06 12:49:03,824 | INFO | (SpringSecurityConfigurationArtifactInstaller:96) - registered OsgiBundleXmlApplicationContext(bundle=matterhorn-kernel, config=file:/etc/opencast/security/mh_default_org.xml): startup date [Thu Jan 01 01:00:00 CET 1970]; root of context hierarchy for mh_default_org
...
And here the code of another try when the spring security context is established right at the beginning of the SOLR indexing operation:
...
2017-09-07 09:46:05,389 | INFO | (SearchServiceImpl:195) - Setting up solr server
2017-09-07 09:46:05,393 | INFO | (SearchServiceImpl:253) - Setting up solr search index at /srv/opencast/solr-indexes/search
2017-09-07 09:46:05,394 | INFO | (SearchServiceImpl:260) - solr config directory doesn't exist. Creating /srv/opencast/solr-indexes/search/conf
2017-09-07 09:46:05,889 | WARN | (SolrCore:389) - Solr index directory '/srv/opencast/solr-indexes/search/data/index' doesn't exist. Creating new index...
2017-09-07 09:46:06,188 | INFO | (SearchServiceImpl:515) - No search index found
2017-09-07 09:46:06,189 | INFO | (SearchServiceImpl:516) - Starting population of search index from database
2017-09-07 09:46:07,045 | WARN | (WorkspaceImpl:367) - Received unexpected response status 403 while trying to download from https://my.opencast.domain:8443/static/mh_default_org/engage-player/006c1836-1579-442a-9a82-449666260fbd/de28a2bc-5ef4-4012-850d-c99e0a29a728/catalog.xml
2017-09-07 09:46:07,136 | WARN | (StaticMetadataServiceDublinCoreImpl:395) - Unable to load metadata from catalog 'https://my.opencast.domain:8443/static/mh_default_org/engage-player/006c1836-1579-442a-9a82-449666260fbd/de28a2bc-5ef4-4012-850d-c99e0a29a728/catalog.xml'
2017-09-07 09:46:07,148 | INFO | (SpringSecurityConfigurationArtifactInstaller:96) - registered OsgiBundleXmlApplicationContext(bundle=matterhorn-kernel, config=file:/etc/opencast/security/mh_default_org.xml): startup date [Thu Jan 01 01:00:00 CET 1970]; root of context hierarchy for mh_default_org
2017-09-07 09:46:07,300 | WARN | (WorkspaceImpl:367) - Received unexpected response status 403 while trying to download from https://my.opencast.domain:8443/static/mh_default_org/engage-player/006c1836-1579-442a-9a82-449666260fbd/c6c07819-8f87-4569-bba4-719bd81774e7/slidetext.xml
2017-09-07 09:46:07,441 | ERROR | (SolrIndexManager:404) - Error loading mpeg7 catalog. Skipping catalog: Unable to load metadata from mpeg7 catalog https://my.opencast.domain:8443/static/mh_default_org/engage-player/006c1836-1579-442a-9a82-449666260fbd/c6c07819-8f87-4569-bba4-719bd81774e7/slidetext.xml
2017-09-07 09:46:08,356 | WARN | (StaticMetadataServiceDublinCoreImpl:395) - Unable to load metadata from catalog 'https://my.opencast.domain:8443/static/mh_default_org/engage-player/01b5eaab-9a63-450a-8d11-392564b89df7/35d4dfe3-ad5e-4770-b343-c90ae473b837/dublincore.xml'
2017-09-07 09:46:08,507 | ERROR | (SolrIndexManager:404) - Error loading mpeg7 catalog. Skipping catalog: Unable to load metadata from mpeg7 catalog https://my.opencast.domain:8443/static/mh_default_org/engage-player/01b5eaab-9a63-450a-8d11-392564b89df7/377565ce-a020-45ce-b1da-91f81dfa7a67/slidetext.xml
2017-09-07 09:46:09,082 | INFO | (WorkspaceImpl:364) - Downloading https://my.opencast.domain:8443/static/mh_default_org/engage-player/01d68fea-9cb4-41ca-9610-e7e19602d052/5e31315c-b117-4a0a-8346-bef4958dec66/catalog.xml to /srv/opencast/workspace/https_my.opencast.domain_8443/static/mh_default_org/engage-player/01d68fea-9cb4-41ca-9610-e7e19602d052/5e31315c-b117-4a0a-8346-bef4958dec66/catalog.xml
2017-09-07 09:46:09,233 | INFO | (WorkspaceImpl:364) - Downloading https://my.opencast.domain:8443/static/mh_default_org/engage-player/01d68fea-9cb4-41ca-9610-e7e19602d052/fe09a1d9-6b16-4aa4-98d1-fe94b9507251/slidetext.xml to /srv/opencast/workspace/https_my.opencast.domain_8443/static/mh_default_org/engage-player/01d68fea-9cb4-41ca-9610-e7e19602d052/fe09a1d9-6b16-4aa4-98d1-fe94b9507251/slidetext.xml
2017-09-07 09:46:09,803 | INFO | (WorkspaceImpl:364) - Downloading https://my.opencast.domain:8443/static/mh_default_org/engage-player/025fded5-cc6a-473b-8c2d-ac0a8b7a38f2/a25726a0-62d4-4870-999b-015796c30111/dublincore.xml to /srv/opencast/workspace/https_my.opencast.domain_8443/static/mh_default_org/engage-player/025fded5-cc6a-473b-8c2d-ac0a8b7a38f2/a25726a0-62d4-4870-999b-015796c30111/dublincore.xml
... (from here everything is OK and the index will be nearly completely rebuild when the indexer has finished!)
Could this be the origin of the problem?
If this issue is that the search service starting the index rebuild too early, then loading the search service later may resolve this too. Ref: https://bitbucket.org/opencast-community/opencast/commits/28a22d31694938cbf1d261ac2d126671593bad1d