Cannot re-build search Solr index

• Ingest two videos: 1x presenter 1x presentation

• Stop Opencast

• Remove data/solr-index/search

• Start Opencast

• Get a lot of errors

• The indexer cannot access all metadata catalogs (403 error)

• Trying to add the slidetexts the event adding will completely fail

Stack trace:

2016-12-22 12:50:22,325 | INFO | (SearchServiceImpl:195) - Setting up solr server
2016-12-22 12:50:22,326 | INFO | (SearchServiceImpl:253) - Setting up solr search index at /home/lars/dev/opencast/build/opencast-dist-allinone-2.3-SNAPSHOT/data/solr-indexes/search
2016-12-22 12:50:22,326 | INFO | (SearchServiceImpl:260) - solr config directory doesn't exist. Creating /home/lars/dev/opencast/build/opencast-dist-allinone-2.3-SNAPSHOT/data/solr-indexes/search/conf
2016-12-22 12:50:22,391 | WARN | (SolrCore:389) - Solr index directory '/home/lars/dev/opencast/build/opencast-dist-allinone-2.3-SNAPSHOT/data/solr-indexes/search/data/index' doesn't exist. Creating new index...
2016-12-22 12:50:22,465 | INFO | (SearchServiceImpl:509) - No search index found
2016-12-22 12:50:22,466 | INFO | (SearchServiceImpl:510) - Starting population of search index from database
2016-12-22 12:50:22,609 | WARN | (WorkspaceImpl:330) - Received unexpected response status 403 while trying to download from http://localhost:8080/static/mh_default_org/engage-player/f25f4cd4-6d1d-40be-b6a5-4e14536a61e1/9352000d-cacb-4287-a9b3-1e5ba4525ea9/dublincore.xml
2016-12-22 12:50:22,614 | WARN | (StaticMetadataServiceDublinCoreImpl:317) - Unable to load metadata from catalog 'http://localhost:8080/static/mh_default_org/engage-player/f25f4cd4-6d1d-40be-b6a5-4e14536a61e1/9352000d-cacb-4287-a9b3-1e5ba4525ea9/dublincore.xml'
2016-12-22 12:50:22,741 | WARN | (WorkspaceImpl:330) - Received unexpected response status 403 while trying to download from http://localhost:8080/static/mh_default_org/engage-player/9c13730c-f13a-46a8-b5f0-7c899d202411/42249fe3-3038-4dc0-a7dc-d32dc1e2c412/dublincore.xml
2016-12-22 12:50:22,742 | WARN | (StaticMetadataServiceDublinCoreImpl:317) - Unable to load metadata from catalog 'http://localhost:8080/static/mh_default_org/engage-player/9c13730c-f13a-46a8-b5f0-7c899d202411/42249fe3-3038-4dc0-a7dc-d32dc1e2c412/dublincore.xml'
2016-12-22 12:50:22,747 | WARN | (WorkspaceImpl:330) - Received unexpected response status 403 while trying to download from http://localhost:8080/static/mh_default_org/engage-player/9c13730c-f13a-46a8-b5f0-7c899d202411/f38c7d38-08f4-4b02-b796-1697d92e9ccf/slidetext.xml
2016-12-22 12:50:22,749 | ERROR | (SearchServiceImpl:535) - Unable to index search instances: {}
org.apache.solr.client.solrj.SolrServerException: java.io.IOException: Unable to load metadata from mpeg7 catalog http://localhost:8080/static/mh_default_org/engage-player/9c13730c-f13a-46a8-b5f0-7c899d202411/f38c7d38-08f4-4b02-b796-1697d92e9ccf/slidetext.xml
at org.opencastproject.search.impl.solr.SolrIndexManager.add(SolrIndexManager.java:348)
at org.opencastproject.search.impl.SearchServiceImpl.populateIndex(SearchServiceImpl.java:533)
at org.opencastproject.search.impl.SearchServiceImpl.activate(SearchServiceImpl.java:236)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)[:1.8.0_111]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)[:1.8.0_111]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)[:1.8.0_111]
at java.lang.reflect.Method.invoke(Method.java:498)[:1.8.0_111]
at org.apache.felix.scr.impl.helper.BaseMethod.invokeMethod(BaseMethod.java:231)
at org.apache.felix.scr.impl.helper.BaseMethod.access$500(BaseMethod.java:39)
at org.apache.felix.scr.impl.helper.BaseMethod$Resolved.invoke(BaseMethod.java:624)
at org.apache.felix.scr.impl.helper.BaseMethod.invoke(BaseMethod.java:508)
at org.apache.felix.scr.impl.helper.ActivateMethod.invoke(ActivateMethod.java:149)
at org.apache.felix.scr.impl.manager.SingleComponentManager.createImplementationObject(SingleComponentManager.java:315)
at org.apache.felix.scr.impl.manager.SingleComponentManager.createComponent(SingleComponentManager.java:127)
at org.apache.felix.scr.impl.manager.SingleComponentManager.getService(SingleComponentManager.java:871)
at org.apache.felix.scr.impl.manager.SingleComponentManager.getServiceInternal(SingleComponentManager.java:838)
at org.apache.felix.scr.impl.manager.SingleComponentManager.getService(SingleComponentManager.java:777)
at org.apache.felix.framework.ServiceRegistrationImpl.getFactoryUnchecked(ServiceRegistrationImpl.java:308)
at org.apache.felix.framework.ServiceRegistrationImpl.getService(ServiceRegistrationImpl.java:219)
at org.apache.felix.framework.ServiceRegistry.getService(ServiceRegistry.java:320)
at org.apache.felix.framework.Felix.getService(Felix.java:3568)[org.apache.felix.framework-4.2.1.jar:]
at org.apache.felix.framework.BundleContextImpl.getService(BundleContextImpl.java:468)
at org.apache.felix.scr.impl.helper.BindMethod.getServiceObject(BindMethod.java:572)
at org.apache.felix.scr.impl.manager.DependencyManager.getServiceObject(DependencyManager.java:2012)
at org.apache.felix.scr.impl.manager.DependencyManager$SingleStaticCustomizer.prebind(DependencyManager.java:1005)
at org.apache.felix.scr.impl.manager.DependencyManager.prebind(DependencyManager.java:1439)
at org.apache.felix.scr.impl.manager.AbstractComponentManager.collectDependencies(AbstractComponentManager.java:1119)
at org.apache.felix.scr.impl.manager.SingleComponentManager.getServiceInternal(SingleComponentManager.java:807)
at org.apache.felix.scr.impl.manager.SingleComponentManager.getService(SingleComponentManager.java:777)
at org.apache.felix.framework.ServiceRegistrationImpl.getFactoryUnchecked(ServiceRegistrationImpl.java:308)
at org.apache.felix.framework.ServiceRegistrationImpl.getService(ServiceRegistrationImpl.java:219)
at org.apache.felix.framework.ServiceRegistry.getService(ServiceRegistry.java:320)
at org.apache.felix.framework.Felix.getService(Felix.java:3568)[org.apache.felix.framework-4.2.1.jar:]
at org.apache.felix.framework.BundleContextImpl.getService(BundleContextImpl.java:468)
at org.apache.felix.fileinstall.internal.FileInstall.addingService(FileInstall.java:150)
at org.osgi.util.tracker.ServiceTracker$Tracked.customizerAdding(ServiceTracker.java:932)[karaf-org.osgi.core.jar:]
at org.osgi.util.tracker.ServiceTracker$Tracked.customizerAdding(ServiceTracker.java:864)[karaf-org.osgi.core.jar:]
at org.osgi.util.tracker.AbstractTracked.trackAdding(AbstractTracked.java:256)[karaf-org.osgi.core.jar:]
at org.osgi.util.tracker.AbstractTracked.track(AbstractTracked.java:229)[karaf-org.osgi.core.jar:]
at org.osgi.util.tracker.ServiceTracker$Tracked.serviceChanged(ServiceTracker.java:894)[karaf-org.osgi.core.jar:]
at org.apache.felix.framework.util.EventDispatcher.invokeServiceListenerCallback(EventDispatcher.java:932)[org.apache.felix.framework-4.2.1.jar:]
at org.apache.felix.framework.util.EventDispatcher.fireEventImmediately(EventDispatcher.java:793)[org.apache.felix.framework-4.2.1.jar:]
at org.apache.felix.framework.util.EventDispatcher.fireServiceEvent(EventDispatcher.java:543)[org.apache.felix.framework-4.2.1.jar:]
at org.apache.felix.framework.Felix.fireServiceEvent(Felix.java:4419)[org.apache.felix.framework-4.2.1.jar:]
at org.apache.felix.framework.Felix.registerService(Felix.java:3423)[org.apache.felix.framework-4.2.1.jar:]
at org.apache.felix.framework.BundleContextImpl.registerService(BundleContextImpl.java:346)
at org.apache.felix.scr.impl.manager.AbstractComponentManager$3.register(AbstractComponentManager.java:1003)
at org.apache.felix.scr.impl.manager.AbstractComponentManager$3.register(AbstractComponentManager.java:992)
at org.apache.felix.scr.impl.manager.RegistrationManager.changeRegistration(RegistrationManager.java:134)
at org.apache.felix.scr.impl.manager.AbstractComponentManager.registerService(AbstractComponentManager.java:1044)
at org.apache.felix.scr.impl.manager.AbstractComponentManager.activateInternal(AbstractComponentManager.java:841)
at org.apache.felix.scr.impl.manager.DependencyManager$SingleStaticCustomizer.addedService(DependencyManager.java:931)
at org.apache.felix.scr.impl.manager.DependencyManager$SingleStaticCustomizer.addedService(DependencyManager.java:895)
at org.apache.felix.scr.impl.manager.ServiceTracker$Tracked.customizerAdded(ServiceTracker.java:1480)[73:org.apache.felix.scr:1.8.2]
at org.apache.felix.scr.impl.manager.ServiceTracker$Tracked.customizerAdded(ServiceTracker.java:1401)[73:org.apache.felix.scr:1.8.2]
at org.apache.felix.scr.impl.manager.ServiceTracker$AbstractTracked.trackAdding(ServiceTracker.java:1210)
at org.apache.felix.scr.impl.manager.ServiceTracker$AbstractTracked.track(ServiceTracker.java:1148)
at org.apache.felix.scr.impl.manager.ServiceTracker$Tracked.serviceChanged(ServiceTracker.java:1432)[73:org.apache.felix.scr:1.8.2]
at org.apache.felix.framework.util.EventDispatcher.invokeServiceListenerCallback(EventDispatcher.java:932)[org.apache.felix.framework-4.2.1.jar:]
at org.apache.felix.framework.util.EventDispatcher.fireEventImmediately(EventDispatcher.java:793)[org.apache.felix.framework-4.2.1.jar:]
at org.apache.felix.framework.util.EventDispatcher.fireServiceEvent(EventDispatcher.java:543)[org.apache.felix.framework-4.2.1.jar:]
at org.apache.felix.framework.Felix.fireServiceEvent(Felix.java:4419)[org.apache.felix.framework-4.2.1.jar:]
at org.apache.felix.framework.Felix.registerService(Felix.java:3423)[org.apache.felix.framework-4.2.1.jar:]
at org.apache.felix.framework.BundleContextImpl.registerService(BundleContextImpl.java:346)
at org.apache.felix.scr.impl.manager.AbstractComponentManager$3.register(AbstractComponentManager.java:1003)
at org.apache.felix.scr.impl.manager.AbstractComponentManager$3.register(AbstractComponentManager.java:992)
at org.apache.felix.scr.impl.manager.RegistrationManager.changeRegistration(RegistrationManager.java:134)
at org.apache.felix.scr.impl.manager.AbstractComponentManager.registerService(AbstractComponentManager.java:1044)
at org.apache.felix.scr.impl.manager.AbstractComponentManager.activateInternal(AbstractComponentManager.java:841)
at org.apache.felix.scr.impl.manager.DependencyManager$SingleStaticCustomizer.addedService(DependencyManager.java:931)
at org.apache.felix.scr.impl.manager.DependencyManager$SingleStaticCustomizer.addedService(DependencyManager.java:895)
at org.apache.felix.scr.impl.manager.ServiceTracker$Tracked.customizerAdded(ServiceTracker.java:1480)[73:org.apache.felix.scr:1.8.2]
at org.apache.felix.scr.impl.manager.ServiceTracker$Tracked.customizerAdded(ServiceTracker.java:1401)[73:org.apache.felix.scr:1.8.2]
at org.apache.felix.scr.impl.manager.ServiceTracker$AbstractTracked.trackAdding(ServiceTracker.java:1210)
at org.apache.felix.scr.impl.manager.ServiceTracker$AbstractTracked.track(ServiceTracker.java:1148)
at org.apache.felix.scr.impl.manager.ServiceTracker$Tracked.serviceChanged(ServiceTracker.java:1432)[73:org.apache.felix.scr:1.8.2]
at org.apache.felix.framework.util.EventDispatcher.invokeServiceListenerCallback(EventDispatcher.java:932)[org.apache.felix.framework-4.2.1.jar:]
at org.apache.felix.framework.util.EventDispatcher.fireEventImmediately(EventDispatcher.java:793)[org.apache.felix.framework-4.2.1.jar:]
at org.apache.felix.framework.util.EventDispatcher.fireServiceEvent(EventDispatcher.java:543)[org.apache.felix.framework-4.2.1.jar:]
at org.apache.felix.framework.Felix.fireServiceEvent(Felix.java:4419)[org.apache.felix.framework-4.2.1.jar:]
at org.apache.felix.framework.Felix.registerService(Felix.java:3423)[org.apache.felix.framework-4.2.1.jar:]
at org.apache.felix.framework.BundleContextImpl.registerService(BundleContextImpl.java:346)
at org.apache.felix.scr.impl.manager.AbstractComponentManager$3.register(AbstractComponentManager.java:1003)
at org.apache.felix.scr.impl.manager.AbstractComponentManager$3.register(AbstractComponentManager.java:992)
at org.apache.felix.scr.impl.manager.RegistrationManager.changeRegistration(RegistrationManager.java:134)
at org.apache.felix.scr.impl.manager.AbstractComponentManager.registerService(AbstractComponentManager.java:1044)
at org.apache.felix.scr.impl.manager.AbstractComponentManager.activateInternal(AbstractComponentManager.java:841)
at org.apache.felix.scr.impl.manager.AbstractComponentManager.enable(AbstractComponentManager.java:419)
at org.apache.felix.scr.impl.config.ConfigurableComponentHolder.enableComponents(ConfigurableComponentHolder.java:376)
at org.apache.felix.scr.impl.BundleComponentActivator.initialize(BundleComponentActivator.java:172)
at org.apache.felix.scr.impl.BundleComponentActivator.<init>(BundleComponentActivator.java:120)
at org.apache.felix.scr.impl.Activator.loadComponents(Activator.java:258)
at org.apache.felix.scr.impl.Activator.access$000(Activator.java:45)
at org.apache.felix.scr.impl.Activator$ScrExtension.start(Activator.java:185)
at org.apache.felix.utils.extender.AbstractExtender.createExtension(AbstractExtender.java:259)
at org.apache.felix.utils.extender.AbstractExtender.modifiedBundle(AbstractExtender.java:232)
at org.osgi.util.tracker.BundleTracker$Tracked.customizerModified(BundleTracker.java:479)[karaf-org.osgi.core.jar:]
at org.osgi.util.tracker.BundleTracker$Tracked.customizerModified(BundleTracker.java:414)[karaf-org.osgi.core.jar:]
at org.osgi.util.tracker.AbstractTracked.track(AbstractTracked.java:232)[karaf-org.osgi.core.jar:]
at org.osgi.util.tracker.BundleTracker$Tracked.bundleChanged(BundleTracker.java:443)[karaf-org.osgi.core.jar:]
at org.apache.felix.framework.util.EventDispatcher.invokeBundleListenerCallback(EventDispatcher.java:868)[org.apache.felix.framework-4.2.1.jar:]
at org.apache.felix.framework.util.EventDispatcher.fireEventImmediately(EventDispatcher.java:789)[org.apache.felix.framework-4.2.1.jar:]
at org.apache.felix.framework.util.EventDispatcher.fireBundleEvent(EventDispatcher.java:514)[org.apache.felix.framework-4.2.1.jar:]
at org.apache.felix.framework.Felix.fireBundleEvent(Felix.java:4403)[org.apache.felix.framework-4.2.1.jar:]
at org.apache.felix.framework.Felix.startBundle(Felix.java:2092)[org.apache.felix.framework-4.2.1.jar:]
at org.apache.felix.framework.BundleImpl.start(BundleImpl.java:955)
at org.apache.felix.framework.BundleImpl.start(BundleImpl.java:942)
at org.apache.karaf.features.internal.FeaturesServiceImpl.startBundle(FeaturesServiceImpl.java:516)
at org.apache.karaf.features.internal.FeaturesServiceImpl.installFeatures(FeaturesServiceImpl.java:474)
at org.apache.karaf.features.internal.BootFeaturesInstaller.installBootFeatures(BootFeaturesInstaller.java:93)
at org.apache.karaf.features.internal.BootFeaturesInstaller.start(BootFeaturesInstaller.java:77)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)[:1.8.0_111]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)[:1.8.0_111]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)[:1.8.0_111]
at java.lang.reflect.Method.invoke(Method.java:498)[:1.8.0_111]
at org.apache.aries.blueprint.utils.ReflectionUtils.invoke(ReflectionUtils.java:299)[15:org.apache.aries.blueprint.core:1.6.1]
at org.apache.aries.blueprint.container.BeanRecipe.invoke(BeanRecipe.java:980)[15:org.apache.aries.blueprint.core:1.6.1]
at org.apache.aries.blueprint.container.BeanRecipe.runBeanProcInit(BeanRecipe.java:736)[15:org.apache.aries.blueprint.core:1.6.1]
at org.apache.aries.blueprint.container.BeanRecipe.internalCreate2(BeanRecipe.java:848)[15:org.apache.aries.blueprint.core:1.6.1]
at org.apache.aries.blueprint.container.BeanRecipe.internalCreate(BeanRecipe.java:811)[15:org.apache.aries.blueprint.core:1.6.1]
at org.apache.aries.blueprint.di.AbstractRecipe$1.call(AbstractRecipe.java:79)[15:org.apache.aries.blueprint.core:1.6.1]
at java.util.concurrent.FutureTask.run(FutureTask.java:266)[:1.8.0_111]
at org.apache.aries.blueprint.di.AbstractRecipe.create(AbstractRecipe.java:88)[15:org.apache.aries.blueprint.core:1.6.1]
at org.apache.aries.blueprint.container.BlueprintRepository.createInstances(BlueprintRepository.java:255)[15:org.apache.aries.blueprint.core:1.6.1]
at org.apache.aries.blueprint.container.BlueprintRepository.createAll(BlueprintRepository.java:186)[15:org.apache.aries.blueprint.core:1.6.1]
at org.apache.aries.blueprint.container.BlueprintContainerImpl.instantiateEagerComponents(BlueprintContainerImpl.java:724)[15:org.apache.aries.blueprint.core:1.6.1]
at org.apache.aries.blueprint.container.BlueprintContainerImpl.doRun(BlueprintContainerImpl.java:411)[15:org.apache.aries.blueprint.core:1.6.1]
at org.apache.aries.blueprint.container.BlueprintContainerImpl.run(BlueprintContainerImpl.java:276)[15:org.apache.aries.blueprint.core:1.6.1]
at org.apache.aries.blueprint.container.BlueprintExtender.createContainer(BlueprintExtender.java:300)[15:org.apache.aries.blueprint.core:1.6.1]
at org.apache.aries.blueprint.container.BlueprintExtender.createContainer(BlueprintExtender.java:269)[15:org.apache.aries.blueprint.core:1.6.1]
at org.apache.aries.blueprint.container.BlueprintExtender.createContainer(BlueprintExtender.java:265)[15:org.apache.aries.blueprint.core:1.6.1]
at org.apache.aries.blueprint.container.BlueprintExtender.modifiedBundle(BlueprintExtender.java:255)[15:org.apache.aries.blueprint.core:1.6.1]
at org.apache.aries.util.tracker.hook.BundleHookBundleTracker$Tracked.customizerModified(BundleHookBundleTracker.java:500)[9:org.apache.aries.util:1.1.1]
at org.apache.aries.util.tracker.hook.BundleHookBundleTracker$Tracked.customizerModified(BundleHookBundleTracker.java:433)[9:org.apache.aries.util:1.1.1]
at org.apache.aries.util.tracker.hook.BundleHookBundleTracker$AbstractTracked.track(BundleHookBundleTracker.java:725)[9:org.apache.aries.util:1.1.1]
at org.apache.aries.util.tracker.hook.BundleHookBundleTracker$Tracked.bundleChanged(BundleHookBundleTracker.java:463)[9:org.apache.aries.util:1.1.1]
at org.apache.aries.util.tracker.hook.BundleHookBundleTracker$BundleEventHook.event(BundleHookBundleTracker.java:422)[9:org.apache.aries.util:1.1.1]
at org.apache.felix.framework.util.SecureAction.invokeBundleEventHook(SecureAction.java:1103)[org.apache.felix.framework-4.2.1.jar:]
at org.apache.felix.framework.util.EventDispatcher.createWhitelistFromHooks(EventDispatcher.java:695)[org.apache.felix.framework-4.2.1.jar:]
at org.apache.felix.framework.util.EventDispatcher.fireBundleEvent(EventDispatcher.java:483)[org.apache.felix.framework-4.2.1.jar:]
at org.apache.felix.framework.Felix.fireBundleEvent(Felix.java:4403)[org.apache.felix.framework-4.2.1.jar:]
at org.apache.felix.framework.Felix.startBundle(Felix.java:2092)[org.apache.felix.framework-4.2.1.jar:]
at org.apache.felix.framework.Felix.setActiveStartLevel(Felix.java:1291)[org.apache.felix.framework-4.2.1.jar:]
at org.apache.felix.framework.FrameworkStartLevelImpl.run(FrameworkStartLevelImpl.java:304)[org.apache.felix.framework-4.2.1.jar:]
at java.lang.Thread.run(Thread.java:745)[:1.8.0_111]
Caused by: java.io.IOException: Unable to load metadata from mpeg7 catalog http://localhost:8080/static/mh_default_org/engage-player/9c13730c-f13a-46a8-b5f0-7c899d202411/f38c7d38-08f4-4b02-b796-1697d92e9ccf/slidetext.xml
at org.opencastproject.search.impl.solr.SolrIndexManager.loadMpeg7Catalog(SolrIndexManager.java:697)
at org.opencastproject.search.impl.solr.SolrIndexManager.createEpisodeInputDocument(SolrIndexManager.java:400)
at org.opencastproject.search.impl.solr.SolrIndexManager.add(SolrIndexManager.java:328)
... 144 more
2016-12-22 12:50:22,775 | ERROR | (SearchServiceImpl:547) - Skipped 1 erroneous search entries while populating the search index