Solr Startup Fails

Steps to reproduce

Steps to reproduce:
1. Start an entire Matterhorn distributed install at the same time (via cssh or something)
2. Observe the issue

Actual Results:
A working solr service

Expected Results:
Sometimes:
2012-02-16 14:12:13 INFO (StaticResourceServlet:79) - serving static files from '/opt/matterhorn/felix/work/opencast/downloads'
2012-02-16 14:12:13 INFO (OrganizationDirectoryServiceImpl:262) - Registering organization 'mh_default_org'
2012-02-16 14:12:17 INFO (RestPublisher:222) - Registered REST endpoint at /org
2012-02-16 14:12:18 INFO (RestPublisher:222) - Registered REST endpoint at /roles
2012-02-16 14:12:18 INFO (SpringSecurityConfigurationArtifactInstaller:115) - registered OsgiBundleXmlApplicationContext(bundle=matterhorn-kernel, config=file:/opt/matterhorn/felix-framework-3.2.2/conf/security/mh_default_org.xml): startup date [Wed Dec 31 18:00:00 CST 1969]; root of context hierarchy for mh_default_org
2012-02-16 14:12:18 INFO (RestPublisher:222) - Registered REST endpoint at /info
2012-02-16 14:12:18 INFO (AnnotationRestService:96) - configured server url is http://devengage.usask.ca:8080
2012-02-16 14:12:18 INFO (RestPublisher:222) - Registered REST endpoint at /annotation
2012-02-16 14:12:19 INFO (ServiceRegistryJpaImpl:612) - Registering http://devengage.usask.ca:8080 with a maximum load of 2
2012-02-16 14:12:19 INFO (ServiceRegistryJpaImpl:255) - Heartbeat disabled
2012-02-16 14:12:19 INFO (WorkspaceImpl:105) - CONFIG org.opencastproject.workspace.rootdir: /opt/matterhorn/felix/work/opencast/workspace
2012-02-16 14:12:19 INFO (SeriesServiceDatabaseImpl:83) - Activating persistence manager for series
2012-02-16 14:12:20 ERROR (?:?) - [org.opencastproject.series.impl.solr.SeriesServiceSolrIndex] The activate method has thrown an exception
java.lang.IllegalStateException: Unable to connect to solr at /opt/matterhorn/felix/work/opencast/seriesindex/series
at org.opencastproject.series.impl.solr.SeriesServiceSolrIndex.activate(SeriesServiceSolrIndex.java:216)
at org.opencastproject.series.impl.solr.SeriesServiceSolrIndex.activate(SeriesServiceSolrIndex.java:187)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at org.apache.felix.scr.impl.helper.BaseMethod.invokeMethod(BaseMethod.java:227)
at org.apache.felix.scr.impl.helper.BaseMethod.access$500(BaseMethod.java:38)
at org.apache.felix.scr.impl.helper.BaseMethod$Resolved.invoke(BaseMethod.java:591)
at org.apache.felix.scr.impl.helper.BaseMethod.invoke(BaseMethod.java:472)
at org.apache.felix.scr.impl.helper.ActivateMethod.invoke(ActivateMethod.java:146)
at org.apache.felix.scr.impl.manager.ImmediateComponentManager.createImplementationObject(ImmediateComponentManager.java:226)
at org.apache.felix.scr.impl.manager.ImmediateComponentManager.createComponent(ImmediateComponentManager.java:118)
at org.apache.felix.scr.impl.manager.AbstractComponentManager$Unsatisfied.activate(AbstractComponentManager.java:997)
at org.apache.felix.scr.impl.manager.AbstractComponentManager.activateInternal(AbstractComponentManager.java:333)
at org.apache.felix.scr.impl.manager.DependencyManager.serviceAdded(DependencyManager.java:283)
at org.apache.felix.scr.impl.manager.DependencyManager.serviceChanged(DependencyManager.java:170)
at org.apache.felix.framework.util.EventDispatcher.invokeServiceListenerCallback(EventDispatcher.java:871)
at org.apache.felix.framework.util.EventDispatcher.fireEventImmediately(EventDispatcher.java:733)
at org.apache.felix.framework.util.EventDispatcher.fireServiceEvent(EventDispatcher.java:662)
at org.apache.felix.framework.Felix.fireServiceEvent(Felix.java:3890)
at org.apache.felix.framework.Felix.access$000(Felix.java:79)
at org.apache.felix.framework.Felix$2.serviceChanged(Felix.java:728)
at org.apache.felix.framework.ServiceRegistry.registerService(ServiceRegistry.java:107)
at org.apache.felix.framework.Felix.registerService(Felix.java:2979)
at org.apache.felix.framework.BundleContextImpl.registerService(BundleContextImpl.java:313)
at org.apache.felix.scr.impl.manager.AbstractComponentManager.registerService(AbstractComponentManager.java:449)
at org.apache.felix.scr.impl.manager.AbstractComponentManager.registerComponentService(AbstractComponentManager.java:460)
at org.apache.felix.scr.impl.manager.AbstractComponentManager$Unsatisfied.activate(AbstractComponentManager.java:1009)
at org.apache.felix.scr.impl.manager.AbstractComponentManager.activateInternal(AbstractComponentManager.java:333)
at org.apache.felix.scr.impl.manager.DependencyManager.serviceAdded(DependencyManager.java:283)
at org.apache.felix.scr.impl.manager.DependencyManager.serviceChanged(DependencyManager.java:170)
at org.apache.felix.framework.util.EventDispatcher.invokeServiceListenerCallback(EventDispatcher.java:871)
at org.apache.felix.framework.util.EventDispatcher.fireEventImmediately(EventDispatcher.java:733)
at org.apache.felix.framework.util.EventDispatcher.fireServiceEvent(EventDispatcher.java:662)
at org.apache.felix.framework.Felix.fireServiceEvent(Felix.java:3890)
at org.apache.felix.framework.Felix.access$000(Felix.java:79)
at org.apache.felix.framework.Felix$2.serviceChanged(Felix.java:728)
at org.apache.felix.framework.ServiceRegistry.registerService(ServiceRegistry.java:107)
at org.apache.felix.framework.Felix.registerService(Felix.java:2979)
at org.apache.felix.framework.BundleContextImpl.registerService(BundleContextImpl.java:313)
at org.apache.felix.scr.impl.manager.AbstractComponentManager.registerService(AbstractComponentManager.java:449)
at org.apache.felix.scr.impl.manager.AbstractComponentManager.registerComponentService(AbstractComponentManager.java:460)
at org.apache.felix.scr.impl.manager.AbstractComponentManager$Unsatisfied.activate(AbstractComponentManager.java:1009)
at org.apache.felix.scr.impl.manager.AbstractComponentManager.activateInternal(AbstractComponentManager.java:333)
at org.apache.felix.scr.impl.manager.DependencyManager.serviceAdded(DependencyManager.java:283)
at org.apache.felix.scr.impl.manager.DependencyManager.serviceChanged(DependencyManager.java:170)
at org.apache.felix.framework.util.EventDispatcher.invokeServiceListenerCallback(EventDispatcher.java:871)
at org.apache.felix.framework.util.EventDispatcher.fireEventImmediately(EventDispatcher.java:733)
at org.apache.felix.framework.util.EventDispatcher.fireServiceEvent(EventDispatcher.java:662)
at org.apache.felix.framework.Felix.fireServiceEvent(Felix.java:3890)
at org.apache.felix.framework.Felix.access$000(Felix.java:79)
at org.apache.felix.framework.Felix$2.serviceChanged(Felix.java:728)
at org.apache.felix.framework.ServiceRegistry.registerService(ServiceRegistry.java:107)
at org.apache.felix.framework.Felix.registerService(Felix.java:2979)
at org.apache.felix.framework.BundleContextImpl.registerService(BundleContextImpl.java:313)
at org.apache.felix.scr.impl.manager.AbstractComponentManager.registerService(AbstractComponentManager.java:449)
at org.apache.felix.scr.impl.manager.AbstractComponentManager.registerComponentService(AbstractComponentManager.java:460)
at org.apache.felix.scr.impl.manager.AbstractComponentManager$Unsatisfied.activate(AbstractComponentManager.java:1009)
at org.apache.felix.scr.impl.manager.AbstractComponentManager.activateInternal(AbstractComponentManager.java:333)
at org.apache.felix.scr.impl.manager.DependencyManager.serviceAdded(DependencyManager.java:283)
at org.apache.felix.scr.impl.manager.DependencyManager.serviceChanged(DependencyManager.java:170)
at org.apache.felix.framework.util.EventDispatcher.invokeServiceListenerCallback(EventDispatcher.java:871)
at org.apache.felix.framework.util.EventDispatcher.fireEventImmediately(EventDispatcher.java:733)
at org.apache.felix.framework.util.EventDispatcher.fireServiceEvent(EventDispatcher.java:662)
at org.apache.felix.framework.Felix.fireServiceEvent(Felix.java:3890)
at org.apache.felix.framework.Felix.access$000(Felix.java:79)
at org.apache.felix.framework.Felix$2.serviceChanged(Felix.java:728)
at org.apache.felix.framework.ServiceRegistry.registerService(ServiceRegistry.java:107)
at org.apache.felix.framework.Felix.registerService(Felix.java:2979)
at org.apache.felix.framework.BundleContextImpl.registerService(BundleContextImpl.java:313)
at org.apache.felix.scr.impl.manager.AbstractComponentManager.registerService(AbstractComponentManager.java:449)
at org.apache.felix.scr.impl.manager.AbstractComponentManager.registerComponentService(AbstractComponentManager.java:460)
at org.apache.felix.scr.impl.manager.AbstractComponentManager$Unsatisfied.activate(AbstractComponentManager.java:1009)
at org.apache.felix.scr.impl.manager.AbstractComponentManager.activateInternal(AbstractComponentManager.java:333)
at org.apache.felix.scr.impl.manager.AbstractComponentManager.enable(AbstractComponentManager.java:157)
at org.apache.felix.scr.impl.config.ConfiguredComponentHolder.enableComponents(ConfiguredComponentHolder.java:256)
at org.apache.felix.scr.impl.BundleComponentActivator.loadDescriptor(BundleComponentActivator.java:253)
at org.apache.felix.scr.impl.BundleComponentActivator.initialize(BundleComponentActivator.java:147)
at org.apache.felix.scr.impl.BundleComponentActivator.<init>(BundleComponentActivator.java:111)
at org.apache.felix.scr.impl.Activator.loadComponents(Activator.java:285)
at org.apache.felix.scr.impl.Activator.bundleChanged(Activator.java:203)
at org.apache.felix.framework.util.EventDispatcher.invokeBundleListenerCallback(EventDispatcher.java:807)
at org.apache.felix.framework.util.EventDispatcher.fireEventImmediately(EventDispatcher.java:729)
at org.apache.felix.framework.util.EventDispatcher.fireBundleEvent(EventDispatcher.java:610)
at org.apache.felix.framework.Felix.fireBundleEvent(Felix.java:3879)
at org.apache.felix.framework.Felix.startBundle(Felix.java:1850)
at org.apache.felix.framework.BundleImpl.start(BundleImpl.java:927)
at org.apache.felix.fileinstall.internal.DirectoryWatcher.process(DirectoryWatcher.java:1175)
at org.apache.felix.fileinstall.internal.DirectoryWatcher.process(DirectoryWatcher.java:1153)
at org.apache.felix.fileinstall.internal.DirectoryWatcher.process(DirectoryWatcher.java:458)
at org.apache.felix.fileinstall.internal.DirectoryWatcher.run(DirectoryWatcher.java:263)
Caused by: org.apache.solr.client.solrj.SolrServerException: java.lang.RuntimeException: Cannot create directory: /opt/matterhorn/felix/work/opencast/seriesindex/series/data/index
at org.opencastproject.solr.SolrServerFactory.newEmbeddedInstance(SolrServerFactory.java:60)
at org.opencastproject.series.impl.solr.SeriesServiceSolrIndex.setupSolr(SeriesServiceSolrIndex.java:268)
at org.opencastproject.series.impl.solr.SeriesServiceSolrIndex.activate(SeriesServiceSolrIndex.java:212)
... 91 more
Caused by: java.lang.RuntimeException: Cannot create directory: /opt/matterhorn/felix/work/opencast/seriesindex/series/data/index
at org.apache.lucene.store.NativeFSLockFactory.acquireTestLock(NativeFSLockFactory.java:74)
at org.apache.lucene.store.NativeFSLockFactory.makeLock(NativeFSLockFactory.java:137)
at org.apache.lucene.store.Directory.makeLock(Directory.java:131)
at org.apache.lucene.index.IndexWriter.init(IndexWriter.java:1563)
at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:1421)
at org.apache.solr.update.SolrIndexWriter.<init>(SolrIndexWriter.java:191)
at org.apache.solr.core.SolrCore.initIndex(SolrCore.java:392)
at org.apache.solr.core.SolrCore.<init>(SolrCore.java:545)
at org.opencastproject.solr.SolrServerFactory.newEmbeddedInstance(SolrServerFactory.java:56)
... 93 more
2012-02-16 14:12:20 ERROR (?:?) - [org.opencastproject.series.impl.solr.SeriesServiceSolrIndex] Component instance could not be created, activation failed
2012-02-16 14:12:20 WARN (StreamingDistributionService:107) - Stream url was not set (org.opencastproject.streaming.url)
2012-02-16 14:12:20 WARN (StreamingDistributionService:114) - Streaming distribution directory must be set (org.opencastproject.streaming.directory)
2012-02-16 14:12:20 INFO (StreamingDistributionService:126) - Streaming distribution directory is null
2012-02-16 14:12:20 INFO (RestPublisher:222) - Registered REST endpoint at /distribution/streaming
2012-02-16 14:12:20 INFO (DownloadDistributionService:115) - Download distribution directory is /opt/matterhorn/felix/work/opencast/downloads
2012-02-16 14:12:20 INFO (RestPublisher:222) - Registered REST endpoint at /distribution/download
2012-02-16 14:12:20 INFO (RestPublisher:222) - Registered REST endpoint at /services
2012-02-16 14:12:34 ERROR (SolrIndexWriter:252) - SolrIndexWriter was not closed prior to finalize(), indicates a bug – POSSIBLE RESOURCE LEAK!!!
2012-02-16 14:12:34 ERROR (SolrCore:740) - REFCOUNT ERROR: unreferenced org.apache.solr.core.SolrCore@64368400 (null) has a reference count of 1
2012-02-16 14:12:34 ERROR (CoreContainer:332) - CoreContainer was not shutdown prior to finalize(), indicates a bug – POSSIBLE RESOURCE LEAK!!!
2012-02-16 14:23:03 INFO (DownloadDistributionService:178) - Distributing catalog-1 to /opt/matterhorn/felix/work/opencast/downloads/ba826d35-7896-4ef3-97d6-0ba4f21deb5b/catalog-1/episode.xml
2012-02-16 14:23:03 INFO (DownloadDistributionService:195) - Finished distribution of http://devadmin.usask.ca:8080/files/mediapackage/ba826d35-7896-4ef3-97d6-0ba4f21deb5b/catalog-1/episode.xml
2012-02-16 14:24:21 INFO (DownloadDistributionService:178) - Distributing catalog-1 to /opt/matterhorn/felix/work/opencast/downloads/66eedfc2-5289-41d6-853b-ff6199887f47/catalog-1/episode.xml
2012-02-16 14:24:21 INFO (DownloadDistributionService:195) - Finished distribution of http://devadmin.usask.ca:8080/files/mediapackage/66eedfc2-5289-41d6-853b-ff6199887f47/catalog-1/episode.xml

Workaround (if any):
Restart the engage server. I've only seen this once, but the directory it's complaining about existed when I checked. I suspect this is a race condition considering where it's happening.

Status

Assignee

Tobias Wunden

Reporter

Greg Logan

Severity

Incorrectly Functioning With Workaround

Tags (folksonomy)

None

Components

Fix versions

Affects versions

1.3

Priority

Critical
Configure