Workflow job dispatching failures

Steps to reproduce

Intermittently, a job fails to be dispatched correctly, so shows as DISPATCHING in the Admin UI jobs, and as queued for the target worker node. But it's never actually dispatched.

Seems to be caused by an OptimisticLockException and then a subsequent NullPointerException in the job dispatching code.

The NPE Is here:

2017-04-12 11:07:59,410 | WARN | pool-15-thread-2 | (JavaLog:289) -
Exception [EclipseLink-5006] (Eclipse Persistence Services - 2.6.0.v20150309-bf26070): org.eclipse.persistence.exceptions.OptimisticLockException
Exception Description: The object [org.opencastproject.job.jpa.JpaJob@2bff3a9f] cannot be updated because it has changed or been deleted since it was last read.
Class> org.opencastproject.job.jpa.JpaJob Primary Key> 58,042,493
at org.eclipse.persistence.exceptions.OptimisticLockException.objectChangedSinceLastReadWhenUpdating(OptimisticLockException.java:144)
at org.eclipse.persistence.descriptors.VersionLockingPolicy.validateUpdate(VersionLockingPolicy.java:790)[95:org.eclipse.persistence.core:2.6.0.v20150309-bf26070]
at org.eclipse.persistence.internal.queries.DatabaseQueryMechanism.updateObjectForWriteWithChangeSet(DatabaseQueryMechanism.java:1086)[95:org.eclipse.persistence.core:2.6.0.v20150309-bf26070]
at org.eclipse.persistence.queries.UpdateObjectQuery.executeCommitWithChangeSet(UpdateObjectQuery.java:84)[95:org.eclipse.persistence.core:2.6.0.v20150309-bf26070]
at org.eclipse.persistence.internal.queries.DatabaseQueryMechanism.executeWriteWithChangeSet(DatabaseQueryMechanism.java:301)[95:org.eclipse.persistence.core:2.6.0.v20150309-bf26070]
at org.eclipse.persistence.queries.WriteObjectQuery.executeDatabaseQuery(WriteObjectQuery.java:58)[95:org.eclipse.persistence.core:2.6.0.v20150309-bf26070]
at org.eclipse.persistence.queries.DatabaseQuery.execute(DatabaseQuery.java:904)[95:org.eclipse.persistence.core:2.6.0.v20150309-bf26070]
at org.eclipse.persistence.queries.DatabaseQuery.executeInUnitOfWork(DatabaseQuery.java:803)[95:org.eclipse.persistence.core:2.6.0.v20150309-bf26070]
at org.eclipse.persistence.queries.ObjectLevelModifyQuery.executeInUnitOfWorkObjectLevelModifyQuery(ObjectLevelModifyQuery.java:108)[95:org.eclipse.persistence.core:2.6.0.v20150309-bf26070]
at org.eclipse.persistence.queries.ObjectLevelModifyQuery.executeInUnitOfWork(ObjectLevelModifyQuery.java:85)[95:org.eclipse.persistence.core:2.6.0.v20150309-bf26070]
at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.internalExecuteQuery(UnitOfWorkImpl.java:2896)[95:org.eclipse.persistence.core:2.6.0.v20150309-bf26070]
at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1857)[95:org.eclipse.persistence.core:2.6.0.v20150309-bf26070]
at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1839)[95:org.eclipse.persistence.core:2.6.0.v20150309-bf26070]
at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1790)[95:org.eclipse.persistence.core:2.6.0.v20150309-bf26070]
at org.eclipse.persistence.internal.sessions.CommitManager.commitChangedObjectsForClassWithChangeSet(CommitManager.java:273)[95:org.eclipse.persistence.core:2.6.0.v20150309-bf26070]
at org.eclipse.persistence.internal.sessions.CommitManager.commitAllObjectsWithChangeSet(CommitManager.java:131)[95:org.eclipse.persistence.core:2.6.0.v20150309-bf26070]
at org.eclipse.persistence.internal.sessions.AbstractSession.writeAllObjectsWithChangeSet(AbstractSession.java:4260)[95:org.eclipse.persistence.core:2.6.0.v20150309-bf26070]
at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.commitToDatabase(UnitOfWorkImpl.java:1441)[95:org.eclipse.persistence.core:2.6.0.v20150309-bf26070]
at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.commitToDatabaseWithChangeSet(UnitOfWorkImpl.java:1531)[95:org.eclipse.persistence.core:2.6.0.v20150309-bf26070]
at org.eclipse.persistence.internal.sessions.RepeatableWriteUnitOfWork.commitRootUnitOfWork(RepeatableWriteUnitOfWork.java:278)[95:org.eclipse.persistence.core:2.6.0.v20150309-bf26070]
at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.commitAndResume(UnitOfWorkImpl.java:1169)[95:org.eclipse.persistence.core:2.6.0.v20150309-bf26070]
at org.eclipse.persistence.internal.jpa.transaction.EntityTransactionImpl.commit(EntityTransactionImpl.java:134)[96:org.eclipse.persistence.jpa:2.6.0.v20150309-bf26070]
at org.opencastproject.serviceregistry.impl.ServiceRegistryJpaImpl.updateInternal(ServiceRegistryJpaImpl.java:930)[172:matterhorn-serviceregistry:2.3.0.SNAPSHOT]
at org.opencastproject.serviceregistry.impl.ServiceRegistryJpaImpl$JobDispatcher.dispatchJob(ServiceRegistryJpaImpl.java:3032)[172:matterhorn-serviceregistry:2.3.0.SNAPSHOT]
at org.opencastproject.serviceregistry.impl.ServiceRegistryJpaImpl$JobDispatcher.dispatchDispatchableJobs(ServiceRegistryJpaImpl.java:2960)[172:matterhorn-serviceregistry:2.3.0.SNAPSHOT]
at org.opencastproject.serviceregistry.impl.ServiceRegistryJpaImpl$JobDispatcher.run(ServiceRegistryJpaImpl.java:2858)[172:matterhorn-serviceregistry:2.3.0.SNAPSHOT]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[:1.8.0_121]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)[:1.8.0_121]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)[:1.8.0_121]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)[:1.8.0_121]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)[:1.8.0_121]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)[:1.8.0_121]
at java.lang.Thread.run(Thread.java:745)[:1.8.0_121]
2017-04-12 11:07:59,500 | WARN | pool-15-thread-2 | (ServiceRegistryJpaImpl$JobDispatcher:2865) - Error dispatching jobs
java.lang.NullPointerException
at org.opencastproject.serviceregistry.impl.ServiceRegistryJpaImpl$JobDispatcher.dispatchJob(ServiceRegistryJpaImpl.java:3096)[172:matterhorn-serviceregistry:2.3.0.SNAPSHOT]
at org.opencastproject.serviceregistry.impl.ServiceRegistryJpaImpl$JobDispatcher.dispatchDispatchableJobs(ServiceRegistryJpaImpl.java:2960)[172:matterhorn-serviceregistry:2.3.0.SNAPSHOT]
at org.opencastproject.serviceregistry.impl.ServiceRegistryJpaImpl$JobDispatcher.run(ServiceRegistryJpaImpl.java:2862)[172:matterhorn-serviceregistry:2.3.0.SNAPSHOT]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[:1.8.0_121]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)[:1.8.0_121]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)[:1.8.0_121]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)[:1.8.0_121]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)[:1.8.0_121]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)[:1.8.0_121]
at java.lang.Thread.run(Thread.java:745)[:1.8.0_121]

Activity

Show:
Stephen Marquard
April 14, 2017, 2:27 PM

Assignee

Stephen Marquard

Reporter

Stephen Marquard

Severity

Incorrectly Functioning With Workaround

Tags (folksonomy)

None

Components

Fix versions

Affects versions

Priority

Major
Configure