Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed and reviewed
    • Affects versions: 2.3.2
    • Fix versions: 2.3.3
    • Components: Backend Software
    • Labels:
      None
    • Severity:
      Incorrectly Functioning With Workaround
    • Steps to reproduce:
      Hide
      Intermittently, a job fails to be dispatched correctly, so shows as DISPATCHING in the Admin UI jobs, and as queued for the target worker node. But it's never actually dispatched.

      Seems to be caused by an OptimisticLockException and then a subsequent NullPointerException in the job dispatching code.

      The NPE Is here:

      {code}
       // We've tried dispatching to every online service that can handle this type of job, with no luck.
            if (triedDispatching) {
              String host = job.getProcessorServiceRegistration().getHost();
      {code}

      2017-04-12 11:07:59,410 | WARN | pool-15-thread-2 | (JavaLog:289) -
      Exception [EclipseLink-5006] (Eclipse Persistence Services - 2.6.0.v20150309-bf26070): org.eclipse.persistence.exceptions.OptimisticLockException
      Exception Description: The object [org.opencastproject.job.jpa.JpaJob@2bff3a9f] cannot be updated because it has changed or been deleted since it was last read.
      Class> org.opencastproject.job.jpa.JpaJob Primary Key> 58,042,493
              at org.eclipse.persistence.exceptions.OptimisticLockException.objectChangedSinceLastReadWhenUpdating(OptimisticLockException.java:144)
              at org.eclipse.persistence.descriptors.VersionLockingPolicy.validateUpdate(VersionLockingPolicy.java:790)[95:org.eclipse.persistence.core:2.6.0.v20150309-bf26070]
              at org.eclipse.persistence.internal.queries.DatabaseQueryMechanism.updateObjectForWriteWithChangeSet(DatabaseQueryMechanism.java:1086)[95:org.eclipse.persistence.core:2.6.0.v20150309-bf26070]
              at org.eclipse.persistence.queries.UpdateObjectQuery.executeCommitWithChangeSet(UpdateObjectQuery.java:84)[95:org.eclipse.persistence.core:2.6.0.v20150309-bf26070]
              at org.eclipse.persistence.internal.queries.DatabaseQueryMechanism.executeWriteWithChangeSet(DatabaseQueryMechanism.java:301)[95:org.eclipse.persistence.core:2.6.0.v20150309-bf26070]
              at org.eclipse.persistence.queries.WriteObjectQuery.executeDatabaseQuery(WriteObjectQuery.java:58)[95:org.eclipse.persistence.core:2.6.0.v20150309-bf26070]
              at org.eclipse.persistence.queries.DatabaseQuery.execute(DatabaseQuery.java:904)[95:org.eclipse.persistence.core:2.6.0.v20150309-bf26070]
              at org.eclipse.persistence.queries.DatabaseQuery.executeInUnitOfWork(DatabaseQuery.java:803)[95:org.eclipse.persistence.core:2.6.0.v20150309-bf26070]
              at org.eclipse.persistence.queries.ObjectLevelModifyQuery.executeInUnitOfWorkObjectLevelModifyQuery(ObjectLevelModifyQuery.java:108)[95:org.eclipse.persistence.core:2.6.0.v20150309-bf26070]
              at org.eclipse.persistence.queries.ObjectLevelModifyQuery.executeInUnitOfWork(ObjectLevelModifyQuery.java:85)[95:org.eclipse.persistence.core:2.6.0.v20150309-bf26070]
              at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.internalExecuteQuery(UnitOfWorkImpl.java:2896)[95:org.eclipse.persistence.core:2.6.0.v20150309-bf26070]
              at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1857)[95:org.eclipse.persistence.core:2.6.0.v20150309-bf26070]
              at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1839)[95:org.eclipse.persistence.core:2.6.0.v20150309-bf26070]
              at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1790)[95:org.eclipse.persistence.core:2.6.0.v20150309-bf26070]
              at org.eclipse.persistence.internal.sessions.CommitManager.commitChangedObjectsForClassWithChangeSet(CommitManager.java:273)[95:org.eclipse.persistence.core:2.6.0.v20150309-bf26070]
              at org.eclipse.persistence.internal.sessions.CommitManager.commitAllObjectsWithChangeSet(CommitManager.java:131)[95:org.eclipse.persistence.core:2.6.0.v20150309-bf26070]
              at org.eclipse.persistence.internal.sessions.AbstractSession.writeAllObjectsWithChangeSet(AbstractSession.java:4260)[95:org.eclipse.persistence.core:2.6.0.v20150309-bf26070]
              at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.commitToDatabase(UnitOfWorkImpl.java:1441)[95:org.eclipse.persistence.core:2.6.0.v20150309-bf26070]
              at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.commitToDatabaseWithChangeSet(UnitOfWorkImpl.java:1531)[95:org.eclipse.persistence.core:2.6.0.v20150309-bf26070]
              at org.eclipse.persistence.internal.sessions.RepeatableWriteUnitOfWork.commitRootUnitOfWork(RepeatableWriteUnitOfWork.java:278)[95:org.eclipse.persistence.core:2.6.0.v20150309-bf26070]
              at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.commitAndResume(UnitOfWorkImpl.java:1169)[95:org.eclipse.persistence.core:2.6.0.v20150309-bf26070]
              at org.eclipse.persistence.internal.jpa.transaction.EntityTransactionImpl.commit(EntityTransactionImpl.java:134)[96:org.eclipse.persistence.jpa:2.6.0.v20150309-bf26070]
              at org.opencastproject.serviceregistry.impl.ServiceRegistryJpaImpl.updateInternal(ServiceRegistryJpaImpl.java:930)[172:matterhorn-serviceregistry:2.3.0.SNAPSHOT]
              at org.opencastproject.serviceregistry.impl.ServiceRegistryJpaImpl$JobDispatcher.dispatchJob(ServiceRegistryJpaImpl.java:3032)[172:matterhorn-serviceregistry:2.3.0.SNAPSHOT]
              at org.opencastproject.serviceregistry.impl.ServiceRegistryJpaImpl$JobDispatcher.dispatchDispatchableJobs(ServiceRegistryJpaImpl.java:2960)[172:matterhorn-serviceregistry:2.3.0.SNAPSHOT]
              at org.opencastproject.serviceregistry.impl.ServiceRegistryJpaImpl$JobDispatcher.run(ServiceRegistryJpaImpl.java:2858)[172:matterhorn-serviceregistry:2.3.0.SNAPSHOT]
              at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[:1.8.0_121]
              at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)[:1.8.0_121]
              at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)[:1.8.0_121]
              at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)[:1.8.0_121]
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)[:1.8.0_121]
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)[:1.8.0_121]
              at java.lang.Thread.run(Thread.java:745)[:1.8.0_121]
      2017-04-12 11:07:59,500 | WARN | pool-15-thread-2 | (ServiceRegistryJpaImpl$JobDispatcher:2865) - Error dispatching jobs
      java.lang.NullPointerException
              at org.opencastproject.serviceregistry.impl.ServiceRegistryJpaImpl$JobDispatcher.dispatchJob(ServiceRegistryJpaImpl.java:3096)[172:matterhorn-serviceregistry:2.3.0.SNAPSHOT]
              at org.opencastproject.serviceregistry.impl.ServiceRegistryJpaImpl$JobDispatcher.dispatchDispatchableJobs(ServiceRegistryJpaImpl.java:2960)[172:matterhorn-serviceregistry:2.3.0.SNAPSHOT]
              at org.opencastproject.serviceregistry.impl.ServiceRegistryJpaImpl$JobDispatcher.run(ServiceRegistryJpaImpl.java:2862)[172:matterhorn-serviceregistry:2.3.0.SNAPSHOT]
              at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[:1.8.0_121]
              at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)[:1.8.0_121]
              at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)[:1.8.0_121]
              at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)[:1.8.0_121]
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)[:1.8.0_121]
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)[:1.8.0_121]
              at java.lang.Thread.run(Thread.java:745)[:1.8.0_121]
      Show
      Intermittently, a job fails to be dispatched correctly, so shows as DISPATCHING in the Admin UI jobs, and as queued for the target worker node. But it's never actually dispatched. Seems to be caused by an OptimisticLockException and then a subsequent NullPointerException in the job dispatching code. The NPE Is here: {code}  // We've tried dispatching to every online service that can handle this type of job, with no luck.       if (triedDispatching) {         String host = job.getProcessorServiceRegistration().getHost(); {code} 2017-04-12 11:07:59,410 | WARN | pool-15-thread-2 | (JavaLog:289) - Exception [EclipseLink-5006] (Eclipse Persistence Services - 2.6.0.v20150309-bf26070): org.eclipse.persistence.exceptions.OptimisticLockException Exception Description: The object [ org.opencastproject.job.jpa.JpaJob@2bff3a9f ] cannot be updated because it has changed or been deleted since it was last read. Class> org.opencastproject.job.jpa.JpaJob Primary Key> 58,042,493         at org.eclipse.persistence.exceptions.OptimisticLockException.objectChangedSinceLastReadWhenUpdating(OptimisticLockException.java:144)         at org.eclipse.persistence.descriptors.VersionLockingPolicy.validateUpdate(VersionLockingPolicy.java:790)[95:org.eclipse.persistence.core:2.6.0.v20150309-bf26070]         at org.eclipse.persistence.internal.queries.DatabaseQueryMechanism.updateObjectForWriteWithChangeSet(DatabaseQueryMechanism.java:1086)[95:org.eclipse.persistence.core:2.6.0.v20150309-bf26070]         at org.eclipse.persistence.queries.UpdateObjectQuery.executeCommitWithChangeSet(UpdateObjectQuery.java:84)[95:org.eclipse.persistence.core:2.6.0.v20150309-bf26070]         at org.eclipse.persistence.internal.queries.DatabaseQueryMechanism.executeWriteWithChangeSet(DatabaseQueryMechanism.java:301)[95:org.eclipse.persistence.core:2.6.0.v20150309-bf26070]         at org.eclipse.persistence.queries.WriteObjectQuery.executeDatabaseQuery(WriteObjectQuery.java:58)[95:org.eclipse.persistence.core:2.6.0.v20150309-bf26070]         at org.eclipse.persistence.queries.DatabaseQuery.execute(DatabaseQuery.java:904)[95:org.eclipse.persistence.core:2.6.0.v20150309-bf26070]         at org.eclipse.persistence.queries.DatabaseQuery.executeInUnitOfWork(DatabaseQuery.java:803)[95:org.eclipse.persistence.core:2.6.0.v20150309-bf26070]         at org.eclipse.persistence.queries.ObjectLevelModifyQuery.executeInUnitOfWorkObjectLevelModifyQuery(ObjectLevelModifyQuery.java:108)[95:org.eclipse.persistence.core:2.6.0.v20150309-bf26070]         at org.eclipse.persistence.queries.ObjectLevelModifyQuery.executeInUnitOfWork(ObjectLevelModifyQuery.java:85)[95:org.eclipse.persistence.core:2.6.0.v20150309-bf26070]         at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.internalExecuteQuery(UnitOfWorkImpl.java:2896)[95:org.eclipse.persistence.core:2.6.0.v20150309-bf26070]         at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1857)[95:org.eclipse.persistence.core:2.6.0.v20150309-bf26070]         at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1839)[95:org.eclipse.persistence.core:2.6.0.v20150309-bf26070]         at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1790)[95:org.eclipse.persistence.core:2.6.0.v20150309-bf26070]         at org.eclipse.persistence.internal.sessions.CommitManager.commitChangedObjectsForClassWithChangeSet(CommitManager.java:273)[95:org.eclipse.persistence.core:2.6.0.v20150309-bf26070]         at org.eclipse.persistence.internal.sessions.CommitManager.commitAllObjectsWithChangeSet(CommitManager.java:131)[95:org.eclipse.persistence.core:2.6.0.v20150309-bf26070]         at org.eclipse.persistence.internal.sessions.AbstractSession.writeAllObjectsWithChangeSet(AbstractSession.java:4260)[95:org.eclipse.persistence.core:2.6.0.v20150309-bf26070]         at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.commitToDatabase(UnitOfWorkImpl.java:1441)[95:org.eclipse.persistence.core:2.6.0.v20150309-bf26070]         at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.commitToDatabaseWithChangeSet(UnitOfWorkImpl.java:1531)[95:org.eclipse.persistence.core:2.6.0.v20150309-bf26070]         at org.eclipse.persistence.internal.sessions.RepeatableWriteUnitOfWork.commitRootUnitOfWork(RepeatableWriteUnitOfWork.java:278)[95:org.eclipse.persistence.core:2.6.0.v20150309-bf26070]         at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.commitAndResume(UnitOfWorkImpl.java:1169)[95:org.eclipse.persistence.core:2.6.0.v20150309-bf26070]         at org.eclipse.persistence.internal.jpa.transaction.EntityTransactionImpl.commit(EntityTransactionImpl.java:134)[96:org.eclipse.persistence.jpa:2.6.0.v20150309-bf26070]         at org.opencastproject.serviceregistry.impl.ServiceRegistryJpaImpl.updateInternal(ServiceRegistryJpaImpl.java:930)[172:matterhorn-serviceregistry:2.3.0.SNAPSHOT]         at org.opencastproject.serviceregistry.impl.ServiceRegistryJpaImpl$JobDispatcher.dispatchJob(ServiceRegistryJpaImpl.java:3032)[172:matterhorn-serviceregistry:2.3.0.SNAPSHOT]         at org.opencastproject.serviceregistry.impl.ServiceRegistryJpaImpl$JobDispatcher.dispatchDispatchableJobs(ServiceRegistryJpaImpl.java:2960)[172:matterhorn-serviceregistry:2.3.0.SNAPSHOT]         at org.opencastproject.serviceregistry.impl.ServiceRegistryJpaImpl$JobDispatcher.run(ServiceRegistryJpaImpl.java:2858)[172:matterhorn-serviceregistry:2.3.0.SNAPSHOT]         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[:1.8.0_121]         at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)[:1.8.0_121]         at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)[:1.8.0_121]         at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)[:1.8.0_121]         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)[:1.8.0_121]         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)[:1.8.0_121]         at java.lang.Thread.run(Thread.java:745)[:1.8.0_121] 2017-04-12 11:07:59,500 | WARN | pool-15-thread-2 | (ServiceRegistryJpaImpl$JobDispatcher:2865) - Error dispatching jobs java.lang.NullPointerException         at org.opencastproject.serviceregistry.impl.ServiceRegistryJpaImpl$JobDispatcher.dispatchJob(ServiceRegistryJpaImpl.java:3096)[172:matterhorn-serviceregistry:2.3.0.SNAPSHOT]         at org.opencastproject.serviceregistry.impl.ServiceRegistryJpaImpl$JobDispatcher.dispatchDispatchableJobs(ServiceRegistryJpaImpl.java:2960)[172:matterhorn-serviceregistry:2.3.0.SNAPSHOT]         at org.opencastproject.serviceregistry.impl.ServiceRegistryJpaImpl$JobDispatcher.run(ServiceRegistryJpaImpl.java:2862)[172:matterhorn-serviceregistry:2.3.0.SNAPSHOT]         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[:1.8.0_121]         at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)[:1.8.0_121]         at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)[:1.8.0_121]         at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)[:1.8.0_121]         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)[:1.8.0_121]         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)[:1.8.0_121]         at java.lang.Thread.run(Thread.java:745)[:1.8.0_121]

      TestRail: Results

        Attachments

          Activity

            People

            • Assignee:
              smarquard Stephen Marquard
              Reporter:
              smarquard Stephen Marquard
            • Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                TestRail: Cases