Job dispatching halts because of http connection hang

Steps to reproduce

Our 3.x (just prior to 3.3) stop dispatching jobs to worker nodes, apparently because the job dispatcher hung indefinitely waiting for this http connection.

Possibly it does not have a timeout set, and should have a timeout: http://www.baeldung.com/httpclient-timeout

"pool-48-thread-2" Id=283 in RUNNABLE (running in native)
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.read(SocketInputStream.java:171)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:160)
at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:84)
at org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:273)
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140)
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261)
at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:283)
at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:251)
at org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:197)
at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272)
at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124)
at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:685)
at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:487)
at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:107)
at org.opencastproject.kernel.http.impl.HttpClientImpl.execute(HttpClientImpl.java:77)
at org.opencastproject.kernel.security.TrustedHttpClientImpl.getRealmAndNonce(TrustedHttpClientImpl.java:605)
at org.opencastproject.kernel.security.TrustedHttpClientImpl.manuallyHandleDigestAuthentication(TrustedHttpClientImpl.java:533)
at org.opencastproject.kernel.security.TrustedHttpClientImpl.execute(TrustedHttpClientImpl.java:393)
at org.opencastproject.kernel.security.TrustedHttpClientImpl.execute(TrustedHttpClientImpl.java:340)
at org.opencastproject.serviceregistry.impl.ServiceRegistryJpaImpl$JobDispatcher.dispatchJob(ServiceRegistryJpaImpl.java:3091)
at org.opencastproject.serviceregistry.impl.ServiceRegistryJpaImpl$JobDispatcher.dispatchDispatchableJobs(ServiceRegistryJpaImpl.java:2987)
at org.opencastproject.serviceregistry.impl.ServiceRegistryJpaImpl$JobDispatcher.run(ServiceRegistryJpaImpl.java:2883)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

Activity

Show:
Stephen Marquard
September 22, 2017, 3:03 PM

Ah, awesome, thanks for the pointer! I had that deja vu feeling that this had been seen and fixed before.

Fixed and reviewed
Your pinned fields
Click on the next to a field label to start pinning.

Assignee

Stephen Marquard

Reporter

Stephen Marquard

Severity

Incorrectly Functioning Without Workaround