Re-connect agent after disconnection

Description

We have seen here in Brngr that the agent can disconnect under heavy load or when there are long GC pauses (here GC pauses are also more than 5s).. The disconnection is most likely initialized by the servers as the data sending is in some way taking too long..

We should try to re-connect the agent. The agent is anyway initialized and collects data, so it's stupid that we are just dropping everything on the first reconnect.

By the way Dscv was also reporting that agents disconnects after some time and there is nothing in the logs, I guess it can also be that this happens maybe after some network problems or similar, so re-connection should solve these and help.

We can implement the re-connect in the keep-alive method, so that with keep-alive it tries to reconnect (have some kind of exponential count so that we don't try with every singal).

Environment

None

Activity

Show:
Technical User
December 2, 2016, 7:50 AM

SUCCESS: Integrated in

inspectIT - Integration #233
INSPECTIT-2243: Reconnection of the agent after loosing connection, (patrice.bouillet: 550adbb333dd5df9624cc5b18f2afa11a20a05b6)

  • (edit) inspectit.shared.all/src/main/java/rocks/inspectit/shared/all/kryonet/TcpConnection.java

  • (edit) inspectit.shared.all/src/test/java/rocks/inspectit/shared/all/storage/nio/stream/SocketExtendedByteBufferInputStreamTest.java

  • (edit) inspectit.agent.java/src/main/java/rocks/inspectit/agent/java/connection/IConnection.java

  • (edit) inspectit.agent.java/src/main/java/rocks/inspectit/agent/java/connection/impl/KeepAliveManager.java

  • (edit) inspectit.agent.java/src/main/java/rocks/inspectit/agent/java/connection/impl/KryoNetConnection.java

  • (edit) inspectit.agent.java/src/test/java/rocks/inspectit/agent/java/connection/impl/KryoNetConnectionTest.java

  • (edit) inspectit.shared.cs/src/main/java/rocks/inspectit/shared/cs/storage/nio/stream/ExtendedByteBufferInputStream.java

  • (edit) inspectit.shared.all/src/main/java/rocks/inspectit/shared/all/storage/nio/stream/SocketExtendedByteBufferInputStream.java

  • (add) inspectit.agent.java/src/test/java/rocks/inspectit/agent/java/connection/impl/KeepAliveManagerTest.java

  • (edit) inspectit.shared.all/src/main/java/rocks/inspectit/shared/all/storage/nio/stream/AbstractExtendedByteBufferInputStream.java

Ivan Senic
November 9, 2016, 5:24 PM

The implementation I provided can reconnect, but it works only if the CMR has not been stop. Let's discuss if we need to handle also situation when CMR is stopped completely. Then we need to ensure that agent can be reconnected (for example ID exists in the H2, class cache is created, status data is refreshed, etc).

Ivan Senic
November 9, 2016, 5:22 PM

Reproducible with

Stefan Siegl
November 8, 2016, 9:45 AM

Ok, lets do the following:
Lets take this topic independently of this ticket here. Right now I think we know what to do to solve this problem directly (increase the timeout/tries). I propose that you Ivan prepare a small decision table what we would benefit (from your pov) with the change to sth like restful communication.

Then we can discuss, all agree (, )?

Ivan Senic
November 8, 2016, 9:21 AM

Yes that's what I had in mind. Without need to keep an open socket connection to the server. I know this is performance wise slower, but also has some advantages, like that we can easily send data from .NET agent or anything else without need to figure out how to handle this kryonet connection etc. Plus we should really think about having this connection via SSL and introduce some security then and I think this is much better than what we have right now.

Fixed

Assignee

Ivan Senic

Reporter

Ivan Senic

Integrator

Patrice Bouillet