IS advanced

This page discusses the requirements of an advanced invocation sequence sensor to be integrated with inspectIT.

Features

The following list contains that necessary features of the advanced invocation sequence tracer.

• Continuous sending of invocation sequence data
• Invocation sequences that span over multiple threads
• Invocation sequences that span over multiple JVMs
• Storage of the invocation sequences in memory and as files (if requested by the user)
• Reduction of method sensor storage (only save method sensors if they are configured)

Overview

Each invocation in Java always occur within one Thread. Within this thread an invocation may use methods of multiple classes. Inner Thread calls are all synchronous (read blocking) calls, thus the sequence is very strictly defined.

It is also possible that one thread calls another thread. This call is most likely an asynchronous call, in which the caller will not wait for the result. There is nearly no application that does not use multiple JVM instances and use communication between these JVMs.

The invocation sequence should provide the concrete user transaction over these boundaries. It thus provides the exact information necessary to understand why and where a concrete invocation is slow.

Analysis

In order to achieve the necessary functionality different topics needs to be addressed.

Unique identification of invocations

There are four different levels that affect the identification of invocations. These are (sequence starting from bottom to top):

Continuous Sending

• Each invocation of a method checks if an invocation sequence is currently active
• The staging area builds up a stack of all methods that are invoked within one "thread"
  • This stack must be unique for each thread identification 
• The data is sent to the CMR. Based on the JVM identification, the platform identification and the thread identification, the CMR can extend its stack of the methods of one invocation by adding the received stack to it. If no information is currently available, the CMR must assume that this was a new invocation.

Inner-Thread

Invocations within one thread are saved as a stack that is unique per thread identification. The invocation of an instrumented method within an invocation sequence will create two different information objects:

• one at the start of the method and (method _start)
• one at the end of the method (method _end)

This functionality can easily be integrated by means of instrumenting the methods with before and after hooks. Additional identification of the methods are not necessary as the invocations within the thread are always synchronous and thus the stack of the invocations will easily allow to reconstruct the invocation sequence.

Example

The following structure shows how the stack may look like (notice that in order to save space, the stack is created from left to right). This structure allows to reconstruct, that M1 called M2 and afterwards called M3.

Inter-Thread

Problems that need to be solved:

• Correlation of the per se independent invocation sequences of two (or more threads)
• Identification which methods within the calling thread invoked a method within the called thread.

Basic: Inter-Thread invocation sequences should provide the location within the invocation that creates the asynchronous call.

Approach 1

Idea: Creation of a Runnable object identifies the location which initializes the asynchronous call to another thread

• The creation of a Runnable is an additional event that can be added to the
• We need to instrument the creation of a Runnable object and provide the following information
  • the current thread identification of the current thread that created the Runnable
  • the counter of how many asynchronous calls were already started within this thread

Scenario:

This scenario should help to describe this approach. Our calling thread has stack that is unique for this Runnable/Thread (see chapter about inner thread invocation sequences). Within this stack the creation of a Runnable is added.

So our m1 method within the thread first invokes the method m2, then we encounter a Runnable object being instantiated, which we understand as the start of an asynchronous call. Then method M2 terminates. After that method m1 starts another asynchronous call and returns. Each creation of a Runnable adds the information about the current thread we are in and the order / number of this asynchronous invocation.

Lets have a look at the first Runnable. Here our first thread called the method M5

Method sensor storage

Usually the analysis of a problem is based on invocation sequences. The invocation sequence is a container for method sensors. But the most likely case is that method sensors are mainly integrated in order to see the invocation sequence and not to create a chart that can be presented all the time. To address this, it would be meaningful to:

1. Allows measure the concrete time that a method within an invocation took (if there is a timer defined) but
2. only add additional graphing data for the timer if this is explicitly requested within the configuration

Braindump

• Each thread has a unique number (per JVM) automatically assigned
  • But threads can and - in fact are - reused. For example thread pools do not instantiate new threads, but instead pass different Runnable instances to the existing threads. Each different runnable is itself a different functionality. Thus the thread ID cannot be safely used for identification
• To build up an invocation sequence, each invoked method must know the method that was invoked before it
  • Innerthread communication can simply make use of the ThreadLocal pattern and build a stack of the concrete methods being called
  • Interthread and inter JVM communication has to connect the stacks of the involved threads. This can be done by an invocation identifier that is passed from the caller to the callee.
    • Invocation identifier must also include the method identification that invoked this call
• Continuous sending of information
  • Each method needs to have a thread-wide identification (stored within the stack)
  • Each invocation of a method must refer to the parent invocation
  • CMR server can match its current call tree with the information sent by the agent. Based on the invocation identification it will find the correct sequence. it will use the method identification to complete the tree

• Active pushing of invocation sequence id from one thread to another
  • hard to realize: how do we know that the next method instrumented will be invoked in a new thread
  • at this point in time threadlocal information of the caller is no longer available
  • It would be necessary to extend the method signature of all methods of all classes that inherit Thread / Runnable with an additional information (the invocation sequence) ... still very unclear
• Alternative would be to use JVM wide identification of the invocation sequence id with the respective thread

Questions

• Aktuell spielt gaukelt der invocation sequence hook einem timer usw vor, dass er ein Coreservice ist, korrekt?
• Wieso misst die invocation sequence selbst zusätzlich die Zeit? (oder tut sie das nur beim initialen Call gegen sich selbst?)

Jira tagging

• INSPECTIT-122 - analysis of the invocation sequence