This page discusses the requirements of an advanced invocation sequence sensor to be integrated with inspectIT.

Features

The following list contains that necessary features of the advanced invocation sequence tracer.

Continuous sending of invocation sequence data
Invocation sequences that span over multiple threads
Invocation sequences that span over multiple JVMs
Storage of the invocation sequences in memory and as files (if requested by the user)
Reduction of method sensor storage (only save method sensors if they are configured)

Overview

Each invocation in Java always occur within one Thread. Within this thread an invocation may use methods of multiple classes. Inner Thread calls are all synchronous (read blocking) calls, thus the sequence is very strictly defined.

It is also possible that one thread calls another thread. This call is most likely an asynchronous call, in which the caller will not wait for the result. There is nearly no application that does not use multiple JVM instances and use communication between these JVMs.

The invocation sequence should provide the concrete user transaction over these boundaries. It thus provides the exact information necessary to understand why and where a concrete invocation is slow.

Analysis

In order to achieve the necessary functionality different topics needs to be addressed.

Unique identification of invocations

There are four different levels that affect the identification of invocations. These are (sequence starting from bottom to top):

Level	Section	Description	Link to the discussion of unique identification
1	Platform	The platform the agent is running on.	IS platform identification
2	JVM	On each platform can be multiple JVMs	IS JVM identification
3	"Thread"	Within each JVM different threads are run	IS thread identification
4	method	Within one thread a method can be invoked multiple times

Continuous Sending

Each invocation of a method checks if an invocation sequence is currently active
The staging area builds up a stack of all methods that are invoked within one "thread"
- This stack must be unique for each thread identification
The data is sent to the CMR. Based on the JVM identification, the platform identification and the thread identification, the CMR can extend its stack of the methods of one invocation by adding the received stack to it. If no information is currently available, the CMR must assume that this was a new invocation.

Inner-Thread

Invocations within one thread are saved as a stack that is unique per thread identification. The invocation of an instrumented method within an invocation sequence will create two different information objects:

one at the start of the method and (method _start)
one at the end of the method (method _end)

This functionality can easily be integrated by means of instrumenting the methods with before and after hooks. Additional identification of the methods are not necessary as the invocations within the thread are always synchronous and thus the stack of the invocations will easily allow to reconstruct the invocation sequence.

Example

The following structure shows how the stack may look like (notice that in order to save space, the stack is created from left to right). This structure allows to reconstruct, that M1 called M2 and afterwards called M3.

M1 start

M2 start

M2 end

M3 start

M3 end

M1 end

Inter-Thread

Problems that need to be solved:

Correlation of the per se independent invocation sequences of two (or more threads)
Identification which methods within the calling thread invoked a method within the called thread.

Basic: Inter-Thread invocation sequences should provide the location within the invocation that creates the asynchronous call.

Approach 1

Idea: Creation of a Runnable object identifies the location which initializes the asynchronous call to another thread

The creation of a Runnable is an additional event that can be added to the
We need to instrument the creation of a Runnable object and provide the following information
- the current thread identification of the current thread that created the Runnable
- the counter of how many asynchronous calls were already started within this thread

Scenario:

This scenario should help to describe this approach. Our calling thread has stack that is unique for this Runnable/Thread (see chapter about inner thread invocation sequences). Within this stack the creation of a Runnable is added.

M1 start

M2 start

Runnable (cur thread id, 1)

M2 end

Runnable (cur thread id, 2)

M1 end

So our m1 method within the thread first invokes the method m2, then we encounter a Runnable object being instantiated, which we understand as the start of an asynchronous call. Then method M2 terminates. After that method m1 starts another asynchronous call and returns. Each creation of a Runnable adds the information about the current thread we are in and the order / number of this asynchronous invocation.

Lets have a look at the first Runnable. Here our first thread called the method M5

M5 start

M6 start

M6 end

M5 end

Method sensor storage

Usually the analysis of a problem is based on invocation sequences. The invocation sequence is a container for method sensors. But the most likely case is that method sensors are mainly integrated in order to see the invocation sequence and not to create a chart that can be presented all the time. To address this, it would be meaningful to:

Allows measure the concrete time that a method within an invocation took (if there is a timer defined) but
only add additional graphing data for the timer if this is explicitly requested within the configuration

tracked as INSPECTIT-155

Braindump

Each thread has a unique number (per JVM) automatically assigned
- But threads can and - in fact are - reused. For example thread pools do not instantiate new threads, but instead pass different Runnable instances to the existing threads. Each different runnable is itself a different functionality. Thus the thread ID cannot be safely used for identification
To build up an invocation sequence, each invoked method must know the method that was invoked before it
- Innerthread communication can simply make use of the ThreadLocal pattern and build a stack of the concrete methods being called
- Interthread and inter JVM communication has to connect the stacks of the involved threads. This can be done by an invocation identifier that is passed from the caller to the callee.
  - Invocation identifier must also include the method identification that invoked this call
Continuous sending of information
- Each method needs to have a thread-wide identification (stored within the stack)
- Each invocation of a method must refer to the parent invocation
- CMR server can match its current call tree with the information sent by the agent. Based on the invocation identification it will find the correct sequence. it will use the method identification to complete the tree

ID	unique across	Buildup
method invocation identification	at least within one invocation sequence within on thread on one jvm two calls to the same method must result in different ids	static counter (within the same classloader static is safe)
invocation sequence identification	everything (JVM, thread)	JVM_ID + Thread ID thread id is not unique (thread pools!)

Active pushing of invocation sequence id from one thread to another
- hard to realize: how do we know that the next method instrumented will be invoked in a new thread
- at this point in time threadlocal information of the caller is no longer available
- It would be necessary to extend the method signature of all methods of all classes that inherit Thread / Runnable with an additional information (the invocation sequence) ... still very unclear
Alternative would be to use JVM wide identification of the invocation sequence id with the respective thread
How do we want to display invocation sequences over multi nodes
- We need to differentiate between synchronous response time (that the user sees) and complete runtime (including the time of all asynchronous calls)
- Dynatrace visualized inter thread/VM calls as an extra node "synchronous call" or "asynchronous call" (which can have multiple children). AppDynamics allows to "jump" to the invocation sequence that represents the connected invocation sequence, meaning that the invocation sequences are not displayed as one, but the user must navigate.

Questions

Aktuell spielt gaukelt der invocation sequence hook einem timer usw vor, dass er ein Coreservice ist, korrekt?
Wieso misst die invocation sequence selbst zusätzlich die Zeit? (oder tut sie das nur beim initialen Call gegen sich selbst?)

Jira tagging

INSPECTIT-122 - analysis of the invocation sequence

Browser not supported