Browser EUM Architecture Documentation

The ke

On this Page we describe the archtiecture of the JS Agent and the data model used for performing Browser End-User-Monitoring.
The goal is to ease the understnading of the concepts and the source code to ease the extension of the Agent with new features.

The page is currently at the state of INSPECTIT-2260 - Getting issue details... STATUS  .

Data Model

In this section we describe on how the data monitored at the client side is modeled at the Browser as well as at the Agent and the CMR.

See the package rocks.inspectit.shared.all.communication.data.eum for details on the different element types.

Element Identification and Referencing

Any type of data sent from the Browser to the CMR via the Agent is uniquely identifiable by a three-part ID as any type of data inherits from AbstractEUMElement:

1. The sessionID: This ID identifies the user session, e.g. a Browser opened at a computer. A session begins when the user starts browsing via our instrumented server and finishes when the last tab of this domain is closed.
   This ID is assigned to the browser by the Server java Agent using a session-Cookie.
2. The tabID: This ID identifies a certain browser tab within a user session. To be more specific, it identifies a running JS-Interpreter which is exactly one per tab.
   This ID also is assigned to the browser by the Agent by specifying the ID in the response sent from the Server Java Agent for the first Beacon. In the following Beacons the JS Agent will specify the ID received this way.
3. The localID: This ID identifies a certain AbstractEUMElement within a browser tab. It is assigned simply using a counter by the JS Agent.
   It is sufficient to store this ID when referencing other AbstractEUMElement, as elements can only reference other elements from the same tab. Therefore, the tabID and the sessionID can be infered from the element holding the reference.

Element / Beacon Serialization

The JS-Agent bundles multiple AbstractEUMElement before using the Beacon class sending them back to the Server Java Agent.
The data is serialized as a JSON String. Element references are serialized by storing the localID of the referenced element.
Note that the sessionID and tabID is not included for each  AbstractEUMElement, instead these are stored in the Beacon Object and assigned back to the individual elements when decoding the Beacon.

Trace Hierarchy Representation

Any kind of element which participates in a Trace inherits from AbstractEUMTraceElement. This type represents a full trace hierarchy using the following schema:

• Trace Elements only reference their parent (using its localID), not their children! This allows us to extend the child list of elements which have already been transferred from the Browser back to the Server java Agent.
• While JavaScript is single-threaded, it has a mechanism allowing the programmer to "asynchronously" issue method calls:
  This mechanism basically adds these calls to a queue and executes them as soon as the JS Interpreter has time for them, but still everything is run in a single thread!
  Especially this mechanism is non-preemptive, meaning that an exeucting function can not be interrupted to executed another function.
  To model such non-blocking calls, AbstractEUMTraceElement contains the "isAsyncCall()" method. If this method returns true, this means that the execution represented by this element was done non-blocking.
• To maintain the order within the children of a certain parent element, we introduced two mechanisms:
  • The "executionOrderIndex":
    As soon as a trace element begins executing, the JS Agent will automatically increment a counter and assign this value to the executionOrderIndex attribute.
    This allows us to exactly order siblings even if they are executed very fast, resulting in non-distinguishable timestamps.
  • The "enterTimestamp" and "exitTimestamp":
    These timestmaps provide timing information about the exeucted trace element. They can be used to order the elements within the trace for elements which lie out side of the scope of the JS Interpreter, for example Resource Loading Requests.
    Resource Loading Requests are executed truly asynchronously without allowing the JS Interpreter to interfer, therefore for the we cannot assign a "executionOrderIndex"

JS Agent Core Architecture

The JS Agent is divided into multiple sub-components, each providing a different service. Some components are accessible for the "Agent Modules", while other are hidden and only accessible from the agent core.
In this section we will walk through the different components and mechanisms provided by the JS-Agent.

Agent Modulization and Configuration

The Agent is freely configurable by selecting a different subset of the modules to use.
The source code sent to the Browser always follows the following layout:

In the HTML:

..
<html>
	<head>
		<script> 
			window.inspectIT_settings = {
				eumManagementServer : "[baseUrl]/inspectIT_beacon_handler"
				..
			}
		</script>
		<script src = "[baseUrl]/inspectit_jsagent_[revision]_[modules].js"/> 
	...
	</head>
...
</html>

This allows us to cache the JS Agent while keeping the settings configurable.

The actual "/inspectit_jsagent_[revision]_[modules].js" file looks like the following:

[Code from src/main/res/js/inspectit_jsagent_base.js]
...
...
//now the code from each selected module, e.g.:
[Code from src/main/res/js/plugins/ajax.js]
[Code from src/main/res/js/plugins/navtimings.js]
...
//finally the initialization invocation
inspectIT.init()

Within their code, each module has to register itself at teh agent core by invoking window.inspectIT.registerPlugin(name, module).
Hereby, name is used to uniquely identify this module to prevent issues in case of a double-injection of the script.
The parameter "module" is just an object storing several callbacks which will be invoked by the agent core. The available callbacks are listed below:

Creating and Managing "AbstractEUMElements"

In this section we describe, how modules can write data which will be sent to Server Java Agent and tehrefore to the CMR using Beacons.
Below is a usage example using which we will explain the available features. 

var metaInfo = inspectIT.createEUMElement("metaInfo");


metaInfo.require("metaInfoData");

metaInfo.language = userLanguage;
metaInfo.device = systemOS;
metaInfo.browser = browserName;

metaInfo.markComplete("metaInfoData");
metaInfo.markRelevant();

First, any type of AbstractEUMElement can be created using the factory-function window.inspectIT.createEUMElement(type) .
The "type" Argument is used to specify the concrete type as javascript is untyped, these are defined in the source of AbstractEUMElement.java.
In oru example, we created an object of type UserSessionInfo, which is identified by the name "metaInfo".
The method then creates a new Object, automatically assigning it a unique localID.

Any kind of data to send back to the server has to be specified as property of this object.
In the example, we set the values of the proeprties "language", "device" and "browser".

Every object created using createEUMElement has the methods require(featureName) and markComplete(featureName).
These are used to prevent the object from being sent when the collection of data has not been completed yet, this is mainly relevant for EUMElements which are shared and modified across multiple modules, such as PageLoadRequest.
To do so, you can call require(featureName) where featureName is just a String used as symbolic identifier for your data which is still missing.
This name has no purpose other than matching the markComplete(featureName) call with the same name, signaling that your data is complete and that from your perspective the object is now allowed to be sent.

To be more precise, an EUMElement is sent as soon as exactly both of the conditions below are met:

• All data is complete, meaning that there are no open require calls for which the corresponding markComplete has not been called yet 
• The element has been marked as "relevant", meaning that this element is possibly of interest for analysis and should be sent back to Java Agent

 The idea behind the "relevant" flag is to prevent the JS Agent from sending lots of unnecessary data, which provide no information gain when analysed.
For example, the "Listener Instrumentation Module" instruments and records every type of DOM event happening at the Browser, of which only a subset is actually of interest and therefore "relevant".
Certain types of EUMElements are always relevant, for example  UserSessionInfo or any type of Request. These can be marked as relevant by calling the "markRelevant()" method as done in the example above.
However, the key benefit of this concept of "relevancy" can be experienced when the EUMElement is part of a trace, and therefore is a subtype of AbstractEUMTraceElement.
When a element within a trace is marked as relevant, this will cause all of its parents in the trace to be marked as relevant too.
For example when a request (which is relevant by nature) occurs within a JSEventListenerExecution (which is not-relevant by nature), this will cause this concrete listener execution to be marked as relevant too, resulting in both elements being sent back to Java Agent.
At the inspectIT UI, it will then be possible to identify which event triggered the request using the JSEventListenerExecution element.

Building Traces

The component inspectIT.traceBuilder is responsible for managing the building of traces. It internally holds its own callstack for this purpose.

To add tracing information to an EUM element, all you have to do is to call inspectIT.traceBuilder.enterChild(element) to start tracing all further calls as children of the EUM element passed as argument.
Also, if the given element does not have a parent element set yet, this method will assign the current parent extracted from the callstack to the element.
This method also is responsible for assigning the executionOrderIndex property of the element passed as argument.
Alternatively, for example for asynchronous calls, you can invoke the method myElement.setParent(parentElement), which is present for any EUMElement created with the createEUMElement method.
To end the tracing of the element, all you have to is call inspectIT.traceBuilder.finishChild() , this will remove the current element from the callstack top.

See the example below, which generates a trace with one parent element, which has three children, of which one is an asynchronous call:

var parent = inspectIT.createEUMElement("someEntryPointType");
var childA = inspectIT.createEUMElement("someSyncMethodCallType");
var childB = inspectIT.createEUMElement("someSyncMethodCallType");
var childC = inspectIT.createEUMElement("someASyncMethodCallType");

inspectIT.traceBuilder.beginChild(parent);

inspectIT.traceBuilder.beginChild(childA); //automatically sets the parent of "childA" to "parent"
inspectIT.traceBuilder.finishChild(); //Pops "childA" from the callstack
//do stuff...
inspectIT.traceBuilder.beginChild(childB);
inspectIT.traceBuilder.finishChild(); //Pops "childB" from the callstack
//do stuff...
inspectIT.traceBuilder.finishChild(); //Pops "parent" from the callstack

//we also have an asynchronous child:
//set the parent manually as for asynchronous calls the parent is not present on the callstack anymore
childC.setParent(parent);
inspectIT.traceBuilder.beginChild(childC);
//do some stuff...
inspectIT.traceBuilder.finishChild(); //Pops "childC" from the callstack

//we say now that the only relevant element is childC:
childC.markRelevant();
//this causes childC and parent to be sent to the client, but NOT childA or childB, as these do not lie on the "relevant path"

The trace builder however is not responsible for assigning timing information to the elements. This has to be done manually through by invoking setEnterTimestamp(timestampInMS) and setExitTimestamp(timestampInMS) on the element.
The current timestamp can be queried through the method timestampMS() which is available alongside with other utility methods in the inspectIT.util component.
As soon as both timestamps have been set through these two methods, this will trigger an automatic "relevancy check": If the duration of the element exceeded the in the inspectIT UI configured "relevancy threshold", the element will be automatically marked as relevant.
This way, long-running javascript can be easily identified when inspecting the data.

As it is a common task to invoke the trace building methods alongside with capturing the timestamps, every element created via the createEUMElement method comes with a utility method for this purpose: buildTrace(captureTimestamps, function).
This method does the following:

1. Call enterChild for this element
2. Call setEnterTimestamp with the current timestamp, if captureTimestamps was set to "true"
3. Invoke the function passed in as argument if it is defined
4. Call setExitTimestamp with the current timestamp, if captureTimestamps was set to "true"
5. Call finishChild to complete the trace

Therefore, the example above can be rewritten using this method to also automatically capture the timestamps:

var parent = inspectIT.createEUMElement("someEntryPointType");
var childA = inspectIT.createEUMElement("someSyncMethodCallType");
var childB = inspectIT.createEUMElement("someSyncMethodCallType");
var childC = inspectIT.createEUMElement("someASyncMethodCallType");

parent.buildTrace(true, function() {
	childA.buildTrace(true, function() {
		//do stuff of child A
	});
	childB.buildTrace(true, function() {
		//do stuff of child B
	});
});

//For the asynchronous child we still have to specify the parent manually
childC.setParent(parent);
childC.buildTrace(true, function() {
	//do some stuff...
});

//we say now that the only relevant element is childC:
childC.markRelevant();
//this causes childC, and parent to be sent to the client, but NOT childA or childB, as these do not lie on the "relevant path"

Using this method improves the readability of your code and also prevents you from forgetting the finishChild call.

Transmitting Elements back to the Java Agent

As previously stated, you do not need to do anything to send your EUMElements back to the java Agent and therefore to the CMR.
Elements are sent automatically as soon as they are bot "relevant" and their data is complete.

The sending happens in the agent core through the hidden beaconService component. This component bundles the completed EUMElements and sends them either though the new HTML beacon API or through normal AJAX requests.
Beacons are sent when for at least 2.5 seconds no new elements have been added to the "send-Queue" or if the last beacon was sent at least 15 seconds ago.
Also, this component communicates with the Java Agent to make sure that a valid tabID and sessionID have been assigned.

The tabID is assigned by sending the first beacon with a tabID of "-1". This lets the Java Agent know that there is a new Tab and responds to the beacon with the tabID which should be used by this tab.

The sessionID is normally already assigned though cookies, however if this did not happen properly it is assigned through the same mechanism as the tabID.

Instrumentation of JavaScript

Instrumentation is done by overriding system functions provided by the Browser, for example for generating AJAX Requests or the setTimeout function.
However, as we use these system functions within our own code, which should not be instrumented, we need a mechanism to temporarily disable the instrumentation.
This is manged by the inspectIT.instrumentation component. See the example below on how it is used  for this purpose:

//example instrumentation of the setTimeout function, normally done by the asnyc.js module:

var originalSetTimeout = window.setTimeout;
window.setTimeout = function(callback, duration) {
	if(inspectIT.instrumentation.isEnabled()) { //check if the instrumentaiton is enabled
		function instrumentedCallback() {
			var timerLog = inspectIT.createEUMElement("timerExecution");
			
			//capture additional data for the tiemr log here..
			...
			
			//build the trace
			timerLog.buildTrace(true, function() {
				callback.apply(originalThis,originalArgs);
			})
			
			...
		}

		//modify the arguments to use our callback instead of the original one
		var modifiedArgs = Array.prototype.slice.call(arguments);
		modifiedArgs[0] = instrumentedCallback;
		
		return originalSetTimeout.apply(this,modifiedArgs);
	} else {
		//if the instrumentaiton is disabled just cal lthe original method
		return originalSetTimeout.apply(this,arguments);
	}		
}

//How to disable instrumentation:
//Variant a) pair of disable() / reenable() calls
window.inspectIT.instrumentation.disable();
setTimeout(...); //This call will not be instrumented
window.inspectIT.instrumentation.reenable();

//Variant b) create a wrapper function for which the instrumentation is automatically disabled
function myFunc() {
	setTimeout(...); //This call would normally be instrumented
}
var myFuncWithoutInstrumentation = window.inspectIT.instrumentation.disableFor(myFunc);
myFuncWithoutInstrumentation(); //calls myFunc with instrumetnation disabled

//Variant c) just run a given function with instrumentation disabled
window.inspectIT.instrumentation.runWithout(function() {
	setTimeout(...); //This call will not be instrumented
});

A reoccurring task is the instrumentation of the addEventListener() function provided by the browser to monitor the performance of event listeners.
For this reason, the instrumentation of this method has been centralized to the JS Agent Base and the instrumentation of event listeners is available through inspectIT.instrumentation.addEventListener(instrumentation) and removeEventListener(instrumentation).

This instrumentation is done by replacing every callback passed to the Browser via the addEventListener() method with a custom callback, which first invokes all listener instrumentations registered using the methods above.
The key benefit is that using this appraoch we do not need to keep an individual list of active instrumentations for each callback passed to addEventListener(), which would result in a high memory footprint and a complex memory management, possibly yielding memory leaks.
A second benefit is by "proactively" instrumenting every event listener, we can apply new instrumentation to event listeners which have been registered before the instrumentation was registered.
The downside however is that with the number of instrumentations and the number of event listener the performance may decrease. For this reason, the UI offers an option to globally disable the instrumentation of event listeners.

See the listener.js module as an example on how to write and use listener instrumentations.