This page provides the general information about the storage for the developers, thus trying to explain in details the structure of storage and processes of their creation and manipulation.

General overview

The purpose of storages development was to allow the performance data to be persisted to disk. Thus, all data contained in a storage is placed on disk. Storages are always created on the CMR and later on can be exported or downloaded via User interface. By default, the root storage folder on both CMR and UI is [root]/storage. Note that the root storage folder can be changed by altering the storage configuration.

When a storage is created the unique ID is given to the storage that will serve as the property storages will be distinguished by. Every storage is placed in the separate folder inside root storage folder. The name of the sub-folder for the storage is same as the ID of the Storage. For example, if we have the storage with the ID=17aa8c9, then it's folder will be [rooStorageFolder]/17aa8c9.

Storage file types

There are several file types that exist for each storage. Here is the overview on those file types:

States

The following picture describes the states the storage can be at one moment in time:

At the moment once storage is finalized it can not be put into the writable state.

Data writing

When CMR is started, one ExecutorService is specially created for the purpose of writing to disk. This executor service is called IOExecutorService and has a purpose of writing bytes to the disk via Java NIO2. 

When storage is put in the writing state, the StorageWriter for that storage is created. Since we want to do the serialization and writing in parallel, each StorageWriter creates it's own ExecutorService that will be used for the operations needed to be done before executing the actual write. Thus, if we have 3 storages in writable state there will be 3 StorageWriters each having own ExecutorService and in addition the previously mentioned IOExecutorService that executes all IO operations for all storages.

Approach

For each data that is written using the StorageWriter a new WriteTask is created and it is given to the writer's executor service for execution. The write task holds the object to be written in a soft reference. This way we ensure that there is no OutOfMemory error if too much writing tasks are created and can not be processed so fast. When executed, the tasks performs the following steps:

1. Check if the data to write is still available via soft reference, if not abort
2. Inform the indexing tree that the write is starting. Indexing tree will return the ID of the channel where the file should be written. This ID is the same as the leaf in the indexing tree where data to be written is indexed.
3. Acquire a serializer from the queue. Wait if no serializer is available.
4. Serialize the data with ExtendedByteBufferOutputStream. Note that this output stream can serialize data of any size. If the data is too large to be serialized in one ByteBuffer, the stream will require additional buffers until data can fit. Additional buffers are provided by ByteBufferProvider.
5. Return the serializer to the queue.
6. Submit the IO writing task to the IOExecutorService passing the channel where write should be done and providing the output stream.

There is no certainty when new IO writing task will be executed. Because of this, the WriteReadCompletionRunnable is also passed to the info.novatec.inspectit.storage.nio.write.WritingChannelManager.write(ExtendedByteBufferOutputStream, Path, WriteReadCompletionRunnable) method. The runnable will be executed when the IO operation is done. If the write was successful, the indexing tree will be informed about the position in the file and the size of written object in bytes. Otherwise, the indexing tree will be informed that the write failed, which will effectifly remove the information about writing data from the indexing tree.

Data processors

In most cases write to the storage is not used directly. Instead, the data is passed to the set of Data processor that perform additional operation on data before it is written, filter out data, etc. The currently available processors are:

Write status

Each storage writer can provide the status of the write based on the amount of created and finished writing tasks since the time writer is created. Currently the status calculation is as follow:

After 

Protection from low disk space

Storage manager regularly checks for the remaining hard disk space and is suspending any write if the space is critically low.

Storage finalization

During the storage finalization following tasks are executed:

1. Suspend any additional writing tasks to be created
2. Wait until all submitted tasks are done
3. When no task is left shut down the Executor service
4. Inform the indexing tree that the write is over and that remaining part of indexing tree should be saved.

In addition to the CmrStorageWriter will write all agents who's data has been saved to stoarge.

Indexing

The indexing is needed because we need to save the start position and the size of the object in a file. Later on we can query the tree for retrieving the information about where the data we want to have is stored. This way one file can have many objects serialized and we can have many data files for one storage.

The tree is very similar to the indexing tree we use for the CMR buffer. The difference is that here tree has information about the file/position/size of the object, while in CMR buffer tree holds the direct reference to the object in memory.

There are two types of leafs that we have in the storage indexing tree:

1. ArrayBasedStorageLeaf - Holds the position and start of the each object that is added to the leaf. Advantage of this leaf is that every object can be referenced later on (needed for example for Invocations). The disadvantage is the size of the leaf that grows lineray with the amount of indexed objects.
2. LeafWithNoDescriptors - This leaf just holds the amount of total bytes in the file it is referring to. This means that when querying only all objects from leaf can be retrieved and there is no single object picking. Of course, the size of the leaf is very small and constant. The leaf can be used for the data types that do not have to be referenced singularly.

The IndexingTreeHandler component is responsible of managing the tree while the writing is in process. There are two main functions of the IndexingTreeHandler:

1. Make sure that the tree does not get too big. If the size of the tree over time is higher than the specified limit, tree will be saved and new tree will be created. At the end we can end up having many trees, that will be reassembled on the User interface when querying is performed.
2. Keep track of all writing tasks that have been started and wait until all task are done, so that information about position/size can be in the tree before the tree is saved to disk.

Storage on the UI

When storage is to be explored on the User interface there is a need to first download all agent and index files and save them locally in the folder for the storage to be explored. This process is called "mounting" of storage. When files are ones downloaded they will remain on the local machine, thus next time storage can be explored without download. In addition, a local storage information file is created and saved on local disk.

When the files are locally available, next step is to create a StorageRepositoryDefintion. During the repository creation all agent and index files will be de-serialized. If more than one indexing file is available they will be combined via CombinedStorageBranch class. The StorageRepositoryDefinition can then provide all Agents for exploring. The indexing tree is passed to the Storage services that use it later on for data retrieval.

Storage can be also completely downloaded. In that case the above described process also includes downloading of all storage data file.

Data querying and retrieving

Storage services are responsible for querying the indexing tree and retrieving data for displaying in view. Index tree is queried with the same queries that are use on the CMR. This way we make sure that if the queries are changed, changes both apply to the CMR buffer and Storage since the structure of trees are same.

Querying the tree returns the list of storage descriptors. Descriptors are the objects holding the information about the file, start position and size of one or more objects that correspond to the given query. Depending if the storage is completely downloaded or not there are two ways how the data is retrieved:

• Retrieving data when storage is fully downloaded
  When all data files are locally available we use the ExtendedByteBufferInputStream to get the correct bytes from the disk. This stream needs the list of descriptors as input. The bytes are then de-serialized and data is returned.

• Retrieving data from remote storage via HTTP
  Situation is a bit more complicated when data needs to be retrieved from remote storage. First we group descriptors based on the file they point to. All descriptors that point to the one file are then processed together with a single HTTP request. The request sets the "Range" header that defines part of the file we want to get. For example with following value with the "Range" header we would get the bytes 10-19 and 90-99 (inclusive):

  bytes=10-19,90-99

  It must be noted that all the files in the root storage folder can be accessed via HTTP. The default Jetty servlet is used to serve them as static content.

Export/Import

When exporting the storage files are simply packed as the ZIP archive. If the storage is not downloaded, files are retrieved via HTTP. It is important to know that if compression is used the compressed content will be pack. Thus, when importing we first need to check if the content in file is of a GZip format. This can be done by checking the that two first bytes are matching the Gzip magic bytes.