...

After 

Protection from low disk space

...

1. Make sure that the tree does not get too big. If the size of the tree over time is higher than the specified limit, tree will be saved and new tree will be created. At the end we can end up having many trees, that will be reassembled on the User interface when querying is performed.
2. Keep track of all writing tasks that have been started and wait until all task are done, so that information about position/size can be in the tree before the tree is saved to disk.

...

Storage on the UI

What do we need for exploring, what files are downloaded and when

Data querying and retrieving

Difference between remote and downloaded storage

...

When storage is to be explored on the User interface there is a need to first download all agent and index files and save them locally in the folder for the storage to be explored. This process is called "mounting" of storage. When files are ones downloaded they will remain on the local machine, thus next time storage can be explored without download. In addition, a local storage information file is created and saved on local disk.

When the files are locally available, next step is to create a StorageRepositoryDefintion. During the repository creation all agent and index files will be de-serialized. If more than one indexing file is available they will be combined via CombinedStorageBranch class. The StorageRepositoryDefinition can then provide all Agents for exploring. The indexing tree is passed to the Storage services that use it later on for data retrieval.

Storage can be also completely downloaded. In that case the above described process also includes downloading of all storage data file.

Data querying and retrieving

Storage services are responsible for querying the indexing tree and retrieving data for displaying in view. Index tree is queried with the same queries that are use on the CMR. This way we make sure that if the queries are changed, changes both apply to the CMR buffer and Storage since the structure of trees are same.

Querying the tree returns the list of storage descriptors. Descriptors are the objects holding the information about the file, start position and size of one or more objects that correspond to the given query. Depending if the storage is completely downloaded or not there are two ways how the data is retrieved:

• Retrieving data when storage is fully downloaded
  When all data files are locally available we use the ExtendedByteBufferInputStream to get the correct bytes from the disk. This stream needs the list of descriptors as input. The bytes are then de-serialized and data is returned.

• Retrieving data from remote storage via HTTP
  Situation is a bit more complicated when data needs to be retrieved from remote storage. First we group descriptors based on the file they point to. All descriptors that point to the one file are then processed together with a single HTTP request. The request sets the "Range" header that defines part of the file we want to get. For example with following value with the "Range" header we would get the bytes 10-19 and 90-99 (inclusive):

  Code Block
  bytes=10-19,90-99

  It must be noted that all the files in the root storage folder can be accessed via HTTP. The default Jetty servlet is used to serve them as static content.

Export/Import

When exporting the storage files are simply packed as the ZIP archive. If the storage is not downloaded, files are retrieved via HTTP. It is important to know that if compression is used the compressed content will be pack. Thus, when importing we first need to check if the content in file is of a GZip format. This can be done by checking the that two first bytes are matching the Gzip magic bytes.