Introduction to the trendminer R package

Alexander Lemm

Source: vignettes/trendminer.Rmd

TrendMiner is an industrial self-service analytics platform for analyzing, monitoring and predicting time-series based process and asset data. The trendminer R package allows access to selected endpoints of the TrendMiner API which is documented here: http://developer.trendminer.com/.

This vignette provides a brief introduction to the trendminer R package, highlighting the following key details of its use:

  • Connecting to a TrendMiner instance from an interactive R session
  • Searching for assets and tags in the asset framework
  • Browsing and visualizing structures in asset trees
  • Fetching time series data of tags
  • Visualizing tag-based time series data

Authentication

All requests to the TrendMiner API require authentication. This is achieved by sending a valid Bearer access token in the request headers. Request tokens are obtained via OAuth2.0 using the Resource Owner Password Credentials flow. That means that any client which likes to interact with the API needs to collect the credentials from the user (username and password) first before passing them together with its own client credentials (client ID and client secret) to the server. The server responds with an access token which the user needs to use for any subsequent API request.

Note: You can request your client ID and client secret by contacting TrendMiner support or your TrendMiner Customer Success Manager.

User credentials, client credentials and the TrendMiner base URL can be passed as arguments to tm_token() for quick testing in interactive mode.

However, it is recommended to call tm_token() without arguments. In this case tm_token() will fetch the credentials and the TrendMiner base URL from the environment variables specified below which you need to store in .Renviron. You can easily edit .Renviron using usethis::edit_r_environ().

For any subsequent API request you just need to pass the token to the respective function as an argument.

Pagination (handling multi-page responses)

TrendMiner responses might be paginated. An internal function manages pagination automatically by combining all paginated search results in a data frame before returning them to the respective exported function that was called. This internal process for checking for further pages and combining the results is completely hidden from the end user.

The ... argument

Each function from the trendminer package comes with a ... (dot-dot-dot) argument. It is used to pass on additional arguments to the underlying HTTP method from the httr package. This might be necessary if you need to set some curl options explicitly via httr::config().

Function naming convention

To play nicely with tab completion, we use consistent prefixes:

  • tm_ for all functions in the package
  • tm_af_ for functions wrapping endpoints of the TrendMiner asset framework API
  • tm_ts_ for functions wrapping endpoints of the TrendMiner time series API

Introduction to the TrendMiner asset framework

The TrendMiner asset framework is constructed around two primary concepts:

  • Nodes
  • Structures

A node either represents an asset (component of the plant) or a tag (attribute of an asset storing timeseries data).

A structure defines the parent-child relationship between assets and attributes or assets and other assets. It defines the logical place of a node in an asset tree. Since a node can be part of various asset trees at the same time, nodes and structures are defined separately.

This distinction between nodes (assets and tags) and structures is reflected accordingly in the trendminer R package: An asset framework related function either returns assets and tags if it is node related or it returns structures.

Since every structure represents a node at a particular level of an asset tree including the entire parent-child relationship path starting at the respective root, functions which return structures actually return two types of information at once:

  • Information on the node the structure presents
  • A single asset tree path including all the notes from the root until the node the structure represents

Searching for assets and tags

tm_search_assets() allows you to search for nodes in the TrendMiner asset framework based on a query you define. There are several search, logical and wildcard operators available which you can use together with selected node properties to construct your query. Below we will give you some examples but you can find all the details in the function’s documentation.

Let us start by searching for all assets and tags which might be reactor-related. We are loading dplyr below only to further process the returned data frames.

If you like to fetch all tags whose name starts with “Temperature” you can do the following:

tm_search_assets() powers two other functions under the hood using pre-defined search queries: tm_assets() which returns the complete set of assets and tm_tags() which returns all tags at once.

Note that in all function calls above TrendMiner only returned a data frame of nodes without any information about their location in the asset trees.

Browsing structures in an asset tree

The best way to start browsing your asset trees is to fetch all available root structures. They give you the entry points to the the different available asset trees:

Next we might want to explore the root structure of site Barcelona a bit more by taking a look at its children. We will use tm_child_structures() for this task which returns child structures by parent structure ID.

We now could go on and continue calling tm_child_structures() on each level with all newly retrieved structure IDs until we reach the leaves to get to know the complete tree structure of the Site Barcelona root structure.

Alternatively, we can make use of tm_descendant_structures() that retrieves the entire asset subtree underneath a given structure. In the example below we would like to get the asset subtree structure underneath the Solvents asset:

Above you see the entire Solvents subtree in table format which was returned by tm_descendant_structures(). Even though all the information is present it is very hard for the human eye to depict the tree structure from the table.

Here we can leverage the data.tree package which allows to convert a data frame that includes tree information in table or network format into a tree structure. Printing the tree structure after the conversion to the console will give us a much better understanding about the asset subtree:

Visualizing asset trees

There are several libraries available which allow you to plot tree structures. In this introduction we will use the network3D package and pass the tree structure we just created with the data.tree package above to the networkD3::diagonalNetwok() function:

library(networkD3)

diagonalNetwork(ToListExplicit(solvents_subtree$Solvents, unname = TRUE), fontSize = 14,
                height = 400, width = 600)

By adding some magrittr and purrr flavor you can even create tree plots for all root structures including all of their descendants in one go. Based on the breadth and depth of your asset hierarchy, however, that might be a very expensive call so use the code example below with caution:

library(purrr)

plots <- tm_af_root_structures(token) %$%
  map(structureId, tm_af_descendant_structures, token = token) %>%
  map(FromDataFrameTable, pathName = "externalId") %>%
  map(~ diagonalNetwork(ToListExplicit(.x, unname = TRUE), fontSize = 14, 
                        height = 500, width = 700))

Fetching and visualizing time series data of tags

All functions and examples we covered so far dealt with extracting and visualizing asset framework data using asset framework API functions. In this section we will now focus on TrendMiner time series API functions for tags.

In the examples below we will use the set of tags which are part of the Reactor 1 sub asset tree of site Barcelona which we already know from above:

tm_ts_interpolated_data() lets you retrieve time series data by tag name. You need to define the start and the end time of the time series you like to fetch as POSIXct objects setting “UTC” as the timezone:

library(lubridate)

start <-  ymd_hms("2019-09-15T03:10:14Z")
end <- ymd_hms("2019-09-15T08:42:15Z")

ba_conc1_data <- tm_ts_interpolated_data(token, "BA:CONC.1", start, end, step = 2)

The maximum number of points tm_ts_interpolated_data() can return is 10.000. The step argument defines the time increment between returned observations. If you define step to be one second which is the default setting, the time interval you can select using the start and end arguments can span 2 hours, 46 minutes and 40 seconds (2 x 60 x 60 + 46 x 60 + 40 = 10.000) at most. Make sure to decrease the resolution by increasing the step argument if you like to fetch cohesive time series data of a tag which spans more than 10.000 seconds.

There are plenty of time series plotting options available in R. Since tm_ts_interpolated_data() returns the time series of the tag in a data frame we will use ggplot in our plotting examples below.

We start by visualizing the time series data we just downloaded for tag BA:CONC.1:

library(ggplot2)

ba_conc1_data$timeSeries %>%
  ggplot(aes(index, value)) +
  geom_line(color = "#09557f") +
  ggtitle(ba_conc1_data$tag$tagName) +
  theme_minimal()

If you know that the tags you are interested in operate on the same scale you can fetch and visualize their data in one go my chaining several purrr::map() calls together. Below, however, we will break the data fetching and visualizing process into two steps to explain some of the underlying details.

We start by selecting three tags from Reactor 1 whose time series data we like to extract from TrendMiner. We fetch their data and store the results in a single combined data frame on top of each other in long format. Because ggplot needs a data frame in long format including a group variable when plotting a multivariate time series, we make sure to add an additional group column tag_name to the data frame:

We can now directly plot our data without further processing:

combined_data %>% 
  ggplot(aes(index, value)) + 
  geom_line(aes(color = tag_name)) +
  ggtitle("Reactor 1 tags")+
  theme_minimal()