About Initial and Incremental Pull¶
MetaMiner Server (MMS) needs to be aware of any data change on your BI landscape. For this, a fast and reliable data exchange between MMS and Tableau server(s) is vastly important.
When you add a new Tableau environment to your MetaMiner configuration, MMTE Server performs the initial pull. It could be slow, may take up to 8 hours, because no information was previously found at MMTE Server disposal.
You can track the progress of the initial pull if you connect to MMTE Server with MMTE Client and run Connection Manager:
Later MMTE Server periodically reads your corporate data from Tableau Server. This is known as an incremental pull, because MMTE Server downloads only changes to the existing data. Therefore it is much faster than the initial pull and usually takes several minutes or even seconds to complete.
There are two worker processes running simultaneously as a part of each MMTE Server for synchronization with Tableau server(s):
- Shallow pull worker
- Deep pull worker
Shallow (or: Regular) pull worker normally runs regularly on schedule. Its time interval is specified in the
tab.env.pull.period property and is 10 minutes by default.
You can also start shallow pull earlier:
Both actions initiate an immediate shallow pull.
On shallow pull, MMTE analyzes objects metadata (mainly from Tableau Repository) and detects objects that are changed (for example, their update timestamp has been changed). If it detects changed objects, it updates their metadata in MetaMiner Database and adds them to the queue for the deep pull worker.
Deep pull worker activates when it finds any object in its queue. Then it downloads the full data of corresponding workbooks/data sources and inspects this data to track all changes in the internal object content.
The Reprocess command initiates deep pull.
Deep pull is slow, it analyzes object files and detail information about them: e. g. utilized table columns. As it is slow and it may fail, there is a special feature for running it individually for a particular BI asset or a set of objects and for tracking deep pull status.
To run deep pull for a particular BI asset or a set of objects:
Open Tableau Explorer.
Select an object or a set of objects.
Right-click and choose Reprocess from the context menu.
Repeat this step for all objects.
- Select the environment where all these objects are located.
Right-click it and run the Force on Demand from the context menu to pull data for a particular environment.
The selected object(s) are added to the deep pull queue. Deep pull worker detects them and starts analyses.
To track deep pull status:
Open Tableau Explorer.
Select an object you want to track.
Right-click and choose Properties from the context menu.
In the Properties dialog window, click the Show deep pull status button.
The Status dialog box is displayed. It shows the current status of deep pull for this object.
The following values are possible:
|UNKNOWN||This workbook/data source was not processed since last launch of MMS, therefore MMTE has no information about process results.|
|PROCESSING||First time deep pull is currently running.|
|REPROCESSING||Deep pull was already run, and later Reprocess was called, and another deep pull is now in progress.|
|SUCCESS||Deep pull has completed successfully.|
|FAILURE||Deep pull has completed with an error.|
Display of the status is designed for Infolytik Technical Support duty, to help Infolytik personnel understand which information is at MMTE disposal and which is not. With it, they can run Reprocess and then view its results.
Monitoring Pull Status¶
You can monitor progress of the current shallow and deep pull commands and also view the log of all previous pulls in the Data Refresh Statistics dialog window:
To open this dialog window: