Skip to content

Collection Schedule

How often the MetaMiner Server (MMS) collects each kind of Power BI, Fabric and Microsoft Entra ID data, and how that data reaches the MetaMiner Client (MMC).

How it works

For each enabled Power BI environment MMS runs two independent loops:

  • Collectors call the Microsoft APIs and write the results to the MetaMiner Database (MMDB). There is one collector per kind of data. Each collector works through its data in steps; when it has finished a full cycle it waits for its interval, then starts again.
  • Snapshot cycle: every 10 minutes MMS reads the collected data from the MMDB and builds the snapshot that MMC shows. The snapshot cycle makes no Microsoft calls.
flowchart LR
  MS[Microsoft Power BI, Fabric, Graph APIs] -->|collectors, own intervals| MMDB[(MMDB)]
  MMDB -->|snapshot cycle, every 10 min| SNAP[Client snapshot]
  SNAP --> MMC[MetaMiner Client]
  MMDB --> WEB[Usage dashboards]

All collectors of an environment run at the same time; one collector never runs two steps at once. A new object therefore appears in MMC after its collector has stored it and the next snapshot cycle has run.

Collector intervals

The interval is the wait after a completed cycle. While a collector still has work (more pages, more workspaces to scan, more hours to catch up) it continues after about one second.

Collector Collects Interval
GET_MODIFIED_WORKSPACES Workspaces changed since the last check 5 minutes
WORKSPACE_SCAN Detailed workspace metadata: reports, semantic models, tables, expressions, dataflows, data sources, users, lineage. Up to 100 workspaces per batch, new or changed ones first. 2 minutes
GET_WORKSPACES Full workspace list 24 hours
ACTIVITY_EVENTS Activity log events, in one-hour windows 10 minutes
REFRESHABLES Semantic model refresh history 10 minutes
APPS Power BI apps 60 minutes
CAPACITIES Capacities 3 hours
FCM Fabric Capacity Metrics app data; see Capacity Metrics Datasets 60 minutes
GATEWAYS On-premises data gateways 3 hours
CONNECTIONS Gateway and cloud connections 1 hour
FABRIC_ITEMS Fabric item inventory 3 hours
FABRIC_ITEM_DEFINITIONS Item definitions of new or changed Fabric items 180 minutes
DATAFLOW_DEFINITIONS Dataflow definitions 3 hours
GRAPH_USERS Entra ID users (changes since the last run) 30 minutes
GRAPH_GROUPS Entra ID groups 30 minutes
GRAPH_GROUP_MEMBERS Group memberships 30 minutes
GRAPH_SERVICE_PRINCIPALS Service principals 30 minutes
GRAPH_APPLICATIONS App registrations 30 minutes
GRAPH_USER_LICENCES User license assignments 30 minutes
GRAPH_GROUP_LICENCES Group license assignments 30 minutes
GRAPH_SUBSCRIBED_SKUS Tenant license subscriptions 24 hours

First collection

On the first run activity events are read back about 28 days, capacity metrics 14 days. Workspace scans start with every workspace and then continue with changed ones. On a large tenant the first full inventory can take several hours.

Microsoft throttling

When Microsoft answers that a limit is reached, the collector waits as long as Microsoft asks, or 20 minutes when no wait is given, then resumes from where it stopped. Other collectors are not affected. Do not run two MetaMiner environments, or two MetaMiner Servers, against the same tenant: they share Microsoft's per-tenant limits.

After a restart

Collectors resume from progress stored in the MMDB (time watermarks, scan queues, sync links). The snapshot is rebuilt from the MMDB at start. Work in flight at the moment of the stop, for example a running workspace scan batch, is requested again.

Refresh the client snapshot now

In MMC, right-click the environment row in the Power BI Explorer and choose Force On Demand Pull.... MMS rebuilds the snapshot from the MMDB immediately instead of waiting for the next 10-minute cycle, and waits up to 20 minutes for the rebuild to finish. It does not make the collectors call Microsoft sooner.

Image of the MMC Power BI Explorer with the environment row context menu open and Force On Demand Pull... highlighted

../../images/admin/pbi-force-on-demand-pull.png

Turn off one collector

You can switch off a single collector for one environment, for example the license collectors if you do not use the licensing views.

  1. Find the environment ID: open server\logs\statereport.log and look under "Configured Power BI Environments". Each line starts with the ID followed by + (enabled) or -.

  2. Add a line to server\MetaMinerServer.properties:

    server\MetaMinerServer.properties
    pbi.<environment ID>.GRAPH_USER_LICENCES.processor.enabled=false

  3. Restart the server.

The state report then lists the collector as DISABLED. Remove the line and restart to turn it on again. Views that depend on the collector stop receiving new data.

Use with Infolytik Support

Turning off WORKSPACE_SCAN, GET_WORKSPACES or GET_MODIFIED_WORKSPACES stops the inventory. Check with Infolytik Support before you turn off a collector.

Other schedule properties
Property Default Effect
pbi.env.pull.period 10 Minutes between snapshot cycles
pbi.fabric.item.definitions.interval.minutes 180 Interval of FABRIC_ITEM_DEFINITIONS

Set them in MetaMinerServer.properties and restart, only on advice from Infolytik Support. The other collector intervals are fixed.