Data loaders

Data loaders gather cluster operation data and load it into tables in a relational database containing raw data. Data loaders are controlled by the Platform loader controller (plc) service.

About data loaders

Data loaders are polling loaders or history data loaders. The data loaders gather data and load this data into specific tables in the relational database containing raw data. Data loaders handle daylight savings automatically by using GMT time when gathering data.

Logging levels

There are logging levels that determine the detail of messages that the data loaders record in the log files. In decreasing level of detail, these levels are ALL (all messages), TRACE, DEBUG, INFO, WARN, ERROR, FATAL, and OFF (no messages).

By default, data loaders log messages of INFO level or higher (that is, all INFO, WARN, ERROR, and FATAL messages).

The data loader log files are located in the dataloader subdirectory of the log directory:

  • UNIX: $PERF_LOGDIR/dataloader

  • Windows: %PERF_LOGDIR%\dataloader

Default behavior

Data loaders gather data from data sources at regular intervals. The following are lists of the data loaders, the specific loader controller configuration file (plc_*.xml), and the default behavior:

LSF host data loaders (plc_coreutil.xml)


Data loader name

Data type

Data gathering interval

Data loads to

Loader type

Host core utilization (hostcoreutilloader)

core utilization

5 minutes

HOST_CORE_UTILIZATION

polling


LSF job data loaders (plc_bjobs-sp012.xml)


Data loader name

Data type

Data gathering interval

Data loads to

Loader type

Bjobs (lsfbjobsloader)

job-related

10 minutes

LSF_BJOBS

polling


LSF data loaders (plc_lsf.xml)


Data loader name

Data type

Data gathering interval

Data loads to

Loader type

Host metrics (hostmetricsloader)

host-related metrics

10 minutes

RESOURCE_METRICS

RESOURCES_RESOURCE_METRICS

polling

Host properties (hostpropertiesloader)

resource properties

1 hour

LSF_RESOURCE_PROPERTIES

polling

Bhosts (lsfbhostsloader)

host utilization and state-related

10 minutes

LSF_BHOSTS

polling

LSF events (lsfeventsloader)

events with a job ID, performance events, resource events, JOB_FINISH2 events

5 minutes

LSB_EVENTS

LSB_EVENTS_EXECHOSTLIST

LSF_PERFORMANCE_METRIC

LSB_JOB_FINISH & LSB_JOB_EXECHOSTS

file

Resource properties (lsfresproploader)

shared resource properties

1 hour

LSF_RESOURCE_PROPERTIES

polling

SLA (lsfslaloader)

SLA performance

5 minutes

LSF_SLA

polling

Shared resource usage (sharedresusageloader)

shared resource usage

5 minutes

SHARED_RESOURCE_USAGE

SHARED_RESOURCE_USAGE_HOSTLIST

polling


LSF advanced data loaders (plc_lsf_advanced.xml)


Data loader name

Data type

Data gathering interval

Data loads to

Loader type

Host group (hostgrouploader)

host group

1 hour

HOST_GROUP

polling

Bqueues (lsfbqueueloader)

queue properties

5 minutes

LSF_BQUEUES

polling

Pending reason (lsfpendingreasonloader)

job pending reasons

15 minutes

JOBS_PENDING_REASON

DPR_BYINTERVAL

polling

User group (usergrouploader)

user group

1 hour

USER_GROUP

polling

Pending Reasons (lsbpendingreasonsloader)

job pending reason - from the LSF data file lsb.pendingreasons

10 minutes

LSB_JOB_PENDINGREASON

file

Job status (lsfjobstatusloader)

job status - from the LSF data file lsb.status

10 minutes

LSB_JOB_STATUS

file


FLEXnet data loaders (plc_license.xml)


Data loader name

Data type

Data gathering interval

Data loads to

Loader type

FLEXnet usage (flexlicusageloader)

license usage

5 minutes

FLEXLM_LICENSE_USAGE

polling

FLEXnet events (flexliceventsloader)

license log file event

5 minutes

FLEXLM_LICENSE_EVENTS

file

FLEXnet Manager (fnmloader) - only supports FLEXnet Manager 11 or later.

license event

30 minutes

FLEXNET_LICENSE_EVENTS

database


Data loader interactions

The loader controller service controls the scheduling of the data loaders. The data loaders store LSF and license data into data tables through the node data sources. Each data loader contains data that is stored in specific data tables in the database.

The following diagram illustrates the interaction between data loaders and other components.

Configuration to modify data loader behavior

After editing the loader controller configuration files, restart the loader controller for your changes to take effect. The specific loader controller configuration file (plc_*.xml) depends on the type of data loader.

These files are located in the loader controller configuration directory:

  • UNIX: $PERF_CONFDIR/plc

  • Windows: %PERF_CONFDIR%\plc


Action

Configuration files

Parameter and syntax

Specify the frequency of data gathering for the specified data loader.

Loader controller configuration files for your data loaders (plc_*.xml).

<DataLoader Name="loader_name" Interval="gather_interval" ... />

where

  • loader_name is the name of your data loader

  • gather_interval is the time interval between data gathering, in seconds

Enable data gathering for the specified data loader.

This is enabled by default.

<DataLoader Name="loader_name
" ... Enable="true" ... />

where

  • loader_name is the name of your data loader

Disable data gathering for the specified data loader.

<DataLoader Name="loader_name
" ... Enable="false" ... />

where

  • loader_name is the name of your data loader

Enable data loss protection for the specified data loader.

This is enabled by default.

Specific data loader configuration file: dataloader_name.xml

File location:

  • UNIX: $PERF_CONFDIR/dataloader
  • Windows: %PERF_CONFDIR%\dataloader

<Writer ... EnableRecover="Y">

Disable data loss protection for the specified data loader.

<Writer ... EnableRecover="N">

Specify the default log level of your data loader log files.

log4j.properties

File location:

  • UNIX: $PERF_CONFDIR
  • Windows: %PERF_CONFDIR%

log4j.logger.${dataloader}=log_level, ${dataloader}

where

  • log_level is the default log level of your data loader log files.

Specify the log level of the log files for the specified data loader.

log4j.logger.dataloader.loader_name=log_level

where

  • loader_name is the name of the data loader.

  • log_level is the log level of the specified data loader.

For example, to set the LSF events data loader (lsfeventsloader) to ERROR, add the following line to log4j.properties:

log4j.logger.dataloader.lsfeventsloader=ERROR

Specify the log level of the log files for the reader or writer area of the specified data loader.

log4j.logger.dataloader.loader_name.area=log_level

where

  • loader_name is the name of the data loader.

  • area is either reader or writer.

  • log_level is the log level of the specified data loader.

For example, to set the LSF events data loader (lsfeventsloader) writer to DEBUG, add the following line to log4j.properties:

log4j.logger.dataloader.lsfeventsloader.writer=ERROR


The data loaders only log messages of the same or lower level of detail as log_level. Therefore, if you change the log level to ERROR, the data loaders will only log ERROR and FATAL messages.

Data loader actions


Action

Command line

View the status and logging levels of the data loaders.

UNIX: plcclient.sh ‑s

Windows: plcclient ‑s

Dynamically change the log level of your data loader log files (temporarily).

UNIX: plcclient.sh ‑n loader_name ‑l log_level

Windows: plcclient ‑n loader_name ‑l log_level

where

  • loader_name is the name of your data loader

  • log_level is the log level of your data loader log files.

If you restart the loader controller, these settings will revert back to the default level.

Dynamically change the log level of the log files for the reader or writer area of the specified data loader (temporarily).

UNIX: plcclient.sh ‑n loader_name ‑l log_level ‑a area

Windows: plcclient ‑n loader_name ‑l log_level ‑a area

where

  • loader_name is the name of your data loader

  • area is either reader or writer.

  • log_level is the log level of your data loader log files.

If you restart the loader controller, these settings will revert back to the default level.


View or dynamically edit the data loader settings

Use the Platform Analytics Console to view or edit the data loader settings. Any changes you make to the settings are permanent (that is, even after restarting the loader controller).

  1. In the navigation tree of the Platform Analytics Console, select Data Collection Nodes.
  2. Right-click the loader controller for your cluster and select Loader properties.
    Note:

    You can only view the data loader properties when the corresponding loader controller is running.

  3. Right-click the data loader you want to view or edit and select Properties.
  4. Edit the data loader parameters, if needed.

    You can edit the following data loader parameters:

    • Parameters: The specific parameters for the data loader. You can only edit the parameters of FLEXnet data loaders (flexlicusageloader and flexliceventsloader).

    • Interval (seconds): The data gathering interval of the data loader, in seconds.

    • Log level: The data loader logs messages of a level specified here and higher.

    • Reader Area: The reader area of the data loader logs messages of a level specified here and higher. Specify Inherit to use the same log level as the entire data loader.

    • Writer Area: The writer area of the data loader logs messages of a level specified here and higher. Specify Inherit to use the same log level as the entire data loader.

    • Description: A description of the data loader.

  5. To save any changes and close the window, click OK.