Skip to main content
Ask Question
andrewhill
Support & Deployment Engineer
Asked a question 3 months ago

How do I sample Isilon metrics from Bright?

Where am I?

In Bright Computing, Inc. you can ask and answer questions and share your experience with others!

andrewhill
Support & Deployment Engineer

Isilon metrics collection script setup

The following steps are tested on Bright 9.0 with CentOS 7

 

1. Install Isilon SDK

Use the pip command on the headnode to download and install the Python Isilon SDK required by the Isilon metrics collection script, the following command will download and install version 8.2.1 of the Isilon SDK. In order to get the SDK working with Python 3.7, we will also need to patch the SDK files with sed:

# module load python37
# pip install isi_sdk_8_2_1
# sed -i 's/\<async\>/x_async/g' /cm/local/apps/python37/lib/python3.7/site-packages/isi_sdk_8_2_1/api/*.py
# sed -i 's/\<async\>/x_async/g' /cm/local/apps/python37/lib/python3.7/site-packages/isi_sdk_8_2_1/api_client.py

 

2. Modify the sample_isilon_clients.conf configuration file and set the IP/hostname of the Isilon and specify the Isilon username and password that will be used to gather the required metrics. You will then need to specify which interfaces on each compute nodes is used to connect to the Isilon which will be used to filter the results from the Isilon

# vi /cm/local/apps/cmd/scripts/metrics/configfiles/sample_isilon_clients.conf
{
  "host": "isilon-test",
  "username": "admin",
  "password": "MyPassword1",
  "interfaces": ["enp0s8"]
}

 

3. Add the sample_isilon metric collection script

The following commands will add the isilon collection script and to sample metrics for the "default" category and is set to run at a 1 minute interval

# cmsh
% monitoring setup
% add collection isilon
% set script /path/to/sample_isilon_clients
% set interval 1m
% nodeexecutionfilters
% active
% exit
% executionmultiplexers
% add category default
% set categories default
% commit

 

4. Verify that the metrics collection is working as expected

Select a compute node that mounts the Isilon storage and try reading/writing some data to the Isilon NFS mount. While reading/writing data to the share, confirm that the metrics collection is working as expected (in/out rate is determined in bytes/s):

# cmsh
% device use node001
% samplenow isilon_in_rate
% samplenow isilon_out_rate
% samplenow isilon_ops_rate

 

Extra Isilon disk sampling script

There is also an additional Isilon sampling script available that will sample specific disk information from each Isilon node in the cluster which may be helpful to the cluster administrator. The script currently samples the following disk metrics from the Isilon nodes:

1. node.disk.access.latency.all

2. node.disk.iosched.queue.all

3. node.disk.xfer.size.in8.all

4. node.disk.xfer.size.out.all

 

This script also relies on the isi_sdk_8_2_1 Python module as per the previous sample_isilon_clients collection script. Setting up this collection script can be done using the following steps:

 

1. Add the Isilon node as a generic device. In the following example we are adding a device called "MyIsilonNode-1" with the IP address 10.141.255.100 on the "myisilonnet" network. Set the userdefinedresources of the device to "IsilonNode" so that we can set the execution multiplexers for the collection script to run for each Isilon node. Repeat this process for each Isilon node in the Isilon storage cluster.

# cmsh
% device
% add genericdevice MyIsilonNode-1
% set ip 10.141.255.100
% set network myisilonnet
% set userdefinedresoures IsilonNode
% commit

 

2. Set the Isilon credentials in the sample_isilon_nodes.conf configuration file so that the collection script is able to authenticate against the Isilon in order to sample the nodes.

# vi /cm/local/apps/cmd/scripts/metrics/configfiles/sample_isilon_nodes.conf
{
 &amp;nbsp;"username": "admin",
 &amp;nbsp;"password": "MyPassword1"
}

Save and quit the editor.

 

3. Add the Isilon node collection script. We set the node execution filter so that the script only runs on the active headnode for each Isilon node. Ensure that the resource set is the same as defined before in the Isilon node user defined resources.

# cmsh
% monitoring setup
% add collection isilon_nodes
% set script /cm/local/apps/cmd/scripts/metrics/sample_isilon_nodes
% nodeexecutionfilters
% active
% executionmultiplexers
% add resource IsilonNode
% set resources IsilonNode
% commit

 

4. Test by running a "samplenow" against one of the Isilon nodes to ensure that the collection script is working as expected.

# cmsh
% device use MyIsilonNode-1
% samplenow