Skip to main content
Asked a question 4 years ago

How to install Gold Allocation Manager on top of Bright Cluster

Where am I?

In Bright Computing, Inc. you can ask and answer questions and share your experience with others!

How to install Gold Allocation Manager on top of Bright Cluster

Gold Allocation Manager is an HPC resource accounting tool. It is open source, and released under a license that seems to be a BSD-style license.

The following instructions to install it were tested on Bright 7.1 with CentOS6u6

Download the latest sources:

(one line:)

# wget -c http://www.clusterresources.com/downloads/gold/gold-2.2.0.5.tar.gz32

Untar the files:

# tar -xzvf /root/tmp-install/gold-2.2.0.5.tar.gz

Configure Gold with MySQL database:

# cd gold-2.2.0.5
# ./configure --prefix=/cm/shared/apps/gold/2.2.0.5 --with-db=mysql

Compile the sources:

To compile the program:

# make

To install the Gold web GUI:

# make gui

Installation:
To install Gold

# make install

To install the Gold web GUI

# make install-gui

** To delete the files created by the Gold installation, you can use 'make uninstall'.

You will also need to generate a secret key which enables secure communication between clients and server. This key is a passphrase consisting of up to 80 characters and can include spaces and the regular visible ASCII characters. Note that if you are using Gold with Moab or the Maui Scheduler, they will need both need to use a shared secret key.

# make auth_key

Enter your secret key (up to 80 characters and can include spaces): mysecret

Install the required Perl modules:

#  yum install perl-Log-Log4perl.noarch
#  yum install perl-Config-Properties.noarch
#  yum install perl-Error.noarch
#  yum install perl-Crypt-CBC.noarch
#  yum install perl-Digest-SHA1.x86_64
#  yum install perl-Digest-HMAC.noarch
# yum install perl-suidperl.x86_64

** for list of Perl Modules: http://docs.adaptivecomputing.com/gold/x385.php33

Create a Gold database:

# mysql -u root -p
mysql> create database gold;

Add database user for Gold:

# mysql -u root -p
mysql> use mysql;
mysql> create user gold;
mysql> grant all on *.* to 'gold'@'localhost21'; 

Initialize the database:

# mysql -u root -p gold < bank.sql

Edit the configuration files and add the database information:

# vim /cm/shared/apps/gold/2.2.0.5/etc/goldd.conf

and inside the file set:

database.user = gold

database.password = Ch@ngeMe

Start Gold:

# /cm/shared/apps/gold/2.2.0.5/sbin/goldd

** a startup script can be placed under /etc/init.d with the name gold in order to start/stop Gold as a service

Now you are now ready to define users, projects, machines, accounts etc. as necessary for your site.

Define Users:

(On one line:)

# gmkuser -n "Bright Support" -E "support@brightcomputing.com18" cmsupport

Define Machines:

# for i in `seq -w 1 100`; do gmkmachine -d "cn$i" cn$i; done

Modify Machines: 

# for i in `seq -w 1 100`; do gchmachine -d "cn$i" -m cn$i; done

Remove Machines:

# grmmachine cn001

Define Projects:

# gmkproject -d "Bright Test" cmsupport

Successfully created 1 Project
Auto-generated Account 1
Add Users to the Projects

# gchproject --addUsers cmsupport cmsupport

Successfully created 1 ProjectUser

Make Deposit

Add 360000000 credits to the cmsupport project, to be used from 2015 till 2016:

# gdeposit -p cmsupport -z 360000000 -s 2015-01-01 -e 2016-01-01
# glsalloc
Id Account Active StartTime  EndTime    Amount    CreditLimit Deposited Description    
-- ------- ------ ---------- ---------- --------- ----------- --------- --------------
1  1       True   -infinity  infinity           0           0         0 Auto-Generated
2  1       True   2015-01-01 2016-01-01 360000000           0 360000000                

 Check Balance:

# gbalance -p cmsupport
Id Name      Amount    Reserved Balance   CreditLimit Available
-- --------- --------- -------- --------- ----------- ---------
1  cmsupport 360000000        0 360000000           0 360000000
# gbalance -p cmsupport -u cmsupport
Id Name      Amount    Reserved Balance   CreditLimit Available
-- --------- --------- -------- --------- ----------- ---------
1  cmsupport 360000000        0 360000000           0 360000000
# gbalance -u cmsupport
Id Name      Amount    Reserved Balance   CreditLimit Available
-- --------- --------- -------- --------- ----------- ---------
1  cmsupport 360000000        0 360000000           0 360000000
# gbalance -u cmsupport -m cn001
Id Name      Amount    Reserved Balance   CreditLimit Available
-- --------- --------- -------- --------- ----------- ---------
1  cmsupport 360000000        0 360000000           0 360000000

** Now you are ready to run some jobs. Before doing so you will need to integrate Gold with your Resource Management System. Although the quotation, reservation, and charge steps will most likely be invoked automatically by your resource management system, it is useful to understand their effects by invoking them manually.

Simulate the lifecycle of a job

Assume the job has the following characteristics:

Job Id:               

Job Name:             
User Name:            cmsupport
Project Name:         cmsupport
Machine Name:         cn001-cn100
Requested Processors: 24
Estimated WallClock:  3600 seconds
Actual WallClock:     1234 seconds

** When a job is submitted, it is useful to check that the user's account has enough funds to run the job. This will be verified when the job starts, but by that point the job may have waited some time in the queue only to find out it never could have run in the first place. The job quotation step can fulfill this function. Additionally, the quote can be used to determine the cheapest place to run, and to guarantee the current rates will be used when the job is charged.

How much it will cost to run the job

# gquote -p cmsupport -u cmsupport -m cn001 -P 24 -t 3600

Successfully quoted 86400 credits

Make Reservation:

When a job starts, the resource management system creates a reservation (or pending charge) against the appropriate allocations based on the estimated wallclock limit specified for the job.

# greserve -J 130.cn-mgmt -p cmsupport -u cmsupport -m cn001 -P 24 -t 3600
Successfully reserved 86400 credits for job 130.cn-mgmt and created gold job id 1
[root@cn-mgmt ~]# gbalance
Id Name      Amount    Reserved Balance   CreditLimit Available
-- --------- --------- -------- --------- ----------- ---------
1  cmsupport 360000000    86400 359913600           0 359913600
 
[root@cn-mgmt ~]# gbalance -p cmsupport
Id Name      Amount    Reserved Balance   CreditLimit Available
-- --------- --------- -------- --------- ----------- ---------
1  cmsupport 360000000    86400 359913600           0 359913600

Charge Jobs:

After a job completes, any associated reservations are removed and a charge is issued against the appropriate allocations based on the actual wallclock time used by the job.

# gcharge -J 130.cn-mgmt -u cmsupport -p cmsupport -m cn001 -P 24 -X WallDuration=2400
# gbalance -p cmsupport --total
Balance   
---------
359942400
The account balance is 359942400 credits
 

Refund Jobs:

# grefund -J 130.cn-mgmt

Successfully refunded 57600 credits for job 130.cn-mgmt

Check Project Accounting Statement:

# gstatement -p cmsupport

Check Project Usage:

# gusage -p cmsupport

Integrate with WLM

Delayed Accounting

In the absence of a dynamic system, some sites enforce allocations by periodically (weekly or nightly) parsing resource manager job logs and then applying debits against the appropriate project accounts. Although Gold can easily support this type of system by the use of the qcharge command in post-processing scripts, this approach allows users or projects to use resources significantly beyond their designated allocation and generally suffers from stale accounting information.

Dynamic Accounting

Gold's design allows it to interact dynamically with your resource management system. Charges for resource utilization can be made immediately when the job finishes (or even incrementally throughout the job). Additionally, reservations can be issued at the start of a job to place a hold against the user's account, thereby ensuring that a job will only start if it has sufficient reserves to complete. The remainder of this document will describe the interactions for dynamic accounting.

Add an appropriate AMCFG line into maui.cfg to tell Maui how to talk to Gold:

# vim /cm/shared/apps/maui/current/spool/maui.cfg 

The AMCFG line is added in this kind of format:

AMCFG[bank] TYPE=GOLD HOST=cm-mgmt.cm.cluster PORT=7112 SOCKETPROTOCOL=HTTP WIREPROTOCOL=XML CHARGEPOLICY=DEBITALLWC JOBFAILUREACTION=IGNORE TIMEOUT=15

Add a CLIENTCFG line into maui-private.cfg to specify the shared secret key. This secret key will be the same secret key specified in the "make auth_key" step:

# cat /cm/shared/apps/maui/current/spool/maui-private.cfg
CLIENTCFG[AM:bank] KEY=Ch@ngeMe AUTHTYPE=HMAC64

** Gold will need to allow the the user id that Maui runs under to perform scheduler related commands (Job Charge, Reserve, Quote, etc).

Allow the maui user to use Gold:

# gmkuser -d "Maui Scheduler" maui

# goldsh RoleUser Create Role=Scheduler Name=maui

Methods of interacting with Gold

http://docs.adaptivecomputing.com/gold/x3877.php23

Interaction Points

http://docs.adaptivecomputing.com/gold/x3841.php31

References:

http://docs.adaptivecomputing.com/gold/30

Downloads:

http://www.clusterresources.com/downloads/gold/25