Viewing Current Jobs in the Scheduler

To view the scheduler’s current queue, log in to the oar scheduler node as root and run the command oarstat:

$ oarstat
Job id     Name           User           Submission Date     S Queue
---------- -------------- -------------- ------------------- - ----------
5725       4825           glassfish      2012-05-01 18:53:32 R default

This provides information on the jobs currently being executed as well as the jobs that are scheduled to be executed once a resource is free.

To view more details of a specific job, log in to the oar scheduler node as root and run the command oarstat and specify the JOB_ID:

$ oarstat --job 5725 --full
Job_Id: 5725
    job_array_id = 5725
    job_array_index = 1
    name = 4825
    project = [% 62, Installing distribution]
    owner = glassfish
    state = Running
    wanted_resources = -l "{type = 'default'}/resource_id=1,walltime=2:0:0"
    types =
    dependencies =
    assigned_resources = 51
    assigned_hostnames = vm
    queue = default
    command = /tmp/USER_DATA/FactoryContainer/images/4825/oar/ISO_4825.sh
    launchingDirectory = /tmp/USER_DATA/FactoryContainer/images/4825/oar
    jobType = PASSIVE
    properties = (nature=6) AND desktop_computing = 'NO'
    reservation = None
    walltime = 2:0:0
    submissionTime = 2012-05-01 18:53:32
    startTime = 2012-05-01 18:53:33
    cpuset_name = glassfish_5725
    initial_request = oarsub -d /tmp/USER_DATA/FactoryContainer/images/4825/oar -E oar_image_job4825.stderr -O oar_image_job4825.stdout -n 4825 --project null --checkpoint=1 --signal=15 -p nature=6 /tmp/USER_DATA/FactoryContainer/images/4825/oar/ISO_4825.sh
    message = FIFO scheduling OK
    scheduledStart = 2012-05-01 18:53:33
    resubmit_job_id = 0
    events =

Viewing the Logs of a Job

The main logs of OAR are stored in: /var/log/oar.log Each job launched on the OAR cluster, whether it be an image generation or publish to a cloud, logs are stored for the job. These include all the traces and error information during the execution of the job. Each job has a unique ID provided to it, which can be recuperated using the oarstat command as shown in Viewing Current Jobs in the Scheduler.

For jobs that generate an image, the log files are stored under: cd <user data mount point>/FactoryContainer/images/<job_name>

For example:

$ oarstat
Job id     Name           User           Submission Date     S Queue
---------- -------------- -------------- ------------------- - ----------
5725       4825           glassfish      2012-05-01 18:53:32 R default

The directory will be:

$ cd <user data mount point>/FactoryContainer/images/4825

Logs for jobs that publish an image to a specific cloud are stored in a sub-directory of the generated image directory. So for example if a user generates an Amazon image, then publishes the machine image to Amazon, the directory structure created is:

/<user data mount point>/FactoryContainer/images
                |
                |
                |-- generated image logs dir --> 4825
                                              |
                                              |
                                              |-- published image logs dir --> publish_<job_name>

To view the logs of a job, log in to the oar scheduler node as root:

$ cd /<user data mount point>/FactoryContainer/images/<job_name>/oar

so for example

$ cd /tmp/USER_DATA/FactoryContainer/images/4825/oar
$ ls -al
total 376
drwxr-xr-x 2 glassfish glassfish   4096 Apr 30 18:21 .
drwxr-xr-x 6 glassfish glassfish   4096 Apr 30 18:22 ..
-rwxr-xr-x 1 glassfish glassfish    980 Apr 30 18:15 ISO_4825.sh
-rwxr-xr-x 1 glassfish glassfish   1088 Apr 30 18:15 cmd_4825.sh
-rwxrwxrwx 1 glassfish glassfish    300 Apr 30 18:18 oar_image_job4825.stderr
-rwxrwxrwx 1 glassfish glassfish 360500 Apr 30 18:21 oar_image_job4825.stdout

To check for suspicious jobs you can run:

$ oarnodes | grep -i suspected