- Infos im HLRS Wiki sind nicht rechtsverbindlich und ohne Gewähr -
- Information contained in the HLRS Wiki is not legally binding and HLRS is not responsible for any damages that might result from its use -

CAE utilities: Difference between revisions

From HLRS Platforms
Jump to navigationJump to search
m (link to)
m (update of help (-h) output)
Line 40: Line 40:
  qgen -h abaqus
  qgen -h abaqus


  qgen    a tool to generate(&submit) pbs jobfiles        (author: Martin Bernreuther <bernreuther@hlrs.de>)
  qgen    a tool to generate(&submit) pbs jobfiles        (version: 2012/03/29, author: Martin Bernreuther <bernreuther@hlrs.de>)
  usage: qgen [-n|--nodes <nodes>] [-t|--walltime <walltime>] [-N|--jobname <jobname>] [-w|--workdir <workdir>] [-o <jobfile>] [--submit|--submitq <queue>|--execute] [-] [<template>] [<template_opt1> <template_opt2>...]
  usage: qgen [-n|--nodes <nodes>] [-t|--walltime <walltime>] [-N|--jobname <jobname>] [-w|--workdir <workdir>] [-o <jobfile>] [--submit|--submitq <queue>|--execute] [-] [<template>] [<template_opt1> <template_opt2>...]
       -n|--nodes <nodes>      [default: 1]   number of nodes
       -n|--nodes <nodes>      [default: 1:nehalem:ppn=8]     number of nodes($QGEN_NODES)
       -t|--walltime <walltime> [default: 12:00:00]      walltime
       -t|--walltime <walltime> [default: 24:00:00]      walltime ($QGEN_WALLTIME)
       -N|--jobname <jobname>  [default: <template><datetime>] job name
       -N|--jobname <jobname>  [default: <template><datetime>] job name ($QGEN_NAME)
       -w|--workdir <workdir>  [default: /scratch2/ws/hpcbern-dynatest-0]       working directory
       -w|--workdir <workdir>  [default: /zhome/academic/HLRS/hlrs/hpcbern]     working directory ($QGEN_WORKDIR)
       -o <jobfile>    [default: <stdout>]      jobfile output
       -o <jobfile>    [default: <stdout>]      save script to <jobfile>
       --submit        submit immediately (to standard queue)
       --submit        submit immediately (to standard queue)
       --submitq <queue>        submit immediately to queue
       --submitq <queue>        submit immediately to queue <queue>
       --execute        execute on current host immediately
       --execute        execute immediately on current host
       -l              list system templates
       -l              list system templates
      -v              be verbose
       -h|--help                print help and exit
       -h|--help                print help and exit
       <template>              template to use
       <template>              template to use
       <template_options>      additional options depending on the template
       <template_options>      additional options depending on and passed directly to the template ($*)
  qgen will create a pbs jobscript based on a template, substituting
  qgen will create a pbs jobscript based on a template, substituting
  $QGEN_NODES, $QGEN_WALLTIME, $QGEN_NAME, $QGEN_WORKDIR and $* (latter with template options)
  $QGEN_NODES, $QGEN_WALLTIME, $QGEN_NAME, $QGEN_WORKDIR and $* (latter with template options)
  Instead of setting the values with command line arguments,"
  Instead of setting the values with command line arguments,
        echo "environment variables might be used (except for template options)
environment variables might be used (except for template options)
  These environment variables might be also set in the configuration file
  These environment variables might be also set in the configuration file
  ~/.qgenrc, which is sourced at the beginning.
  /zhome/academic/HLRS/hlrs/hpcbern/.qgenrc, which is sourced at the beginning.
  Templates are searched for in $QGEN_TEMPLATE_PATH, actually set to:
  Templates are searched for in $QGEN_TEMPLATE_PATH, actually set to:
  ~/qgen_templates /app/rus/struct/bin/qgen_templates
  /zhome/academic/HLRS/hlrs/hpcbern/qgen_templates /opt/cae/bin/qgen_templates
   
   
  abaqus template_options: options for abaqus, e.g. job=<jobname>
  abaqus template_options: [<version>] <abaqus-options>
                          cpus and mp_mode are set automatically
                examples: job=<jobname>
                          692 job=<jobname> inp=<inputfile> user=<usrprgfile> scratch=.
ABAQUS options:
                job=job-name
                [input=input-file]
                [user={source-file | object-file}]
                [oldjob=oldjob-name]
                [fil={append | new}]
                [globalmodel={results file-name | output database file-name}]
                [domains=number-of-domains]
                [dynamic_load_balancing]
                [standard_parallel={all | solver}]
                [gpus=number-of-gpgpus]
                [memory=memory-size]
                [interactive |  background | queue=[queue-name][after=time]]
                [double={explicit | both | off | constraint}]
                [scratch=scratch-dir]
                [output_precision={single | full} ]
                [field={odb | exodus | nemesis} ]
                [history={odb | csv} ]
                [madymo=MADYMO-input-file]
                [port=co-simulation port-number]
                [host=co-simulation hostname]
                [listenerport=Co-Simulation Engine listener port-number]
                [remoteconnections=Co-Simulation Engine host:port-number, remote job host:port-number]
                [timeout=co-simulation timeout value in seconds]
                [unconnected_regions={yes | no}]
(Nutzung dieses Templates in eigener Verantwortung - use this template on your own responsibility)


Submitting an ABAQUS job on 4 nodes is as easy as executing
Submitting an ABAQUS job on 4 nodes is as easy as executing
Line 138: Line 170:
  qcat -h
  qcat -h


  usage: /app/rus/struct/bin/qcat [o|e] [<JOBID>] [-c] [-t [<N>]]
  usage: /opt/cae/bin/qcat [o|e] [<JOBID>] [-c] [-t [<N>]]
        o|e      show output or error output file        default:o
      o|e      show output or error output file        default:o
        <JOBID>  OpenPBS job ID                          default: last own job
      <JOBID>  OpenPBS job ID                          default: last own job
        -c      copy file to final destination          default: no copy
      -c      copy file to final destination          default: no copy
        -t [<N>] show tail (only <N> last lines)        default: show all
      -t [<N>] show tail (only <N> last lines)        default: show all




Line 160: Line 192:
  ws_exchange -h
  ws_exchange -h


/app/rus/struct/bin/ws_exchange: option requires an argument -- h
  usage: ws_exchange [-s <secret subdirectory name>] [-d <workspace duration>] [-w <workspacename>] [-f <workspace filesystem>] [-p <permissions>]
  usage: ws_exchange [-s <secret subdirectory name>] [-d <workspace duration>] [-w <workspacename>] [-f <workspace filesystem>] [-p <permissions>]
         -s <secret subdirectory name> [default: random name]
         -s <secret subdirectory name> [default: random name]
         -d <workspace duration [d]>  [default: 0 days (24 hours)]
         -d <workspace duration [d]>  [default: 1 days (48 hours)]
         -w <workspacename>            [default: exchange%Y%m%d%H%M%S]
         -w <workspacename>            [default: exchange%Y%m%d%H%M%S]
         -f <workspace filesystem>    [default: default of ws_allocate]
         -f <workspace filesystem>    [default: default of ws_allocate]
         -p <permissions>             [default: go+rwx]
         -p <subdirectory permissions> [default: go+rwx]
  ws_exchange will create a workspace, which is executable but not readable for the world.
  ws_exchange will create a workspace, which is executable but not readable for the world.
  Its contents are hidden and a additional created subdirectory, which name serves as a password, could be open to the world.
  Its contents are hidden and a additional created subdirectory, which name serves as a password, could be open to the world.

Revision as of 11:04, 10 May 2016

A collection of small helper scripts.

module load cae

will you take care that these are found when called through the commandline or e.g. a job script.

qgen

qgen will generate a job script based on a template.

qgen -h

will print some help and

qgen -l

will list all available templates found in the QGEN_TEMPLATE_PATH One of these templates is "test", which doesn't need any further arguments.

qgen test

A job file will be printed. Save this job file to "test.pbs" and compare it with the template:

qgen -o test.pbs test
tkdiff test.pbs /app/rus/struct/bin/qgen_templates/test

(If there's no X avaiable, replace tkdiff with e.g. sdiff)

qgen just replaces some placeholders within the template. Everyone can expand the system writing own templates. The default path for these files is ~/qgen_templates

Now the job script could be submitted with "qsub test.pbs", but... (approx.) the same (for 2 requested nodes) can also be achieved with:

qgen -n 2 --submit test

However, omitting the -o option, the executed job script will not be saved for the user.

In general, templates require additional options. The help might give you a hint at the end, e.g.

qgen -h abaqus
qgen    a tool to generate(&submit) pbs jobfiles        (version: 2012/03/29, author: Martin Bernreuther <bernreuther@hlrs.de>)
usage: qgen [-n|--nodes <nodes>] [-t|--walltime <walltime>] [-N|--jobname <jobname>] [-w|--workdir <workdir>] [-o <jobfile>] [--submit|--submitq <queue>|--execute] [-] [<template>] [<template_opt1> <template_opt2>...]
      -n|--nodes <nodes>       [default: 1:nehalem:ppn=8]      number of nodes($QGEN_NODES)
      -t|--walltime <walltime> [default: 24:00:00]      walltime ($QGEN_WALLTIME)
      -N|--jobname <jobname>   [default: <template><datetime>] job name ($QGEN_NAME)
      -w|--workdir <workdir>   [default: /zhome/academic/HLRS/hlrs/hpcbern]     working directory ($QGEN_WORKDIR)
      -o <jobfile>     [default: <stdout>]      save script to <jobfile>
      --submit         submit immediately (to standard queue)
      --submitq <queue>        submit immediately to queue <queue>
      --execute        execute immediately on current host
      -l               list system templates
      -v               be verbose
      -h|--help                print help and exit
      <template>               template to use
      <template_options>       additional options depending on and passed directly to the template ($*)
qgen will create a pbs jobscript based on a template, substituting
$QGEN_NODES, $QGEN_WALLTIME, $QGEN_NAME, $QGEN_WORKDIR and $* (latter with template options)
Instead of setting the values with command line arguments,
environment variables might be used (except for template options)
These environment variables might be also set in the configuration file
/zhome/academic/HLRS/hlrs/hpcbern/.qgenrc, which is sourced at the beginning.
Templates are searched for in $QGEN_TEMPLATE_PATH, actually set to:
/zhome/academic/HLRS/hlrs/hpcbern/qgen_templates /opt/cae/bin/qgen_templates

abaqus template_options: [<version>] <abaqus-options>
                         cpus and mp_mode are set automatically
               examples: job=<jobname>
                         692 job=<jobname> inp=<inputfile> user=<usrprgfile> scratch=.

ABAQUS options:
                job=job-name
                [input=input-file]
                [user={source-file | object-file}]
                [oldjob=oldjob-name]
                [fil={append | new}]
                [globalmodel={results file-name | output database file-name}]
                [domains=number-of-domains]
                [dynamic_load_balancing]
                [standard_parallel={all | solver}]
                [gpus=number-of-gpgpus]
                [memory=memory-size]
                [interactive |  background | queue=[queue-name][after=time]]
                [double={explicit | both | off | constraint}]
                [scratch=scratch-dir]
                [output_precision={single | full} ]
                [field={odb | exodus | nemesis} ]
                [history={odb | csv} ]
                [madymo=MADYMO-input-file]
                [port=co-simulation port-number]
                [host=co-simulation hostname]
                [listenerport=Co-Simulation Engine listener port-number]
                [remoteconnections=Co-Simulation Engine host:port-number, remote job host:port-number]
                [timeout=co-simulation timeout value in seconds]
                [unconnected_regions={yes | no}]

(Nutzung dieses Templates in eigener Verantwortung - use this template on your own responsibility)

Submitting an ABAQUS job on 4 nodes is as easy as executing

qgen -n 4 --submit abaqus job=<jobname>

with <jobname> as ABAQUS inputfile (absolute path or relative path to the current directory). Typically qgen is executed within a writable directory and a workspace is used here. As mentioned before, more control is gained, if the PBS jobfile is generated first and submitted afterwards, like with this LS-Dyna example:

qgen -o lsdyna.pbs -n 8 dyna i=<inputfile>
qsub lsdyna.pbs

Before the submission with qsub, the jobfile might be changed/tuned.

The qgen command can also be used within LSOPT for LS-Dyna optimization/DoE runs. After loading the necessary modules (module load cae lstc) start "lsoptui" to do the configuration. Within the "Solvers" tab, after choosing "LS-DYNA" as "Solver Package Name", a "Command" like "qgen --submit -n 1:ppn=8 -o job.pbs dyna" will generate and submit the LS-Dyna jobs, controlled by lsopt.

There's also a simple template to execute a single command:

qgen -N cmdtoptest --submit cmd top -b -n 1

The -N option sets the jobname and affects the names of the stdout/stderr files. The output of the command "top -b -n 1" can thus be found in the cmdtoptest.o* file. The error output cmdtoptest.e* is hopefully empty...

qgen customization

The preset qgen templates can be augmented through user defined ones. qgen will check the directory ~/qgen_templates for templates by default and files in this directory will be preferred over the preset templates. There's also the environment variable $QGEN_TEMPLATE_PATH to define, which directories qgen should include, searching for the given template name.

To start with a first example, a simple helloworld template will be created (if you're not familar with vi and you have X enabled, use another editor like nedit,kwrite,...):

mkdir ${HOME}/qgen_templates
vi ${HOME}/qgen_templates/helloworld

The content of ~/qgen_templates/helloworld might look like

#!/bin/bash
#QGENHELP pbs jobscript file for testing/demonstration purposes
#QGENHELP helloworld template_options: <name>
#PBS -l nodes=$QGEN_NODES
#PBS -l walltime=$QGEN_WALLTIME
#PBS -N $QGEN_NAME
cd $QGEN_WORKDIR
echo -e "Hello world!\nThe current working directory is $PWD and my name is...\n$*"

The $QGEN_XXX strings will be replaced by qgen with some job specific data according to default values and the qgen options set.

Now we can use this template to submit a first job (the cae module has to be loaded only once - otherwise the full path to qgen has be typed...):

module load cae
qgen -h helloworld
qgen -o helloworld.pbs --submit helloworld Rumpelstilzchen

The help message will also display the QGENHELP part of the template at the end.

qcat

Whereas the qsub option "-k oe" causes the system to create a *.o* and *.e* file for the standard and error output in the home directory right from the start of a job run, the default behaviour is that these files are kept in a spool directory and moved at the end to the working directory. qcat will show these files using cat or copy the actual version to the working directory.

qcat -h
usage: /opt/cae/bin/qcat [o|e] [<JOBID>] [-c] [-t [<N>]]
      o|e      show output or error output file        default:o
      <JOBID>  OpenPBS job ID                          default: last own job
      -c       copy file to final destination          default: no copy
      -t [<N>] show tail (only <N> last lines)         default: show all


ws_exchange procedure

Exchanging data between users can be done using a workspace. If the users don't share a common group it is not advisable to just create such a workspace with permissions set to read (or/and write) and execute to world (others) since everyone on the systems gets access to the data. ws_exchange uses such a workspace with a slight modification: the workspace directory permissions are set to execute for all, but only the user is allowed to read its contents. A readable(/writable) subdirectory is therefore "invisble" to others. This approach is not secure, but better than the "open to all" approach.

ws_exchange

Looking at the help first:

ws_exchange -h
usage: ws_exchange [-s <secret subdirectory name>] [-d <workspace duration>] [-w <workspacename>] [-f <workspace filesystem>] [-p <permissions>]
       -s <secret subdirectory name> [default: random name]
       -d <workspace duration [d]>   [default: 1 days (48 hours)]
       -w <workspacename>            [default: exchange%Y%m%d%H%M%S]
       -f <workspace filesystem>     [default: default of ws_allocate]
       -p <subdirectory permissions> [default: go+rwx]
ws_exchange will create a workspace, which is executable but not readable for the world.
Its contents are hidden and a additional created subdirectory, which name serves as a password, could be open to the world.

Executing ws_exchange with default options will create a one-day workspace in the default file system with a name starting with "exchange" followed by the date and time and a subdirectory with random name and execute/read/write permissions:

ws_exchange
Workspace created on gerris
exchange20081018113236                                      Oct 18 11:32:51  0 days 23 hours
                 /scratch2/ws/hpcstruc-exchange20081018113236-0
 exchange directory ----------------------------------------
/scratch2/ws/hpcstruc-exchange20081018113236-0/Yj2mskvAuC6
------------------------------------------------------------
you might ws_cp2exchange to copy your files to exchange20081018113236

Concerned other user have to be notified about the path ("/scratch2/ws/hpcstruc-exchange20081018113236-0/Yj2mskvAuC6" in the example) e.g. by email. Another example:

ws_exchange -w myexchangews -s secretsubdir -d 30 -f lustre -p go+rx
Workspace created on gerris
myexchangews                                                Oct 18 12:43:19  0 days 23 hours
                 /scratch2/ws/hpcstruc-myexchangews-0
 exchange directory ----------------------------------------
/scratch2/ws/hpcstruc-myexchangews-0/kkE2WVIY
------------------------------------------------------------
you might ws_cp2exchange to copy your files to myexchangews

Now the workspace (on the lustre file system) named myexchangews lasts for 30 days.

ws_cp2exchange

Files also have permissions and especially copying files to a ws_exchange workspace these permissions are automatically set too restrictive. It's also a good idea first to change the current directory to the ws_exchange subdirectory and copy to this directory than in the other direction. To simplify this, "ws_cp2exchange" was created.

ws_cp2exchange -h
usage: ws_cp2exchange [cp|mv] [<options>] <src> <wsname>

To copy a file "testfile" to the ws_exchange workspace subdirectory created above simply write

ws_cp2exchange testfile exchange20081018113236
working directory: /scratch2/ws/hpcstruc-exchange20081018113236-0/Yj2mskvAuC6
cp  /DDN1/HLRS/hlrs/hpcstruc/testfile .
change permissions to g+rwx,o+rwx
done

Copying whole directories needs an option (see "man cp"):

ws_cp2exchange cp -r testdir1 myexchangews
working directory: /scratch2/ws/hpcstruc-myexchangews-0/kkE2WVIY
cp  -r /DDN1/HLRS/hlrs/hpcstruc/testdir1 .
change permissions to g+rwx,o+rwx
done

(The "cp" can be omitted, since copy is the default mode.)

To move a directory (or file) use the "mv" mode:

ws_cp2exchange mv testdir2 myexchangews
working directory: /scratch2/ws/hpcstruc-myexchangews-0/kkE2WVIY
mv  /DDN1/HLRS/hlrs/hpcstruc/testdir2 .
change permissions to g+rwx,o+rwx
done