Dynamic aperture
Prerequisites
Dynamic aperture studies are managed using the SixDesk environment. It can be downloaded to local from github, but unless you plan to develop SixDesk functionalities we recommend using the version on afs via lxplus.
Step 1: Prepare Study
Environmental variables
Set up by setting following environmental variable:
export SixDeskTools=/afs/cern.ch/project/sixtrack/SixDesk_utilities/pro/utilities/bash
The environmental variable $SixDeskTools represents a link to all SixDesk scripts which might be used for generating the simulation input, submitting the simulation and retreiving the results. The SixDesk environment is hence available without need of downloading the SixDesk codes to the local user space.
Set up SixDesk environment
Let us create a new environment in a given directory. For example, let's say we want to have input and output in our private afs work directory. For team member John Doe with CERN user account name jdoe, this directory can be reached in lxplus via
cd /afs/cern.ch/work/j/jdoe/private/
da_studies:
mkdir `da_studies`
cd da_studies
We can now create the SixDesk environment using the following command
$SixDeskTools/set_env.sh -N scratch/workspace01
scratch space with one work space named workspace01. We can use several workspaces, for example if we want to study b1/b2 or inj/col optics and so on. Now we can navigate to the directory in which we have to adjust the simulation input for our needs:
cd workspace01/sixjobs
In this directory we will find
-
The
sixdeskenvfile, which specifies the.maskfile to be used, the machine state (injection or collision), the seeds to be used for the simulation, and many more (see in the Appendix), -
The
sysenvfile which provides environmental variables, for example the path to MAD-X, -
The directory to store the
.maskfile with the MAD-X input.
The business of creating a mask file is a complex topic. Ideally, a mask file for the machine and optics to be studied is already available and can be re-used. Let's assume we have a mask file with the name LHC_ats_2018_qp16.mask in the mask directory. The paramter LHCDescrip in the sixdeskenv must be set to LHC_ats_2018_qp16 in this case:
# The name of the "study" .le. 16 characters ([a-z A-Z 0-9 + - _ . ,]
# It must NOT contain a "%" (replacing "#") nor "_""_"
# It now includes the Version info e.g. v64lhc
# (If running mad6t you must have a $LHCDescrip.mask file in the subdirectory mask)
export LHCDescrip=LHC_ats_2018_qp16
sixdeskenv, .mask and sysenv files are ready, we can load the study:
$SixDeskTools/set_env.sh -s
Wed 10/03/21 15:20:02 set_env.sh STUDY [LHC_ats_2018_qp16]
Wed 10/03/21 15:20:02 set_env.sh WSPACE [wTest]
Wed 10/03/21 15:20:02 set_env.sh PLATFORM [boinc]
Wed 10/03/21 15:20:02 set_env.sh HOSTNAME [lxplus7116.cern.ch]
Wed 10/03/21 15:20:02 set_env.sh no BNL flag
Wed 10/03/21 15:20:02 set_env.sh APPNAME/EXE [sixtrack - /afs/cern.ch/project/sixtrack/build/50205/sixtrack - 50205]
The platform in this case is BOINC (which is set in the sixdeskenv file), we could select HTCondor alternatively.
Run MAD-X to generate the SixTrack input files
With this setup we are ready to run MAD-X to generate the optics input (fort.2) for the DA simulation. We can simply run the script mad6t.sh to submit the 60 MAD-X runs to HTCondor where they will be executed in parallel:
$SixDeskTools/mad6t.sh -s
And we receive the following output:
Wed 10/03/21 15:25:26 mad6t.sh Using madx Version madx in /afs/cern.ch/user/m/mad/bin
Wed 10/03/21 15:25:26 mad6t.sh STUDY LHC_ats_2018_qp16
Wed 10/03/21 15:25:26 mad6t.sh RUNTYPE inj
Wed 10/03/21 15:25:26 mad6t.sh SEEDS [1:60]
Wed 10/03/21 15:25:26 mad6t.sh PLATFORM htcondor
Wed 10/03/21 15:25:26 mad6t.sh QUEUE microcentury
Wed 10/03/21 15:25:27 mad6t.sh Using junktmp: /afs/cern.ch/work/p/pahermes/private/006_DA_simulations/sixtrack_input/wTest/ats2018_b1_QP_16_IOCT_65_56D50_4MVRF/mad.mad6t.sh.lxplus7116.cern.ch.2dwB4Y
Submitting job(s)............................................................
60 job(s) submitted to cluster 4833133.
Side note: If you are a new joiner in the team, you are likely to receive the following warning:
/WARNING: QMGMT:0:You have no Accounting Group membership, assigning to
low quota default.
LHC-COLL-LSF-users e-group, such that you benefit from the higher quota available for members of the team.
We can now check the status of our MAD-X runs, by typing
condor_q
OWNER BATCH_NAME SUBMITTED DONE RUN IDLE TOTAL JOB_IDS
jdoe mad/wTest/LHC_ats_2018_qp16 3/10 15:25 _ 59 1 60 4833133.0-59
Total for query: 60 jobs; 0 completed, 0 removed, 1 idle, 59 running, 0 held, 0 suspended
Depending on the traffic on the batch system, this set of simulations is completed rather quickly. We can check if all of our jobs were correctly submitted:
$SixDeskTools/mad6t.sh -c
There are now two possible outcomes:
- Not all jobs were correctly finished. We will receive the following screen output:
Wed 10/03/21 15:39:28 mad6t.sh Checking that a fort.2_??.gz exists for each MADX seed requested...
Wed 10/03/21 15:39:29 mad6t.sh ...discrepancy!!! Found 59 fort.2_??.gz (expected 60)
Wed 10/03/21 15:39:29 mad6t.sh Checking that a fort.8_??.gz exists for each MADX seed requested...
Wed 10/03/21 15:39:31 mad6t.sh ...discrepancy!!! Found 59 fort.8_??.gz (expected 60)
Wed 10/03/21 15:39:31 mad6t.sh Checking that a fort.16_??.gz exists for each MADX seed requested...
Wed 10/03/21 15:39:32 mad6t.sh ...discrepancy!!! Found 59 fort.16_??.gz (expected 60)
Wed 10/03/21 15:39:32 mad6t.sh generating list of missing MADX seed in mad.mad6t.sh.lxplus7116.cern.ch.2dwB4Y/jobs.list
Wed 10/03/21 15:39:32 mad6t.sh all mother files are there
Wed 10/03/21 15:39:32 mad6t.sh Problems with MADX runs! - error: 3
condor_q), we need to re-submit the missing seeds. The mad6t.sh script provides a very simple way of doing this:
$SixDeskTools/mad6t.sh -w
-w (for wrong seeds) will take care of it.
- All jobs were correctly finished. The output of the
$SixDeskTools/mad6t.sh -ccommand is as follows:
Wed 10/03/21 15:43:46 mad6t.sh Checking that a fort.2_??.gz exists for each MADX seed requested...
Wed 10/03/21 15:43:47 mad6t.sh ...found 60 fort.2_??.gz (as expected)
Wed 10/03/21 15:43:48 mad6t.sh average dimension (uncompressed): 415.739 kB - sigma: 0 kB
Wed 10/03/21 15:43:49 mad6t.sh Checking that a fort.8_??.gz exists for each MADX seed requested...
Wed 10/03/21 15:43:50 mad6t.sh ...found 60 fort.8_??.gz (as expected)
Wed 10/03/21 15:43:51 mad6t.sh average dimension (uncompressed): 5.5 kB - sigma: 0 kB
Wed 10/03/21 15:43:52 mad6t.sh Checking that a fort.16_??.gz exists for each MADX seed requested...
Wed 10/03/21 15:43:53 mad6t.sh ...found 60 fort.16_??.gz (as expected)
Wed 10/03/21 15:43:59 mad6t.sh average dimension (uncompressed): 4267.72 kB - sigma: 0 kB
Wed 10/03/21 15:44:00 mad6t.sh all mother files are there
Wed 10/03/21 15:44:00 mad6t.sh All the mad6t jobs appear to have completed successfully using madx -X Version madx in /afs/cern.ch/user/m/mad/bin
Now we are ready to go and can submit our DA simulation.
Step 2: Submit DA simulation to BOINC
We can now submit our simulation to the BOINC platform. This is achieved by using the script run_six.sh as follows:
$SixDeskTools/run_six.sh -a
Depending on the number of amplitudes, seeds and angles used in the simulation, this step can take quite a moment. If all jobs were correctly submitted, your submission should conclude with an output similar to this:
Wed 10/03/21 17:12:46 run_six.sh done.
Wed 10/03/21 17:12:46 run_six.sh GENERATED 2640 directories
Wed 10/03/21 17:12:46 run_six.sh CHECKED 2640 directories
Wed 10/03/21 17:12:46 run_six.sh SUBMITTED 2640 jobs
Wed 10/03/21 17:12:46 run_six.sh CONSIDERED 2640 jobs
Now we have to wait. If submitting to BOINC, it can take several days until the results are ready. If you have submitted to HTCondor (only recommended for small studies) it should be done within a couple of hours / a few days.
Step 3: Check the progress
While waiting for the results you can check the progress of your simulation by using the command
$SixDeskTools/run_status
Wed 10/03/21 17:55:14 run_status Backup status 2640 cases, 964 complete (1676 incomplete)
Step 4: Get the results
We can download the results from the BOINC spool server, using the command:
$SixDeskTools/run_results
If we run it early after the submission, no jobs have been processed yet, the result will look as follows:
================================================================================
Wed 10/03/21 17:19:15 run_results No zipped results to process
================================================================================
Wed 10/03/21 17:19:15 run_results Before this run of run_results: 2640 total cases, 0 complete (2640 incomplete)
Wed 10/03/21 17:19:15 run_results There were no results available
Wed 10/03/21 17:19:15 run_results Present status of regular DB: 2640 cases, 0 complete (2640 incomplete)
Wed 10/03/21 17:19:15 run_results Present status of platform DB: 2640 cases, 0 complete (2640 incomplete)
Step 5: Submit missing cases
If you run the simulation on BOINC, not all tracking simulations will be successful. We will instead have some missing cases. In order to get the simulation results for these cases, we don't carry out another submission to BOINC, this would take too long. We can selectively submit them to the batch system using run_six.sh with the option for incomplete cases -i:
$SixDeskTools/run_six.sh -i
Appendix A: sixdeskenv file
Parameters to be set in the sixdeskenv file
| name | Possible value | Description |
|---|---|---|
LHCDescrip |
any | Name of the study (must also be used for the .mask file) |
pmass |
any | Particle mass in MeV/c^2. Default for protons is 938.272013 |
platform |
HTCondor / BOINC |
Platform on which the simulation shall be conducted (HTCondor or BOINC) |
istamad |
1 |
Starting seed for MAD-X calculation |
iendmad |
60 |
End seed for MAD-X calculation |
runtype |
col / inj |
Run type (collision or injection) |
beam |
"" / b1 / b2 |
The beam can be set to "" or b1 or B1 for beam1, or to b2 or B2 for beam2 |
emit |
for example 2.5 |
Normalized emittance in \mum rad |
Team Contacts
Frederik, Malte, Pascal