Skip to content

SixTrack-FLUKA Coupling (Protons)

Applications

  • Loss map simulation (collimation cleaning inefficiency)

  • Input for energy deposition studies

Detailed Manual

A. Mereghetti: Hands-on Coupling Manual

Simplified Manual

Step 1: Prepare environmental variables [only once]

First we have to load the environmental variables. We can do this using the following code below. To load the environment for compiling Fluka: 

source /cvmfs/sft.cern.ch/lcg/contrib/gcc/9.2.0/x86_64-centos7/setup.sh

The environment for compiling SixTrack can be loaded using 

# cmake for SixTrack
export PATH=/cvmfs/sft.cern.ch/lcg/contrib/CMake/3.14.2/Linux-x86_64/bin:$PATH
export CMAKE_ROOT=/cvmfs/sft.cern.ch/lcg/contrib/CMake/3.14.2/Linux-x86_64
The path to the FLUKA installation ${FLUKA} and the subpath ${FLUSPC} to the FLUKA afs project space are loaded using the following. We decide to use a local version of fluka that we have downloaded:

export FLUSPC=/afs/cern.ch/project/fluka **Deprecated**
export FLUSPC=/eos/project-f/flukafiles/fluka-coupling
export FLUKA=/afs/cern.ch/work/p/pahermes/private/tools/fluka4-1.1

We can also add useful paths e.g. to flair (a FLUKA GUI, see here), pt (periodic table with Tkinter), fm (fluka manual) 

export PATH=${FLUSPC}:${FLUKA}/bin:${FLUSPC}/flair:$PATH

To access the fluka_coupling repository and the fedb_coupling repository (FLUKA element database), we run:

export CPLPATH=/afs/cern.ch/work/p/pahermes/private/tools/fluka_coupling

export FEDBPATH=/afs/cern.ch/work/p/pahermes/private/tools/fedb_coupling

Step 2: Prepare the input files

Setting up the directory structure

Before we start the study, we create a new directory in the afs work folder in which we prepare and run the simulation: 

cd /afs/cern.ch/work/p/pahermes/private/

mkdir coupling_simulation_LHC_runII

We assume that the relevant input files are already available: 

allapert.b1
CollDB_RunII.data
CollPositionsRunII.b1.dat
fort.2
fort.3
SurveyWithCrossing_XP_lowb.dat

For simulations that were already carried out by the team, we can find them in the gitlab repository:

https://gitlab.cern.ch/collteam/coupling_simulation_material

For new setups we need to create our own input files, according to our needs. Let's now create the folder raw_input in which we store the raw input (equivalent to the nominal SixTrack input without coupling):

cd coupling_simulation_LHC_runII
mkdir raw_input
Now we need to copy the relevant input files into clean input. Let's assume that before we have cloned the coupling_simulation_material repository into our afs folder, for example into 
/afs/cern.ch/work/p/pahermes/private/simulation_material/coupling_simulation_material
If, for example, we are interested in simulating Beam 1 in the RunII configuration of 2018 with \beta^*=30 cm, a crossing angle of 160\murad, we can find the relevant input here:

/afs/cern.ch/work/p/pahermes/private/simulation_material/sixtrack_simulation_material/Simulation_scenarios/LHC/RunII/6p5TeV_2018_30cm_160urad/B1_cleaning

We can simply copy over everything from that directory into our raw_input directory:

cp -r /afs/cern.ch/work/p/pahermes/private/simulation_material/sixtrack_simulation_material/Simulation_scenarios/LHC/RunII/6p5TeV_2018_30cm_160urad/B1_cleaning/* .
To keep a clear directory structure we are now going to create a new folder that we call preprocessed_input:

cd ..
mkdir preprocessed_input
cd preprocessed_input

That's were we are going to create our new input files for the coupling. In order to have the input files for the preprocessing easily available we are going to link them into this folder:

ln -s ../raw_input/* .
For simplicity let's also link the preprocessing folder of the fluka_coupling repository into this directory: 

ln -s /afs/cern.ch/work/p/pahermes/private/tools/fluka_coupling/tools/preprocess .

Finally, we need to have access to a prototypes file, which is a list of the different FLUKA prototypes that might be included in the simulation. We have a template available in the fluka_coupling repository. We can also link it to our local preprocessed_input:

ln -s /afs/cern.ch/work/p/pahermes/private/tools/fluka_coupling/tools/template_inputs/prototypes.lbp .

Now we have the symbolic link to all relevant inputs and preprocessing scripts in this folder and we can start preparing the actual input for the simulation.

Create FLUKA input

The first step is to produce the FLUKA input. The python script FLUKA_builder.py in the preprocessing directory will take care of it. We have two options:

  • Based on the CollDB.data file (note: only the old format of this file is compatible)
  • Based on the collgaps.dat file (requires a previous run with non-coupling SixTrack)

If we decide to use the CollDB file, we use the FLUKA builder script as follows:

python2.7 preprocess/FLUKA_builder.py --CollDB CollDB_RunII.data --EMIn 3.5E-06 --pc 6500 --TFile MADX/twiss_ats_30cm_thin.b1.tfs

The description of the individual output files (copied 1:1 from the Hands-on-Manual by A. Mereghetti) is as follows: - lhc_coupling.inp: the actual FLUKA input file. - lhc_coupling.insertion.txt: the list of injection/extraction roto-translations to transform the local reference system of SixTrack to the global reference system of FLUKA. It is a mandatory file for the coupling to run. - lhc_coupling.fort3.list: FLUKA block for the fort.3 SixTrack file. It contains the list of collimators, the name of the initial and final single elements in the accelerator lattice, a unique index labelling the FLUKA insertion region, and the synchronous length. The synchronous length is the length travelled by the synchronous particle; in the case of the collimators, it corresponds to the distance between the injection and extraction points, including the collimator tank, so it might be different from the jaw length. - lhc_coupling.log: log file of FLUKA_builder.py. - new collgaps.dat: generated by FLUKA_builder.py only in case a CollDB is used as input.

The file lhc_coupling.insertion.txt must be re-named to insertion.txt:

mv lhc_coupling.insertion.txt insertion.txt

Customize your FLUKA input

The file lhc_coupling.inp does contain include statements referring to template files for the different FLUKA geometries and physics inputs. In general this content must be adjusted to reflect the specific needs of the user. In order to 

cp /afs/cern.ch/work/p/pahermes/private/tools/fluka_coupling/tools/template_inputs/include_templates/*.inp .

We can now adjust the individual input files (with extension .inp) according to our needs. We integrate the physics options into our main FLUKA input file using the expand.sh script in the fedb_coupling repository:

/afs/cern.ch/work/p/pahermes/private/tools/fedb_coupling/tools/expand.sh lhc_coupling.inp

Indicate collimators to produce touches and first impacts

The SixTrack-FLUKA coupling can be used to produce the input for subsequent detailed shower propagation and energy deposition studies in FLUKA. The required input file touches.dat is created for a specified set of collimators, and contains the coordinates where the particles enter the physical material of the collimator. For example, if we want to perform such a simulation for IR7 and we have to identify the ID of all collimators in IR7. We can find this information in the collgaps.dat file or in the CollPositions file. Assume that the collimators with the IDs between 27 and 45 are those of IR7, we create a new file relcol.dat with the following content: 

19
27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45

Where the number 19 in the first row specifies the total number of collimators to take into account for the touches file. The second row contains the IDs of these 19 collimators.

Note that only the first touch per particle per turn will be recorded, to avoid double counting. If one is interested in e.g. the TCTs, the IR7 collimators should be removed from the relcol file, or there will be very few recorded touches in the TCTs.

Prepare SixTrack input

Online aperture check

The online aperture check to be used with the coupling must be prepared by adjusting the fort.2 file and creating new fort3.limi file. In a first step, we include dedicated markers at the extraction/injection locations of the collimators. These are the locations at which the particles are sent from the tracking (SixTrack) to the particle-matter interaction simulation (FLUKA) and vice versa. To include the markers into the fort.2 file, we use the following command:

python2.7 /afs/cern.ch/work/p/pahermes/private/tools/fluka_coupling/tools/preprocess/fort2_markers.py fort.2 lhc_coupling.fort3.list
The execution of the script should be completed within a few seconds. The first argument fort.2 is the standard file for SixTrack simulations. The file lhc_coupling.fort3.list is an output of the FLUKA_builder.py script. The file created in this step is called fort_w_markers.2 and it is of the same structure as the standard fort.2 file with the injection/extraction markers included.

Now we use this file to generate the final fort.2 for the coupling also with aperture markers and the fort3.limi file with aperture definitions which are used by SixTrack to check if particles are outside of the available aperture. These two input files are generated as follows:

python2.7 /afs/cern.ch/work/p/pahermes/private/tools/fluka_coupling/tools/preprocess/generate_sixtrack_apertures.py fort_w_markers.2 allapert.b1 SurveyWithCrossing_XP_lowb.dat

Note: this step is not necessary if one has already generated the fort3.limi file directly from MAD-X. In that case, simply rename the fort_w_markers.2 into fort.2, and use it together with the MAD-X generated fort3.limi.

Create final fort.3

The fort.3 file must be adjusted to incorporate new blocks. The new FLUKA input block is the content of the file lhc coupling.fort3.list:

tcl.4r1.b1           ecl.4r1.b1              1    1.48200 
tcl.5r1.b1           ecl.5r1.b1              2    1.48200 

...

tctpv.4l8.b1         ectpv.4l8.b1           47    1.48200 
tctph.4l1.b1         ectph.4l1.b1           48    1.48200 
tctpv.4l1.b1         ectpv.4l1.b1           49    1.48200

For the online aperture check we do need to include the LIMI block into the fort.3:

LIMI 
LOAD fort3.limi
NEXT

If a distribution shall be loaded via the DIST block, we also have to include the following:

DIST
READ "initial.dat"
NEXT
and the second input line in the tracking block has to be adjusted such that the third and the fifth attribute are set to one. For example:

TRAC
10 0 150 0.0 0.0 0 1
1 1 1 0 1
0 0 100000 100000 100000 100000 1
NEXT
Finally, the SYNC block contains the mass of the reference particle, which must be adjusted to match with the reference mass in FLUKA (for protons 938.27231):

SYNC
35640 .0003225 16 0 26658.883200 938.27231 1
1 1
NEXT

Set initial distribution parameters

If the initial distribution should be created using gpdist, we have to adjust the configuration file to reflect our needs. A template of the configuration file can be found in the fluka_coupling repository tools/template_inputs/config_gpdist.txt.

A detailed description of the configuration is given in [A. Mereghetti: Hands-on Coupling Manual]. Most notably, the user must specify:

  • The betatron functions at the location at which the distribution is initialized (by default IP1)
  • The emittance to be used
  • The type of halo to be sampled
  • The reference orbit at the location at which the distribution is initialized.

Run the simulation

Create directory with simulation input

We can now generate a directory with the simulation input. For example, we can call it coupling_input:

mkdir ../coupling_input
cp fort.2 fort.3 lhc_coupling_exp.inp insertion.txt config_gpdist.txt fort3.limi relcol.dat ../coupling_input

Submit to batch system

We prepare the run to be executed in the subdirectory run via 

./control.sh -p -d run -i clean_input
where we indicate that clean_input contains the input for the simulation. Now the simulation input is ready and we can submit to HTCondor:

./control.sh -s -d run
The screen output looks as follows:

   echo of input:
   - dirName=run
   - dirNameCompare=
   - inputDir=.
   - iMin=1
   - iMax=5
   - clusterID=
   - lPrepare=false
   - lCompare=false
   - lSubmit=true
   - lRetrieve=false
   - lRemove=false
   - lRelease=
   - nMaxRetrial=10

command: condor_submit -spool -batch-name run htcondor.sub - run at Wed Mar 17 14:36:23 CET 2021 - trial 1
Submitting job(s).....
5 job(s) submitted to cluster 4903758.
...done.

We will need the cluster ID to retrieve back the simulation output. We can also find out the cluster ID if we run condor_q:

-- Schedd: bigbird13.cern.ch : 
OWNER    BATCH_NAME    SUBMITTED   DONE   RUN    IDLE  TOTAL JOB_IDS
pahermes run          3/17 14:36      _      _      5      5 4903758.0-4

If the jobs are done, we can retreive the result using the following command:

./control.sh -r -R -C 4903758

Literature

E. Skordis and V. Vlachoudis and R. Bruce and F. Cerutti and A. Ferrari and A. Lechner and A. Mereghetti and P.G. Ortega and S. Redaelli and D. Sinuela Pastor, FLUKA coupling to Sixtrack, CERN-2018-011-CP, Proceedings of the ICFA Mini-Workshop on Tracking for Collimation, CERN, Geneva, Switzerland, 2018. Weblink.

Team Contacts

Andrey, Roderik, James, Laurie, Pascal