Running Ansys Electronics Desktop 2022 R1 using an LSF Job Scheduler (command line; without GUI).
More information about the Load Sharing Facility (LSF) job scheduler is provided here.
CMC’s CAD Compute Cluster nodes can be displayed by typing lshosts at a command prompt in the environment as shown in Figure 1. You will see four login nodes (uwlogin*), two management nodes (uwlsfm*; 8 cores and 63 GB each), and eight compute nodes in the list (uwmhpc*; 32 cores and hundreds of GB).
Figure 1: The Nodes in the CAD Compute Cluster
When you run a simulation on the Cluster, you submit your simulation request to an LSF queue. This step is common to all CAD software installed on CMC’s Cluster – Ansys, COMSOL, Xilinx, etc. The job scheduler will try to assign the hardware (CPU cores and RAM) you want to your simulation run, up to a limit of 32 cores. This limit has been set by CMC IT staff, to prevent abuse of Cluster privileges.
Figure 2: A Block Diagram of Cluster Operation
Use the bqueues command to discover queue names. The letter ‘b’ in the command bqueues means that these queues run simulations in batches, or batch mode. Any command starting with ‘b’ also means that LSF will recognize it as one it should perform.
Figure 3: The Queues in the CAD Compute Cluster
Summary: It is possible to run Ansys Electronics Desktop (EDT) software by creating a script containing all the commands that are to be given to Ansys, and then submitting this script to the LSF job scheduler for completion.
Here are the steps that have been tested in the Cluster to show how this is done.
For these instructions, a prepared project supplied by Ansys support staff was used (a bandpass filter, with filename bp_filter.aedt). You will find it in the list of examples included with your local Ansys EDT installation.
- Set the number of cores to a maximum of 32.
Note that you will have to define the use of Ansys HPC pool licenses in a registry.txt file in the batchoptions segment of the script. See Appendix A below for an example.
- Log into the Cluster using one of the VCAD Cloud instances called CAD Compute Cluster.
- Enter module load ansys/22.1.cadconnect in your terminal window, then type module list to see if the software has been loaded into memory.
- Use the mpitest command to verify message-passing interface (MPI) software function.
If you want to know more about the role MPI plays in simulation, your Ansys Help files have this information.
Figure 4: Summary of Set-up Commands to Run Ansys 2022 R1
- Use CMC’s setsshkey utility under /home/scripts to generate an SSH passcode.
You only have to do this once, when you first set up your workspace on the Cluster. For simplicity, do not enter a password and leave the result in the default location. This is usually your home directory on a Cluster login node.
Figure 5: Generate an ssh Passkey for Each Node Using the Script
- Create a shell file to enter LSF batch submission (bsub) and the Ansys batch commands.
See the Appendix for an example (“my_ansys_hfss.sh”).
- Upload your project file(s) to your working directory. Instructions are provided for using MobaXterm to perform a secure file transfer (sftp) upload on the Cluster pages.
- Upload your shell file (e.g. “my_ansys_hfss.sh”) to same directory.
Ensure that the body of your shell file does not contain extraneous Windows editor characters (e.g. a carriage return).
- Use the command bsub < my_ansys_hfss.sh at a prompt to run a simulation.
During simulation runs, use commands such as bjobs -l and bhosts to monitor your simulation progress. For example:
~$ bjobs -l
Any text after a single # is visible to the LSF Job Scheduler and will be read as a possible command by LSF.
Any command after ## is treated as a comment by both LSF and Ansys.
If using this as an example, insert your assigned CMC Cluster user name in place of *user_name* below.
## embedded options to bsub – start with #BSUB
## — Name of the job —
#BSUB -J ansys_HFSS_example
## — specify queue from the bqueues list —
#BSUB -q adept
# — specify the number of processors —
#BSUB -n 32
## — Specify the output and error files. %J is the job ID —
## — -o and -e mean append, -oo and-eo mean overwrite —
#BSUB -oo BPFilter_%J.out
#BSUB -eo BPFilter_%J.err
## — example of ansys command line call —
/CMC/tools/ansys/ansys.2022r1/AnsysEM22.1/v221/Linux64/ansysedt -distributed -machinelist numcores=32 -auto NumDistributedVariations=1 -monitor -ng -batchoptions registry.txt -batchsolve /home/*user_name*/Ansoft/bp_filter.aedt
The registry.txt file included above contains a direct reference to the anshpc license feature ‘pool’. Use of a registry file is described in the ANSYS 2022 R1 help files.
Note also that there is a “/scratch” directory variable in this file. With each iteration of a solution, Ansys creates temporary files where it stores data after it finishes one set of calculations. This data becomes the starting points for the next round of calculations, and so on, until a solution is reached. These files are overwritten by fresh data each time, and they can be quite large/numerous, so a scratch disk is provided on the Cluster for this activity.
Here is a copy of the file contents.
## — include other desired registry entries here —