Note: This document has been updated to demonstrate the use of ANSYS 2020 R2 on CMC Microsystems’ CAD Compute cluster.
Introduction to the Use of Ansys EDT in the CMC CAD Cluster
This is the main webpage describing the CAD Cluster Cluster on CMC Microsystems’ main website. Step-by-step instructions for logging into and using the CAD Computer Cluster can be found in this instruction document.
Special documents such as this one explain how to configure CAD tools for HPC/cluster simulations. In this one, we describe a way to configure an HFSS project. The instructions you see below have been developed for and run successfully in the CAD Compute Cluster. However, project settings must be tailored by a student for each simulation run, as all projects will be different.
Note that image captures and settings in this document are taken from ANSYS EDT 2020 R2. To navigate in the CAD Compute Cluster and find out which CAD tools reside there, use Load Share Facility (LSF) module commands described in Appendix A below. We also list some of them here.
Ensure that the projects you upload to the cluster are compatible with the installed CAD tool version(s) or contact CMC Microsystems’ staff to ask for advice.
Uploading an ANSYS EDT Simulation to the CAD Compute Cluster
The basic method for starting a cluster simulation from your local desktop is outlined in Figure 1. It is possible to use a Secure Copy Protocol (SCP) transfer of your project from your desktop to the CAD cluster using MobaXterm. Our step-by-step instructions show how this is done.
CentOS 7 LINUX is the underlying operating system in the CAD Compute Cluster. We advise those considering the use of this cluster to learn a few basic LINUX commands: To change directories; to list files in a directory; to find the present working directory; and so on. CentOS help is a useful CentOS resource. It is helpful to be able to use a LINUX command line text editor as well, for making quick changes to scripts in your working directory. Uploading a Windows-edited file to a LINUX environment can sometimes cause problems, as certain Windows inserts like carriage returns can be interpreted wrongly by LINUX.
Figure 1. The basic method for starting a cluster simulation.
Note that we enforce a queue system to submit simulation jobs to the cluster. The purpose of queuing is twofold: It prevents some ANSYS modules (EDT, MAPDL) from occupying an entire node while using less hardware, as they are currently designed to do; and it permits software from different vendors to run simulations on nodes while sharing the hardware.
Our default queue for users is the “adept” queue.
Figure 2. A list of queues configured in the CAD Compute Cluster.
An LSF Job Scheduler command starting with the letter ‘b’ means that the instruction is used for batch (command line) jobs. More useful LSF commands are listed here.
Setting up ANSYS EDT in an LSF Server for Batch Mode/Command Line Use
These instructions describe the steps to be taken after you have uploaded a project for simulation.
The CAD Compute Cluster nodes can be displayed by typing lshosts at a command prompt in the environment as shown in Figure 3. You will see four login nodes (uwlogin*), two management nodes (uwlsfm*) and eight compute nodes in the list (uwmhpc*).
Figure 3. CAD Compute Cluster environment for testing ANSYS simulations
When you log into the Cluster, your account will reside on a login node. You will upload your project and other required files to your account.
For this example, we use a prepared test file (a bandpass filter, with filename bp_filter.aedt) supplied by ANSYS support staff. It is from one of its standard tutorials and can be extended in simulation frequency so that multi-core capability can be studied. You will find it in the list of examples supplied with your local ANSYS EDT installation. Specific settings are shown in Appendix A at the end of these instructions.
- Extend the frequency simulation range to involve more cores and RAM. Set number of cores greater than four (4) to engage anshpc licences. Note that you will have to tell LSF to use the ANSYS pool licences for multi-core simulations in a registry.txt file in the batchoptions segment of the script. See Appendix A below for an example of its inclusion.
The utility setsshkey is used to circumvent password checking on uwmhpc nodes (SSH security). Be sure to run this utility before starting your first simulation. When running this utility, do not use a passphrase, and leave the result in the default directory. We discuss the importance of this SSH key in this document.
You now load Ansys software into your environment.
For Ansys 2020 R1:
~$ module load ansys/20.1.cadconnect
For Ansys 2020 R2:
~$ module load ansys/20.2.cadconnect
Use the following command to ensure you have loaded the software correctly:
~$ module list
For Ansys 2020 R2, you should see a reply from the LSF Job Scheduler that looks like the one in Figure 4. A module in the LSF environment corresponds to a CAD software program.
Figure 4. Check to see if your ANSYS 2020 R2 software has loaded correctly.
When you loaded ANSYS, you also loaded Message Passing Interface (MPI) software, as both Intel and IBM MPI programs are bundled with ANSYS EDT. This is another layer of software in the CAD Compute Cluster that plays a part in simulation runs. Use the following command to ensure that the MPI software is ready to use:
~$ mpitest (for the Intel version)
Figure 5. Test for the installation of the Intel Message Passing Interface (MPI) software
~$ mpitest –ibmmpi (for the IBM version)
Figure 6. Test for the installation of the IBM Message Passing Interface (MPI) software
At this point, you should have your *.aedt project file, your submission script in a *.sh file and any other data or registry files that your simulation might need for execution in your Cluster working directory (workdir). If you are not already in it, change your present working (pwd) directory so that you are in your workdir. At the prompt ~$ in a terminal window, enter the command:
~$ bsub < (script_name).sh
… to start your simulation run. The command bsub means “batch submission”. For example, if your script was called my_ansys_hfss.sh, you would enter:
~$ bsub < my_ansys_hfss.sh
Use the command bjobs to follow the progress of your submission. Use the extension bjobs –l (small letter L) for a detailed job description, including a running tally of CPU time.
General information about writing job scripts for LSF:
Appendix A: Creating an HPC Simulation: ANSYS EDT 2020 R2 Settings and Building a LINUX Script on the Cluster
This Appendix describes some recommended settings we would declare to create a batch command script.
1) On your computer, launch Electronics Desktop. For these instructions, we use a test file (e.g. bp_filter.aedt) supplied by ANSYS staff. It is from a tutorial and can be extended in simulation frequency so that multi-core capability can be exercised. Find and load this example.
Next, go to the menu Tools | Options |General Options… and set Product Licensing to “Use Electronics Pro, Premium, Enterprise product licensing” as shown in Figure 7.
Figure 7. Select “Use Electronics Pro, Premium, Enterprise product licensing”
2) Open Tools | Options | HPC and Analysis Options… | Configurations tab… | Add… Use this GUI to specify your simulation details. The following are examples, not recommendations.
Analysis Configuration: Machines Tab
Configuration name: CAD_cluster
Press the Add Machine to List button. In the window Machines for Distributed Analysis, put in your configuration.
RAM limit: 90%
Figure 8. The Analysis Configuration window
Click OK to save and close. Make the selection Active; this will happen automatically if it is the only selection.
Figure 9. The HPC and Analysis Option window showing the Configurations tab.
Click OK again to save and close.
3) Open Tools | Options |HPC and Analysis Options |Options tab. Use this GUI to specify the type of simulation to be run by the LSF job scheduler.
Figure 10. The HPC and Analysis Options window showing the Options tab.
HPC License: Pool
Options for Design Type: HFSS
Under Distributed Memory
MPI vendor: Intel
Remote Spawn Command: SSH
Under HPC Licensing
Enable GPU: False
Enable GPU for SBR+ Solve: False
Select OK to save these changes and close this window.
6) Save your project for transfer to the Cluster. Close your project. Reserve an SSH terminal in our vcad.cmc.ca menu. Transfer your *.aedt file to the Cluster using a method outlined in the Cluster documents in our Community pages.
The remainder of these instructions are handled in your home account on the Cluster.
7) To create a *.sh script to be run by the LSF Job Scheduler, use the structure outlined in the example at the end of these instructions as a template.
Load and check ANSYS and run an MPI test per the earlier steps.
At the prompt in a LINUX terminal window, enter the command:
~$ bsub < [script_name].sh
For example, if your script was called my_ansys_hfss.sh, you would enter:
~$ bsub < my_ansys_hfss.sh
Here is an example of a script. Any text after a single # is visible to the LSF Job Scheduler and will be read as a possible command by LSF. Any command after ## is treated as a comment by both LSF and ANSYS.
## embedded options to bsub – start with #BSUB
# — Name of the job —
#BSUB -J ansys_HFSS_example
# — specify queue —
#BSUB -q adept
# — specify wall clock time to run as hh:mm —
##BSUB -W 04:00
# — specify the number of processors —
#BSUB -n 48
# — specify the number of nodes —
##BSUB -R “span[hosts=]”
# — user e-mail address —
##BSUB – email@example.com or other address
# — mail notification —
# — at start —
# — at completion —
# — Specify the output and error files. %J is the job ID —
# — -o and -e mean append, –oo and-eo mean overwrite —
#BSUB –oo BPFilter_%J.out
#BSUB –eo BPFilter_%J.err
# — example of an ansys command line call —
/CMC/tools/ansys/ansys.2020r2/AnsysEM20.2/Linux64/ansysedt -distributed –machinelist numcores=48 -auto -monitor -ng –useElectronicsPPE –batchoptions registry.txt –batchsolve /eng/home/guest/ansoft/bp_filter.aedt
The registry.txt file included above contains a direct reference to the anshpc feature ‘pool’. If using this as an example, insert your assigned Cluster user name in place of <user_name> below. Here is a copy of the file contents:
## — include other desired registry entries here —
Use of this registry.txt file is described in the ANSYS 2020 R2 help files.