The CFD-ACE+ solver can be run in parallel mode by using domain decomposition, whereby you divide up the data among multiple processors. The number of zones is equated to the number of processes or in other words, each processor would solve for a single zone of the overall mesh.
The parallel feature of CFD-ACE+ provides several methods and options for decomposition of the domain. In this tip we will talk about these methods and options and their usage. In many cases, the manner of decomposition influences the parallel efficiency and speed-up. For better speed-ups it is important that the workload is balanced between processors, in other words, all the processors gets nearly equal number of cells. The user has to use his own intuition as to which decomposition would be most appropriate for his/her simulation before attempting to do the domain decomposition.
Syntax:
The domain decomposition in CFD-ACE+ is done by executing the dtf_decompose script. When you type dtf_decompose at the command prompt, you can get its full usage.
The syntax for the command is:
dtf_decompose [-version] [-metis | -cell_groups | -orig_topo | -x | -y | -z | -material | -wavefront] [-even] [-combine] [-keepFF] [-file_out outFile.DTF] [-restart] inFile.DTF sim# num_procs
Method 1 # -metis
This is the default method for dtf_decompose. So if the user does not provide any other method, -metis option is used. METIS is multi-level graph partitioning scheme used for partitioning unstructured meshes. It provides high quality partitions with less CPU time. This decomposition method does workload balancing. Depending on the number of partitions, METIS uses two different ways of partitions.
-- K-way Partitioning for N > 8
-- Recursive Bisection for N < 8
For more details on these partitioning schemes, the user can refer to the following web site. http://www-users.cs.umn.edu/~karypis/metis/
The syntax is:
$ dtf_decompose -metis model.DTF sim# num_procs
where, sim# is the simulation number and, num_procs refer to the number of processors, which is equal to number of decomposed zones.
For example,
$ dtf_decompose -metis test_metis.DTF 1 4
This indicates that simulation # 1 would be decomposed into 4 zones. The decomposed domain is stored in simulation # 2, which is stored in the same DTF file. When you open the file in CFD-ACE-GUI or CFD-VIEW, it prompts the user select the appropriate simulation number in the DTF file.
Figure 1 shows the original interface between two zones, and Figure 2 shows the new interfaces created after decomposition into four zones
Method 2 # -cell_groups
This method can be used to decompose the domain along cell groups. So if a simulation has 4 cell groups, it will decompose the simulation into 4 zones. Decomposing by cell groups may be useful for special cases such as materials grouping. Note that since this is done by cell groups, there is no attempt to load balance the zones.
Method 3 # -orig_topo
This method allows the user to partition the domains along the original topology as it was created in CFD-GEOM. If the user has taken care of load balancing in GEOM, this method can be used. Note that since the decomposition is done along the original zones, there is no attempt to load balance the zones.
Figure 3 shows the same original topology after decomposition.
This method allows the user to decompose the domain along the X, Y or Z-axis. In this case, the decomposition is workload balanced. In other words, the decomposition will be done in such a way that the number of cells for each processor is nearly equal.
Figure 4 shows the original domain with 2 zones, one is coarse and the other one is refined. After decomposing this into 4 zones, in Figure 5, we can see that the z decomposition is workload balanced as there are more number of cuts in the refined zone.
In this method, the decomposition will take place along the material interfaces. So if your simulation domain contains fluid and solid zones, then this method would allow you to group all the fluid zones into one decomposed zone and all the solid zones into another decomposed zones. Note that since the decomposition is done along the material zones, there is no attempt to load balance the zones.
Figure 6 shows the original domain and Figure 7 shows the domain after decomposition into 2 zones (i.e. solid and fluid zone).
This method is based on the Wavefront Element Reordering scheme, which can be used to improve the efficiency of the global matrix decomposition process performed during an analysis. It minimizes the number of operations needed to decompose the assembled global matrices during an analysis. This method is used rarely. This decomposition method provides workload balancing.
There are some other options, which are described below, that can be used with the above methods.
Option 1 # -even
This option allows the user to do an even x, y or z decomposition, which means it will be spatially balanced (not workload balanced). In other words, the decomposition will be done in such a way that the number of cells for each processor is nearly equal. Note that this option is only used with x, y or z decomposition.
Figure 8, 9, and 10 shows example of even –x, -y, -z decompositions.
This option will allow the user to combine the Volume and Boundary Conditions with identical properties to be decomposed into one zone. This option reduces the number of volume and boundary conditions after decomposition and can be used with any method described above.
Option 3 # -keepFF
This option allows the user to keep the original Fluid-Fluid interfaces inside of a zone. This becomes useful if the user would like to retain the Fluid-Fluid interface to output Mass Flow Summary or the Heat Flow Summary at the interface. This option can be used with any method described above.
Option 4 # -restart
This option can be used with other methods, when the user would like to do a restart from the serial solution. It provides mapping of face data from a simulation to the decomposed simulation, which is needed for restarts.
In addition to the above options in V2004, we can run FSI (Fluid Structure Interaction) cases in parallel. This is done by decomposing all the stress volume conditions into one zone and decomposing all the other zones (fluid + non-stress solid zones) into (n-1) processors. The decomposition for FSI cases is done in the same way as illustrated above. The decomposition algorithm identifies the stress volume conditions automatically and decomposes it into one domain.
This summarizes all the methods and options that can be used for running dtf_decompose prior to submitting the case in parallel by running the cfd-ace-mpi script.
If you have any questions about this feature or would like us to discuss some other topic in the future, please let us know.
Amit Saxena
Customer Support Engineer
ESI Group