Parallel

During a simulation with distributed parallel, the communication among processors is crucial. In KitAMR.jl, the communication is supported by MPI.jl.

Ghost layers

KitAMR.jl decomposes the computational domain in physical space. With explicit spatial discretization, the update of a cell only depends on its neighboring cells. If some of these neighboring cells lie on other processors, they are locally marked as ghost. The variables of the ghost cells are updated by MPI communication. Related functions are

KitAMR.data_exchange! — Function

data_exchange!(
    p4est::Union{Ptr{KitAMR.P4est.LibP4est.p4est}, Ptr{KitAMR.P4est.LibP4est.p8est}},
    ka::KA{DIM, NDF}
)

Update df in GhostVsData.

source

KitAMR.slope_exchange! — Function

slope_exchange!(
    p4est::Union{Ptr{KitAMR.P4est.LibP4est.p4est}, Ptr{KitAMR.P4est.LibP4est.p8est}},
    ka::KA{DIM, NDF}
)

Update sdf in GhostVsData.

source

KitAMR.solid_exchange! — Function

solid_exchange!(
    p4est::Union{Ptr{KitAMR.P4est.LibP4est.p4est}, Ptr{KitAMR.P4est.LibP4est.p8est}},
    ka::KA{DIM, NDF}
)

Update df in GhostVsData for immersed-boundary cells only.

Only ghost cells with bound_enc < 0 (solid-adjacent, non-InsideSolid) are communicated; all other ghost cells are skipped, reducing both message sizes and the number of communicated elements compared to a full data_exchange!.

source

In contrast to traditional CFD solver, KitAMR.jl also discretizes velocity space. The resolution of this discretization varies as time and physical coordinates changes. This results in the size of communicated data changing. Currently, the size of the communicated data is unified to hold the largest velocity space that exists in ghost layers. The size is obtained by

KitAMR.get_vs_num — Function

get_vs_num(
    forest::Union{Ptr{KitAMR.P4est.LibP4est.p4est}, Ptr{KitAMR.P4est.LibP4est.p8est}},
    ghost::Union{Ptr{KitAMR.P4est.LibP4est.p4est_ghost_t}, Ptr{KitAMR.P4est.LibP4est.p8est_ghost_t}}
) -> Any

Get the globally largest number of the velocity cells among all local and ghost quadrants.

source

Whether a unified communication size decreases the efficiency noticeably still requires testing.

Partition

After an AMR process, the grids density on different processors may changes a lot. To balance the load, partition can be performed by calling

KitAMR.partition! — Function

partition!(
    p4est::Ptr{KitAMR.P4est.LibP4est.p4est}
) -> Tuple{Vector{Int64}, Vector{Int64}, Int32, Int32}
partition!(
    p4est::Ptr{KitAMR.P4est.LibP4est.p4est},
    weight::Function
) -> Tuple{Vector{Int64}, Vector{Int64}, Int32, Int32}

source

The grids in physical space are encoded as a 1-dimensional sequence by Morton code, and then partitioned according to the weight provided by

KitAMR.partition_weight — Function

partition_weight(
    p4est::Union{Ptr{KitAMR.P4est.LibP4est.p4est}, Ptr{KitAMR.P4est.LibP4est.p8est}},
    which_tree,
    quadrant::Union{Ptr{KitAMR.P4est.LibP4est.p4est_quadrant}, Ptr{KitAMR.P4est.LibP4est.p8est_quadrant}}
) -> Any

source

The backend functions are provided by p4est, insuring high efficiency.

When the partition is finished, a data exchange process is performed to transfer all of the data to where it is required. The communication only occurs to those partitioned cells.

All of the functionalities are wrapped into

KitAMR.ps_partition! — Function

ps_partition!(
    p4est::Union{Ptr{KitAMR.P4est.LibP4est.p4est}, Ptr{KitAMR.P4est.LibP4est.p8est}},
    ka::KA
)

Re-partition grids in physical space to balance the computational load according to the weight function.

source