Shared Memory Arrays

Introduction

Two different implementations are available: one implementation with all segments contiguous and with a constant size, defined when the object is constructed, and another implementation with non-contiguous segments and a variable size.

Implementation with Contiguous Segments

This implementation allows the creation of a memory window whose segments are all contiguous. It is thus quite simple to re-slice the segments during use (for example, to balance a calculation).

Usage

This part is managed by the Arcane::ContigMachineShMemWin class.

This class can use three implementations of IContigMachineShMemWinBase, one for each type of Arcane::IParallelMng. It is therefore possible to use this class regardless of whether you have an MpiParallelMng, a SequentialParallelMng, a SharedMemoryParallelMng, or a HybridParallelMng (Choosing the Message Exchange Manager).

The creation of an object of this type is collective. An instance of this class will create a memory window composed of several segments (one per subdomain).

Access to the elements of the segments is not collective. Concurrent access to an element is possible using semaphores, mutexes, or std::atomic. For std::atomic, the operations must be address-free:

bool is_lock_free = std::atomic<Real>{}.is_lock_free();

When this object is constructed, each subdomain provides a segment size. The window size will be equal to the sum of the segment sizes.

    constexpr Integer nb_elem = 14;
 
    IParallelMng* pm = m_parallel_mng;
    Integer my_rank = pm->commRank();
 
    if (!ParallelMngUtils::isMachineShMemWinAvailable(pm)) {
      // Problem with MPI. May occur if MPICH is compiled in ch3:sock mode.
      // We do not crash the tests in this case.
      warning() << "Shared memory not supported";
      return;
    }
 
    ContigMachineShMemWin<Integer> window(pm, nb_elem);

Remarks

It is possible that MPI does not support shared memory (if using MPICH in ch3:sock mode, for example). To check this, you can use the function ParallelMngUtils::isMachineShMemWinAvailable(IParallelMng* pm).

To access its segment, you can use the method Arcane::ContigMachineShMemWin::segmentView().

    {
      Span av_my_segment(window.segmentView());
 
      Integer iter = 0;
      for (Integer& elem : av_my_segment) {
        elem = iter * (my_rank + 1);
        iter++;
      }
    }
    window.barrier();

Once the segment has been modified, you can perform a barrier to ensure that everyone has written to their segment before using it.

To find out which subdomains share a window on the node, you can retrieve an array of ranks.

    ConstArrayView<Int32> machine_ranks(window.machineRanks());
    Integer machine_nb_proc = machine_ranks.size();

The position of the ranks in this array corresponds to the position of their segment in the window.

To read the segments of other subdomains on the node, you can use the method Arcane::ContigMachineShMemWin::segmentConstView().

    for (Int32 rank : machine_ranks) {
      Span av_segment(window.segmentConstView(rank));
 
      for (Integer i = 0; i < nb_elem; ++i) {
        if (av_segment[i] != i * (rank + 1)) {
          ARCANE_FATAL("Bad element in memory window -- Expected : {0} -- Found : {1}", (i * (rank + 1)), av_segment[i]);
        }
      }
    }

The window size cannot be modified. However, the implementation in Arcane allows resizing the segments collectively (provided that the new window size is less than or equal to the original size).

Note

An implementation with memory windows having non-contiguous segments could be more performant but would make this functionality impossible.

    window.barrier();
 
    constexpr Integer nb_elem_div = nb_elem / 2;
 
    window.resizeSegment(nb_elem_div);

Remarks

The elements of the window are not modified during resizing.

Since the window is contiguous, access to the entire window is possible for all subdomains.

    if (my_rank == machine_ranks[0]) {
      Span av_window(window.windowView());
      for (Integer j = 0; j < machine_nb_proc; ++j) {
        for (Integer i = 0; i < nb_elem; ++i) {
          av_window[i + (j * nb_elem)] = machine_ranks[j];
        }
      }
    }
    window.barrier();
 
    {
      Span av_window(window.windowConstView());
      for (Integer j = 0; j < machine_nb_proc; ++j) {
        for (Integer i = 0; i < nb_elem; ++i) {
          if (av_window[i + (j * nb_elem)] != machine_ranks[j]) {
            ARCANE_FATAL("Bad element in memory window -- Expected : {0} -- Found : {1}", machine_ranks[j], av_window[i + (j * nb_elem)]);
          }
        }
      }
    }
    window.barrier();

Implementation with Non-Contiguous Segments

This implementation is quite different from the previous one. Here, the segments of the memory windows are no longer contiguous. Furthermore, with this implementation, it is possible to resize the segments like a classic dynamic array.

Nevertheless, this operation is collective, which contaminates most of the implementation's methods.

Usage

This part is managed by the Arcane::MachineShMemWin class.

As with the previous implementation, this one is compatible with all Arcane parallelism modes.

The creation of an object of this type is collective. An instance of this class will create a memory window composed of several segments (one per subdomain).

Like a UniqueArray, it is possible to specify an initial size (here 5):

    IParallelMng* pm = m_parallel_mng;
 
    if (!ParallelMngUtils::isMachineShMemWinAvailable(pm)) {
      // Problem with MPI. May occur if MPICH is compiled in ch3:sock mode.
      // We do not crash the tests in this case.
      warning() << "Shared memory not supported";
      return;
    }
 
    Integer my_rank = pm->commRank();
    MachineShMemWin<Integer> window(pm, 5);
    ConstArrayView machine_ranks(window.machineRanks());

And it is possible not to specify an initial size.

      MachineShMemWin<Integer> window2(pm);

The method Arcane::MachineShMemWin::machineRanks() is available and returns the same array as the Arcane::ContigMachineShMemWin implementation.

To explore our segment or the segment of another subdomain, you can use the same methods as before:

    {
      Span av_my_segment(window.segmentView());
      Integer iter = 0;
      for (Integer& elem : av_my_segment) {
        elem = iter * (my_rank + 1);
        iter++;
      }
    }
    window.barrier();
    for (Int32 rank : machine_ranks) {
      Span av_segment(window.segmentConstView(rank));
      for (Integer i = 0; i < 5; ++i) {
        if (av_segment[i] != i * (rank + 1)) {
          ARCANE_FATAL("Bad element in memory window -- Expected : {0} -- Found : {1}", (i * (rank + 1)), av_segment[i]);
        }
      }
    }

However, since the segments are not contiguous, the windowView() methods are not available.

The segments have a size that can be increased or decreased over time.

It is possible to add elements using the method Arcane::MachineShMemWin::add(Arcane::Span<const Type> elem):

    Integer pos_in_machine_ranks = -1;
    for (Integer i = 0; i < machine_ranks.size(); ++i) {
      if (my_rank == machine_ranks[i]) {
        pos_in_machine_ranks = i;
        break;
      }
    }
    // Note: here, pos_in_machine_ranks corresponds to the rank of the process
    // in the MPI communicator "machine".
    {
      UniqueArray<Integer> buf;
      if (pos_in_machine_ranks == 0) {
        for (Integer i = 0; i < 10; ++i) {
          buf.add(i);
        }
      }
      window.add(buf);
    }

This method is collective; all subdomains on a node must call it. If a subdomain does not wish to add elements to its segment, it can call the add() method with an empty array or without arguments (Arcane::MachineShMemWin::add()).

This operation can be costly due to memory reallocation. It is therefore advisable to add a large quantity of elements at once rather than element by element.

If element-by-element addition is indispensable, the method Arcane::MachineShMemWin::reserve(Arcane::Int64 new_capacity) is available to avoid reallocating a segment multiple times:

    window.reserve(20);

In this piece of code, we will reserve space for 20 Integers for all subdomains. This value can be different for each subdomain (if a subdomain does not want to reserve more space, it can call Arcane::MachineShMemWin::reserve()).

Note

With this method, you cannot reserve less space than already reserved (calling reserve(0) has no effect). To reduce the reserved space, the method Arcane::MachineShMemWin::shrink() is available.

Warning

As with UniqueArray, the method Arcane::MachineShMemWin::reserve(Arcane::Int64 new_capacity) does not have the same function as the method Arcane::MachineShMemWin::resize(Arcane::Int64 new_nb_elem). The former only reserves memory space, but this space remains inaccessible without add() or without resize(). The latter changes the number of elements in the segment and calls reserve() if necessary.

    window.resize(12);
    info() << window.segmentConstView().size();

In our example, resize() will increase the number of elements in all segments except for the subdomain that previously performed the add() operations (which has 15 elements, compared to 5 for the others). This subdomain will go from 15 elements to 12.

Like the reserve() method, each subdomain can set the value it wants.

It is also possible to add elements to the segment of another subdomain using the collective method Arcane::MachineShMemWin::addToAnotherSegment(Arcane::Int32 rank, Arcane::Span<const Type> elem).

    Int32 voisin = -1;
    if (my_rank % 2 == 0 && pos_in_machine_ranks + 1 < machine_ranks.size()) {
      voisin = machine_ranks[pos_in_machine_ranks + 1];
    }
    else if (my_rank % 2 == 1 && pos_in_machine_ranks - 1 >= 0) {
      voisin = machine_ranks[pos_in_machine_ranks - 1];
    }
 
    // Clear the elements already present in the segments.
    window.resize(0);
 
    // If there are no neighbors, add nothing.
    if (voisin == -1) {
      window.addToAnotherSegment();
    }
    else {
      UniqueArray<Integer> buf;
      for (Integer i = 0; i < 10; ++i) {
        buf.add(my_rank);
      }
      window.addToAnotherSegment(voisin, buf);
    }
    info() << "Final segment : " << window.segmentConstView();
    window.shrink();

Warning

It is impossible to mix calls to add() and addToAnotherSegment(). If a subdomain calls the addToAnotherSegment() method, all subdomains must collectively call addToAnotherSegment() (with or without parameters) and not add().

The functionality is almost identical to the add() method but with an extra parameter to designate the rank of the subdomain possessing the segment to be modified.

Warning

Two subdomains cannot add elements to the same segment (at the same time) (which prevents concurrency issues).

// Not possible:
if (my_rank == 0){
  window.addToAnotherSegment();
}
else if (my_rank == 1){
  window.addToAnotherSegment(0, mon_tableau);
}
else if (my_rank == 2){
  window.addToAnotherSegment(0, mon_tableau);
}

// Possible:
if (my_rank == 0){
  window.addToAnotherSegment();
  window.addToAnotherSegment();
}
else if (my_rank == 1){
  window.addToAnotherSegment();
  window.addToAnotherSegment(0, mon_tableau);
}
else if (my_rank == 2){
  window.addToAnotherSegment(0, mon_tableau);
  window.addToAnotherSegment();
}

Shared Memory Between Processes

Inter-process shared memory should not be seen as multithreaded shared memory. This sharing only occurs on a part of the memory, not on all of the memory.

Consider this structure:

struct MaStruct
{
    MaStruct()
    : array_integer(10)
    {}
    
    UniqueArray<Integer> array_integer;
};

It can be used like this:

MaStruct ma_struct;
ma_struct.array_integer[0] = 123 * (my_rank+1);

If this structure is used in a window, it would look like this:

ContigMachineShMemWin<MaStruct> win_struct(pm, 1);
Span<MaStruct> my_span = win_struct.segmentView();
new (my_span.data()) MaStruct();
 
my_span[0].array_integer[0] = 123 * (my_rank+1);

You can display the value you assigned, and it works correctly:

debug() << "Elem : " << my_span[0].array_integer[0];

But if you want to display the value of another process:

window.barrier();
 
Span<MaStruct> other_span = win_struct.segmentView(machine_ranks[(my_rank + 1) % machine_nb_proc]);
debug() << "Elem : " << other_span[0].array_integer[0];
 
my_span[0].~MaStruct();

In multi-process mode (launching with mpirun -n 2 ...), the program will crash (segfault), whereas it will not crash in multithreading (launching with -A,S=2).

In multi-process mode, the attributes of the UniqueArray<Integer> array_integer; array in the structure are not allocated in shared memory (the new or malloc calls are made in local memory), so other processes do not have access to them.

It is also important to note that the same memory location in shared memory is addressed differently between processes. Therefore, if you provide a shared memory allocator to the UniqueArray, the addresses used will only be valid locally.

---

Shared Memory Windows in Multi-Process Variables in Shared Memory