.. _random_numbers:

Randomness in NEST Simulations
==============================

.. admonition:: osx-arm64: missing random number generators

   Due to a cross-compiling issue in the conda NEST package, some random number
   generators are not available if you are using macOS arm64 architecture.
   The available generators are the Mersenne Twister generators `mt19937` and `mt19937_64`.

Random numbers in network simulations
-------------------------------------

Random numbers are used for a variety of purposes in neuronal network
simulations, e.g.,

1. during node creation to randomize node parameters;

#. during connection creation

   a. to select connections to be created,
   #. to parameterize connections;

#. during simulation

   a. to generate stochastic input (randomized spike trains, noisy currents),
   #. to support stochastic spike generation.

This document discusses how NEST provides random numbers for these
purposes, how you can choose which random number generator (RNG) to
use, and how to set the seed of RNGs in NEST. We use the term "random
number" here for ease of writing, even though we are always talking
about pseudorandom numbers generated by some algorithm.

.. admonition:: Compiler dependency

   NEST uses random generators provided by the C++ Standard Library to
   obtain random deviates, i.e., random numbers following different distributions
   such as normal, binomial, or Poissonian. Different versions of the C++
   Standard Library use different algorithms to generate these deviates.
   Therefore, simulation results can differ in detail depending on the compiler
   and the C++ Standard Library implementation used. Note in particular that
   GCC and Clang come with different library implementations.

.. _working_with_rngs:

How to work with randomness in NEST
-----------------------------------

NEST makes randomness easy: Setting a single seed gives you full
control over randomness throughout a simulation, and you can choose seed values
at liberty. NEST's powerful mechanism to :ref:`parameterize nodes and synapses <param_ex>`
and NEST's :ref:`probabilistic connection rules <conn_rules>` allow you
to construct your network with parameters with a wide range of random properties,
from uniformly distributed numbers drawn from a fixed interval to mathematical
expressions combining spatial information with probability distributions. Neuron, device,
and synapse models apply randomness during simulation as described in the model documentation.

This section provides a practical introduction into managing random number
generators and seeds in NEST, followed by an :ref:`overview of the nest.random module <nest_random>`
and :ref:`examples of using randomness <random_examples>`.  A separate section provides
:ref:`suggestions for additional randomization from the Python level <python_rand>`.

Programs that are entirely serial typically use a single source of random numbers.
In parallel simulations, such a single source of random numbers would form an
unacceptable bottleneck. To avoid this, NEST uses parallel streams of random numbers.
See section :ref:`Random number internals <random_internals>` if you are interested
in the technical details.


Select the type of random number generator
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

NEST provides different random number generators, presently taken from the
C++ Standard Library and the `Random123 <https://www.deshawresearch.com/resources_random123.html>`__
library. To see all available generators, run

::

    nest.rng_types

This currently gives the following generators:

::

     'Philox_32', 'Philox_64',
     'Threefry_32', 'Threefry_64',
     'mt19937', 'mt19937_64'

The Philox and Threefry generators are cryptographic generators from Random123, while mt19937
is the `Mersenne Twister generator <http://www.math.sci.hiroshima-u.ac.jp/~m-mat/MT/emt.html>`__.
Each generator comes in a 32- and a 64-bit implementation. The number of bits has no consequence
for the quality of randomness, but depending on computer and compiler, one or the other may show
better performance. More generators may be added in the future. Currently, only a small number of
generators is available because we require generators with a sufficiently long period as discussed in
:ref:`Random number internals <random_internals>`.

The default random number generator set in NEST is ``mt19937_64``. To choose a different generator,
set it in the following way:

::

     nest.rng_type = 'Philox_32'

It is a good idea to cross-check your simulation results using a different random number generator
type. Even though generators and our understanding of them has become much better in recent years,
there always remains a risk of :hxt_ref:`RNG` artifacts affecting simulations.


Seed the random number generator
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

NEST uses a built-in default seed if you do not specify one. This means that unless you explicitly
set the seed, all simulations will be run with the same sequence of random numbers. So *do not forget
to seed your simulations*!

You can use any number :math:`s` with :math:`1\leq s \leq 2^{31}-1` as seed:

::

    nest.rng_seed = 12345

As long as you use different seed values, NEST will ensure that all random number streams in a
simulation are seeded properly; see :ref:`Random number internals <random_internals>` for details.

You can inspect the :hxt_ref:`RNG` type and seed value used with

::

    print(nest.rng_type, nest.rng_seed)

Any simulation run with the same seed shall return identical results (provided the same
compiler/C++ Standard Library was used).


.. _nest_random:

The NEST random module
----------------------

The ``nest.random`` module provides a range of random distributions that
can be used to specify parameters for neurons, synapses, and connection
rules. See :ref:`below for examples <random_examples>` on how to use them in
practice and :ref:`some details on randomizing delays <random_delays>`.

.. automodule:: nest.random.hl_api_random
    :members:
    :noindex:

.. _random_examples:

Examples of using randomness
----------------------------


Randomize the membrane potential
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

To set the membrane potential at creation, just pass a random distribution as the :hxt_ref:`V_m` value.

::

    nest.Create('iaf_psc_alpha', 10000, {'V_m': nest.random.normal(mean=-60.0, std=10.0)})

You may also set the membrane potential after creation.

::

   n = nest.Create('iaf_psc_alpha', 10000)
   n.V_m = nest.random.normal(mean=-60.0, std=10.0)

Randomize other node parameters
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Other parameters may be randomized in the same way as the membrane potential.

::

   nest.Create('iaf_psc_alpha', 10000,
               {'C_m': nest.random.uniform(min=240.0, max=260.0),
                'I_e': nest.random.uniform(min=0.0, max=5.0)})

Randomize connection parameters
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Likewise, synapse parameters can be specified using the random distributions.

::

   nest.Connect(n, n, syn_spec={'weight': nest.random.normal(mean=0., std=1.),
                                'delay': nest.random.uniform(min=0.5, max=1.5)})

.. _random_delays:

Rounding effects when randomizing delays
........................................

Connection delays in NEST are rounded to the nearest multiple of the simulation resolution,
even if delays are drawn from a continuous distribution. This will work as expected if the
distribution has infinite support, for example, the normal distribution. For the uniform distribution,
though, this rounding will usually lead to lower probabilities for the delays at the edges of
the distribution. This then also applies to a truncated normal distribution.
Consider the following case:

::

    nest.resolution = 0.1
    n = nest.Create('iaf_psc_alpha', 100)
    nest.Connect(n, n, syn_spec={'delay': nest.random.uniform(min=1, max=2)})

This will create 10000 connections in total with delay values 1.0, 1.1, ..., 2.0. But while
the interior delay values 1.1, ..., 1.9 will occur approximately 1000 times each, the first and last
cases, 1.0 and 2.0, will occur only approximately 500 times each. This happens because NEST
first draws the delay uniformly from :math:`[1, 2)` and then rounds to a fixed delay. Thus,
any number from :math:`[1.05, 1.15)` will be rounded to 1.1, but only numbers in
:math:`[1.0, 1.05)` will be rounded to 1.0.

To achieve equal probabilities of the first and last values, you can either extend the interval
of the uniform distribution by half the resolution:

::

    nest.Connect(n, n, syn_spec={'delay': nest.random.uniform(min=1 - 0.5 * nest.resolution,
                                                              max=2 + 0.5 * nest.resolution)})

or use the uniform integer distribution. Since it always draws numbers beginning with zero,
this is slightly more cumbersome:

::

    nest.Connect(n, n, syn_spec={'delay': 1 + 0.1 * nest.random.uniform_int(11)})

An in-depth analysis of delay rounding is available in a
`master thesis <https://hdl.handle.net/11250/3012689>`_.

Randomize spatial positions
~~~~~~~~~~~~~~~~~~~~~~~~~~~

Spatial positions of neurons may be randomized using the same random distributions.
Note that we have to specify the number of dimensions we want when specifying a single
distribution.

::

   nest.Create('iaf_psc_alpha', 1000,
               positions=nest.spatial.free(nest.random.uniform(min=0.0, max=10.0),
                                           num_dimensions=2))

We can also specify a separate distribution for each dimension.

::

   nest.Create('iaf_psc_alpha', 1000,
               positions=nest.spatial.free([nest.random.uniform(min=0.0, max=10.0),
                                            nest.random.normal(mean=5.0, std=10.0)])

Combine random distributions
~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Random distributions are NEST parameter objects which support arithmetic with constants.

::

   vm = -50.0 + nest.random.exponential(beta=2.0)
   nest.Create('iaf_psc_alpha', 100, {'V_m': vm})

They also support arithmetic with other NEST parameters.

::

   n = nest.Create('iaf_psc_alpha', 1000,
                   positions=nest.spatial.free(nest.random.uniform(min=0.0, max=10.0),
                                               num_dimensions=2))

   nest.Connect(n, n,
                conn_spec={'rule': 'fixed_indegree', 'indegree': 1},
                syn_spec={'delay': nest.spatial.distance + nest.random.normal(mean=1.0, std=0.5)})

.. _python_rand:

Randomization from the Python level
-----------------------------------

In rare occasions, the capabilities provided by NEST for parameterizing your network will
not cover your needs and require additional randomization at the Python level. This requires
some additional consideration when performing simulations in parallel, especially when using
MPI parallelization. You may want to consult the section on :ref:`Random number internals <random_internals>`
before continuing.

A key principle of parallel simulation in NEST is that a simulation performed with a fixed
number of virtual processes :math:`N_{\text{vp}} = M \times T` shall produce identical results
independent of the number of :hxt_ref:`MPI` processes :math:`M` and threads :math:`T` that go into each virtual process.
To observe this principle when also randomizing from the Python level, it is essential
to create one Python random number generator per virtual process and use the random number
generator for the virtual process to which a node belongs (for synapses: the VP of the target
node).

We consider first an example setting a random membrane potential.
We use the `modern random package introduced with NumPy 1.17 <https://numpy.org/doc/stable/reference/random/>`__.

.. warning:: Don't do this in Python!

   The randomization example below is shown only to demonstrate how you in principle could
   use random values generated at the Python level in a NEST simulation in a way consistent
   with NEST's approach to random numbers in parallel simulations. You should only need to
   use this approach under very particular circumstances (e.g., if the distribution you want
   to use is not available in ``nest.random``).

   The proper way to randomize the membrane potential in NEST with uniform random numbers as
   in the example below is

   .. code-block:: ipython

       nest.Create('iaf_psc_alpha', 12, params={'V_m': nest.random.uniform()})


.. code-block:: ipython

    import numpy as np

    n_vp = 4
    py_seed = 987654

    nest.total_num_virtual_procs = n_vp
    nrns = nest.Create('iaf_psc_alpha', 12)

    rngs = {vp: np.random.default_rng(py_seed + vp) for vp in nest.GetLocalVPs()}

    for n in nest.GetLocalNodeCollection(nrns):
        n.set({'V_m': rngs[n.get('vp')].uniform()})

After creating the neurons, we create ``n_vp`` random number generators. We then use
``nest.GetLocalNodeCollection()`` to obtain all those neurons that belong to the local
MPI rank, since each rank can only access local nodes. We then use ``n.get('vp')`` to obtain
the virtual processes responsible for the node and use it to pick out the correct RNG,
from which we draw the random membrane potential.

To parameterize a connection, we need to use the random generator for the virtual process of
the target neuron. Continuing from the example above, we can randomize the connection weight
as follows:

.. code-block:: ipython

    nest.Connect(nrns, nrns, 'one_to_one')

    for c in nest.GetConnections():
        c.set({'weight': rngs[c.get('vp')].uniform()})

.. _random_internals:

Random number internals
-----------------------

Global and local generators
~~~~~~~~~~~~~~~~~~~~~~~~~~~

As described above, random numbers are consumed in NEST

1. during node creation to randomize node parameters;

#. during connection creation

   a. to select connections to be created,
   #. to parameterize connections;

#. during simulation

   a. to generate stochastic input (randomized spike trains, noisy currents),
   #. to support stochastic spike generation.

In all cases except certain sub-cases of 2.a, randomization is tied to a node,
which in turn is assigned to a specific virtual process (VP), either because the node is
concerned (cases 1, 3) or because the node is the target of a connection
(case 2). NEST guarantees that nodes assigned to different VPs
can be updated independently of each other, and the same is true for
connections if created or parameterized from a target-node perspective.
Therefore, for all these purposes, one random number stream per VP is
required and sufficient to ensure that different VPs can operate
independently.

The only exception are certain sub-cases of 2.a and 3 which require
identical random number streams either per MPI process or per virtual process.
This pertains in particular to ``fixed_outdegree`` and ``fixed_total_number``
connection rules and the `mip_generator`.

NEST therefore provides three kinds of random number streams

1. a rank-synchronized (global) random number stream to be used only by the
   main thread of each MPI rank; all MPI ranks must draw equally many numbers
   from this stream and discard those they do not need.
#. a VP-synchronized (global) random number stream to be used in parallel
   by all VPs.
#. one asynchronous random number stream per virtual process.

This results in a total of :math:`N_{\text{vp}}+2` random number streams.
To avoid unnecessary complications in the code using random numbers,
serial simulations also use all three kinds. The generators for all streams
are of the same type. If the :hxt_ref:`RNG` type is changed, the change
applies to all generators.

NEST regularly checks during a simulation that the rank- and VP-synchronized
random number streams indeed stay in sync.


Quantity of random numbers required
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Considering the future with exascale computers, we need to prepare for
:math:`N_{\text{vp}}=\mathcal{O}(10^7)` random streams. We can
estimate the number of random numbers required per stream as follows:

- Based on current practice, we assume :math:`N = 1000` neurons per stream;
  this results in a total of :math:`\mathcal{O}(10^{11})` neurons, on the scale
  of the human brain and thus a sensible upper limit.
- Each neuron receives excitatory and inhibitory Poisson background input, i.e.,
  two Poisson distributed random numbers per time step per neuron, each of which
  typically requires multiple uniform random numbers; we assume for simplicity
  two uniform numbers per Poisson number, i.e., four plain random numbers per
  time step per neuron.
- A time step is 0.1ms and a typical simulation duration 100s simulated time
  resulting in :math:`\mathcal{O}(10^6)` time steps.
- In total, a simulation will consume :math:`\sim 1000 \times 4 \times 10^6 = 4 \times 10^{10}`
  random numbers per stream (consumption during network construction will usually be negligible).
- To allow for a margin of error, we assume that :math:`\mathcal{O}(10^{11})`
  random numbers are consumed per stream during a single simulation.


Collision risk of parallel random number streams
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

`L'Ecuyer et al (2017) <https://dx.doi.org/10.1016/j.matcom.2016.05.005>`__ provide a
recent account of random number generation in highly parallel settings.
The main approach to providing independent random number streams in
parallel simulations is to use a high-quality random number generator
with long period and to seed it with a different seed for each stream. They
then argue that if we have :math:`s` streams of length :math:`l` and a generator
with period :math:`r`, the probability that two streams seeded with
different seeds will overlap is given by

.. math::

   p_{\text{overlap}} \sim s^2\frac{l}{r}\; .

We have

.. math::

   s = N_{\text{vp}} = 10^7 \sim 2^{23}\quad\text{and}\quad l = 10^{11} \sim 2^{37}\;,

so assuming an :hxt_ref:`RNG` period of :math:`r = 2^{128}` we obtain

.. math::

   p_{\text{overlap}} \sim \left(2^{23}\right)^2 \times 2^{37} / 2^{128} = 2^{-45} \sim 3 \times 10^{-14}

while for a period of :math:`r = 2^{256}` we obtain

.. math::

   p_{\text{overlap}} \sim 2^{-173} \sim 10^{-52}\; .

The probability of stream collisions is thus negligibly small, provided
we use random number generators with a period of at least :math:`2^{128}`.

`L'Ecuyer (2012, Ch 3.6) <https://doi.org/10.1007/978-3-642-21551-3_3>`__ points out
that certain random number generator classes will show too much
regularity in their output if more than :math:`r^{1/3}` numbers are used (this
relation is based on exercises in Knuth, TAOCP vol 2, see `L'Ecuyer and
Simard (2001) <https://doi.org/10.1016/S0378-4754(00)00253-6>`__). While
it is not certain that that analysis also applies to (short)
subsequences of the period of an RNG, one might still re-consider the
analysis above, by using :math:`l^3 \sim 2^{111}` instead of :math:`l` when
computing :math:`p_{\text{overlap}}`. We then obtain

============================= ===============================
Period :math:`r`              :math:`p_{\text{overlap}}`
============================= ===============================
:math:`2^{128} \sim 10^{38}`  :math:`>> 1`
:math:`2^{256} \sim 10^{77}`  :math:`2^{-99} \sim 10^{-30}`
:math:`2^{512} \sim 10^{154}` :math:`2^{-355} \sim 10^{-107}`
============================= ===============================

Thus, for generators with periods of at least :math:`2^{256}`, we have
negligible collision probability also under this tightened criterium.

Among the generators offered by the C++ Standard Library, only the
Mersenne Twister generator has a sufficiently long period, so only this
generator is included. The generators from the Random123 library are
cryptographic generators and thus no period.


Seed parallel random number streams
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

In NEST, the user provides a single seed :math:`1 \leq S \leq 2^{31}-1` to initialize
all :math:`N_{\text{vp}}+2`  random number generators in a parallel simulation.
NEST uses a seed sequence  generator to derive seeds for the individual RNGs from
the user-provided seed :math:`S`.

Since ``std::seed_seq`` from the C++ Standard Library `has weaknesses <https://www.pcg-random.org/posts/cpp-seeding-surprises.html>`__,
we use `Melissa O'Neill's improved seed sequence generator <https://www.pcg-random.org/posts/simple-portable-cpp-seed-entropy.html>`__.



Further background
~~~~~~~~~~~~~~~~~~

For more background on the NEST random number generation architecture,
see also `NEST Github issue #1440 <https://github.com/nest/nest-simulator/issues/1440>`__.
