Targeting a non-Julian model
Suppose you have some code implementing vanilla MCMC, written in an arbitrary "foreign" language such as C++, Python, R, Java, etc. You would like to turn this vanilla MCMC code into a Parallel Tempering algorithm able to harness large numbers of cores, including distributing this algorithm over MPI. However, you do not wish to learn anything about MPI/multi-threading/Parallel Tempering.
Surprisingly, it is very simple to bridge such code with Pigeons. The only requirement on the "foreign" language is that it supports reading the standard in and writing to the standard out, hence virtually any languages can be interfaced in this fashion. Based on this minimalist "standard stream bridge" with worker processes running foreign code (one such process per replica; not necessarily running on the same machine), Pigeons will coordinate the execution of an adaptive non-reversible parallel tempering algorithm.
This behaviour is implemented in StreamTarget, see its documentation for details. In a nutshell, there will be one child process for each PT chain. These processes will not necessarily be on the same machine: indeed distributed sampling is the key use case of this bridge. Pigeons will do some lightweight coordination with these child processes to orchestrate non-reversible parallel tempering. Interprocess communication only involves pigeons telling each child process to perform exploration at a pigeons-provided annealing parameter.
StreamTarget implements log_potential and explorer by invoking worker processes via standard stream communication. The standard stream is less efficient than alternatives such as protobuff, but it has the advantage of being supported by nearly all programming languages in existence. Also in many practical cases, since the worker process is invoked only three times per chain per iteration, it is unlikely to be the bottleneck (overhead is in the order of 0.1ms per interprocess call).
Usage example
To demonstrate this capability, we show here how it enables running Blang models in pigeons. Blang is a Bayesian modelling language designed for sampling combinatorial spaces such as phylogenetic trees.
We first setup Blang as follows (assuming Java 11 is accessible in the PATH variable):
using Pigeons
redirect_stdout(devnull) do
Pigeons.setup_blang("blangDemos")
end┌ Warning: Using a precompiled build for blangDemos: double check it is up to date
└ @ Pigeons ~/work/Pigeons.jl/Pigeons.jl/src/targets/BlangTarget.jl:141
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
0 0 0 0 0 0 0 0 --:--:-- 0:00:01 --:--:-- 0
0 0 0 0 0 0 0 0 --:--:-- 0:00:02 --:--:-- 0
0 152M 0 523k 0 0 160k 0 0:16:16 0:00:03 0:16:13 160k
1 152M 1 2109k 0 0 494k 0 0:05:16 0:00:04 0:05:12 494k
3 152M 3 5070k 0 0 961k 0 0:02:42 0:00:05 0:02:37 1064k
20 152M 20 31.7M 0 0 5169k 0 0:00:30 0:00:06 0:00:24 6805k
34 152M 34 53.2M 0 0 7478k 0 0:00:20 0:00:07 0:00:13 10.5M
42 152M 42 64.4M 0 0 7966k 0 0:00:19 0:00:08 0:00:11 12.7M
46 152M 46 70.6M 0 0 7790k 0 0:00:20 0:00:09 0:00:11 13.6M
50 152M 50 76.9M 0 0 7653k 0 0:00:20 0:00:10 0:00:10 14.3M
53 152M 53 81.9M 0 0 7432k 0 0:00:21 0:00:11 0:00:10 10.0M
55 152M 55 85.6M 0 0 7130k 0 0:00:21 0:00:12 0:00:09 6625k
57 152M 57 87.9M 0 0 6777k 0 0:00:23 0:00:13 0:00:10 4809k
58 152M 58 90.2M 0 0 6464k 0 0:00:24 0:00:14 0:00:10 4001k
60 152M 60 92.2M 0 0 6173k 0 0:00:25 0:00:15 0:00:10 3129k
61 152M 61 93.6M 0 0 5883k 0 0:00:26 0:00:16 0:00:10 2396k
62 152M 62 94.9M 0 0 5632k 0 0:00:27 0:00:17 0:00:10 1918k
63 152M 63 96.3M 0 0 5404k 0 0:00:28 0:00:18 0:00:10 1735k
63 152M 63 97.7M 0 0 5196k 0 0:00:30 0:00:19 0:00:11 1550k
64 152M 64 99.2M 0 0 5011k 0 0:00:31 0:00:20 0:00:11 1436k
65 152M 65 100M 0 0 4853k 0 0:00:32 0:00:21 0:00:11 1476k
67 152M 67 102M 0 0 4720k 0 0:00:33 0:00:22 0:00:11 1577k
68 152M 68 104M 0 0 4608k 0 0:00:33 0:00:23 0:00:10 1696k
69 152M 69 107M 0 0 4516k 0 0:00:34 0:00:24 0:00:10 1896k
71 152M 71 109M 0 0 4442k 0 0:00:35 0:00:25 0:00:10 2131k
73 152M 73 112M 0 0 4366k 0 0:00:35 0:00:26 0:00:09 2292k
74 152M 74 113M 0 0 4279k 0 0:00:36 0:00:27 0:00:09 2316k
75 152M 75 115M 0 0 4197k 0 0:00:37 0:00:28 0:00:09 2291k
77 152M 77 118M 0 0 4130k 0 0:00:37 0:00:29 0:00:08 2265k
78 152M 78 120M 0 0 4075k 0 0:00:38 0:00:30 0:00:08 2226k
80 152M 80 122M 0 0 4008k 0 0:00:39 0:00:31 0:00:08 2132k
81 152M 81 124M 0 0 3939k 0 0:00:39 0:00:32 0:00:07 2081k
82 152M 82 126M 0 0 3877k 0 0:00:40 0:00:33 0:00:07 2070k
83 152M 83 128M 0 0 3827k 0 0:00:40 0:00:34 0:00:06 2050k
85 152M 85 130M 0 0 3785k 0 0:00:41 0:00:35 0:00:06 2027k
86 152M 86 132M 0 0 3737k 0 0:00:41 0:00:36 0:00:05 2038k
88 152M 88 134M 0 0 3700k 0 0:00:42 0:00:37 0:00:05 2155k
89 152M 89 137M 0 0 3668k 0 0:00:42 0:00:38 0:00:04 2273k
90 152M 90 139M 0 0 3628k 0 0:00:43 0:00:39 0:00:04 2258k
92 152M 92 141M 0 0 3595k 0 0:00:43 0:00:40 0:00:03 2254k
94 152M 94 143M 0 0 3570k 0 0:00:43 0:00:41 0:00:02 2356k
95 152M 95 146M 0 0 3548k 0 0:00:44 0:00:42 0:00:02 2419k
97 152M 97 148M 0 0 3523k 0 0:00:44 0:00:43 0:00:01 2407k
98 152M 98 151M 0 0 3496k 0 0:00:44 0:00:44 --:--:-- 2469k
100 152M 100 152M 0 0 3485k 0 0:00:44 0:00:44 --:--:-- 2540kNext, we run a Blang implementation of our usual unidentifiable toy example:
using Pigeons
blang_unidentifiable_example(n_trials, n_successes) =
Pigeons.BlangTarget(
`$(Pigeons.blang_executable("blangDemos", "demos.UnidentifiableProduct")) --model.nTrials $n_trials --model.nFails $n_successes`
)
pt = pigeons(target = blang_unidentifiable_example(100, 50))┌ Info: Neither traces, disk, nor online recorders included.
│ You may not have access to your samples (unless you are using a custom recorder, or maybe you just want log(Z)).
└ To add recorders, use e.g. pigeons(target = ..., record = [traces; record_default()])
Preprocess {
2 samplers constructed with following prototypes:
RealScalar sampled via: [RealSliceSampler]
} [ endingBlock=Preprocess blockTime=435.2ms blockNErrors=0 ]
Inference {
────────────────────────────────────────────────────────────────────────────
scans Λ time(s) allc(B) log(Z₁/Z₀) min(α) mean(α)
────────── ────────── ────────── ────────── ────────── ────────── ──────────
2 0.992 5.43 1.6e+07 -4.05 0.621 0.89
4 1.42 0.038 8.74e+06 -4.43 0.192 0.842
8 1.44 0.0744 1.97e+07 -5.35 0.581 0.84
16 1.7 0.107 3.54e+07 -4.72 0.577 0.811
32 1.51 0.233 6.99e+07 -4.8 0.602 0.833
64 1.52 0.373 1.4e+08 -4.95 0.727 0.831
128 1.56 0.472 2.66e+08 -4.93 0.713 0.827
256 1.52 1.07 5.49e+08 -5.06 0.78 0.831
512 1.52 1.6 1.06e+09 -4.97 0.793 0.831
1.02e+03 1.54 2.73 2.1e+09 -4.97 0.789 0.829
────────────────────────────────────────────────────────────────────────────As shown above, create a StreamTarget amounts to specifying which command will be used to create a child process.
To terminate the child processes associated with a stream target, use:
Pigeons.kill_child_processes(pt)