~/logs/realtime.logs.prod
- 1770653693418211 + [[ -n '' ]] event body
- 1770653696838223 + echo 'Seeding selfhosted Realtime' event body
- 1770653699361768 info ["$kind": :gen_rpc_dispatcher_start] event body
- 1770653701128669 notice SYN[nonode@nohost|registry<realtime_postgres_cdc_2>] Discovering the cluster event body
- 1770653701128954 notice SYN[nonode@nohost] Creating tables for scope <realtime_postgres_cdc_3> event body
- 1770653701129212 notice SYN[nonode@nohost|registry<realtime_postgres_cdc_3>] Discovering the cluster event body
- 1770653701130696 info Starting MetricsCleaner event body
- 1770653701608981 Starting Realtime event body
- 1770653704236150 info ["$kind": :gen_rpc_dispatcher_start] event body
- 1770653704280723 notice SYN[realtime@127.0.0.1|registry<Elixir.Realtime.Tenants.Connect>] Discovering the cluster event body
- 1770653704282069 notice SYN[realtime@127.0.0.1] Creating tables for scope <users_1> event body
- 1770653704282339 notice SYN[realtime@127.0.0.1|registry<users_1>] Discovering the cluster event body
- 1770653704283306 notice SYN[realtime@127.0.0.1] Adding node to scope <users_3> event body
- 1770653704414707 notice SYN[realtime@127.0.0.1|registry<realtime_postgres_cdc_3>] Discovering the cluster event body
- 1770653704415518 info Starting MetricsCleaner event body
- 1770799249116833 info Janitor started event body
- 1770884994476946 info Janitor started event body
- 1771076215336413 info Janitor started event body
- 1771168598799812 info Janitor started event body
- 1771280166952561 info Janitor started event body
- 1771408614077018 info Janitor started event body
- 1771506998885382 Could not write to file "pg_notify/0063" at offset 106496: No space left on device. event body
- 1771507048906961 error Postgrex.Protocol (#PID<0.3371.0>) disconnected: ** (Postgrex.Error) ERROR 53100 (disk_full) could not access status of transaction 0 event body
- 1771507053909297 error Postgrex.Protocol (#PID<0.3371.0>) disconnected: ** (Postgrex.Error) ERROR 53100 (disk_full) could not access status of transaction 0 event body
- 1771507061953731 (stdlib 6.2.1) proc_lib.erl:329: :proc_lib.init_p_do_apply/3 event body
- 1771507062054961 error Postgrex.Protocol (#PID<0.335969.0>) failed to connect: ** (Postgrex.Error) FATAL 57P03 (cannot_connect_now) the database system is in recovery mode event body
- 1771507062082096 error Postgrex.Protocol (#PID<0.336006.0>) failed to connect: ** (Postgrex.Error) FATAL 57P03 (cannot_connect_now) the database system is in recovery mode event body
- 1771507062087916 error Postgrex.Protocol (#PID<0.336016.0>) failed to connect: ** (Postgrex.Error) FATAL 57P03 (cannot_connect_now) the database system is in recovery mode event body
- 1771507062096711 error Postgrex.Protocol (#PID<0.336032.0>) failed to connect: ** (Postgrex.Error) FATAL 57P03 (cannot_connect_now) the database system is in recovery mode event body
- 1771507064242891 + '[' '!' -z 10000 ']' event body
- 1771507065071978 (elixir 1.18.4) src/elixir.erl:386: :elixir.eval_external_handler/3 event body
- 1771507065973503 + ulimit -n event body
- 1771507065973607 + ERL_CRASH_DUMP=/tmp/erl_crash.dump event body
- 1771507066742084 ** (RuntimeError) Failed to detect IP version for DB_HOST: nxdomain event body
- 1771507066742119 (elixir 1.18.4) src/elixir.erl:386: :elixir.eval_external_handler/3 event body
- 1771507066742151 (stdlib 6.2.1) erl_eval.erl:663: :erl_eval.expr/6 event body
- 1771507067736180 + echo 'Setting RLIMIT_NOFILE to 10000' event body
- 1771507067736204 + ERL_CRASH_DUMP=/tmp/erl_crash.dump event body
- 1771507067736572 + [[ -n '' ]] event body
- 1771507068619538 (stdlib 6.2.1) erl_eval.erl:919: :erl_eval.do_apply/7 event body
- 1771507068619545 (stdlib 6.2.1) erl_eval.erl:663: :erl_eval.expr/6 event body
- 1771507069941868 + '[' '!' -z 10000 ']' event body
- 1771507070736559 ** (RuntimeError) Failed to detect IP version for DB_HOST: nxdomain event body
- 1771507070736595 (stdlib 6.2.1) erl_eval.erl:663: :erl_eval.expr/6 event body
- 1771507070736604 (elixir 1.18.4) src/elixir.erl:364: :elixir.eval_forms/4 event body
- 1771507070983955 Crash dump is being written to: /tmp/erl_crash.dump...done event body
- 1771507072867786 + ulimit -n event body
- 1771507072867915 + ERL_CRASH_DUMP=/tmp/erl_crash.dump event body
- 1771507073859349 Crash dump is being written to: /tmp/erl_crash.dump...done event body
- 1771507077343390 Running migrations event body
- 1771507078063161 (stdlib 6.2.1) erl_eval.erl:919: :erl_eval.do_apply/7 event body
- 1771507085027969 + [[ -n '' ]] event body
- 1771507085767005 (stdlib 6.2.1) erl_eval.erl:919: :erl_eval.do_apply/7 event body
- 1771507085767042 (elixir 1.18.4) lib/code.ex:572: Code.validated_eval_string/3 event body
- 1771507099912493 /app/releases/2.72.0/runtime.exs:96: (file) event body
- 1771507126074814 + echo 'Setting RLIMIT_NOFILE to 10000' event body
- 1771507126074852 + ERL_CRASH_DUMP=/tmp/erl_crash.dump event body
- 1771507178780063 + '[' '!' -z 10000 ']' event body
- 1771507178780150 Setting RLIMIT_NOFILE to 10000 event body
- 1771507178780430 + echo 'Running migrations' event body
- 1771507178780438 + sudo -E -u nobody /app/bin/migrate event body
- 1771507179564402 (elixir 1.18.4) lib/module/parallel_checker.ex:120: Module.ParallelChecker.verify/1 event body
- 1771507179564410 (elixir 1.18.4) lib/code.ex:572: Code.validated_eval_string/3 event body
- 1771507240153232 1024 event body
- 1771507240914166 ERROR! Config provider Config.Reader failed with: event body
- 1771507302210131 Runtime terminating during boot ({#{message=><<"Failed to detect IP version for DB_HOST: nxdomain">>,'__struct__'=>'Elixir.RuntimeError','__exception__'=>true},[{elixir_eval,'__FILE__',1,[{file,"/app/releases/2.72.0/runtime.exs"},{line,96}]},{elixir,eval_external_handler,3,[{file,"src/elixir.erl"},{line,386},{error_info,#{module=>'Elixir.Exception'}}]},{erl_eval,do_apply,7,[{file,"erl_eval.erl"},{line,919}]},{erl_eval,expr,6,[{file,"erl_eval.erl"},{line,663}]},{erl_eval,exprs,6,[{file,"erl_eval.erl"},{line,271}]},{elixir,eval_forms,4,[{file,"src/elixir.erl"},{line,364}]},{'Elixir.Module.ParallelChecker',verify,1,[{file,"lib/module/parallel_checker.ex"},{line,120}]},{'Elixir.Code',validated_eval_string,3,[{file,"lib/code.ex"},{line,572}]}]}) event body
- 1771507363496025 Runtime terminating during boot ({#{message=><<"Failed to detect IP version for DB_HOST: nxdomain">>,'__struct__'=>'Elixir.RuntimeError','__exception__'=>true},[{elixir_eval,'__FILE__',1,[{file,"/app/releases/2.72.0/runtime.exs"},{line,96}]},{elixir,eval_external_handler,3,[{file,"src/elixir.erl"},{line,386},{error_info,#{module=>'Elixir.Exception'}}]},{erl_eval,do_apply,7,[{file,"erl_eval.erl"},{line,919}]},{erl_eval,expr,6,[{file,"erl_eval.erl"},{line,663}]},{erl_eval,exprs,6,[{file,"erl_eval.erl"},{line,271}]},{elixir,eval_forms,4,[{file,"src/elixir.erl"},{line,364}]},{'Elixir.Module.ParallelChecker',verify,1,[{file,"lib/module/parallel_checker.ex"},{line,120}]},{'Elixir.Code',validated_eval_string,3,[{file,"lib/code.ex"},{line,572}]}]}) event body
- 1771507363733717 Crash dump is being written to: /tmp/erl_crash.dump...done event body
- 1771507424018140 + '[' '!' -z 10000 ']' event body
- 1771507425119765 Crash dump is being written to: /tmp/erl_crash.dump...done event body
- 1771507486446555 Crash dump is being written to: /tmp/erl_crash.dump...done event body
- 1771507546706913 + ERL_CRASH_DUMP=/tmp/erl_crash.dump event body
- 1771507546707295 Running migrations event body
- 1771507547453390 (stdlib 6.2.1) erl_eval.erl:271: :erl_eval.exprs/6 event body
- 1771507547711055 Crash dump is being written to: /tmp/erl_crash.dump...done event body
- 1771507607988663 + ulimit -n event body
- 1771507607988712 + echo 'Setting RLIMIT_NOFILE to 10000' event body
- 1771507607988719 + ulimit -Sn 10000 event body
- 1771507608743924 (elixir 1.18.4) src/elixir.erl:386: :elixir.eval_external_handler/3 event body
- 1771507608743936 (stdlib 6.2.1) erl_eval.erl:271: :erl_eval.exprs/6 event body
- 1771507669281864 + ulimit -n event body
- 1771507670133612 event body
- 1771507730772712 1024 event body
- 1771507731594672 /app/releases/2.72.0/runtime.exs:96: (file) event body
- 1771507792215771 + ERL_CRASH_DUMP=/tmp/erl_crash.dump event body
- 1771507794561780 info ["$kind": :gen_rpc_dispatcher_start] event body
- 1771507794753955 notice :alarm_handler: {:set, {{:disk_almost_full, ~c"/"}, []}} event body
- 1771507794933176 notice SYN[nonode@nohost] Adding node to scope <Elixir.RegionNodes> event body
- 1771507794951718 notice SYN[nonode@nohost|registry<users_1>] Discovering the cluster event body
- 1771507794952662 notice SYN[nonode@nohost|registry<users_2>] Discovering the cluster event body
- 1771507795497198 notice SYN[nonode@nohost|pg<realtime_postgres_cdc_1>] Discovering the cluster event body
- 1771507795497343 notice SYN[nonode@nohost] Adding node to scope <realtime_postgres_cdc_2> event body
- 1771507795497576 notice SYN[nonode@nohost] Creating tables for scope <realtime_postgres_cdc_2> event body
- 1771507798014648 info ["$kind": :gen_rpc_dispatcher_start] event body
- 1771507798062235 notice SYN[realtime@127.0.0.1|registry<Elixir.RegionNodes>] Discovering the cluster event body
- 1771507798066106 notice SYN[realtime@127.0.0.1|pg<users_4>] Discovering the cluster event body
- 1771507798140605 info Beacon[realtime@127.0.0.1|users] Starting event body
- 1771507798197127 notice SYN[realtime@127.0.0.1] Adding node to scope <realtime_postgres_cdc_2> event body
- 1771507798198621 notice SYN[realtime@127.0.0.1|registry<realtime_postgres_cdc_3>] Discovering the cluster event body
- 1771507798199226 notice SYN[realtime@127.0.0.1] Creating tables for scope <realtime_postgres_cdc_4> event body
random(tips): Match a field with regex (e.g. `m.browser:~"Firefox 5\d"`).