[Opensearch] Opensearch never goes healthy and just crashes #2691
Description
Bug Report
Which version of the demo you are using?
commit 0a84166 (HEAD -> main, origin/main, origin/HEAD)
Symptom
Opensearch does not come up with clean environment, and the logs do not explain why.
Command to start application:
docker compose up -d
docker-compose up --force-recreate --remove-orphans --detach
What is the expected behavior?
Opensearch comes up and is healthy.
What do you expect to see?
What is the actual behavior?
Opensearch never goes healthy.
Reproduce
Could you provide the minimum required steps to resolve the issue you're seeing?
We will close this issue if:
With the application as is, it can start up opensearch and go healthy.
Additional Context
Docker versions:
Client:
Version: 28.5.1
API version: 1.51
Go version: go1.25.2 X:nodwarf5
Git commit: e180ab8ab8
Built: Wed Oct 8 17:17:33 2025
OS/Arch: linux/amd64
Context: default
Server:
Engine:
Version: 28.5.1
API version: 1.51 (minimum version 1.24)
Go version: go1.25.2 X:nodwarf5
Git commit: f8215cc266
Built: Wed Oct 8 17:17:33 2025
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: v2.1.4
GitCommit: 75cb2b7193e4e490e9fbdc236c0e811ccaba3376.m
runc:
Version: 1.3.2
GitCommit:
docker-init:
Version: 0.19.0
GitCommit: de40ad0
Logs:
opensearch | [2025-10-28T03:42:32,136][INFO ][o.o.n.Node ] [demo-node] JVM arguments [-Xshare:auto, -Dopensearch.networkaddress.cache.ttl=60, -Dopensearch.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,CLDR, -Xms1g, -Xmx1g, -XX:+UseG1GC, -XX:G1ReservePercent=25, -XX:InitiatingHeapOccupancyPercent=30, -Djava.io.tmpdir=/tmp/opensearch-10577306881741522463, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, --add-modules=jdk.incubator.vector, -javaagent:agent/opensearch-agent.jar, --add-opens=java.base/java.nio=org.apache.arrow.memory.core,ALL-UNNAMED, -Dorg.apache.lucene.store.MMapDirectory.sharedArenaMaxPermits=1, -Dclk.tck=100, -Djdk.attach.allowAttachSelf=true, -Djava.security.policy=/usr/share/opensearch/config/opensearch-performance-analyzer/opensearch_security.policy, --add-opens=jdk.attach/sun.tools.attach=ALL-UNNAMED, -Dopensearch.cgroups.hierarchy.override=/, -Xms400m, -Xmx400m, -XX:MaxDirectMemorySize=209715200, -Dopensearch.path.home=/usr/share/opensearch, -Dopensearch.path.conf=/usr/share/opensearch/config, -Dopensearch.distribution.type=tar, -Dopensearch.bundled_jdk=true]
opensearch | [2025-10-28T03:42:32,152][WARN ][o.o.p.PluginsService ] [demo-node] Missing plugin [workload-management], dependency of [opensearch-security]
opensearch | [2025-10-28T03:42:32,152][WARN ][o.o.p.PluginsService ] [demo-node] Some features of this plugin may not function without the dependencies being installed.
opensearch |
opensearch | [2025-10-28T03:42:32,233][WARN ][stderr ] [demo-node] Oct 28, 2025 3:42:32 AM org.apache.lucene.internal.vectorization.PanamaVectorizationProvider <init>
opensearch | [2025-10-28T03:42:32,233][WARN ][stderr ] [demo-node] INFO: Java vector incubator API enabled; uses preferredBitSize=512; FMA enabled
opensearch | [2025-10-28T03:42:32,581][INFO ][o.o.s.s.t.SSLConfig ] [demo-node] SSL dual mode is disabled
opensearch | [2025-10-28T03:42:32,582][WARN ][o.o.s.OpenSearchSecurityPlugin] [demo-node] OpenSearch Security plugin installed but disabled. This can expose your configuration (including passwords) to the public.
opensearch | [2025-10-28T03:42:32,612][INFO ][o.o.t.g.GrpcPlugin ] [demo-node] No QueryBuilderProtoConverter extensions found from other plugins
opensearch | [2025-10-28T03:42:32,614][INFO ][o.o.i.r.ReindexModulePlugin] [demo-node] ReindexPlugin reloadSPI called
opensearch | [2025-10-28T03:42:32,614][INFO ][o.o.i.r.ReindexModulePlugin] [demo-node] Unable to find any implementation for RemoteReindexExtension
opensearch | [2025-10-28T03:42:32,617][INFO ][o.o.j.JobSchedulerPlugin ] [demo-node] Loaded scheduler extension: opendistro-index-management, index: .opendistro-ism-config
opensearch | [2025-10-28T03:42:32,618][INFO ][o.o.j.JobSchedulerPlugin ] [demo-node] Loaded scheduler extension: async-query-scheduler, index: .async-query-scheduler
opensearch | [2025-10-28T03:42:32,621][INFO ][o.o.p.PluginsService ] [demo-node] loaded module [aggs-matrix-stats]
opensearch | [2025-10-28T03:42:32,621][INFO ][o.o.p.PluginsService ] [demo-node] loaded module [analysis-common]
opensearch | [2025-10-28T03:42:32,621][INFO ][o.o.p.PluginsService ] [demo-node] loaded module [cache-common]
opensearch | [2025-10-28T03:42:32,621][INFO ][o.o.p.PluginsService ] [demo-node] loaded module [geo]
opensearch | [2025-10-28T03:42:32,621][INFO ][o.o.p.PluginsService ] [demo-node] loaded module [ingest-common]
opensearch | [2025-10-28T03:42:32,621][INFO ][o.o.p.PluginsService ] [demo-node] loaded module [ingest-geoip]
opensearch | [2025-10-28T03:42:32,621][INFO ][o.o.p.PluginsService ] [demo-node] loaded module [ingest-user-agent]
opensearch | [2025-10-28T03:42:32,621][INFO ][o.o.p.PluginsService ] [demo-node] loaded module [lang-expression]
opensearch | [2025-10-28T03:42:32,621][INFO ][o.o.p.PluginsService ] [demo-node] loaded module [lang-mustache]
opensearch | [2025-10-28T03:42:32,621][INFO ][o.o.p.PluginsService ] [demo-node] loaded module [lang-painless]
opensearch | [2025-10-28T03:42:32,621][INFO ][o.o.p.PluginsService ] [demo-node] loaded module [mapper-extras]
opensearch | [2025-10-28T03:42:32,621][INFO ][o.o.p.PluginsService ] [demo-node] loaded module [opensearch-dashboards]
opensearch | [2025-10-28T03:42:32,621][INFO ][o.o.p.PluginsService ] [demo-node] loaded module [parent-join]
opensearch | [2025-10-28T03:42:32,621][INFO ][o.o.p.PluginsService ] [demo-node] loaded module [percolator]
opensearch | [2025-10-28T03:42:32,621][INFO ][o.o.p.PluginsService ] [demo-node] loaded module [rank-eval]
opensearch | [2025-10-28T03:42:32,621][INFO ][o.o.p.PluginsService ] [demo-node] loaded module [reindex]
opensearch | [2025-10-28T03:42:32,621][INFO ][o.o.p.PluginsService ] [demo-node] loaded module [repository-url]
opensearch | [2025-10-28T03:42:32,621][INFO ][o.o.p.PluginsService ] [demo-node] loaded module [rule-framework]
opensearch | [2025-10-28T03:42:32,621][INFO ][o.o.p.PluginsService ] [demo-node] loaded module [search-pipeline-common]
opensearch | [2025-10-28T03:42:32,621][INFO ][o.o.p.PluginsService ] [demo-node] loaded module [store-subdirectory]
opensearch | [2025-10-28T03:42:32,621][INFO ][o.o.p.PluginsService ] [demo-node] loaded module [systemd]
opensearch | [2025-10-28T03:42:32,622][INFO ][o.o.p.PluginsService ] [demo-node] loaded module [transport-grpc]
opensearch | [2025-10-28T03:42:32,622][INFO ][o.o.p.PluginsService ] [demo-node] loaded module [transport-netty4]
opensearch | [2025-10-28T03:42:32,622][INFO ][o.o.p.PluginsService ] [demo-node] loaded plugin [opensearch-index-management]
opensearch | [2025-10-28T03:42:32,622][INFO ][o.o.p.PluginsService ] [demo-node] loaded plugin [opensearch-job-scheduler]
opensearch | [2025-10-28T03:42:32,622][INFO ][o.o.p.PluginsService ] [demo-node] loaded plugin [opensearch-security]
opensearch | [2025-10-28T03:42:32,622][INFO ][o.o.p.PluginsService ] [demo-node] loaded plugin [opensearch-sql]
opensearch | [2025-10-28T03:42:32,671][INFO ][o.o.e.NodeEnvironment ] [demo-node] using [1] data paths, mounts [[/ (overlay)]], net usable_space [2.9tb], net total_space [5.3tb], types [overlay]
opensearch | [2025-10-28T03:42:32,671][INFO ][o.o.e.NodeEnvironment ] [demo-node] heap size [400mb], compressed ordinary object pointers [true]
opensearch | [2025-10-28T03:42:32,688][INFO ][o.o.n.Node ] [demo-node] node name [demo-node], node ID [FAbLADUuS1-Dgpy_wXCm-g], cluster name [demo-cluster], roles [ingest, remote_cluster_client, data, cluster_manager]
opensearch | [2025-10-28T03:42:32,703][INFO ][o.o.e.ExtensionsManager ] [demo-node] ExtensionsManager initialized