[RAGFlow] Connection refused on localhost port 9200 when bringing up RS docker image on Ubuntu EC2 instance

[bepisorconk]

Hi,
I am new to EC2. When bringing up the docker container for ES, I am seeing this error:

ubuntu@ip-<>:~/ragflow/docker$ sudo docker inspect --format "{{json .State.Health }}" ragflow-es-01 | jq
{
  "Status": "starting",
  "FailingStreak": 0,
  "Log": [
    {
      "Start": "2024-06-23T03:59:52.150793945Z",
      "End": "2024-06-23T03:59:59.146252217Z",
      "ExitCode": 7,
      "Output": "  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current\n                                 Dload  Upload   Total   Spent    Left  Speed\n\r  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0\r  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0\ncurl: (7) Failed to connect to localhost port 9200: Connection refused\n"
    },...
  ]
}

Some System info:
EC2 Instance: ubuntu-noble-24.04-amd64-server
vm.max_map_count = 262144

Docker info:
STACK_VERSION=8.11.3
ES_PORT=1200
CLUSTER_NAME=rag_flow
MEM_LIMIT=8073741824

  es01:
    container_name: ragflow-es-01
    image: docker.elastic.co/elasticsearch/elasticsearch:${STACK_VERSION}
    volumes:
      - esdata01:/usr/share/elasticsearch/data
    ports:
      - ${ES_PORT}:9200
    environment:
      - node.name=es01
      - cluster.name=${CLUSTER_NAME}
      - cluster.initial_master_nodes=es01
      - ELASTIC_PASSWORD=${ELASTIC_PASSWORD}
      - bootstrap.memory_lock=false
      - xpack.security.enabled=false
      - cluster.max_shards_per_node=4096
      - TZ=${TIMEZONE}
    mem_limit: ${MEM_LIMIT}
    ulimits:
      memlock:
        soft: -1
        hard: -1
    healthcheck:
      test: ["CMD-SHELL", "curl http://localhost:9200"]
      interval: 10s
      timeout: 10s
      retries: 120
    networks:
      - ragflow
    restart: always

I have already modified the security group settings to allow All TCP, SSH, HTTP, HTTPS (both inbound and outbound) [NOTE- ONLY TEMPORARILY FOR TRIAGE]. However the port 9200 isn't open.

ubuntu@ip-~/ragflow/docker$ nmap -p- localhost
Starting Nmap 7.94SVN ( https://nmap.org ) at 2024-06-22 07:12 UTC
Nmap scan report for localhost (127.0.0.1)
Host is up (0.029s latency).
Not shown: 65528 closed tcp ports (conn-refused)
PORT     STATE SERVICE
22/tcp   open  ssh
80/tcp   open  http
1200/tcp open  scol
5455/tcp open  apc-5455
6379/tcp open  redis
9000/tcp open  cslistener
9001/tcp open  tor-orport

Nmap done: 1 IP address (1 host up) scanned in 35.03 seconds

These are the docker ps that are running:

ubuntu@ip-<>:~/ragflow/docker$ sudo docker compose ps
nmap WARN[0001] The "MACOS" variable is not set. Defaulting to a blank string. 
NAME            IMAGE                                                  COMMAND                  SERVICE   CREATED          STATUS                                 PORTS
ragflow-es-01   docker.elastic.co/elasticsearch/elasticsearch:8.11.3   "/bin/tini -- /usr/l…"   es01      55 minutes ago   Up About a minute (health: starting)   9300/tcp, 0.0.0.0:1200->9200/tcp, :::1200->9200/tcp
ragflow-minio   quay.io/minio/minio:RELEASE.2023-12-20T01-00-02Z       "/usr/bin/docker-ent…"   minio     22 hours ago     Up About a minute                      0.0.0.0:9000-9001->9000-9001/tcp, :::9000-9001->9000-9001/tcp
ragflow-mysql   mysql:5.7.18                                           "docker-entrypoint.s…"   mysql     22 hours ago     Up About a minute (healthy)            0.0.0.0:5455->3306/tcp, :::5455->3306/tcp
ragflow-redis   redis:7.2.4                                            "docker-entrypoint.s…"   redis     22 hours ago     Up About a minute                      0.0.0.0:6379->6379/tcp, :::6379->6379/tcp

docker compose logs just state this:

ragflow-es-01  | Created elasticsearch keystore in /usr/share/elasticsearch/config/elasticsearch.keystore
ragflow-es-01  | 
ragflow-es-01  | ERROR: Elasticsearch exited unexpectedly, with exit code 137

This is the elastisearch.yml from within the container

root@aefbd24acf01:/usr/share/elasticsearch/config# cat elasticsearch.yml 
cluster.name: "docker-cluster"
network.host: 0.0.0.0

And these are the logs from within the docker container itself:

root@88edb4308cdb:/usr/share/elasticsearch# cat /usr/share/elasticsearch/logs/gc.log.14
[2024-06-23T04:16:33.223+0000][143][info][gc,init] CardTable entry size: 512
[2024-06-23T04:16:33.345+0000][143][info][gc     ] Using G1
[2024-06-23T04:16:33.378+0000][143][info][gc,init] Version: 21.0.1+12-29 (release)
[2024-06-23T04:16:33.378+0000][143][info][gc,init] CPUs: 1 total, 1 available
[2024-06-23T04:16:33.378+0000][143][info][gc,init] Memory: 957M
[2024-06-23T04:16:33.378+0000][143][info][gc,init] Large Page Support: Disabled
[2024-06-23T04:16:33.378+0000][143][info][gc,init] NUMA Support: Disabled
[2024-06-23T04:16:33.378+0000][143][info][gc,init] Compressed Oops: Enabled (32-bit)
[2024-06-23T04:16:33.378+0000][143][info][gc,init] Heap Region Size: 1M
[2024-06-23T04:16:33.378+0000][143][info][gc,init] Heap Min Capacity: 8M
[2024-06-23T04:16:33.378+0000][143][info][gc,init] Heap Initial Capacity: 16M
[2024-06-23T04:16:33.378+0000][143][info][gc,init] Heap Max Capacity: 240M
[2024-06-23T04:16:33.378+0000][143][info][gc,init] Pre-touch: Disabled
[2024-06-23T04:16:33.378+0000][143][info][gc,init] Parallel Workers: 1
[2024-06-23T04:16:33.378+0000][143][info][gc,init] Concurrent Workers: 1
[2024-06-23T04:16:33.378+0000][143][info][gc,init] Concurrent Refinement Workers: 1
[2024-06-23T04:16:33.378+0000][143][info][gc,init] Periodic GC: Disabled
[2024-06-23T04:16:34.461+0000][143][info][gc,metaspace] CDS archive(s) mapped at: [0x0000762653000000-0x0000762653ca5000-0x0000762653ca5000), size 13258752, SharedBaseAddress: 0x0000762653000000, ArchiveRelocationMode: 1.
[2024-06-23T04:16:34.461+0000][143][info][gc,metaspace] Compressed class space mapped at: 0x0000762654000000-0x0000762694000000, reserved size: 1073741824
[2024-06-23T04:16:34.461+0000][143][info][gc,metaspace] Narrow klass base: 0x0000762653000000, Narrow klass shift: 0, Narrow klass range: 0x100000000
[2024-06-23T04:16:36.370+0000][143][info][safepoint   ] Safepoint "Cleanup", Time since last: 1037759845 ns, Reaching safepoint: 75738001 ns, Cleanup: 21623 ns, At safepoint: 10956 ns, Total: 75770580 ns
[2024-06-23T04:16:37.571+0000][143][info][safepoint   ] Safepoint "Cleanup", Time since last: 1200769188 ns, Reaching safepoint: 148555 ns, Cleanup: 23065 ns, At safepoint: 6308 ns, Total: 177928 ns
[2024-06-23T04:16:37.810+0000][143][info][gc,heap,exit] Heap
[2024-06-23T04:16:37.810+0000][143][info][gc,heap,exit]  garbage-first heap   total 18432K, used 2077K [0x00000000f1000000, 0x0000000100000000)
[2024-06-23T04:16:37.810+0000][143][info][gc,heap,exit]   region size 1024K, 1 young (1024K), 0 survivors (0K)
[2024-06-23T04:16:37.810+0000][143][info][gc,heap,exit]  Metaspace       used 200K, committed 448K, reserved 1114112K
[2024-06-23T04:16:37.810+0000][143][info][gc,heap,exit]   class space    used 5K, committed 128K, reserved 1048576K
root@88edb4308cdb:/usr/share/elasticsearch# cat /usr/share/elasticsearch/logs/gc.log.13
[2024-06-23T04:14:12.909+0000][189][info][gc,init] CardTable entry size: 512
[2024-06-23T04:14:12.945+0000][189][info][gc     ] Using G1
root@88edb4308cdb:/usr/share/elasticsearch# cat /usr/share/elasticsearch/logs/gc.log.12
[2024-06-23T04:14:08.125+0000][165][info][gc,init] CardTable entry size: 512
[2024-06-23T04:14:08.164+0000][165][info][gc     ] Using G1
[2024-06-23T04:14:08.206+0000][165][info][gc,init] Version: 21.0.1+12-29 (release)
[2024-06-23T04:14:08.206+0000][165][info][gc,init] CPUs: 1 total, 1 available
[2024-06-23T04:14:08.206+0000][165][info][gc,init] Memory: 957M
[2024-06-23T04:14:08.206+0000][165][info][gc,init] Large Page Support: Disabled
[2024-06-23T04:14:08.206+0000][165][info][gc,init] NUMA Support: Disabled
[2024-06-23T04:14:08.206+0000][165][info][gc,init] Compressed Oops: Enabled (32-bit)
[2024-06-23T04:14:08.206+0000][165][info][gc,init] Heap Region Size: 1M
[2024-06-23T04:14:08.206+0000][165][info][gc,init] Heap Min Capacity: 382M
[2024-06-23T04:14:08.207+0000][165][info][gc,init] Heap Initial Capacity: 382M
[2024-06-23T04:14:08.207+0000][165][info][gc,init] Heap Max Capacity: 382M
[2024-06-23T04:14:08.207+0000][165][info][gc,init] Pre-touch: Disabled
[2024-06-23T04:14:08.207+0000][165][info][gc,init] Parallel Workers: 1
[2024-06-23T04:14:08.207+0000][165][info][gc,init] Concurrent Workers: 1
[2024-06-23T04:14:08.207+0000][165][info][gc,init] Concurrent Refinement Workers: 1
[2024-06-23T04:14:08.207+0000][165][info][gc,init] Periodic GC: Disabled
[2024-06-23T04:14:08.525+0000][165][info][gc,metaspace] CDS archive(s) mapped at: [0x000070907f000000-0x000070907fca5000-0x000070907fca5000), size 13258752, SharedBaseAddress: 0x000070907f000000, ArchiveRelocationMode: 1.
[2024-06-23T04:14:08.525+0000][165][info][gc,metaspace] Compressed class space mapped at: 0x0000709080000000-0x00007090c0000000, reserved size: 1073741824
[2024-06-23T04:14:08.525+0000][165][info][gc,metaspace] Narrow klass base: 0x000070907f000000, Narrow klass shift: 0, Narrow klass range: 0x100000000
[2024-06-23T04:14:10.480+0000][165][info][safepoint   ] Safepoint "Cleanup", Time since last: 1118812583 ns, Reaching safepoint: 79697048 ns, Cleanup: 11676 ns, At safepoint: 11010 ns, Total: 79719734 ns
[2024-06-23T04:14:11.640+0000][165][info][safepoint   ] Safepoint "Cleanup", Time since last: 1080013631 ns, Reaching safepoint: 79996268 ns, Cleanup: 14267 ns, At safepoint: 10853 ns, Total: 80021388 ns
[2024-06-23T04:14:12.000+0000][165][info][gc,heap,exit] Heap
[2024-06-23T04:14:12.000+0000][165][info][gc,heap,exit]  garbage-first heap   total 391168K, used 3101K [0x00000000e8200000, 0x0000000100000000)
[2024-06-23T04:14:12.000+0000][165][info][gc,heap,exit]   region size 1024K, 2 young (2048K), 0 survivors (0K)
[2024-06-23T04:14:12.000+0000][165][info][gc,heap,exit]  Metaspace       used 190K, committed 384K, reserved 1114112K
[2024-06-23T04:14:12.000+0000][165][info][gc,heap,exit]   class space    used 5K, committed 128K, reserved 1048576K
root@88edb4308cdb:/usr/share/elasticsearch# cat /usr/share/elasticsearch/logs/gc.log.11
[2024-06-23T04:14:04.772+0000][146][info][gc,init] CardTable entry size: 512
[2024-06-23T04:14:04.932+0000][146][info][gc     ] Using G1
[2024-06-23T04:14:05.011+0000][146][info][gc,init] Version: 21.0.1+12-29 (release)
[2024-06-23T04:14:05.011+0000][146][info][gc,init] CPUs: 1 total, 1 available
[2024-06-23T04:14:05.011+0000][146][info][gc,init] Memory: 957M
[2024-06-23T04:14:05.011+0000][146][info][gc,init] Large Page Support: Disabled
[2024-06-23T04:14:05.011+0000][146][info][gc,init] NUMA Support: Disabled
[2024-06-23T04:14:05.011+0000][146][info][gc,init] Compressed Oops: Enabled (32-bit)
[2024-06-23T04:14:05.011+0000][146][info][gc,init] Heap Region Size: 1M
[2024-06-23T04:14:05.011+0000][146][info][gc,init] Heap Min Capacity: 8M
[2024-06-23T04:14:05.011+0000][146][info][gc,init] Heap Initial Capacity: 16M
[2024-06-23T04:14:05.011+0000][146][info][gc,init] Heap Max Capacity: 240M
[2024-06-23T04:14:05.011+0000][146][info][gc,init] Pre-touch: Disabled
[2024-06-23T04:14:05.011+0000][146][info][gc,init] Parallel Workers: 1
[2024-06-23T04:14:05.011+0000][146][info][gc,init] Concurrent Workers: 1
[2024-06-23T04:14:05.011+0000][146][info][gc,init] Concurrent Refinement Workers: 1
[2024-06-23T04:14:05.011+0000][146][info][gc,init] Periodic GC: Disabled
[2024-06-23T04:14:05.492+0000][146][info][gc,metaspace] CDS archive(s) mapped at: [0x00007798f3000000-0x00007798f3ca5000-0x00007798f3ca5000), size 13258752, SharedBaseAddress: 0x00007798f3000000, ArchiveRelocationMode: 1.
[2024-06-23T04:14:05.492+0000][146][info][gc,metaspace] Compressed class space mapped at: 0x00007798f4000000-0x0000779934000000, reserved size: 1073741824
[2024-06-23T04:14:05.492+0000][146][info][gc,metaspace] Narrow klass base: 0x00007798f3000000, Narrow klass shift: 0, Narrow klass range: 0x100000000
[2024-06-23T04:14:07.049+0000][146][info][safepoint   ] Safepoint "Cleanup", Time since last: 1037539805 ns, Reaching safepoint: 40996801 ns, Cleanup: 12533 ns, At safepoint: 40104178 ns, Total: 81113512 ns
[2024-06-23T04:14:07.366+0000][146][info][gc,heap,exit] Heap
[2024-06-23T04:14:07.366+0000][146][info][gc,heap,exit]  garbage-first heap   total 18432K, used 2077K [0x00000000f1000000, 0x0000000100000000)
[2024-06-23T04:14:07.366+0000][146][info][gc,heap,exit]   region size 1024K, 1 young (1024K), 0 survivors (0K)
[2024-06-23T04:14:07.366+0000][146][info][gc,heap,exit]  Metaspace       used 200K, committed 448K, reserved 1114112K
[2024-06-23T04:14:07.366+0000][146][info][gc,heap,exit]   class space    used 5K, committed 128K, reserved 1048576K
root@88edb4308cdb:/usr/share/elasticsearch# cat /usr/share/elasticsearch/logs/gc.log.10
[2024-06-23T04:11:42.428+0000][178][info][gc,init] CardTable entry size: 512
[2024-06-23T04:11:42.787+0000][178][info][gc     ] Using G1

Am I missing something obvious? I am new to dockers and EC2.

Would appreciate any help.
Thank you

[aopstudio]

Your 'es-01' service is not started. It is constantly starting. Maybe it's because your memory is too small. You need at least 16GB of memory to run RAGFlow.
By the way, 9200 is the port number in Docker container. If you want to connect the service outside the Docker, you should use 1200, which is set by ES_PORT in .env file