Merge pull request #121 from OneBusAway/claude

Add support for launching the server from a local GTFS zip file

Author: aaronbrethorst

.gitignore

@@ -1,3 +1,5 @@
 ci-bundle
 bundle
 .DS_Store
+.claude
+example-local-gtfs/*.zip

CLAUDE.md

@@ -0,0 +1,105 @@
+# CLAUDE.md
+
+This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
+
+## Overview
+
+This repository contains Docker images for running OneBusAway Application Suite v2, a transit data platform. The system consists of three main components:
+
+1. **Bundler Service**: Processes GTFS data and creates transit data bundles
+2. **OBA App Service**: Runs the OneBusAway API and transit data federation webapps
+3. **Database Service**: MySQL (default) or PostgreSQL for storing application data
+
+## Common Commands
+
+### Building and Running
+
+```bash
+# Build the app server
+docker compose build oba_app
+
+# Build a bundle with custom GTFS
+GTFS_URL=https://example.com/gtfs.zip docker compose up oba_bundler
+
+# Build with default test data (Unitrans)
+docker compose up oba_bundler
+
+# Run the OBA server
+docker compose up oba_app
+
+# Run validation tests
+./bin/validate.sh
+
+# Clean up
+docker compose down -v
+```
+
+### Testing
+
+```bash
+# Run validation script to test API endpoints
+./bin/validate.sh
+
+# Build and test Docker images locally
+docker compose build
+docker compose up
+```
+
+## Architecture
+
+### Directory Structure
+
+- `/bundler/`: Docker setup for building transit data bundles from GTFS feeds
+  - Uses Maven to fetch OneBusAway dependencies
+  - Includes gtfstidy for cleaning/optimizing GTFS data
+  - Outputs to `/bundle/` directory
+
+- `/oba/`: Docker setup for the OneBusAway application server
+  - Runs on Tomcat 8.5 with Java 11
+  - Template-based configuration for database connections
+  - Supports GTFS-RT feeds
+  - Includes Prometheus JMX exporter for monitoring
+
+- `/bundle/`: Shared volume containing processed transit data
+  - Contains serialized Java objects (.obj files)
+  - Lucene search indices
+  - Processed GTFS data
+
+### Key Technologies
+
+- **Build System**: Maven-based Java project
+- **OneBusAway Version**: v2.6.0 (configurable via OBA_VERSION)
+- **Runtime**: Tomcat 8.5.100 with JDK 11
+- **Databases**: MySQL 8.0 or PostgreSQL 16
+- **GTFS Processing**: gtfstidy (Go-based optimizer)
+- **Template Engine**: Custom Handlebars renderer (Go)
+
+### Environment Variables
+
+Database configuration:
+- `JDBC_URL`, `JDBC_DRIVER`, `JDBC_USER`, `JDBC_PASSWORD`
+
+GTFS configuration:
+- `GTFS_URL`: URL to GTFS zip file
+- `TZ`: Timezone for the transit agency
+
+GTFS-RT configuration:
+- `TRIP_UPDATES_URL`, `VEHICLE_POSITIONS_URL`, `ALERTS_URL`
+- `REFRESH_INTERVAL`: Update frequency in seconds
+- `AGENCY_ID`: Transit agency identifier
+- `FEED_API_KEY`, `FEED_API_VALUE`: Authentication headers
+
+### API Endpoints
+
+When running locally:
+- API webapp: http://localhost:8080/
+- Example: http://localhost:8080/api/where/agencies-with-coverage.json?key=TEST
+- Transit data federation: http://localhost:8080/onebusaway-transit-data-federation-webapp
+
+### Docker Images
+
+Published to Docker Hub:
+- `opentransitsoftwarefoundation/onebusaway-bundle-builder`
+- `opentransitsoftwarefoundation/onebusaway-api-webapp`
+
+Multi-architecture support: x86_64, ARM64

README.md

@@ -63,6 +63,10 @@ You will then have two web apps available:
 
 When done using this web server, you can use the shell-standard `^C` to exit out and turn it off. If issues persist across runs, you can try using `docker compose down -v` and then `docker compose up oba_app` to refresh the Docker containers and services.
 
+### Using local GTFS files
+
+If you have a local GTFS file instead of downloading from a URL, see the [`example-local-gtfs/`](example-local-gtfs/) directory for a complete example that demonstrates how to build bundles using local GTFS files.
+
 ### Inspecting the database
 
 The Docker Compose database service should remain up after a call of `docker compose up oba_app`. Otherwise, you can always invoke it using `docker compose up oba_database`.

bin/validate.sh

@@ -9,47 +9,82 @@ else
     exit 1
 fi
 
-output=$(curl -s "http://localhost:8080/api/where/agencies-with-coverage.json?key=test" | jq '.data.list[0].agencyId')
+# Get the first agency from agencies-with-coverage
+agency_response=$(curl -s "http://localhost:8080/api/where/agencies-with-coverage.json?key=test")
+agency_count=$(echo "$agency_response" | jq '.data.list | length')
 
-if [[ ! -z "$output" && "$output" == "\"unitrans\"" ]]; then
-    echo "agencies-with-coverage.json endpoint works."
+if [[ "$agency_count" -gt 0 ]]; then
+    echo "agencies-with-coverage.json endpoint works (found $agency_count agencies)."
+    AGENCY_ID=$(echo "$agency_response" | jq -r '.data.list[0].agencyId')
+    echo "Using agency: $AGENCY_ID"
 else
-    echo "Error: agencies-with-coverage.json endpoint is not working: $output"
+    echo "Error: agencies-with-coverage.json endpoint is not working or no agencies found: $agency_count"
     exit 1
 fi
 
-output=$(curl -s "http://localhost:8080/api/where/routes-for-agency/unitrans.json?key=test" | jq '.data.list | length')
-if [[ $output -gt 10 ]]; then
-    echo "routes-for-agency/unitrans.json endpoint works."
+# Get routes for the agency
+routes_response=$(curl -s "http://localhost:8080/api/where/routes-for-agency/${AGENCY_ID}.json?key=test")
+route_count=$(echo "$routes_response" | jq '.data.list | length')
+if [[ "$route_count" -gt 0 ]]; then
+    echo "routes-for-agency/${AGENCY_ID}.json endpoint works (found $route_count routes)."
+    ROUTE_ID=$(echo "$routes_response" | jq -r '.data.list[0].id')
+    echo "Using route: $ROUTE_ID"
 else
-    echo "Error: routes-for-agency/unitrans.json is not working: $output"
+    echo "Error: routes-for-agency/${AGENCY_ID}.json is not working or no routes found: $route_count"
     exit 1
 fi
 
-output=$(curl -s "http://localhost:8080/api/where/stops-for-route/unitrans_C.json?key=test" | jq '.data.entry.routeId')
-if [[ ! -z "$output" && "$output" == "\"unitrans_C\"" ]]; then
-    echo "stops-for-route/unitrans_C.json endpoint works."
+# Get stops for the route
+stops_response=$(curl -s "http://localhost:8080/api/where/stops-for-route/${ROUTE_ID}.json?key=test")
+route_id_check=$(echo "$stops_response" | jq -r '.data.entry.routeId')
+if [[ ! -z "$route_id_check" && "$route_id_check" == "$ROUTE_ID" ]]; then
+    echo "stops-for-route/${ROUTE_ID}.json endpoint works."
+    STOP_ID=$(echo "$stops_response" | jq -r '.data.entry.stopIds[0]')
+    echo "Using stop: $STOP_ID"
 else
-    echo "Error: stops-for-route/unitrans_C.json endpoint is not working: $output"
+    echo "Error: stops-for-route/${ROUTE_ID}.json endpoint is not working: $route_id_check"
     exit 1
 fi
 
-output=$(curl -s "http://localhost:8080/api/where/stop/unitrans_22182.json?key=test" | jq '.data.entry.code')
-if [[ ! -z "$output" && "$output" == "\"22182\"" ]]; then
-    echo "stop/unitrans_22182.json endpoint works."
+# Get stop details
+stop_response=$(curl -s "http://localhost:8080/api/where/stop/${STOP_ID}.json?key=test")
+stop_id_check=$(echo "$stop_response" | jq -r '.data.entry.id')
+if [[ ! -z "$stop_id_check" && "$stop_id_check" == "$STOP_ID" ]]; then
+    echo "stop/${STOP_ID}.json endpoint works."
+    # Extract coordinates for stops-for-location test
+    STOP_LAT=$(echo "$stop_response" | jq -r '.data.entry.lat')
+    STOP_LON=$(echo "$stop_response" | jq -r '.data.entry.lon')
+    echo "Using coordinates: $STOP_LAT, $STOP_LON"
 else
-    echo "Error: stop/unitrans_22182.json endpoint is not working: $output"
+    echo "Error: stop/${STOP_ID}.json endpoint is not working: $stop_id_check"
     exit 1
 fi
 
-output=$(curl -s "http://localhost:8080/api/where/stops-for-location.json?lat=38.555308&lon=-121.735991&key=test" | jq '.data.outOfRange')
-if [[ ! -z "$output" && "$output" == "false" ]]; then
-    echo "stops-for-location/unitrans_false.json endpoint works."
+# Test stops-for-location using coordinates from the stop
+LOCATION_URL="http://localhost:8080/api/where/stops-for-location.json?lat=${STOP_LAT}&lon=${STOP_LON}&key=test"
+location_response=$(curl -s "$LOCATION_URL")
+out_of_range=$(echo "$location_response" | jq '.data.outOfRange')
+stops_found=$(echo "$location_response" | jq '.data.list | length')
+if [[ ! -z "$out_of_range" && "$out_of_range" == "false" && "$stops_found" -gt 0 ]]; then
+    echo "stops-for-location.json endpoint works (found $stops_found stops)."
 else
-    echo "Error: stops-for-location/unitrans_false.json endpoint is not working: $output"
+    echo "Error: stops-for-location.json endpoint is not working: outOfRange=$out_of_range, stops=$stops_found"
+    echo "URL: $LOCATION_URL"
+    echo "Response: $location_response"
     exit 1
 fi
 
-# todo: add support for arrivals-and-departures-for-stop endpoint.
-# however, it doesn't seem that the unitrans_22182 stop has arrivals and departures on the weekend, so we'll need
-# something else to test with. However, for now, this is still a great step forward.
+# Test arrivals-and-departures-for-stop endpoint
+arrivals_response=$(curl -s "http://localhost:8080/api/where/arrivals-and-departures-for-stop/${STOP_ID}.json?key=test")
+arrivals_stop_id=$(echo "$arrivals_response" | jq -r '.data.entry.stopId')
+arrivals_count=$(echo "$arrivals_response" | jq '.data.entry.arrivalsAndDepartures | length // 0')
+
+if [[ "$arrivals_stop_id" == "$STOP_ID" ]]; then
+    if [[ "$arrivals_count" -gt 0 ]]; then
+        echo "arrivals-and-departures-for-stop/${STOP_ID}.json endpoint works (found $arrivals_count arrivals/departures)."
+    else
+        echo "arrivals-and-departures-for-stop/${STOP_ID}.json endpoint works but no arrivals/departures at this time."
+    fi
+else
+    echo "Warning: arrivals-and-departures-for-stop/${STOP_ID}.json endpoint may not be working correctly."
+fi

bundler/build_bundle.sh

@@ -16,8 +16,14 @@
 # limitations under the License.
 #
 
-if [ -z "$GTFS_URL" ]; then
-    echo "GTFS_URL is not set"
+# Check that either GTFS_URL or GTFS_ZIP_FILENAME is set, but not both
+if [ -n "$GTFS_URL" ] && [ -n "$GTFS_ZIP_FILENAME" ]; then
+    echo "Error: Both GTFS_URL and GTFS_ZIP_FILENAME are set. Please provide only one."
+    exit 1
+fi
+
+if [ -z "$GTFS_URL" ] && [ -z "$GTFS_ZIP_FILENAME" ]; then
+    echo "Error: Neither GTFS_URL nor GTFS_ZIP_FILENAME is set. Please provide one."
     exit 1
 fi
 
@@ -35,17 +41,33 @@ TDF_BUILDER_JAR=${TDF_BUILDER_JAR:-/oba/libs/onebusaway-transit-data-federation-
 # -D: drop erroneous entries from feed
 GTFS_TIDY_ARGS=${GTFS_TIDY_ARGS:-OscRCSmeD}
 
-GTFS_ZIP_FILENAME="gtfs_pristine.zip"
+# Set default filename if using GTFS_URL
+if [ -n "$GTFS_URL" ]; then
+    GTFS_ZIP_FILENAME="gtfs_pristine.zip"
+fi
 
 echo "OBA Bundle Builder Starting"
-echo "GTFS_URL: $GTFS_URL"
+if [ -n "$GTFS_URL" ]; then
+    echo "GTFS_URL: $GTFS_URL"
+else
+    echo "GTFS_ZIP_FILENAME: $GTFS_ZIP_FILENAME"
+fi
 echo "OBA Version: $OBA_VERSION"
 echo "GTFS Tidy Args: $GTFS_TIDY_ARGS"
 echo "TDF_BUILDER_JAR: $TDF_BUILDER_JAR"
 
 cd /bundle
 
-wget -O ${GTFS_ZIP_FILENAME} ${GTFS_URL}
+# Download GTFS file if URL is provided, otherwise use local file
+if [ -n "$GTFS_URL" ]; then
+    wget -O ${GTFS_ZIP_FILENAME} ${GTFS_URL}
+else
+    # Check if the local file exists
+    if [ ! -f "$GTFS_ZIP_FILENAME" ]; then
+        echo "Error: GTFS file not found: $GTFS_ZIP_FILENAME"
+        exit 1
+    fi
+fi
 
 gtfstidy -${GTFS_TIDY_ARGS} ${GTFS_ZIP_FILENAME}
 

example-local-gtfs/.dockerignore

@@ -0,0 +1,10 @@
+# Ignore everything except GTFS files and Docker-related files
+*
+!*.zip
+!*.gtfs
+!Dockerfile
+!docker-entrypoint.sh
+
+# But still ignore these even if they're zip files
+!backup*.zip
+!old*.zip

example-local-gtfs/Dockerfile

@@ -0,0 +1,27 @@
+# Example Dockerfile for using local GTFS files with OneBusAway
+# This builds on top of the main OBA Docker image
+
+# First, build the base OBA image if not already built:
+# docker build -t oba-base:latest -f ../oba/Dockerfile ../oba
+
+FROM oba-base:latest
+
+# Copy your local GTFS file into the container
+# We copy to /tmp first because /bundle gets mounted over
+COPY *.zip /tmp/
+
+# Set the GTFS_ZIP_FILENAME environment variable
+# This tells the build_bundle.sh script to use the local file instead of downloading
+ENV GTFS_ZIP_FILENAME=gtfs.zip
+
+# Copy the entrypoint script
+COPY docker-entrypoint.sh /docker-entrypoint.sh
+RUN chmod +x /docker-entrypoint.sh
+
+ENTRYPOINT ["/docker-entrypoint.sh"]
+
+# Optional: Override other environment variables if needed
+# ENV TZ=America/Los_Angeles
+ENV GTFS_TIDY_ARGS=OscRCSmeD
+
+# The entrypoint and other configurations are inherited from the base image