Mercurial > repos > enis > gcp_batch_netcat
changeset 11:fe0bf22037a5 draft
planemo upload for repository https://github.com/afgane/gcp_batch_netcat commit f730cbb207e028a5d4fd982fe65ece7345af4879
| author | enis |
|---|---|
| date | Thu, 14 Aug 2025 16:39:36 +0000 |
| parents | cdfa2e1a7ef4 |
| children | 56543de39954 |
| files | README.md gcp_batch_netcat.py log1.txt |
| diffstat | 3 files changed, 333 insertions(+), 144 deletions(-) [+] |
line wrap: on
line diff
--- a/README.md Tue Aug 12 16:22:41 2025 +0000 +++ b/README.md Thu Aug 14 16:39:36 2025 +0000 @@ -1,36 +1,33 @@ -# GCP Batch Netcat Galaxy Tool +# GCP Batch NFS and CVMFS Connectivity Test Tool -A Galaxy tool that submits a job to Google Cloud Platform (GCP) Batch service to test network connectivity to various services including NFS servers. This tool is predominantly intended for use with Galaxy deployments using the Galaxy Helm chart, where it can verify network connectivity between GCP Batch workers and critical services. +A Galaxy tool that submits a job to Google Cloud Platform (GCP) Batch service to +test comprehensive network connectivity to NFS storage and CVMFS repositories. +This tool is specifically designed for Galaxy deployments using the Galaxy Helm +chart, where it validates connectivity between GCP Batch workers and critical +Galaxy infrastructure services. ## Overview -This tool creates and submits a GCP Batch job that runs comprehensive network connectivity tests using various tools. It's particularly useful for: -- Testing network connectivity between GCP Batch compute nodes and NFS storage -- Validating connectivity to Galaxy web services -- Testing Kubernetes DNS resolution and external connectivity +This tool creates and submits a GCP Batch job that runs comprehensive +connectivity tests using a custom VM image with CVMFS support. It's particularly +useful for: +- Testing network connectivity and mounting capabilities for NFS storage servers +- Validating access to Galaxy's CVMFS reference data repositories +- Verifying NFSv4.2 mounting with proper security contexts +- Testing specific Galaxy reference data file access (e.g., Arabidopsis TAIR10) - Troubleshooting connectivity issues in Galaxy deployments on Kubernetes -- Debugging firewall rules and network configuration - -**New Features:** -- **Auto-Discovery**: Automatically discovers NFS LoadBalancer external IP addresses via Kubernetes API -- **Multiple Test Types**: NFS, Galaxy web, Kubernetes DNS, Google DNS, and custom targets -- **Enhanced Debugging**: Comprehensive network diagnostics with detailed logging +- Debugging firewall rules, NFS export configurations, and CVMFS client setup +- Comprehensive Network Diagnostics: DNS resolution, routing, and external connectivity +- Custom VM Integration: Uses galaxy-k8s-boot-v2025-08-12 image with pre-configured CVMFS client The tool is available in the Main Tool Shed at: https://toolshed.g2.bx.psu.edu/view/enis/gcp_batch_netcat/ ## For use with the Galaxy Helm Chart -This tool is specifically designed for Galaxy deployments using the Galaxy Helm chart on Google Kubernetes Engine (GKE). A sample deployment can be obtained using the [galaxy-k8s-boot repository](https://github.com/galaxyproject/galaxy-k8s-boot/). - -## Auto-Discovery of NFS LoadBalancer - -**Important**: The tool now intelligently discovers the correct NFS IP address for external access: - -1. **LoadBalancer Discovery** (Primary): Uses `kubectl` to find NFS services with LoadBalancer type and external IPs -2. **Mount Detection** (Fallback): Extracts NFS IP from Galaxy's local mount (returns ClusterIP - may not work for external jobs) - -For reliable operation, ensure your NFS server is exposed via a LoadBalancer service. See `NFS_LOADBALANCER_GUIDE.md` for detailed setup instructions. +This tool is specifically designed for Galaxy deployments using the Galaxy Helm +chart. A sample deployment can be obtained using the [galaxy-k8s-boot +repository](https://github.com/galaxyproject/galaxy-k8s-boot/). ## Input Parameters Reference @@ -38,16 +35,6 @@ ### Required Parameters -#### **Test Type** -- **Galaxy Label**: "Test Type" -- **Options**: - - NFS Server (port 2049) - Default - - Galaxy Web Service (port 80/443) - - Kubernetes DNS (kubernetes.default.svc.cluster.local:443) - - Google DNS (8.8.8.8:53) - - Custom Host/Port -- **Description**: Type of connectivity test to perform - #### **GCP Batch Region** - **Galaxy Label**: "GCP Batch Region" - **Description**: The GCP region where the Batch job will be submitted @@ -65,6 +52,13 @@ - **Description**: The name of the subnet in which Galaxy runs - **Examples**: `default` +#### **NFS Server Address** +- **Galaxy Label**: "NFS Server Address" +- **Description**: LoadBalancer external IP address of the NFS service that GCP Batch jobs should connect to. This is typically a private IP within your VPC network that's accessible to Batch jobs but not the public internet. This must be the LoadBalancer external IP, not the ClusterIP that Galaxy pods use internally. +- **Example**: `10.150.0.60` (LoadBalancer external IP - private within VPC) +- **Important**: This is different from the internal ClusterIP (e.g., `10.43.86.187`) that Galaxy pods use +- **How to find**: `kubectl get svc -n nfs-provisioner` and look for the EXTERNAL-IP of the LoadBalancer service + #### **GCP Service Account Key File** - **Galaxy Label**: "GCP Service Account Key File" - **Format**: JSON file @@ -80,13 +74,6 @@ ### Optional Parameters -#### **NFS Server Address** -- **Galaxy Label**: "NFS Server Address" -- **Description**: IP address or hostname of the NFS server to test connectivity to. This is the same address as Galaxy is using. -- **Auto-detection**: If not supplied, the tool attempts to detect the NFS server from Galaxy's database mount. This is the preferred mode of operation. -- **Example**: `10.0.0.100` -- **When to specify**: Use when auto-detection fails or when testing a different NFS server - #### **GCP Project ID** - **Galaxy Label**: "GCP Project ID" - **Description**: The ID of the GCP project where the Batch job should be created @@ -98,9 +85,24 @@ ### What Happens The tool will: -- Submit a lightweight job to GCP Batch in your specified region and network -- Test connectivity to the NFS server on port 2049 using `netcat` -- Return a report showing whether the connection was successful +1. **Submit a GCP Batch job** using a custom VM image (galaxy-k8s-boot-v2025-08-12) with CVMFS client pre-installed +2. **Trigger CVMFS mount** on the host VM by accessing the Galaxy data repository +3. **Run a container** with bind-mounted CVMFS from the host to test comprehensive connectivity +4. **Test NFS connectivity** including: + - Basic network connectivity to port 2049 + - NFS volume mounting via GCP Batch (avoiding container capability issues) + - Directory listing and dynamic PVC discovery + - Search for actual Galaxy directories (jobs_directory, shed_tools, objects, tools, cache) + - Write access testing to verify Batch job permissions +5. **Test CVMFS access** including: + - Repository accessibility and directory listing + - Specific file access (Arabidopsis TAIR10 reference genome) + - Galaxy reference data directory validation +6. **Comprehensive network diagnostics** including: + - DNS resolution testing + - Network routing analysis + - External connectivity verification +7. **Generate detailed report** with success/failure status and troubleshooting information ## Setup Requirements @@ -111,17 +113,21 @@ - A VPC network and subnet where both Galaxy and the NFS server can communicate - A service account with "Batch Job Editor" role - Downloaded JSON key file for the service account +- Access to the custom VM image: e.g., `galaxy-k8s-boot-v2025-08-12` ### Network Configuration -- Firewall rule allowing traffic from the Batch subnet to NFS server on port 2049 for the specified network: +- Firewall rule allowing traffic from the Batch subnet to NFS server (all required NFS ports): ``` gcloud compute firewall-rules create allow-nfs-from-batch \ --network=NETWORK_NAME \ - --allow=tcp:2049 + --allow=tcp:2049,tcp:111,tcp:20048,udp:111,udp:2049 ``` +**Important**: Port 111 (RPC portmapper) is critical for NFS mounting to work. Without it, you'll get "operation not permitted" errors even if port 2049 is accessible. -### NFS server Setup -- The Ganesha NFS service needs to use an internal LoadBalancer +### NFS Server Setup +- The NFS service must be accessible via LoadBalancer with external IP (typically private within VPC) +- NFS server should support NFSv4.2 with sec=sys security +- Exports should be configured to allow access from GCP Batch subnet ``` apiVersion: v1 kind: Service @@ -130,8 +136,44 @@ namespace: nfs-provisioner annotations: cloud.google.com/load-balancer-type: "Internal" - ... spec: type: LoadBalancer - ... + ports: + - name: nfs + port: 2049 + protocol: TCP + - name: rpcbind + port: 111 + protocol: TCP + - name: mountd + port: 20048 + protocol: TCP ``` + +### CVMFS Requirements +- The custom VM image includes a pre-configured CVMFS client +- CVMFS repositories are mounted via autofs on first access +- Galaxy data repository (`data.galaxyproject.org`) should be accessible +- Network connectivity to CVMFS stratum servers + +## Key Features + +### NFS Volume Mounting +The tool uses GCP Batch's native NFS volume support instead of attempting to mount from within containers. This approach: +- **Eliminates capability issues**: No need for privileged containers or CAP_SYS_ADMIN +- **Improves reliability**: Batch handles the mount with full host privileges +- **Simplifies debugging**: Clear separation between network connectivity and mount issues + +### Dynamic Galaxy Directory Discovery +The tool automatically discovers Galaxy installations regardless of PVC structure: +- **Smart PVC detection**: Finds all `pvc-*` directories under `/export` +- **Flexible directory mapping**: Adapts to actual Galaxy directory structure (`jobs_directory`, `shed_tools`, etc.) +- **Comprehensive validation**: Tests both read and write access to Galaxy directories +- **Detailed reporting**: Shows directory contents, sizes, and file counts for verification + +### Comprehensive Testing +- **Network connectivity**: Basic port testing and DNS resolution +- **NFS functionality**: Mount verification and directory access +- **CVMFS access**: Repository mounting and file availability +- **Galaxy structure**: Specific directory and file validation +- **Write permissions**: Actual file creation tests to verify Batch job capabilities
--- a/gcp_batch_netcat.py Tue Aug 12 16:22:41 2025 +0000 +++ b/gcp_batch_netcat.py Thu Aug 14 16:39:36 2025 +0000 @@ -88,7 +88,7 @@ echo "Target: {target_host}:{target_port}" echo "Timestamp: $(date)" echo "Container hostname: $(hostname)" -echo "Host VM Image: galaxy-k8s-boot-v2025-08-10" +echo "Host VM Image: galaxy-k8s-boot-v2025-08-12" echo "Container Image: afgane/gcp-batch-netcat:0.3.0" echo "" @@ -153,65 +153,149 @@ ip route echo "" -# NFS Mount Test -echo "=== NFS Mount Test ===" -MOUNT_POINT="/tmp/nfs_test_mount" -echo "Creating mount point: $MOUNT_POINT" -mkdir -p "$MOUNT_POINT" - -echo "Attempting to mount NFS share..." -echo "Command: mount -t nfs -o vers=3,tcp {target_host}:/ $MOUNT_POINT" - -# Try mounting the NFS share +# NFS Mount Test - Check if Batch mounted it for us +echo "=== NFS Mount Test (via Batch Volume) ===" +NFS_MOUNT_POINT="/mnt/nfs" mount_result=1 -if mount -t nfs -o vers=3,tcp {target_host}:/ "$MOUNT_POINT" 2>&1; then - mount_result=0 - echo "✓ NFS mount successful!" + +echo "Checking if NFS is mounted by Batch at $NFS_MOUNT_POINT..." +if [ -d "$NFS_MOUNT_POINT" ]; then + echo "✓ NFS mount point exists" + + # Check if it's actually mounted + if mount | grep "$NFS_MOUNT_POINT"; then + mount_result=0 + echo "✓ NFS mounted by Batch successfully!" + + echo "" + echo "=== NFS Share Contents ===" + echo "Long listing of NFS share:" + ls -la "$NFS_MOUNT_POINT" 2>/dev/null || echo "Could not list directory contents" + + echo "" + echo "Disk usage of NFS share:" + df -h "$NFS_MOUNT_POINT" 2>/dev/null || echo "Could not get disk usage" + + # Look for export subdirectories + echo "" + echo "=== Looking for export directories ===" + if [ -d "$NFS_MOUNT_POINT/export" ]; then + echo "✓ Found: export directory" + ls -la "$NFS_MOUNT_POINT/export" | head -10 2>/dev/null || echo "Could not list export contents" + + # Look for PVC subdirectories + echo "Looking for PVC directories in export..." + find "$NFS_MOUNT_POINT/export" -name "pvc-*" -type d | head -5 2>/dev/null || echo "No PVC directories found" + else + echo "✗ No export directory found" + fi - echo "" - echo "=== NFS Share Contents ===" - echo "Long listing of NFS share root:" - ls -la "$MOUNT_POINT" 2>/dev/null || echo "Could not list directory contents" + # Try to find common Galaxy directories + echo "" + echo "=== Looking for Galaxy directories ===" + + # First check if they exist directly in the NFS root + galaxy_dirs_in_root=0 + for dir in "jobs_directory" "shed_tools" "objects" "tools" "cache" "config"; do + if [ -d "$NFS_MOUNT_POINT/$dir" ]; then + echo "✓ Found in root: $dir" + ls -la "$NFS_MOUNT_POINT/$dir" | head -5 + galaxy_dirs_in_root=$((galaxy_dirs_in_root + 1)) + fi + done - echo "" - echo "Disk usage of NFS share:" - df -h "$MOUNT_POINT" 2>/dev/null || echo "Could not get disk usage" + if [ $galaxy_dirs_in_root -eq 0 ]; then + echo "✗ No Galaxy directories found in NFS root" + else + echo "✓ Found $galaxy_dirs_in_root Galaxy directories in NFS root" + fi - echo "" - echo "Mount information:" - mount | grep "$MOUNT_POINT" || echo "Mount info not found" + # Then check inside any PVC directories under export + if [ -d "$NFS_MOUNT_POINT/export" ]; then + echo "" + echo "=== Checking PVC directories for Galaxy structure ===" + + # Find all PVC directories + pvc_count=0 + for pvc_dir in $(find "$NFS_MOUNT_POINT/export" -name "pvc-*" -type d 2>/dev/null); do + pvc_count=$((pvc_count + 1)) + echo "" + echo "Checking PVC ($pvc_count): $(basename $pvc_dir)" + echo " Full path: $pvc_dir" - # Try to find common Galaxy directories - echo "" - echo "=== Looking for Galaxy directories ===" - for dir in "database" "database/files" "database/objects" "tools" "shed_tools"; do - if [ -d "$MOUNT_POINT/$dir" ]; then - echo "✓ Found: $dir" - ls -la "$MOUNT_POINT/$dir" | head -10 + # Show directory listing of PVC + echo " Contents:" + ls -la "$pvc_dir" | head -10 | sed 's/^/ /' + + # Check for Galaxy directories inside this PVC + galaxy_dirs_found=0 + for dir in "jobs_directory" "shed_tools" "objects" "tools" "cache" "config" "deps" "tmp"; do + if [ -d "$pvc_dir/$dir" ]; then + echo " ✓ Found Galaxy directory: $dir" + # Show a sample of contents + ls -la "$pvc_dir/$dir" 2>/dev/null | head -3 | sed 's/^/ /' + galaxy_dirs_found=$((galaxy_dirs_found + 1)) + fi + done + + # Check for Galaxy-specific files + galaxy_files_found=0 + for file in "galaxy.yml" "universe_wsgi.ini" "config/galaxy.yml" "results.sqlite" "celery-beat-schedule"; do + if [ -f "$pvc_dir/$file" ]; then + echo " ✓ Found Galaxy file: $file" + galaxy_files_found=$((galaxy_files_found + 1)) + fi + done + + total_indicators=$((galaxy_dirs_found + galaxy_files_found)) + if [ $total_indicators -gt 0 ]; then + echo " 🎯 This PVC contains $galaxy_dirs_found Galaxy directories and $galaxy_files_found Galaxy files" + + # Test write access + test_file="$pvc_dir/.batch_test_file_$(date +%s)" + if echo "test" > "$test_file" 2>/dev/null; then + echo " ✓ Write access confirmed" + rm -f "$test_file" 2>/dev/null + else + echo " ✗ No write access" + fi + + # Test specific Galaxy directories access + if [ -d "$pvc_dir/jobs_directory" ]; then + echo " � Jobs directory details:" + du -sh "$pvc_dir/jobs_directory" 2>/dev/null | sed 's/^/ /' || echo " Could not get size" + job_count=$(find "$pvc_dir/jobs_directory" -mindepth 1 -maxdepth 1 -type d 2>/dev/null | wc -l) + echo " Job subdirectories: $job_count" + fi + + if [ -d "$pvc_dir/shed_tools" ]; then + echo " 🔧 Shed tools directory details:" + du -sh "$pvc_dir/shed_tools" 2>/dev/null | sed 's/^/ /' || echo " Could not get size" + tool_count=$(find "$pvc_dir/shed_tools" -name "*.py" -o -name "*.xml" 2>/dev/null | wc -l) + echo " Tool files (py/xml): $tool_count" + fi + else + echo " ✗ No Galaxy directories or files found in this PVC" + fi + done + + if [ $pvc_count -eq 0 ]; then + echo "✗ No PVC directories found in export" + else + echo "" + echo "📊 Summary: Found $pvc_count PVC directories in export" + fi else - echo "✗ Not found: $dir" + echo "" + echo "✗ No export directory found in NFS mount" fi - done - - echo "" - echo "Unmounting NFS share..." - umount "$MOUNT_POINT" 2>/dev/null && echo "✓ Unmount successful" || echo "✗ Unmount failed" + else + echo "✗ NFS mount point exists but is not mounted" + echo "This suggests Batch volume configuration may be incorrect" + fi else - echo "✗ NFS mount failed" - echo "Mount error details above" - - # Try alternative mount options - echo "" - echo "Trying alternative NFS mount options..." - echo "Command: mount -t nfs -o vers=4,tcp {target_host}:/ $MOUNT_POINT" - if mount -t nfs -o vers=4,tcp {target_host}:/ "$MOUNT_POINT" 2>&1; then - mount_result=0 - echo "✓ NFS v4 mount successful!" - ls -la "$MOUNT_POINT" 2>/dev/null || echo "Could not list directory contents" - umount "$MOUNT_POINT" 2>/dev/null && echo "✓ Unmount successful" || echo "✗ Unmount failed" - else - echo "✗ NFS v4 mount also failed" - fi + echo "✗ NFS mount point $NFS_MOUNT_POINT does not exist" + echo "This suggests Batch volume was not configured" fi # CVMFS Mount Test @@ -235,8 +319,8 @@ ls -la "/cvmfs/data.galaxyproject.org" 2>/dev/null | head -10 || echo "Could not list directory contents" echo "" - echo "Checking for Galaxy reference data directories:" - for dir in "byhand" "location" "tool-data" "genomes"; do + echo "Listing Galaxy reference data directories:" + for dir in "byhand" "managed"; do if [ -d "/cvmfs/data.galaxyproject.org/$dir" ]; then echo "✓ Found CVMFS directory: $dir" ls "/cvmfs/data.galaxyproject.org/$dir" | head -5 2>/dev/null || echo "Could not list contents" @@ -246,6 +330,22 @@ done echo "" + echo "=== CVMFS File Access Test ===" + echo "Testing access to specific Galaxy reference file..." + echo "File: /cvmfs/data.galaxyproject.org/byhand/Arabidopsis_thaliana_TAIR10/seq/Arabidopsis_thaliana_TAIR10.fa.fai" + + CVMFS_TEST_FILE="/cvmfs/data.galaxyproject.org/byhand/Arabidopsis_thaliana_TAIR10/seq/Arabidopsis_thaliana_TAIR10.fa.fai" + if [ -f "$CVMFS_TEST_FILE" ]; then + echo "✓ File exists, reading first 10 lines:" + head "$CVMFS_TEST_FILE" 2>/dev/null || echo "Could not read file contents" + else + echo "✗ File not found" + echo "Checking if parent directories exist:" + [ -d "/cvmfs/data.galaxyproject.org/byhand/Arabidopsis_thaliana_TAIR10" ] && echo " ✓ Arabidopsis_thaliana_TAIR10 directory exists" || echo " ✗ Arabidopsis_thaliana_TAIR10 directory missing" + [ -d "/cvmfs/data.galaxyproject.org/byhand/Arabidopsis_thaliana_TAIR10/seq" ] && echo " ✓ seq directory exists" || echo " ✗ seq directory missing" + fi + + echo "" echo "CVMFS mount information from host:" mount | grep cvmfs || echo "CVMFS mount info not visible from container" else @@ -261,6 +361,7 @@ echo "Expected: /cvmfs from host VM bind-mounted into container" fi + echo "" echo "=== Final Result ===" if [ $nc_result -eq 0 ] && [ $mount_result -eq 0 ]; then @@ -277,7 +378,6 @@ echo "This suggests:" echo "- NFS server is reachable but may not be properly configured" echo "- NFS export permissions may be incorrect" - echo "- NFS version mismatch (tried NFSv3 and NFSv4)" echo "- Firewall may allow port 2049 but block other NFS ports (111, 20048)" if [ $cvmfs_result -eq 0 ]; then echo "✓ CVMFS repository mount was successful" @@ -305,17 +405,56 @@ # Define the job using the Python client library objects logger.info("Building job specification...") - runnable = batch_v1.Runnable() - runnable.container = batch_v1.Runnable.Container() - runnable.container.image_uri = "afgane/gcp-batch-netcat:0.3.0" + + # Escape the test script for use in docker command (outside f-string to avoid backslash issues) + escaped_test_script = test_script.replace("'", "'\"'\"'") + + # Create a host script that triggers CVMFS mount and then runs the container + host_script = f'''#!/bin/bash +set -e +echo "=== Pre-Container Host Script ===" +echo "Timestamp: $(date)" +echo "Host VM Image: galaxy-k8s-boot-v2025-08-12" +echo "Running on host before container starts..." +echo "" + +echo "=== Triggering CVMFS Mount ===" +echo "Checking CVMFS autofs status:" +mount | grep cvmfs || echo "No CVMFS mounts yet" + +echo "" +echo "Triggering CVMFS mount by accessing repository:" +ls /cvmfs/data.galaxyproject.org/ || echo "Could not access CVMFS repository" + +echo "" +echo "After access - checking CVMFS mounts:" +mount | grep cvmfs || echo "Still no CVMFS mounts visible" - # Bind mount /cvmfs from the host VM (which has CVMFS client) into the container - # Use the docker-style volume syntax for bind mounting host paths - runnable.container.volumes = ["/cvmfs:/cvmfs:ro"] +echo "" +echo "Testing specific file access from host:" +if [ -f "/cvmfs/data.galaxyproject.org/byhand/Arabidopsis_thaliana_TAIR10/seq/Arabidopsis_thaliana_TAIR10.fa.fai" ]; then + echo "✓ CVMFS file accessible from host" + head -3 "/cvmfs/data.galaxyproject.org/byhand/Arabidopsis_thaliana_TAIR10/seq/Arabidopsis_thaliana_TAIR10.fa.fai" +else + echo "✗ CVMFS file not accessible from host" +fi + +echo "" +echo "=== Starting Container ===" +echo "Running container with bind-mounted CVMFS and NFS..." - runnable.container.entrypoint = "/bin/bash" - runnable.container.commands = ["-c", test_script] - logger.debug(f"Container config: image={runnable.container.image_uri}, with /cvmfs bind mount from custom VM") +# Run the container with the test script and volume mounts +docker run --rm \\ + -v /cvmfs:/cvmfs:ro \\ + -v /mnt/nfs:/mnt/nfs:rw \\ + afgane/gcp-batch-netcat:0.3.0 \\ + /bin/bash -c '{escaped_test_script}' +''' + + runnable = batch_v1.Runnable() + runnable.script = batch_v1.Runnable.Script() + runnable.script.text = host_script + logger.debug(f"Host script configured to trigger CVMFS mount and run container") task = batch_v1.TaskSpec() task.runnables = [runnable] @@ -324,6 +463,16 @@ task.compute_resource.memory_mib = 1024 logger.debug(f"Compute resources: CPU={task.compute_resource.cpu_milli}m, Memory={task.compute_resource.memory_mib}MiB") + # Configure NFS volume in the task + volume = batch_v1.Volume() + volume.nfs = batch_v1.NFS() + volume.nfs.server = target_host + volume.nfs.remote_path = "/" # Root of the NFS export + volume.mount_path = "/mnt/nfs" + + task.volumes = [volume] + logger.debug(f"NFS volume configured: {target_host}:/ -> /mnt/nfs") + task_group = batch_v1.TaskGroup() task_group.task_count = 1 task_group.parallelism = 1 @@ -344,7 +493,7 @@ instance_policy = batch_v1.AllocationPolicy.InstancePolicy() instance_policy.machine_type = "e2-medium" # Specify machine type for custom image instance_policy.boot_disk = batch_v1.AllocationPolicy.Disk() - instance_policy.boot_disk.image = f"projects/{project_id}/global/images/galaxy-k8s-boot-v2025-08-10" + instance_policy.boot_disk.image = f"projects/{project_id}/global/images/galaxy-k8s-boot-v2025-08-12" instance_policy.boot_disk.size_gb = 99 logger.debug(f"Using custom VM image: {instance_policy.boot_disk.image}")
--- a/log1.txt Tue Aug 12 16:22:41 2025 +0000 +++ b/log1.txt Thu Aug 14 16:39:36 2025 +0000 @@ -1,31 +1,29 @@ -2025-08-11 19:47:27.404 WGST -is only available from another source -2025-08-11 19:47:27.404 WGST -2025-08-11 19:47:27.407 WGST -E: Package 'docker-ce' has no installation candidate -2025-08-11 19:47:27.407 WGST -E: Package 'docker-ce-cli' has no installation candidate -2025-08-11 19:47:27.407 WGST -E: Unable to locate package containerd.io -2025-08-11 19:47:27.407 WGST -E: Couldn't find any package by glob 'containerd.io' -2025-08-11 19:47:27.407 WGST -E: Couldn't find any package by regex 'containerd.io' -2025-08-11 19:47:27.407 WGST -E: Unable to locate package docker-compose-plugin -2025-08-11 19:47:27.408 WGST -Task action/STARTUP/0/0/group0 runnable 0 wait error: exit status 100 -2025-08-11 19:47:27.408 WGST -Task action/STARTUP/0/0/group0 runnable 0 exited with status 100 -2025-08-11 19:47:27.408 WGST -Task action/STARTUP/0/0/group0 runnable 0 execution err: command failed with exitCode 100 -2025-08-11 19:47:27.408 WGST -Task action/STARTUP/0/0/group0 background runnables all exited on their own. -2025-08-11 19:47:27.408 WGST -Task action/STARTUP/0/0/group0: failed runnables [0] -2025-08-11 19:47:27.408 WGST -Scheduler reported task "action/STARTUP/0/0/group0" -2025-08-11 19:47:27.408 WGST -report agent state: metadata:{parent:"projects/526897014808/locations/us-east4" zone:"us-east4-b" instance:"netcat-job-bf1b499-5d135274-e581-45210-group0-0-mpfx" instance_id:447085315223971911 creation_time:{seconds:1754945238 nanos:445714106} creator:"projects/526897014808/regions/us-east4/instanceGroupManagers/netcat-job-bf1b499-5d135274-e581-45210-group0-0" version:"cloud-batch-agent_20250723.00_p00" os_release:{key:"ID" value:"ubuntu"} os_release:{key:"NAME" value:"Ubuntu"} os_release:{key:"VERSION" value:"24.04.3 LTS (Noble Numbat)"} os_release:{key:"VERSION_CODENAME" value:"noble"} os_release:{key:"VERSION_ID" value:"24.04"} machine_type:"e2-highcpu-2"} agent_info:{state:AGENT_RUNNING job_id:"netcat-job-bf1b499-5d135274-e581-45210" user_project_num:526897014808 tasks:{task_id:"action/STARTUP/0/0/group0" task_status:{state:FAILED status_events:{type:"ASSIGNED" description:"task action/STARTUP/0/0/group0 ASSIGNED" event_time:{seconds:1754945238 nanos:621564382} task_state:ASSIGNED} status_events:{type:"RUNNING" description:"task action/STARTUP/0/0/group0 RUNNING" event_time:{seconds:1754945238 nanos:621571338} task_state:RUNNING} status_events:{type:"FAILED" description:"failed runnables [0]" event_time:{seconds:1754945247 nanos:408496955} task_execution:{exit_code:100 stderr_snippet:"e-plugin\nouldn't find any package by regex 'containerd.io'\nE: Unable to locate package docker-compos"} task_state:FAILED}}} report_time:{seconds:1754945247 nanos:408751935} task_group_id:"group0"} agent_timing_info:{boot_time:{seconds:1754945215 nanos:397254370} script_startup_time:{seconds:1754945232 nanos:497254370} agent_startup_time:{seconds:1754945238 nanos:445714106}} -2025-08-11 19:47:27.484 WGST -Server response for instance 447085315223971911: tasks:{task:"action/STARTUP/0/0/group0" status:{state:FAILED status_events:{type:"ASSIGNED" description:"task action/STARTUP/0/0/group0 ASSIGNED" event_time:{seconds:1754945238 nanos:621564382} task_state:ASSIGNED} status_events:{type:"RUNNING" description:"task action/STARTUP/0/0/group0 RUNNING" event_time:{seconds:1754945238 nanos:621571338} task_state:RUNNING} status_events:{type:"FAILED" description:"failed runnables [0]" event_time:{seconds:1754945247 nanos:408496955} task_execution:{exit_code:100 stderr_snippet:"e-plugin\nouldn't find any package by regex 'containerd.io'\nE: Unable to locate package docker-compos"} task_state:FAILED}} intended_state:ASSIGNED job_uid:"netcat-job-bf1b499-5d135274-e581-45210" task_group_id:"group0" location:"us-east4" job_id:"netcat-job-bf1b4994-8ab8-4361-91b3-c7b90ce3a02f"} use_batch_monitored_resource:true. +2025-08-14 14:20:55.181 WGST +✓ Found: export directory +2025-08-14 14:20:55.187 WGST +total 4 +2025-08-14 14:20:55.187 WGST +drwxr-xr-x 3 nobody nogroup 0 Aug 12 15:49 . +2025-08-14 14:20:55.187 WGST +drwxrwsrwx 14 nobody nogroup 4096 Aug 14 15:12 pvc-aa9a2d4e-2066-40ec-85de-8eb13c8cb9a5 +2025-08-14 14:20:55.188 WGST +Looking for PVC directories in export... +2025-08-14 14:21:06.333 WGST +report agent state: metadata:{parent:"projects/526897014808/locations/us-east4" zone:"us-east4-b" instance:"netcat-job-9b31e9b-e30f48bf-ed6a-41470-group0-0-6rs8" instance_id:3148160217266536671 creation_time:{seconds:1755184743 nanos:205126122} creator:"projects/526897014808/regions/us-east4/instanceGroupManagers/netcat-job-9b31e9b-e30f48bf-ed6a-41470-group0-0" version:"cloud-batch-agent_20250723.00_p00" os_release:{key:"ID" value:"ubuntu"} os_release:{key:"NAME" value:"Ubuntu"} os_release:{key:"VERSION" value:"24.04.3 LTS (Noble Numbat)"} os_release:{key:"VERSION_CODENAME" value:"noble"} os_release:{key:"VERSION_ID" value:"24.04"} machine_type:"e2-medium"} agent_info:{state:AGENT_RUNNING job_id:"netcat-job-9b31e9b-e30f48bf-ed6a-41470" user_project_num:526897014808 tasks:{task_id:"action/STARTUP/0/0/group0" task_status:{state:SUCCEEDED status_events:{type:"ASSIGNED" description:"task action/STARTUP/0/0/group0 ASSIGNED" event_time:{seconds:1755184743 nanos:486748362} task_state:ASSIGNED} status_events:{type:"RUNNING" description:"task action/STARTUP/0/0/group0 RUNNING" event_time:{seconds:1755184743 nanos:486756403} task_state:RUNNING} status_events:{type:"SUCCEEDED" description:"succeeded" event_time:{seconds:1755184743 nanos:953481712} task_state:SUCCEEDED}}} tasks:{task_id:"task/netcat-job-9b31e9b-e30f48bf-ed6a-41470-group0-0/0/0" task_status:{state:RUNNING status_events:{type:"ASSIGNED" description:"task task/netcat-job-9b31e9b-e30f48bf-ed6a-41470-group0-0/0/0 ASSIGNED" event_time:{seconds:1755184746 nanos:423966399} task_state:ASSIGNED} status_events:{type:"RUNNING" description:"task task/netcat-job-9b31e9b-e30f48bf-ed6a-41470-group0-0/0/0 RUNNING" event_time:{seconds:1755184746 nanos:423969481} task_state:RUNNING}}} report_time:{seconds:1755184866 nanos:332761176} task_group_id:"group0"} agent_timing_info:{boot_time:{seconds:1755184712 nanos:999352912} script_startup_time:{seconds:1755184734 nanos:429352912} agent_startup_time:{seconds:1755184743 nanos:205126122}} +2025-08-14 14:21:06.416 WGST +Server response for instance 3148160217266536671: tasks:{task:"action/STARTUP/0/0/group0" status:{state:SUCCEEDED status_events:{type:"ASSIGNED" description:"task action/STARTUP/0/0/group0 ASSIGNED" event_time:{seconds:1755184743 nanos:486748362} task_state:ASSIGNED} status_events:{type:"RUNNING" description:"task action/STARTUP/0/0/group0 RUNNING" event_time:{seconds:1755184743 nanos:486756403} task_state:RUNNING} status_events:{type:"SUCCEEDED" description:"succeeded" event_time:{seconds:1755184743 nanos:953481712} task_state:SUCCEEDED}} intended_state:ASSIGNED job_uid:"netcat-job-9b31e9b-e30f48bf-ed6a-41470" task_group_id:"group0" location:"us-east4" job_id:"netcat-job-9b31e9b3-b4ac-4a1c-8eb6-ed0104b17750"} tasks:{task:"task/netcat-job-9b31e9b-e30f48bf-ed6a-41470-group0-0/0/0" status:{state:RUNNING status_events:{type:"ASSIGNED" description:"task task/netcat-job-9b31e9b-e30f48bf-ed6a-41470-group0-0/0/0 ASSIGNED" event_time:{seconds:1755184746 nanos:423966399} task_state:ASSIGNED} status_events:{type:"RUNNING" description:"task task/netcat-job-9b31e9b-e30f48bf-ed6a-41470-group0-0/0/0 RUNNING" event_time:{seconds:1755184746 nanos:423969481} task_state:RUNNING}} intended_state:ASSIGNED job_uid:"netcat-job-9b31e9b-e30f48bf-ed6a-41470" task_group_id:"group0" location:"us-east4" job_id:"netcat-job-9b31e9b3-b4ac-4a1c-8eb6-ed0104b17750"} use_batch_monitored_resource:true. +2025-08-14 14:21:15.570 WGST +/mnt/nfs/export/pvc-aa9a2d4e-2066-40ec-85de-8eb13c8cb9a5 +2025-08-14 14:21:15.570 WGST +2025-08-14 14:21:15.570 WGST +=== Looking for Galaxy directories === +2025-08-14 14:21:15.571 WGST +✗ Not found: database +2025-08-14 14:21:15.571 WGST +✗ Not found: database/files +2025-08-14 14:21:15.571 WGST +✗ Not found: database/objects +2025-08-14 14:21:15.576 WGST +✗ Not found: tools +2025-08-14 14:21:15.576 WGST +✗ Not found: shed_tools