Debug Tests

This guide explains how to troubleshoot and debug Garden Linux tests when they fail or behave unexpectedly.

Debugging Approaches

The test framework provides several debugging approaches:

• Debug logs - Understand what the test framework and plugins are doing internally
• Login scripts - Interactively investigate the test environment (QEMU VMs or cloud instances)
• Dev mode - Rapidly iterate on tests with automatic synchronization
• Manual test execution - Run tests directly inside the test environment

Choose the approach based on your needs: inspect framework behavior, investigate system state, or manually verify test conditions.

Debug Logs

Enable debug-level logging to see detailed information about plugin operations, system interactions, and test execution flow.

Enable Debug Logging

Enable debug logging for all components:

./test --test-args "--log-cli-level=DEBUG" .build/image.raw

Enable debug logging for a specific test:

./test --test-args "--log-cli-level=DEBUG test_ssh.py" .build/image.raw

:::important Tests need to implement debug output. Results may vary depending on the test. :::

Debug Output in Tests

For information on adding debug output to tests, see Developing Tests.

Login Scripts

Use login scripts to access running VMs for interactive investigation. Once connected, you can explore the filesystem, check service status, examine logs, and run tests manually.

QEMU Environment

Connect to a running QEMU VM:

# Get an SSH session
./util/login_qemu.sh

# Get an SSH session with custom user
./util/login_qemu.sh --user admin

# Execute a command directly
./util/login_qemu.sh "uname -a"

# Run tests manually after login
cd /run/gardenlinux-tests && ./run_tests --system-booted --allow-system-modifications --expected-users gardenlinux

# Run tests with sudo if you need root privileges
cd /run/gardenlinux-tests && sudo ./run_tests --system-booted --allow-system-modifications --expected-users gardenlinux

:::important Login to QEMU VMs is only possible if --ssh --skip-cleanup is passed when starting the test. Secure Shell Daemon (SSHD) is reachable on 127.0.0.1:2222 with the user gardenlinux. The QEMU VM stays open in the shell that started the test and can be stopped with ctrl + c. :::

Common debugging workflow:

1. Start tests with --ssh --skip-cleanup to keep the VM running
2. In a separate terminal, use ./util/login_qemu.sh to connect
3. Investigate the system state, check logs, or run tests manually
4. Return to the original terminal and stop the VM with ctrl + c when done

Cloud Environment

Connect to a cloud VM:

# Basic SSH connection
./util/login_cloud.sh .build/aws-gardener_prod-amd64-today-13371337.raw

# Execute a command directly
./util/login_cloud.sh .build/aws-gardener_prod-amd64-today-13371337.raw "uname -a"

# Run tests manually after login
cd /run/gardenlinux-tests && ./run_tests --system-booted --allow-system-modifications --expected-users gardenlinux

# Run tests with sudo if you need root privileges
cd /run/gardenlinux-tests && sudo ./run_tests --system-booted --allow-system-modifications --expected-users gardenlinux

:::important Cloud VMs use the SSH user and Internet Protocol (IP) address from the OpenTofu output. The login script automatically retrieves this information from the Terraform state. :::

Common debugging workflow:

1. Start tests with --skip-cleanup to keep the cloud instance running
2. Use ./util/login_cloud.sh with the image file to connect
3. Investigate the system state, check logs, or run tests manually
4. Re-run tests without --skip-cleanup or use --only-cleanup to clean up resources when done

Dev Mode (Rapid Iteration)

Use the --dev flag to quickly iterate on tests with automatic file synchronization and re-execution.

QEMU Dev Mode

Start a QEMU VM in dev mode:

./test --dev .build/aws-gardener_prod-amd64-today-local.raw

With specific tests:

./test --dev --test-args "test_ssh.py -v" .build/aws-gardener_prod-amd64-today-local.raw

What happens:

1. Starts a QEMU VM
2. Syncs .build/dist.tar.gz into the VM
3. Syncs the current tests/ tree
4. Runs pytest inside the VM with the provided --test-args
5. Waits for file changes in tests/ and features/
6. Re-syncs changed tests into the VM
7. Re-runs pytest with your given arguments after each change

Stop watch mode at any time with Ctrl+C.

:::note --dev is shorthand for --ssh --skip-cleanup --skip-tests --watch. :::

Cloud Dev Mode

Start a cloud VM in dev mode:

./test --dev --cloud azure .build/azure-gardener_prod-amd64-today-local.raw

Or use an already uploaded image:

./test --dev --cloud azure \
  --cloud-image --image-requirements-file .build/azure-gardener_prod-amd64-today-local.requirements \
  /CommunityGalleries/gardenlinux-13e998fe-534d-4b0a-8a27-f16a73aef620/Images/gardenlinux-nvme-gardener_prod-amd64/Versions/2150.0.0

With specific tests:

./test --dev --cloud azure --test-args "test_ssh.py -v" .build/image.raw

What happens:

1. Deploys a cloud VM
2. Syncs .build/dist.tar.gz into the VM
3. Syncs the current tests/ tree
4. Runs pytest inside the VM with the provided --test-args
5. Waits for file changes in tests/ and features/
6. Re-syncs changed tests into the VM
7. Re-runs pytest with your given arguments after each change

Stop watch mode at any time with Ctrl+C.

:::note --dev is shorthand for --skip-cleanup --skip-tests --watch in cloud mode. :::

Chroot Dev Mode

The --dev flag works the same way in chroot environments:

./test --dev .build/image.tar

Common Debugging Scenarios

Test Fails Unexpectedly

1. Enable debug logging to see what is happening:

   ./test --test-args "--log-cli-level=DEBUG test_failing.py" .build/image.raw

2. Run the test in isolation to rule out dependencies:

   ./test --test-args "test_failing.py::test_specific_function -v" .build/image.raw

3. Use dev mode to iterate quickly:

   ./test --dev --test-args "test_failing.py -v" .build/image.raw

Test Passes Locally But Fails in Continuous Integration (CI)

1. Test in the same environment as CI (cloud testing):

   ./test --cloud aws .build/image.raw

2. Check for environment-specific issues (markers, dependencies)

3. Enable debug logging in CI by modifying test arguments

Need to Inspect System State

1. Start VM with cleanup disabled:

   ./test --ssh --skip-cleanup .build/image.raw

2. Login to inspect:

   ./util/login_qemu.sh

3. Check logs, services, files:

   journalctl -xe
   systemctl status
   ls -la /etc/

Test Takes Too Long to Run

1. Run only the specific test:

   ./test --test-args "test_specific.py" .build/image.raw

2. Use chroot for faster iteration (if applicable):

   ./test .build/image.tar

3. Skip expensive tests during development:

   ./test --test-args "-m 'not slow'" .build/image.raw

Troubleshooting Tips

Connection Issues

If you cannot connect to a VM:

• Verify --ssh flag is used for QEMU
• Check that --skip-cleanup is used
• Verify VM is still running
• Check firewall rules for cloud VMs

File Synchronization Issues in Dev Mode

If changes are not synced in dev mode:

• Ensure files are saved
• Check file paths match expected locations
• Verify inotify-tools is installed
• Look for errors in the watch mode output

Permission Issues

If tests fail with permission errors:

• Check if tests require root privileges (should use @pytest.mark.root)
• Use sudo when running tests manually if needed
• Verify file permissions in the test environment

Test Environment Differences

If tests behave differently across environments:

• Check test markers (@pytest.mark.booted, @pytest.mark.feature)
• Verify the test is appropriate for the environment
• Check for environment-specific configurations

Related Topics

• Testing Garden LinuxGet started with the Garden Linux Testing Framework
• Test Framework ArchitectureUnderstanding the Garden Linux Test Framework Architecture
• Test OrganizationUnderstanding how Garden Linux tests are organized and structured
• Run TestsHow to run Garden Linux tests in chroot, QEMU, cloud, OCI, and Kubernetes environments
• Setup Test EnvironmentInstall prerequisites and dependencies for running Garden Linux tests
• Test in CloudRun Garden Linux tests on cloud providers with authentication and configuration guides
• Developing TestsComprehensive guidelines for developing and maintaining tests in the Garden Linux test framework
• Test Coverage MarkersGuide to test coverage markers for establishing traceability between features and tests
• Test CLI ReferenceComplete command-line interface reference for the Garden Linux test runner