Linux Forwarder Troubleshooting

This guide covers common Linux forwarder issues and how to resolve them. For log file locations and configuration paths, see What to Check First.

Service fails to start

Symptoms: After installation, systemctl status stairwell shows failed or the service exits immediately.

Steps:

1. Check the service status and recent logs:
   Bash

   systemctl status stairwell
   journalctl -u stairwell -n 50 --no-pager

2. Look for startup errors in the failsafe log (RHEL 6/early startup failures):
   Bash

   cat /var/log/stairwell_failsafe.log

3. Confirm that the configuration file is present and valid:
   Bash

   cat /etc/stairwell/config.json

   The file should contain a valid environmentId and registrationToken. If it is empty or missing these fields, re-run the installer with the correct parameters.
4. Verify the forwarder binary has execute permission:
   Bash

   ls -la /usr/bin/stairwell

Common cause: SELinux or AppArmor is blocking the forwarder binary or its access to monitored directories. See the SELinux section below.

SELinux or AppArmor is blocking the forwarder

Symptoms: Service starts but logs show permission denied errors, or file collection stops on enforcing systems.

SELinux:

1. Check whether SELinux is enforcing:
   Bash

   getenforce

2. Review denials related to the forwarder:
   Bash

   ausearch -m avc -ts recent | grep stairwell

3. If denials are present, generate and apply a local policy module:
   Bash

   ausearch -m avc -ts recent | audit2allow -M stairwell_local
   semodule -i stairwell_local.pp

4. Contact Stairwell support with the denial output if a persistent policy exception is needed.

AppArmor:

1. Check for AppArmor denials:
   Bash

   dmesg | grep apparmor | grep stairwell

2. If denials are present, set the profile to complain mode temporarily to confirm the issue:
   Bash

   aa-complain /usr/bin/stairwell

3. Work with your security team to create an appropriate AppArmor exception for the forwarder binary and its monitored paths.

Registration failure

Symptoms: Forwarder service runs but the asset never appears in the Stairwell platform. Logs show repeated registration errors.

Steps:

1. Confirm the environment ID and registration token in /etc/stairwell/config.json.
2. Test connectivity to Stairwell endpoints from the host:
   Bash

   curl -v https://api.stairwell.com/health 2>&1 | head -30

   A successful connection returns an HTTP response (even an error response). A timeout or connection refused indicates a firewall block.
3. Check whether a proxy is required. If the host routes outbound traffic through a proxy, configure the forwarder accordingly:
   Bash

   # In /etc/stairwell/config.json
   "proxy": "http://proxy.example.com:8080"

4. Verify that the registration token has not expired. Tokens can be revoked and regenerated in Stairwell under Settings → Environment → Forwarder Tokens.

Backscan is stuck or disabled

Symptoms: File collection appears to have stopped. The asset is active but sightings are not arriving.

Steps:

1. Check the backscan state:

   Bash

   cat /var/lib/stairwell/scansession.json

   The status field maps to: 0 = not started, 1 = complete, 2 = in progress, 3 = stopped, 4 = failed, 5 = disabled.

2. If status is 3 (stopped) or 4 (failed), restart the service to resume:

   Bash

   sudo systemctl restart stairwell

3. If status is 5 (disabled), backscan has been explicitly disabled via policy. Review your policy configuration in the Stairwell platform.

4. If backscan status is 2 (in progress) but no activity is observed for more than 12 hours on a standard server, check disk I/O saturation. The forwarder throttles its own I/O, but on heavily loaded systems you can also add high-volume directories to deny paths to reduce scope.

Exporting logs for support

To export recent logs for Stairwell support (RHEL 7+):

journalctl -u stairwell -S "24 hours ago" --no-tail > stairwell_service.log

For RHEL 6:

cp /var/log/stairwell/fileshipper.log ./stairwell_service.log

Also include:

• /etc/stairwell/config.json (remove or redact the registrationToken before sharing)
• /var/lib/stairwell/runtimeconfig.json (contains asset ID and registration state)
• /var/lib/stairwell/scansession.json (backscan state)
• OS version: cat /etc/os-release
• Forwarder package version: rpm -q stairwell or dpkg -l | grep stairwell