Linux Forwarder Troubleshooting

On Linux, check auditd first. The forwarder needs the audit daemon running to see files as they are created and changed. Without it the package still installs, the service still starts, and the initial backscan still collects what is already on disk -- so the asset looks healthy and simply stops gaining new sightings. An asset with a completed backscan and no recent activity is this, until proven otherwise. The check is one command, so rule it out before anything else.

sudo systemctl status auditd

If it is not running, enable it and start it, then restart the forwarder.

sudo systemctl enable --now auditd
sudo systemctl restart stairwell

On RHEL 6 the equivalents are sudo service auditd status, sudo service auditd start, and sudo chkconfig auditd on.

Work through What to Check First as well, for the connectivity and TLS causes that apply on every platform.

Where does the Linux forwarder keep its state?

Four locations, and between them they answer most questions without a log.

Logs go to the system journal under the stairwell unit on RHEL 7 and later and on the Debian family. On RHEL 6 they are files under /var/log/stairwell, with fileshipper.log the most recent.

The service will not start

Read the unit's status and its recent output, in that order.

systemctl status stairwell
journalctl -u stairwell -n 50 --no-pager

If the journal has nothing, the failure happened before logging started, and the failsafe log is where it went.

cat /var/log/stairwell_failsafe.log

Then check the configuration file. It is JSON, and a trailing comma or an unclosed brace stops the service from starting rather than starting it with defaults. The fields the forwarder needs are its environment and its token:

{
  "logger": { "loglevel": "info" },
  "asset": {
    "EnvId": "YOUR_ENVIRONMENT_ID",
    "Token": "YOUR_TOKEN"
  }
}

If asset.EnvId or asset.Token is empty or absent, the forwarder has nothing to register with. Re-run the installer with the correct values, or fill them in and restart.

Two causes to consider next, in this order: auditd, per the top of this page, and then SELinux or AppArmor, below.

SELinux or AppArmor is blocking the forwarder

The symptom is a service that starts and then logs permission denied on paths it should be able to read, or collection that stops on an enforcing host while the service stays up.

For SELinux, confirm the mode and then look for denials naming the forwarder.

getenforce

sudo ausearch -m avc -ts recent | grep stairwell

If there are denials, generate a local policy module from them and load it.

sudo ausearch -m avc -ts recent | audit2allow -M stairwell_local
sudo semodule -i stairwell_local.pp

Read what audit2allow produced before you load it. It generates permissions from whatever was denied in the window you gave it, which can be broader than the forwarder needs. If you would rather have a considered exception than a generated one, send the denial output to Stairwell support.

For AppArmor, check for denials and, if you need to confirm the diagnosis, put the profile into complain mode temporarily.

sudo dmesg | grep apparmor | grep stairwell

sudo aa-complain /usr/bin/stairwell

Complain mode is a diagnostic, not a fix. Once it confirms that AppArmor is the cause, work with whoever owns your profiles on a proper exception for the forwarder binary and the paths it reads.

The forwarder is running and the asset never appeared

Registration needs three things to be true: the credentials are right, the token is still valid, and the host can reach Stairwell.

1. Confirm asset.EnvId and asset.Token in /etc/stairwell/config.json.
2. Confirm the token has not been revoked or regenerated in Stairwell since the host was installed.
3. Prove the network from this host with swell verify connectivity. This is the step that catches an egress rule or a TLS-inspecting proxy, and it distinguishes "the path is blocked" from "the credentials are wrong", which look identical from the outside. See Pre-Deployment Check.

If the host reaches the internet only through a proxy, set proxyURL in the configuration file. Proxy Support covers the format.

The backscan is stuck or nothing is arriving

Check the Backscan status column in the Assets view first, because it is what the platform actually believes. The local state file is the second opinion, and useful when the host cannot reach Stairwell at all.

cat /var/lib/stairwell/scansession.json

Its status field maps to:

What to do with each. 3 and 4 are the ones to act on: restart the service to resume.

sudo systemctl restart stairwell

5 means backscans are switched off for this asset in policy, so the fix is a policy change rather than anything on the host. 2 with no visible progress for a long stretch on a busy server is usually disk saturation or a resource limit rather than a fault: the forwarder limits its own I/O, and the policy may be limiting it further. See Throttles, and consider excluding high-volume directories that will never hold anything security-relevant, which shortens the scan and the ongoing upload volume together.

If you need a full pass now, Force or Skip a Backscan runs one from the Assets view without touching the host.

How do I export logs for support?

On RHEL 7 and later, and on the Debian family:

journalctl -u stairwell -S "24 hours ago" --no-tail > stairwell_service.log

On RHEL 6:

cp /var/log/stairwell/fileshipper.log ./stairwell_service.log

Send these alongside it:

• /etc/stairwell/config.json, with the token removed or redacted first.
• /var/lib/stairwell/runtimeconfig.json.
• /var/lib/stairwell/scansession.json.
• The distribution and version, from cat /etc/os-release.
• The installed package version, from rpm -q stairwell or dpkg -l | grep stairwell.

Redacting the token is not a formality. It is a credential that can register assets into your environment, and a log bundle is a file that gets forwarded.

What should I read next?

• Linux v2.5.1, for the current packages and the distributions they were tested against.
• What to Check First, for the causes that are not Linux-specific.
• Change Logging Level, when the default logs do not carry enough detail.
• Force or Skip a Backscan, to run a full pass over a machine's existing files.