Documentation

Setting up a
collector

Blackholing tells you traffic stopped. It does not tell you which machine was sending it, and that is the only fact an incident actually turns on. The collector runs inside your network and answers that question.

Current release v0.3.3 · downloads and tokens are in the console

What the collector does

It sits where your blackhole routes point — either as the sinkhole next-hop itself, or reading NetFlow, IPFIX or sFlow from the router that discards the traffic — and turns every blocked beacon into a named internal host, a named user, and the threat category that put the route there.

It runs entirely on your premises. The only thing it asks BlackHawk is what a destination address is, over ordinary HTTPS to this console. Internal addresses, hostnames and usernames never leave your network.

Choosing a deployment

All of these run the same binary and serve the same dashboard.

DeploymentUse it whenPacket capture
Linux (systemd)You have a VM or host to give it. The default.Yes
Proxmox / LXCYou run a Proxmox estate and want a 1 GB container.Privileged only
VMware / ESXiDeploy the .ova — not the bare .vmdk beside it.Yes
KVM / Proxmox VMImport the .qcow2.Yes
DockerYou already run containers and want no host changes.Sinkhole profile only
WindowsThe only spare box is Windows. Runs as a real service.No — flow mode

Flow mode is the one most sites should pick

Sinkhole mode is higher fidelity — every attempt, with ports, at packet granularity — but it puts the collector on the data path. Flow mode changes nothing about how traffic moves and is usually the only proposal that survives a change board. The dashboard is identical either way.

Installing it

Mint a token first — Console → Collectors → Register a collector. It is shown once. Everything below assumes you have it.

Linux

curl -fsSL https://blackhole.blackhawk11.com/install.sh | sudo bash -s -- --token bhc_YOUR_TOKEN

Adds a service account, a systemd unit and configuration under /etc/blackhawk-collector. Re-run it to upgrade; configuration and data are preserved. Add --sinkhole eth1 for sinkhole mode.

Windows

iex "& { $(irm https://blackhole.blackhawk11.com/install.ps1) } -Token bhc_YOUR_TOKEN"

Proxmox

Download the -pct-archive from the console, copy it to your Proxmox host’s template store, then:

pct create 3001 local:vztmpl/blackhawk-collector-v0.3.3-pct-amd64.tar.xz \
  --hostname bhc1 --memory 1024 --unprivileged 1 \
  --net0 name=eth0,bridge=vmbr0,ip=dhcp
pct start 3001

Proxmox and LXD do not take the same file

The -pct- archive is a plain root filesystem, which is what pct expects. The -lxc- archive is a unified image for LXD and Incus and has a different internal layout.

Give Proxmox the wrong one and the container will not start, with sync_wait: An error occurred in another process (expected sequence number 7) — which sounds like a kernel fault and is only the wrong archive.

VMware / ESXi

Download the .ova and use Deploy OVF Template. It carries the hardware definition — LSI Logic SAS controller, E1000 adapter, 2 vCPU, 2 GB — all of which have in-tree Debian drivers, so it boots and gets an address with no VMware Tools work.

Do not attach the bare .vmdk to a VM

The .vmdk beside it is streamOptimized: compressed and read-only by design, because it is the format that lives inside an OVA. ESXi cannot open it read-write, and powering on a VM with it attached fails with

Failed to lock the file · Cannot open the disk … Module ‘Disk’ power on failed

which names neither the format nor the fix. Deploy the OVA instead. If you already have the bare disk on a datastore, convert it in place on the host:

Converting a bare .vmdk on the ESXi host

vmkfstools -i blackhawk-collector-v0.3.3.vmdk -d thin collector-disk.vmdk

LXD or Incus

lxc image import blackhawk-collector-v0.3.3-lxc-amd64.tar.xz --alias blackhawk-collector
lxc launch blackhawk-collector bhc1

Reaching the dashboard

The dashboard listens on port 8443 over HTTPS with a self-signed certificate. Your browser will warn about it; that is expected on a first boot, and you can replace the certificate later with listen.tls_cert and listen.tls_key.

Containers and appliances have no SSH and no root password

This is deliberate. A container is reached through its host, so it ships with no SSH server and a locked root account — there is nothing listening to attack and no password to guess. It also means you cannot SSH in, and console login will refuse you.

Use your hypervisor’s exec instead. That is the supported way in.

1. Get a shell

Proxmox

pct enter 3001

LXD / Incus

lxc exec bhc1 -- bash

VM appliance, or a Linux install

# the VM appliance does have SSH; a native install is just the host itself
ssh youruser@<address>

2. Find its address

ip -4 addr show eth0 | awk '/inet /{print $2}'

From the Proxmox host without entering the container: pct exec 3001 -- ip -4 addr show eth0

3. Get the dashboard password

An admin account is generated on first start and its password is printed once, to the service log:

journalctl -u blackhawk-collector | grep 'initial dashboard account'

If that has already scrolled away or the log was rotated, set a new one — this works offline and needs no existing password:

blackhawk-collector passwd --user admin

4. Open it

https://<the address from step 2>:8443/

Change the password

Sign in and change it on the System page. The dashboard will keep telling you to until you do.

Enrolling it

A collector that is installed but not enrolled captures traffic and cannot classify any of it — it has no credential to ask this console what an address is. The install commands above enrol as they go. If you installed without a token, or you are bringing up an appliance, do it now:

blackhawk-collector enroll --server https://blackhole.blackhawk11.com --token bhc_YOUR_TOKEN
systemctl restart blackhawk-collector

In a container, use the full path if that is not found

pct enter and lxc exec attach to the container instead of starting a login shell, so /etc/profile never runs and /usr/local/bin may be missing from your PATH. The binary is installed and fine — it just is not being found:

/opt/blackhawk-collector/bin/blackhawk-collector enroll --server https://blackhole.blackhawk11.com --token bhc_...

enroll verifies the credential before storing it, so a typo fails here rather than silently every five minutes afterwards.

Pre-provisioned appliances enrol themselves

Hardware shipped to you, or a VM built for you, carries an activation code already bound to your organisation. Read the code from the appliance’s own dashboard or its label, then enter it under Console → Collectors → Activate an appliance. It collects its credential on its next check-in. Nothing is typed into a terminal.

Giving it traffic to see

A freshly enrolled collector shows an empty dashboard, because nothing is reaching it yet. Point your router’s flow export at it.

Cisco IOS-XE

flow exporter BLACKHAWK
 destination <collector address>
 transport udp 2055
 export-protocol netflow-v9

The collector listens on 2055/udp for NetFlow v5, v9 and IPFIX, and 6343/udp for sFlow. Both are enabled by default.

Then tell it what your internal address space actually is. The defaults are the RFC 1918 ranges, which is wrong if you use public addressing internally — and it is what separates “one of ours reached out” from the rest of the internet talking to you. Edit /etc/blackhawk-collector/collector.yml:

network:
  internal:
    - 10.0.0.0/8
    - 192.168.0.0/16

dns:
  enabled: true
  # Your resolvers. The system stub usually cannot answer for internal
  # reverse zones, and then no host ever gets a name.
  servers:
    - 10.10.0.10

Set your own resolvers

Leaving dns.servers empty is the most common reason a working collector shows IP addresses instead of hostnames. The whole point is naming the machine.

Sinkhole mode: be the next-hop

Flow mode tells you a conversation happened. Sinkhole mode gives you the packets: every attempt, with source port, destination port and timing, at packet granularity. It is the higher-fidelity answer and it is fully supported.

Nothing changes on our side. Blackhole routes are already advertised to you with a discard next-hop — 192.0.2.1 for IPv4 and 100::1 for IPv6. Normally you point that at a discard interface and the traffic dies there. Point it at the collector instead and the same traffic arrives somewhere that can identify who sent it.

1. Repoint the discard route

This is the whole change, and it happens on your router.

Cisco IOS — before (discard)

ip route 192.0.2.1 255.255.255.255 Null0

Cisco IOS — after (sinkhole)

ip route 192.0.2.1 255.255.255.255 <collector address>

Juniper

set routing-options static route 192.0.2.1/32 next-hop <collector address>

Linux / VyOS / FRR

ip route add 192.0.2.1/32 via <collector address>

2. Put the collector on that path

The collector must be reachable at the address you just pointed at, on an interface it can capture from. Install it in sinkhole mode:

curl -fsSL https://blackhole.blackhawk11.com/install.sh | sudo bash -s -- --token bhc_YOUR_TOKEN --sinkhole eth1

Or set it in the config directly:

sensors:
  sinkhole:
    enabled: true
    interface: "eth1"

What sinkhole mode requires

Linux only. It needs CAP_NET_RAW, which the systemd unit grants. Windows has no equivalent that works without a third-party kernel driver.

Containers need more than the default. Docker needs --cap-add=NET_RAW and host networking — that is the sinkhole compose profile. An LXC container must be privileged; an unprivileged one cannot capture. Create it with --unprivileged 0, and expect a change board to want to know why.

It reads headers only and never stores payload.

You are now absorbing hostile traffic

The collector becomes the destination for everything your network tries to send to a blackholed address. That is the point — it is also a real change to your data path, and the reason flow mode exists. Size the box for your beacon volume, put it somewhere its failure is not your failure, and do not give it a route back out.

3. Confirm packets are arriving

Work down this list. Each step tells you whether the problem is above or below it, and none of them needs a packet capture tool inside the container.

a. Is the sinkhole sensor even running?

journalctl -u blackhawk-collector | grep -i sinkhole

sinkhole sensor capturing — it is running; the problem is upstream, keep going.
AF_PACKET socket: operation not permitted — the container is unprivileged and cannot capture. Recreate it with --unprivileged 0.
Nothing at all — the sensor is not enabled. This is the common one: routing traffic at the collector does not switch capture on, and nothing reports an error because nothing was asked to capture. Uncomment the sinkhole block in /etc/blackhawk-collector/collector.yml and restart.

b. Are packets reaching the container at all?

No tcpdump needed — the kernel counts every frame. Run it twice a few seconds apart and compare RX.

ip -s link show eth0

RX climbing while the dashboard stays empty means capture is the problem — go back to (a). RX flat means nothing is arriving, so the problem is the network — continue.

c. Capture from the Proxmox host instead

The host has tcpdumpand can see the container’s veth directly, so you never need to install anything inside it. Find the interface, then watch it:

# on the Proxmox host
grep -H '' /sys/class/net/veth*/ifindex | head        # or: brctl show vmbr0
tcpdump -ni veth3001i0 -c 20

Traffic on the host’s veth but not in the container is unusual and points at the container’s own configuration. Nothing on the veth means it is not getting that far — check the bridge itself with tcpdump -ni vmbr0 host <collector address>.

d. Or just install it

apt-get update && apt-get install -y tcpdump
tcpdump -ni eth0 -c 20

Proxmox nested on VMware needs portgroup changes

This one accounts for a lot of “the routing is right and nothing arrives”. A container behind a Proxmox bridge has its ownMAC address, which is not the MAC ESXi registered for the Proxmox VM’s vNIC.

Inbound frames addressed to the container are unknown-unicast to ESXi and get dropped, and outbound frames carrying the container’s MAC look forged. On the portgroup or vSwitch carrying that VM, set all three to Accept:

Promiscuous mode · MAC address changes · Forged transmits

They default to Reject, and the symptom is exactly this: routes correct, counters at zero, nothing in any capture inside the guest. On a vDS, MAC Learning is the better alternative to promiscuous mode.

journalctl -u blackhawk-collector | grep -i sinkhole
# and, on the collector, watch the interface you named

Hosts appear on the collector dashboard within a minute of the first blocked beacon. If nothing arrives, the discard route is still pointing at Null0 somewhere upstream — that is nearly always what it is.

Checking it works

Three things, in order.

1 — the service is up

systemctl status blackhawk-collector

2 — it can reach this console

journalctl -u blackhawk-collector | grep -i upstream

3 — it is checking in

# the console is the authority here

Open Console → Collectors. A collector that has checked in shows Online with a last-heartbeat time, its version and its platform. “Awaiting first heartbeat” means it has not yet reached us — that is an enrolment or connectivity problem, not a capture problem.

When it does not

ESXi: Failed to lock the file / Cannot open the disk / Module 'Disk' power on failed

Why: A bare streamOptimized .vmdk was attached to a VM. That format is compressed and read-only — it exists to live inside an OVA, and ESXi cannot open it read-write.

Fix: Deploy the .ova instead. To reuse a disk already on a datastore: vmkfstools -i <src>.vmdk -d thin <dst>.vmdk

Container will not start; sync_wait / expected sequence number 7

Why: Proxmox was given the LXD unified image instead of the pct root filesystem archive.

Fix: Destroy the container and recreate it from blackhawk-collector-v0.3.3-pct-amd64.tar.xz.

blackhawk-collector: No such file or directory, but the collector is installed

Why: pct enter and lxc exec attach to the container rather than starting a login shell, so /etc/profile never runs and PATH can be missing /usr/local/bin.

Fix: Run it by full path: /opt/blackhawk-collector/bin/blackhawk-collector enroll ... — or upgrade to v0.3.2, where the binary is also linked into /usr/bin, which is on every PATH.

SSH refused, and console login will not accept any password

Why: Containers ship with no SSH server and a locked root account, by design.

Fix: Use pct enter 3001 on the Proxmox host, or lxc exec bhc1 -- bash on LXD.

Service restarts every few seconds; permission denied on collector.yml

Why: A build before v0.3.1 left the config file unreadable by the service account.

Fix: Upgrade to v0.3.1 or later. As an immediate fix: chgrp blackhawk-collector /etc/blackhawk-collector/collector.yml

Dashboard loads but every host is an IP address

Why: No internal resolvers configured, so reverse lookups go nowhere.

Fix: Set dns.servers in collector.yml to your own resolvers and restart.

Traffic routed at the collector but the dashboard stays empty

Why: Most often the sinkhole sensor is not enabled — routing packets at the collector does not switch capture on, and nothing errors because nothing was asked to capture.

Fix: Check journalctl -u blackhawk-collector | grep -i sinkhole. No output means enable the sinkhole block in collector.yml and restart.

Sinkhole enabled, log says AF_PACKET socket: operation not permitted

Why: Packet capture needs CAP_NET_RAW, which an unprivileged LXC container cannot hold.

Fix: Recreate the container with --unprivileged 0. On Docker, add --cap-add=NET_RAW and host networking.

Proxmox nested on VMware: routes correct, RX counters at zero

Why: The container's MAC is not the one ESXi registered for the Proxmox VM, so inbound frames are unknown-unicast and outbound ones look forged.

Fix: Set Promiscuous mode, MAC address changes and Forged transmits to Accept on the portgroup carrying that VM. All three default to Reject.

Dashboard is empty and the console says Online

Why: Enrolment is fine; nothing is being exported to it.

Fix: Check the router is exporting to the collector on 2055/udp and that network.internal matches your real address space.

Console shows Awaiting first heartbeat

Why: It cannot reach this console, or it was never enrolled.

Fix: Run blackhawk-collector enroll --server https://blackhole.blackhawk11.com --token bhc_... and confirm outbound HTTPS is permitted.

Still stuck? The service log is the first thing we will ask for: journalctl -u blackhawk-collector -n 200 --no-pager