Server Provisioning

This document is outdated and does not represent ASL Infrastructure as of 2021-01-17

This is intended to be a reference for setting up a VM or Server for AllStarLink.

Server Overview

Basic Requirements

AllStarLink has standardized on Ubuntu 16.04 LTS for it's servers.

The minimum configuration of any server will be 2 cores of 2 GHz or faster, 4 GiB of ram and 40 GiB of Disk.

All new servers shall support IPv6, or have it available from the hosting provider.

Reverse IP's shall be delegated via a CNAME to $NAME.PTR.allstarlink.org, where $NAME is the name of the server.

All servers shall be partitioned to use / as the only partition unless a specific configuration is required.

Install guide

When provisioning a new server

• check the VM is setup (cpu/mem/disk) as it should be, if not contact the provider

  cat /proc/cpuinfo |grep processor
  processor       : 0
  processor       : 1
  processor       : 2
  processor       : 3

• on the server install python (apt-get install python). This is needed for the ansibile provisioning
• setup the server in the infrastructure configs and push the users and keys to it.

Mandatory Software

All servers require this software

apt-get install ntp ntpdate python vim screen ipsec-tools strongswan fail2ban snmp haveged libacl1-dev python3-dev libssl-dev gcc g++ fio pbzip2 ncdu

Mandatory Configs

Ubuntu 18 Config

Ubuntu 18 uses the net netplan config. It sucks badly.

First you need to disable the resolved service:

sudo systemctl disable systemd-resolved.service
sudo systemctl stop systemd-resolved
rm /etc/resolv.conf 
sudo touch /etc/cloud/cloud-init.disabled
sudo apt-get purge cloud-init

echo "nameserver 1.1.1.1" > /etc/resolv.conf
apt-get install ifupdown

Configure /etc/network/interfaces

systemctl unmask networking
systemctl enable networking
systemctl restart networking

systemctl stop systemd-networkd.socket systemd-networkd networkd-dispatcher systemd-networkd-wait-online
systemctl disable systemd-networkd.socket systemd-networkd networkd-dispatcher systemd-networkd-wait-online
systemctl mask systemd-networkd.socket systemd-networkd networkd-dispatcher systemd-networkd-wait-online
apt-get --assume-yes purge nplan netplan.io

Network Config

• The network should be configured to use /etc/network/interfaces, and add DNS and the firewall to it and search in the allstarlink.org domain

# The primary network interface
auto eth0
iface eth0 inet6 static
   address 9805:0900:0340:1000::2600/64
   autoconf 0
   accept_ra 2	
iface eth0 inet static
    address 44.103.0.49
    netmask 255.255.255.0
    network 44.103.0.0
    broadcast 44.103.0.255
    gateway 44.103.0.1 
    dns-nameservers 44.103.0.4 1.1.1.1
    dns-search allstarlink.org
    up /etc/network/firewall.sh

Persistent Interface Names

To ensure that interface names are persistent (e.g., ethX) you can choose using either UDEV or Grub.

The Grub method tends to be easier and less prone to locking yourself out of a VM due to an interface naming conflict at reboot.

UDEV method

• There is typically only one network interface, and it will be named dynamically. We must setup this using udev to be persistent

   root@server# ifconfig |grep HWaddr
   eth0      Link encap:Ethernet  HWaddr 52:54:00:73:86:06  

Now take this HWaddr and put it in the config file

   echo 'SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="52:54:00:73:86:06", ATTR{dev_id}=="0x0", ATTR{type}=="1", NAME="eth0"' >/etc/udev/rules.d/70-persistent-net.rules

Grub method

• Edit the /etc/default/grub file:

  vim /etc/default/grub

• Look for "GRUB_CMDLINE_LINUX" and add the following "net.ifnames=0 biosdevname=0"

  GRUB_CMDLINE_LINUX="net.ifnames=0 biosdevname=0"

• Generate new grub config file

  sudo grub-mkconfig -o /boot/grub/grub.cfg

Other required config items

• configure screen to use the scroll back buffer

  vim /etc/screenrc
  uncomment "termcapinfo xterm|xterms|xs|rxvt ti@:te@"

• configure bash completion for interactive shells

   vim /etc/bash.bashrc
   uncomment the stuff below 
   # enable bash completion in interactive shells 

• set the host name

   echo "domain.allstarlink.org" >/etc/hostname

• set the default editor

   update-alternatives --config editor 
   Then select #3 vim.basic

• setup a firewall as /etc/network/firewall.sh and chmod +x it. You'll need to edit this based on the machine. Note the stuff in tampa uses a firewall on the HV too.

   #!/bin/bash
   
   INET_IF=eth0
   
   #Flush and zero all tables
   modprobe ip_tables
   modprobe ipt_limit
   modprobe iptable_mangle
   modprobe ipt_state
   modprobe ipt_LOG
   modprobe iptable_filter
   
   iptables -F INPUT
   iptables -F FORWARD
   iptables -t nat -F POSTROUTING
   iptables -t nat -F PREROUTING
   
   #init the log-and-drop chain
   iptables -F log-and-drop
   iptables -X log-and-drop
   iptables -N log-and-drop
   #init log-and-reject
   iptables -F log-and-reject
   iptables -X log-and-reject
   iptables -N log-and-reject
   
   echo "all tables flushed and dropped"
   # Specific chain used for logging packets before blocking them
   iptables -A log-and-drop -j LOG --log-prefix "[IPTables] Drop "
   iptables -A log-and-drop -j DROP
   
   # Specific chain used for logging packets before blocking them
   iptables -A log-and-reject -j LOG --log-prefix "[IPTables] Reject "
   iptables -A log-and-reject -j REJECT
   echo "logging chains setup"
   
   # The packets having the TCP flags activated are dropped
   # and so for the ones with no flag at all (often used with Nmap scans)
   iptables -A FORWARD -p tcp --tcp-flags ALL ALL -j log-and-drop
   iptables -A FORWARD -p tcp --tcp-flags ALL NONE -j log-and-drop
   
   #Global blocks
   #iptables -t filter -A INPUT -j DROP -s 119.118.232.185/24
   
   # allow IPSEC from other boxes
   IPSECsrc='199.47.174.150,44.98.254.151,44.103.0.48,44.103.0.49,44.98.254.145,44.72.21.13,44.72.21.12'
   #Technically the next two are not needed as we have the policy
   iptables -A INPUT -i $INET_IF -p 50 -j ACCEPT --src "$IPSECsrc"
   iptables -A INPUT -i $INET_IF -p 51 -j ACCEPT --src "$IPSECsrc"
   iptables -A INPUT -i $INET_IF -p udp --dport 500 -j ACCEPT --src "$IPSECsrc"
   # this is needed to allow all ipsec packets when it's host to host
   iptables -A INPUT -m policy --dir in --pol ipsec -j ACCEPT --src "$IPSECsrc" 
   
   # allow all ssh in
   iptables -t filter -A INPUT -j ACCEPT --protocol tcp --dport 22
   
   #allow http and https
   #iptables -t filter -A INPUT -j ACCEPT --protocol tcp --dport 80
   #iptables -t filter -A INPUT -j ACCEPT --protocol tcp --dport 443
   # allow asterisk 4569
   #iptables -t filter -A INPUT -j ACCEPT --protocol udp --dport 4569
   # allow DNS
   #iptables -t filter -A INPUT -j ACCEPT --protocol tcp --dport 53
   #iptables -t filter -A INPUT -j ACCEPT --protocol udp --dport 53
   
   echo "end of services"
   # allow ping at 2 per sec
    iptables -t filter -A INPUT -j ACCEPT --in-interface $INET_IF --protocol icmp --icmp-type echo-request --match limit --limit 4/s --limit-burst 3
    iptables -t filter -A INPUT -j log-and-drop  --in-interface $INET_IF --protocol icmp --icmp-type echo-request
   # allow responces to local initated connections
   #iptables -A INPUT -i  $INET_IF --match state --state NEW,INVALID -j log-and-drop
   #iptables -A FORWARD -i $INET_IF  --match state --state NEW,INVALID -j log-and-drop
   iptables -t filter -A INPUT -j ACCEPT --match state --state RELATED,ESTABLISHED
   # Set rp_filter to 2
   for i in `find /proc/sys/net/ipv*/conf -name rp_filter`
   do
           echo "2" >$i
   done
   # setup a default deny rule for outside traffic
   iptables -t filter -A INPUT --in-interface $INET_IF -j log-and-drop

• setup fail2ban for ssh and have it null route offenders. edit ignoreip as needed

   vi /etc/fail2ban/jail.conf
   ignoreip = 127.0.0.1/8 199.47.172.0/22 44.98.254.0/24 44.72.21.0/24 44.103.0.0/24
   bantime  = 3600
   # A host is banned if it has generated "maxretry" during the last "findtime"
   # seconds.
   findtime  = 3600
   # "maxretry" is the number of failures before a host get banned.
   maxretry = 2
   banaction = route

• Set the TimeZone to UTC

   sudo timedatectl set-timezone UTC

• Set the server up in forward and reverse DNS
  • for reverse have the provider do a CNAME in their reverse file pointing to $DOMAIN.PTR.allstarlink.org. In the allstarlink.org DNS zone add an entry

   example:
   stats             IN      PTR     stats.allstarlink.org.

This will do a lookup on 130.254.98.44.in-addr.arpa. and return a CNAME pointing to stats.PTR.allstarlink.org, which has a PTR record pointing to stats.allstarlink.org.

Configure IPSEC

AllStarLink servers use strong crypto using host to host IPSEC between them for protection of services. This is configured only between servers that need it, as we don't have dynamic tunneling enabled, and each server needs a config for each tunnel. This can quickly add up to lots of configrations.

This example will show two servers, 1 and 2 with IP 44.1.1.1 and 44.2.2.2 respectively.

Server 1

We need to provision the ipsec tools to know about the connections and configure a pre shared key (PSK). Note the left server is always the local server.

   /etc/ipsec.conf
   conn one-to-two
           authby=secret
           #auto=start enabled the tunnel to come up even if there is not traffic for it.  
           auto=start
           keyexchange=ike
           left=4.1.1.1
           right=4.2.2.2
           leftikeport=500
           rightikeport=500   
           type=transport
           esp=aes128gcm16!
           dpddelay=5
           dpdtimeout=20
           dpdaction=restart

   vim /etc/ipsec.secrets
   44.1.1.1 44.2.2.2 : PSK "This is the AllStarLink PSK"

Then do an 'ipsec restart' on the server.

Server 2

/etc/ipsec.conf

   conn two-to-one
           authby=secret
           #auto=start enabled the tunnel to come up even if there is not traffic for it.  
           auto=start
           keyexchange=ike
           left=4.2.2.2
           right=4.1.1.1
           leftikeport=500
           rightikeport=500   
           type=transport
           esp=aes128gcm16!
           dpddelay=5
           dpdtimeout=20
           dpdaction=restart

   vim /etc/ipsec.secrets
   44.2.2.2 44.1.1.1 : PSK "This is the AllStarLink PSK"

Then do an 'ipsec restart' on the server.

Verify IPsec

The 'ipsec' command is used to verify the tunnel is up between the servers

   root@server# ipsec status
   two-to-one[839]: ESTABLISHED 98 minutes ago, 44.1.1.1[44.1.1.1]...44.2.2.2[44.2.2.2]
   two-to-one{13209}:  INSTALLED, TRANSPORT, reqid 695, ESP SPIs: c824e4db_i c1e4bf5c_o
   two-to-one{13209}:   44.1.1.1/32 === 44.2.2.2/32

If they are not up, check /var/log/syslog and restart ipsec on both servers. Some times a server can get in a bad status if there is a mis-config. Also it's worth noting that IPSEC is processed by iptables once it's decrypted, the iptables -A INPUT -m policy --dir in --pol ipsec -j ACCEPT --src "$IPSECsrc" line in the firewall allows all IPsec packets once decrypted to bypass the firewall. This is able to prevent traffic between unencrypted services on the servers (e.g. mysql will not connect if the ipsec is down).

Configure Postfix

Postfix / other MTAs are no longer in use for ASL Infrastructure

Postfix is installed to forward mail for root to a smtp host.

apt-get install postfix mailutils

This will run an installer with a curses interface and you must select Satallite System. Check the System mail name is the hostname of the server, and the SMTP relay host is morty.keekles.org. Root and postmaster mail should be rootmail@allstarlink.org.

Should you need to reconfigure this use:

dpkg-reconfigure postfix

other aliases are setup in /etc/aliases. You must run newaliases after this is updated for them to take effect.

Verification

It's important to verify the server provisiong before being put into production.

Items to check

• reboot the server/vm, do all services start properly?
• Is the IP address configured on the server on eth0?
• Is the hostname set?
• Is it configured in DNS both forward and reverse?
• Is the firewall active (try netcat on a non-permitted port)
• Does Screen work in an xterm with scroll back?
• Is the time set via ntp ntptime and is the timezone set to UTC?
• Is fail2ban working? Make a couple test connections and see if the IP is null routed ip route show

You may need to check your other services on this server now.

Network Monitoring

It's time to hand off the server to the NMS team. Please ensure SNMP is configured and an IPSEC tunnel is built to nms.allstarlink.org Logging will be sucked up by graylog.

Please ensure it's being watched in librenms by asking on the admin list or in the slack.