I’d like to publish more of my writing in both book form and here on my old blog, but I feel at the current intersection of AI generated slop and an overabundance of content – helpful tech blogs, email lists, and youtube videos – there really isn’t a point any more. And there hasn’t been for a long time for me. I write documentation and advice nearly everyday for DevOps/SRE/Security teams I work with, but it’s so so niche that I don’t think anyone would find it helpful or meaningful to read.

Ultimately, I’m leaving the blog in place as a historical marker of my own personal journey throught the early years of my career. Otherwise the information is out of date and irrelevant.

Cheers, and happy new years!

from Author GIFs via Gfycat

The Three phases of great documentation are Personal, Professional, and Presentable

Phase 1) Personal Documentation

The main point of personal documentation should be to gather as much information, notes, logs, output steps, etc; and slam them into a plain text format of your choice. Do not worry about formatting or anyone else seeing this documentation. It is for your eyes only, and as long as you can follow it, then continue to build that repository of information.

from Author GIFs via Gfycat

Naturally, when you take time to place and organize your own notes, your brain is going to make a pseudo index of that information. Key details may be missing from what it is you are recording, but you will most always remember where you put that detail, and then when you need it again, can easily retrieve the information.

Some scholarly folks have even invented an entire “Second Brain” called a zettelkasten around this concept of continually taking notes, indexing them with tags and/or interlinks, and this can be more than just notes at work. These Second Brain’s are for everything; recipes, meetings, shopping lists, and just about anything.

Phase 2) Professional Documentation

This phase is contingent on having some previous notes from the personal documentation phase. This is where you are formally recording notes that you want to share with your organization. Many teams use a form of Wiki or Shared documents to go about this.

from Author GIFs via Gfycat

The documentation needs to be accurate, true, and above all readable to a wide audience - You never know who in the future may need to understand the whys and whats of your instructions.

Things that a professional documentation system requires:

• Searchability - If you can’t easily search for ANY text within the docs, then it will be impossible for someone else to find your article
• Tagging - Not all documentation fits in a purely hierarchy structure, nor should it be forced into one, so tags are a way to cut across hierarchy and create a new grouping of closely related topics.
• Viewable - Everyone in the organization should be able to view your documentation. Likewise, you should make your documentation understandable to a wide audience, and also be careful of revealing secret information.
• Mixed Media - The wiki system should make it easy to include images, videos, diagrams, whatever is needed for those making and reading to understand the concept.

Phase 3) Presentable Documentation

Most teams stop at step 2, with the thought that so long as it’s in the wiki someone can find it. I’m here today, to tell you that most other people, do not read documentation until forced to. Does that mean this is all a waste of time and effort? NO. The previous phases were for your understanding, phase 1 is explaining it to yourself, phase 2 is explaining it to your team, now phase 3 is you distilling everything down into few slides of a presentation and verbally explaining it to your team or larger development organization.

via Gfycat

At my current place of employment, every friday a team from the Engineering department presents on a new or innovative topic that they want to share with the rest of the organization. These are great, people listen and ask questions, you can physically see engagement go from close to zero to a ton of side conversations spinning out as people absorb and then start to imagine using the new tool/concept/idea in their own work.

Likewise, anytime our team builds something new, we do a show and tell, running through both the documentation on our wiki and live demoing the new feature so that everyone understand how it works, why it was made, and what problem it solves. These presentations are recorded for future viewers and are by far the most effective way to spread the word and get team members involved and engaged with the new concept.

Plus, if you can accurately explain a technology or idea to someone else, that means you fundamentally have learned this new skill and you can be proud of the information you have and know that people want to learn from you.

This was written during the height of CentOS 6. Some concepts may be stale and out of date.

Summary

There are basically an infinite amount of tools and ways to resolve an issue. 10 times out of 10, if you have a problem someone else has already encountered it and documented how to resolve the issue somewhere out there via google.com.

• EVERYTHING on Linux is a file, understanding this fact is key to understanding linux, permissions, access, drivers, kernels, and debugging
• Follow through! Do not just hit enter and walk away, understand the consequences (good and bad) of what you are doing before you change something.
• Start small, the first thing every aspiring Linux Admin should build is a LAMP or LAPP Stack and then move on to more complicated services/setups
• Take notes, so that you remember where you got stuck and what you did so it can be a repeatable process that should be able to be duplicated many times over (this is what the wiki is for)
• Don’t Panic! when you mess up, learn how to fix it this will make you a better admin, trial by fire is the way of life for all of us.
• Take snapshots, backups, SVN/GIT, copy files before any major changes, always start at Testing/DEV and work your way up to BTA/PROD deployments.
• Ask questions, discuss what you are doing before/after you change something(even if it is just research), measure twice/cut once: This allows everyone to know where you are and what you are doing, that way if something bad happens we can quickly triage the situation.

Bash - Shell Scripting

Bash scripting is the bread and butter of systems administration. Too often we have to repeat a set of commands across many machines, and the goal of everyone should be to complete that goal with as much automation as possible. Any command that you can type on the CMD line is valid in bash scripting along with basic if/elif/else logic, for loops, variable substitution, and mathematical expressions.

#!/bin/bash

Wikipedia:Bash (Unix shell)

Man Pages

If you are ever unsure of what a command does man pages are the way to locally read and understand how to use any command in Linux. It is literally the command manual in text format.

man ifconfig
 IFCONFIG(8)                Linux Programmer’s Manual               IFCONFIG(8)
 
 NAME
        ifconfig - configure a network interface
 
 SYNOPSIS
        ifconfig [interface]
        ifconfig interface [aftype] options | address ...
 
 DESCRIPTION
        Ifconfig  is  used to configure the kernel-resident network interfaces.
        It is used at boot time to set up interfaces as necessary.  After that,
        it  is  usually  only  needed  when  debugging or when system tuning is
        needed.
 
        If no arguments are given, ifconfig displays the  status  of  the  cur-
        rently  active interfaces.  If a single interface argument is given, it
        displays the status of the given interface only; if a single  -a  argu- 
        ment  is  given,  it  displays the status of all interfaces, even those
        that are down.  Otherwise, it configures an interface.
  [there is much more to this document in full]

/var/log/messages

/var/log/messages is the default catch-all for any system errors that may be occurring. There are sometimes logs setup for specific services. Always try to see if a specific service has its own log file or directory (httpd,cron,secure,php-fpm,mysql,postgresql) and if it does not exist or is empty, check /var/log/messages it will probably have the errors, warning, INFO, emergencies, kernel panics, and any other frightening logs that you need to resolve any issues that may occur.

journalctl

The command journalctl can be used to read system and application logs through systemd’s journaling service. This is the newer, sexier way to quickly access logs, but by default journalctl only keep logs around for a few days or until you max out the reserved amount.

$ journalctl -f # follow the logs
$ journlactl -n100 # look back last 100 lines of the log file

If you have checked both the service log and /var/log/messages and still cannot find any information about the service you are trying to fix, you may have to enable debugging mode(s) or level(s) on the service itself. This is different for all services and can usually be found in the products documentation.

Machine Info

The following is a collection of commands and their result to gather basic information about the machine.

Get name, kernel # and date of build with uname

$ uname -a
Linux devsql005.example.com 3.10.0-123.8.1.el7.x86_64 #1 SMP Mon Sep 22 19:06:58 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux

OS release info can be found in the following files

$ cat /etc/redhat-release; # shows official CentOS/Redhat version number
  CentOS Linux release 7.0.1406 (Core)

or for Ubuntu

$ cat /etc/lsb-release                                             
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=20.04
DISTRIB_CODENAME=focal
DISTRIB_DESCRIPTION="Ubuntu 20.04.1 LTS"

Use Uptime to get the amount of time since the server has started, and also load averages

$ uptime
 20:40:43 up 1 day, 22:36,  0 users,  load average: 0.23, 0.16, 0.18

Ip Address and network information

$ ip addr; 
 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
     link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
     inet 127.0.0.1/8 scope host lo
        valid_lft forever preferred_lft forever
     inet6 ::1/128 scope host
        valid_lft forever preferred_lft forever
 2: ens160: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
     link/ether 00:50:56:84:59:79 brd ff:ff:ff:ff:ff:ff
     inet 10.0.0.[?]/24 brd 10.0.0.255 scope global ens160
        valid_lft forever preferred_lft forever
     inet6 fe80::250:56ff:fe84:5979/64 scope link
        valid_lft forever preferred_lft forever

$ ifconfig;
 ens160: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
         inet 10.0.0.[?]  netmask 255.255.255.0  broadcast 10.0.50.255
         inet6 fe80::250:56ff:fe84:5979  prefixlen 64  scopeid 0x20<link>
         ether 00:50:56:84:59:79  txqueuelen 1000  (Ethernet)
         RX packets 1675151509  bytes 296413707167 (276.0 GiB)
         RX errors 0  dropped 235945  overruns 0  frame 0
         TX packets 1428044385  bytes 471991954342 (439.5 GiB)
         TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
 
 lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
         inet 127.0.0.1  netmask 255.0.0.0
         inet6 ::1  prefixlen 128  scopeid 0x10<host>
         loop  txqueuelen 0  (Local Loopback)
         RX packets 409040766  bytes 330694444196 (307.9 GiB)
         RX errors 0  dropped 0  overruns 0  frame 0
         TX packets 409040766  bytes 330694444196 (307.9 GiB)
         TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

Find the directory you are in

$ pwd 
/var/lib/pgsql/9.3/data

Show local users

$ cat /etc/passwd;
 root:x:0:0:root:/root:/bin/bash
 bin:x:1:1:bin:/bin:/sbin/nologin
 daemon:x:2:2:daemon:/sbin:/sbin/nologin
 adm:x:3:4:adm:/var/adm:/sbin/nologin
 lp:x:4:7:lp:/var/spool/lpd:/sbin/nologin
 sync:x:5:0:sync:/sbin:/bin/sync
 shutdown:x:6:0:shutdown:/sbin:/sbin/shutdown
 halt:x:7:0:halt:/sbin:/sbin/halt
 mail:x:8:12:mail:/var/spool/mail:/sbin/nologin
 ....

Show local groups

$ cat /etc/group;
 root:x:0:
 bin:x:1:
 daemon:x:2:
 sys:x:3:
 adm:x:4:
 tty:x:5:
 disk:x:6:
 ....

Find which groups the user ‘apache’ is in

$ groups apache;
 apache : apache svc_account1

grep/awk/sed

Learn these three tools, they are the swiss army knife to the system’s administrator’s war chest. They can be used to do most anything and you will see many example combinations and use cases in the sections below.

You can be like captain planet and when you combine the power of all three of these tools you can create very powerful and flexible scripts that do all the heavy lifting for you.

grep

grep is a tool mainly used for searching or weeding out information.

$ grep -r mysql; # will search the entire directory tree and every file in those directories for lines containing the word mysql
$ cat /var/log/maillog | grep user@fqdn.com

https://en.wikipedia.org/wiki/Grep

awk

I generally use awk piped at the end of a command to filter parts of the returned information. But awk is it’s own programming language and can be just as powerful as perl or any other scripting language.

$ df -h -t nfs -P | grep /vol/ |  awk '{ print $5 " " $6}'; 
# df -h gives me a listing of the mounts, I search for /vol/ to get mounted drives, then I awk for result $5 and $6 of each returned line which is always used percentage and the partition that is in question.
$ grep "request ID" history | awk 'match($0,"is"){print substr($0,RSTART+3,50)}'`; 
# this searches for "request ID" in a file called history and awk looks for the word "is" then prints 50 characters, 3 characters after "is", which is always the request ID in this example. 

http://en.wikipedia.org/wiki/AWK

sed

sed is used for inline text editing and manipulation

$ sed -n 51,61p sbr/index.html | sed -i '50r /dev/stdin' testsed;
# this takes lines 51-61 of index.html and appends them after line 50 in testsed
 
IP=`nslookup $HOSTNAME | grep Address | grep -v "#53"| awk '{ print $2}'`;
# looks up hostname using DNS and greps for address, ignores anything with #53,prints second value on returned line
$ echo "$IP"
$ sed -i 's/0.0.0.0/'$IP'/g' /etc/monitrc; - # reads file, greps for 0.0.0.0, then replaces 0.0.0.0 with result of $IP in script.

http://www.bashoneliners.com/oneliners/oneliner/popular/

$ curl -s http://cbsg.sourceforge.net/cgi-bin/live | grep -Eo '^<li>.*</li>' | sed s,\</\\?li\>,,g | shuf -n 1

https://en.wikipedia.org/wiki/Sed

Disk is full

The following are commands and explanation of the many things you can do to find and remove large log/temporary files, unusual file names, and similar scenarios.

Below is a list of the most likely to be filled areas and should be the first place(s) one looks for space that can be reclaimed

1. /var/log
2. /var/spool/{clientmqueue,mqueue,mail}
3. /tmp
4. /var/tmp

df

df -h; shows one the local and mounted filesystems and the amount of space, used vs available.

$ df -h
Filesystem                              Size  Used Avail Use% Mounted on
/dev/sda3                                12G  4.9G  6.7G  43% /
/dev/sda1                               497M  115M  383M  24% /boot
10.10.0.248:/vol/remote_data            1.9T  1.2T  628G  66% /mnt/remote_data

du

du-sh /var/log/; will count the size of each file within a directory and total it.

$ du -sh /var/log
113M    /var/log
$ du -shx /var/log/*
4.0K    /var/log/alternatives.log
100K    /var/log/apt
0       /var/log/btmp
4.0K    /var/log/dist-upgrade
92K     /var/log/dpkg.log
4.0K    /var/log/journal
4.0K    /var/log/landscape
4.0K    /var/log/lastlog
4.0K    /var/log/unattended-upgrades
0       /var/log/wtmp

Clearing large Log files

Do not just rm -rf logfile; if a process, say apache, is still writing to the logfile and you remove the file from existence the process will crash. Remember we wish to have an uptime of 24/7/365. Instead you can clear out the file to size 0 and allows the process to continue writing to it.

$ cat /dev/null > /var/log/messages

No Space left to Delete

Rarely, but it does happen a volume will fill up and be so full that rm will not work as it creates temporary records while it deletes files. There is another way to delete the file by finding it’s inode number and using find to delete the file

let’s assume you did all steps above and found the large offending file.

$ rm /var/tmp/3a8066e5-a90c-4ae5-bdc6-47e117acf354.error
 rm: remove regular file ‘/var/tmp/3a8066e5-a90c-4ae5-bdc6-47e117acf354.error’? y
 rm: cannot remove ‘/var/tmp/3a8066e5-a90c-4ae5-bdc6-47e117acf354.error’: No space left on device
$ ls -li /var/tmp/3a8066e5-a90c-4ae5-bdc6-47e117acf354.error
 56436168 -rw-r--r-- 1 gerbn308 zxdev 0 May 13 11:17 /var/tmp/3a8066e5-a90c-4ae5-bdc6-47e117acf354.error
find . -inum 56436168 -delete

find

Find is really really useful and below are some of the ways find has solved strange issues for me

find date range

$ ll -tr;
# will list out files in time/date order newest -> oldest (remove the r if you would like oldest -> newest)
$ find . -type f -newer file_xyz.txt ! -newer recent_zzy.txt -exec ls -l {} \; -print &> output.txt;
# make sure I got the correct range of data greped for earliest date and latest date
$ find . type -newer file_xyz.txt ! -newer recent_zzy.txt delete; 

find a specific file type

In this example we are finding all mp3 files and moving them to a mounted usb drive on /mnt/mp3

$ grep *.mp3 
# lets assume there is a mix of file types with no standard naming onvention and there are thousands of them
$ find / -iname "*.mp3" -exec mv {} /mnt/mp3 ;
# you can have any command in the -exec section

find anything older than 12 hours

Sometimes you have to alleviate some pressure so the file system does not fill up and you do not want to clear out files that may be actively being worked on.

ls -R | wc -l; 
# will list number of files in directory
find . -type f -mmin +720 -delete; 
# find and delete anything over 12 hours old

find like grep

find can be used much in the same way as grep to search for the name of files in a directory.

cd /usr/local/lib;
find ./ -name "*gdal*" -print; 
# prints out all files containing the word gdal in their name
find ./ -name "*gdal*" -delete; 
# deletes all the files containing the word gdal in their name

Server seems slow

This is the holy grail of complaints as there are literally millions of things that could cause a server to be slow.

top

top gives one a task explorer like peak into the activity of the server. When one is using top you can press u; key and sort by specific username. c; allows one to get more detail on the processes running. top by default tries to list everything in order by highest use of %CPU.

$ top -u apache
  top - 08:37:14 up 41 days, 20:30,  3 users,  load average: 0.00, 0.04, 0.08
  Tasks: 140 total,   3 running, 136 sleeping,   0 stopped,   1 zombie
  %Cpu(s):  1.7 us,  2.0 sy,  0.0 ni, 95.9 id,  0.0 wa,  0.0 hi,  0.3 si,  0.0 st
  KiB Mem:   1885520 total,  1339856 used,   545664 free,        0 buffers
  KiB Swap:  4095996 total,    52368 used,  4043628 free.   462444 cached Mem
  
  PID USER      PR  NI    VIRT    RES    SHR S %CPU %MEM     TIME+ COMMAND
  11462 apache    20   0 1005776  12212   4072 S  0.0  0.6   1:13.79 httpd
  11492 apache    20   0 1005640  12688   4208 S  0.0  0.7   1:13.22 httpd
  11493 apache    20   0 1005636  12124   4104 S  0.0  0.6   1:13.60 httpd
  12802 apache    20   0  254124   1636    836 S  0.0  0.1   0:00.00 httpd
  12803 apache    20   0  255300   1532    672 S  0.0  0.1   0:14.62 httpd
  12804 apache    20   0  255300   1508    652 S  0.0  0.1   0:00.18 httpd
  12806 apache    20   0 1005940  12892   3548 S  0.0  0.7   4:35.21 httpd
  19201 apache    20   0  656156  13036   1832 S  0.0  0.7   0:00.03 php-fpm
  19202 apache    20   0  656156  13036   1832 S  0.0  0.7   0:00.03 php-fpm

free

The free command will give one a snapshot of the current memory and swap usage. Remember to subtract buffered and cached memory from used to get an actual representation of the amount of RAM in use.

$ free
              total       used       free     shared    buffers     cached
 Mem:       3924876    3663056     261820          0     298528    1755512
 -/+ buffers/cache:    1609016    2315860
 Swap:      4194296     235456    3958840

ps

The ps or process command can be used to get a very detailed account of every single process running at the exact moment you enter the command, it does not refresh like top, but can clue you into process trees and zombie/dead/defunct processes that might not show up in top. I will truncate the output below as it can be quite lengthy.

$ ps fax
  PID TTY      STAT   TIME COMMAND
    1 ?        Ss     0:18 /sbin/init
  480 ?        S<s    0:00 /sbin/udevd -d
 5799 ?        S<     0:00  \_ /sbin/udevd -d
 1057 ?        S      1:29 /opt/chef-server/embedded/service/bookshelf/erts-5.9.3.1/bin/epmd -daemon
 1349 ?        Sl   174:03 /usr/sbin/vmtoolsd
 1836 ?        S<sl   0:47 auditd
 1854 ?        Ss     0:00 /sbin/portreserve
 ...
 2194 ?        Ssl    1:07 hald
 2195 ?        S      0:00  \_ hald-runner
 2224 ?        S      0:00      \_ hald-addon-input: Listening on /dev/input/event2 /dev/input/event0
 2235 ?        S      0:00      \_ hald-addon-acpi: listening on acpid socket /var/run/acpid.socket
 2255 ?        Ssl    1:44 automount --pid-file /var/run/autofs.pid
 2307 ?        Ss     0:00 rpc.rquotad
 3078 tty2     Ss+    0:00 /sbin/mingetty /dev/tty2
 3081 tty3     Ss+    0:00 /sbin/mingetty /dev/tty3
 3083 tty4     Ss+    0:00 /sbin/mingetty /dev/tty4
 3092 tty5     Ss+    0:00 /sbin/mingetty /dev/tty5
 3095 tty6     Ss+    0:00 /sbin/mingetty /dev/tty6
 4846 tty1     Ss+    0:00 /sbin/mingetty /dev/tty1
21685 ?        Ss     3:27 /usr/sbin/httpd
 7578 ?        S      0:18  \_ /usr/sbin/httpd
 7579 ?        S      0:00  \_ /usr/sbin/httpd
 7580 ?        S      0:00  \_ /usr/sbin/httpd
 7581 ?        S      0:00  \_ /usr/sbin/httpd
 7582 ?        S      0:00  \_ /usr/sbin/httpd
 7583 ?        S      0:00  \_ /usr/sbin/httpd
 7584 ?        S      0:00  \_ /usr/sbin/httpd
 7585 ?        S      0:00  \_ /usr/sbin/httpd
 7586 ?        S      0:00  \_ /usr/sbin/httpd
 6584 ?        Sl    24:30 /usr/bin/monit
 6133 ?        Ss     0:07 /usr/sbin/sssd -f -D
 6135 ?        S      0:02  \_ /usr/libexec/sssd/sssd_nss --debug-to-files
 6136 ?        S      0:02  \_ /usr/libexec/sssd/sssd_pam --debug-to-files
 6137 ?        S      0:01  \_ /usr/libexec/sssd/sssd_ssh --debug-to-files
 6166 ?        S      0:05  \_ /usr/libexec/sssd/sssd_be --domain default --debug-to-files

ps faux # the u will give one system usage along with the process tree

USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root      2951  0.0  0.0 117296  1256 ?        Ss    2014   1:35 crond
root      6584  0.0  0.0 179628  2872 ?        Sl   Mar17  24:30 /usr/bin/monit
jenkins   5454  1.4 14.7 2503936 577244 ?      Ssl  Apr09 700:00 /etc/alternatives/java -Djava.awt.headless=true
root     28121  0.0  0.0 101428  3364 ?        Ss   May11   0:09 /var/cfengine/bin/cf-execd
root     28130  0.6  0.1 366888  4236 ?        Ss   May11  24:15 /var/cfengine/bin/cf-serverd
root     28141  0.0  0.1  35624  5284 ?        Ss   May11   0:59 /var/cfengine/bin/cf-monitord
root     17905  0.0  0.0  22180   988 ?        Ss   May12   0:00 xinetd -stayalive -pidfile /var/run/xinetd.pid
dhcpd    18334  0.0  0.1  49000  4376 ?        Ss   May12   0:02 /usr/sbin/dhcpd -user dhcpd -group dhcpd
root      6133  0.0  0.0 199608  2372 ?        Ss   May12   0:07 /usr/sbin/sssd -f -D
root      6135  0.0  0.3 201872 14588 ?        S    May12   0:02  \_ /usr/libexec/sssd/sssd_nss --debug-to-files
root      6136  0.0  0.0 192212  2808 ?        S    May12   0:02  \_ /usr/libexec/sssd/sssd_pam --debug-to-files
root      6137  0.0  0.0 189892  2688 ?        S    May12   0:01  \_ /usr/libexec/sssd/sssd_ssh --debug-to-files
root      6166  0.0  0.1 229764  6916 ?        S    May12   0:05  \_ /usr/libexec/sssd/sssd_be --domain default

kill

Now is the time to learn how to stop runaway processes. Always first try to do a service reset as it will exit “correctly”; and start again and if there are more problems you should be able to see them in a logfile, instead if you just kill it. The process will not have time to show you errors.

$ kill PID PID2 PID3;
# you can kill any number of specific processes as long as you get their PID number from either top or ps fax.
$ kill -9 PID PID2 PID3; 
# this <b>REALY</b> kills it if the above does not work, sometime you have to resort to the most drastic measure to get a zombie process out of there. Use sparingly as regular kill alows the process to stop and let go of files before exiting. adding -9 kills it immediately and may leave behind file locks.
$ killall httpd; 
# will kill all processes with the word httpd in the name
$ killall -u user1; 
# will kill all processes running as the user1 user

Kill - my favorite one liner

ps fax | grep httpd | awk '{print $1}' | xargs kill 
# substitue any process name in the grep and kill a bunch at one time, good for when things are going really crazy

Networking

Often times you will have to determine if a service is working correctly by whether or not it is listening on the correct port or if it is responding at all

ping

The most common network debugging tool, ping. It is good for a quick up/down test of the machine, but NOTE if the machine is super busy it will not respond to ping right away and you could be seeing slow ping times and it could have nothing to do with the network or the NIC.

$ ping -t 8.8.8.8 
# -t will continuously ping, never stop until you quit

telnet

Telnet is a good way to query a specific remote listening port to see if a service is responding as it should

$ telnet sql_host 5432
Trying sql_host...
Connected to sql_host.
Escape character is '^]'.
^]
exit

netstat

netstat produces a list of listening interfaces, ports, and connections to and from the machine.

$ netstat -an
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State
tcp        0      0 127.0.0.1:25            0.0.0.0:*               LISTEN
tcp        0      0 0.0.0.0:5308            0.0.0.0:*               LISTEN
tcp        0      0 0.0.0.0:46341           0.0.0.0:*               LISTEN
tcp        0      0 127.0.0.1:9000          0.0.0.0:*               LISTEN
tcp6       0      0 :::443                  :::*                    LISTEN
tcp6       0      0 :::34459                :::*                    LISTEN
tcp6       0      0 :::22                   :::*                    LISTEN
udp        0      0 0.0.0.0:53348           0.0.0.0:*
udp        0      0 0.0.0.0:111             0.0.0.0:*
udp        0      0 0.0.0.0:123             0.0.0.0:*
udp        0      0 0.0.0.0:36748           0.0.0.0:*
udp6       0      0 :::111                  :::*
udp6       0      0 :::55414                :::*
udp6       0      0 :::48913                :::*
raw6       0      0 :::58                   :::*                    7

$ netstat -anetu | grep 514; 
tcp        0      0 0.0.0.0:514                 0.0.0.0:*                   LISTEN
tcp        0      0 :::514                      :::*                        LISTEN
tcp        0      0 :::5514                     :::*                        LISTEN
tcp        0      0 ::ffff:127.0.0.1:44946      ::ffff:127.0.0.1:9300       ESTABLISHED 497        3320514
udp        0      0 0.0.0.0:514                 0.0.0.0:*                       
udp        0      0 :::514                      :::*                            
udp        0      0 :::5514                     :::*       

ss

the ss command can do the same and more as netstat and should be used in the future. netstat has become deprecated/obsolete to ss since CentOS 6.4

$ ss -apnetu | grep 443;
 tcp    LISTEN     0      128                   :::443                  :::*      ino:202599741 sk:ffff88013beba100
 tcp    ESTAB      0      0      ::ffff:10.0.20.63:443    ::ffff:70.198.42.193:2400   timer:(keepalive,119min,0) uid:48 ino:228626831 sk:ffff880011629880
 tcp    TIME-WAIT  0      0      ::ffff:10.0.20.63:443    ::ffff:70.198.42.193:2426   timer:(timewait,48sec,0) ino:0 sk:ffff88013cb3c940
 tcp    ESTAB      0      0      ::ffff:10.0.20.63:443    ::ffff:70.198.42.193:2421   timer:(keepalive,119min,0) uid:48 ino:228626830 sk:ffff8800027380c0
 tcp    TIME-WAIT  0      0      ::ffff:10.0.20.63:443    ::ffff:70.198.42.193:2410   timer:(timewait,47sec,0) ino:0 sk:ffff88013cb3ca80
 tcp    ESTAB      0      0      ::ffff:10.0.20.63:443    ::ffff:70.198.42.193:2414   timer:(keepalive,119min,0) uid:48 ino:228626829 sk:ffff88006031c100
 tcp    TIME-WAIT  0      0      ::ffff:10.0.20.63:443    ::ffff:70.198.42.193:2401   timer:(timewait,48sec,0) ino:0 sk:ffff880017550e80

Find all incoming connections from unique IP Addresses

$ netstat -tapn | awk '{print $5}' | sed 's/::ffff://' | sed 's/:.*//' | sort | uniq -c | sort;
     1 10.0.20.63
     1 10.0.50.232
     1 10.0.50.234
     1 10.0.50.244
     1 10.0.5.54
     1 10.10.1.231
     2 10.0.5.154
     2 129.82.224.115
    22 129.82.224.137
     3 10.0.20.201
    39 10.0.20.52
     4 10.0.20.30
     9 0.0.0.0

Find all outgoing connections to unique IP Addresses

$ ss -tapw | awk '{print $5}' | sed 's/::ffff://' | sed 's/:.*//' | sort | uniq -c | sort;
  1 10.10.1.63
  12 10.0.20.63
  1 Local
  3 10.0.50.63

lsof

lsof can also be used to determine which process is using a specific port. We once had an issue with a process running on a sendmail port which would not allow sendmail to start on one of our production web servers. This was key in tracking that down and has become a quicker way to check specific ports than netstat.

$ lsof -i :514
COMMAND   PID USER   FD   TYPE  DEVICE SIZE/OFF NODE NAME
rsyslogd 4835 root    1u  IPv4 3354631      0t0  TCP *:shell (LISTEN)
rsyslogd 4835 root    2u  IPv6 3354632      0t0  TCP *:shell (LISTEN)
rsyslogd 4835 root    3u  IPv4 3354623      0t0  UDP *:syslog
rsyslogd 4835 root    4u  IPv6 3354624      0t0  UDP *:syslog

Copying large number of files

rsync

rsync is the method to use when copying a large number of files big or small. The / in the from where to where are very important and is the diference between copying the directory or just everything within the directory to a new location.

$ rsync -avr /nfs/data othermachine:/new/dataarea/ --progress;
# copies local directory and all of it's content data to othermachine:/new/dataarea/data
$ rsync -avr othermachine:/new/dataarea/ /nfs/data --progress; 
# copies all data FROM the othermachine under dataarea/ to /nfs/data/

screen

Highly recommended to do any long running operations in a screen session so that if your connection to the machine timesout, the operation does not end. This is also a way to do things privetly on a server no one can see within a screen session, unless you allow it

$ screen -S copyfiles; 
# start a new session, named copyfiles
$ rsync -avr /nfs/data othermachine:/new/dataarea/ --progress; 
## example command
Ctrl-a d 
# While the rsync is running, Control-a d will detach but leave running the operation and you can continue doing what ever you want
$ screen -r copyfiles; 
# will reattach to the the rsync screen session
$ screen -d R session_share; 
# allows you to <a class="external text" href="http://technonstop.com/screen-commands-for-terminal-sharing" rel="nofollow">share your session</a> with someone else, useful for training.
$ screen -x session_share; 
 # allows your friend on the same machine to connect to your session
Ctrl-d - while inside the screen will detach and terminate the session

I need my script to run at startup

Scenario: On many of our servers we need NFS mounts to other servers/NAS devices to be present at start up so that users/services/websites can get to their data.

$ vim /usr/local/bin/nfs_mounts.sh
# insert nfs mount command to new file nfs_mounts.sh
$ chmod 700 /usr/local/bin/nfs_mounts.sh
# changes permissions so only root user can execute.
$ vim /etc/rc.local
# add the line at the end
$ /usr/local/bin/nfs_mounts.sh &
# make sure their is an &; symbol after the command
$ chmod -x /etc/rc.local 
# the rc.local file should be set to executable permissions

sidenote: Do not use fstab/mtab to auto-mount NFS volumes if they are not present at boot time then the system will hang indefinetly until it is available. our nfs_mounts shell script is the way to get around that.

PATH

Sometimes a problem occurs when one build a project via source and the executable are not located in /usr/bin or /usr/local/bin, the filesystem does not automatically know where to find them via name so one has to type out the full path to interact with those files.

env

$ env | grep PATH
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/var/cfengine/bin:/root/bin

Add new location to PATH

To temporarily add a new location to path type the following:

$ export PATH=/home/user/new_path:$PATH;  
# adding $PATH will keep the old PATH along with the new one, order matters, IF you forget $PATH nothing will work, reboot

To permanently add a new location to path there are 2 options:

User Specific paths

If the user always logs onto the same machine and needs certain files.

$ vim /nethome/username/.bashrc
add export command to the file, examples of some below
 # .bashrc
 export SVN_EDITOR=vim
 export PATH=/usr/lib64/qt-3.3/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin:/opt/grads-  2.0.1.oga.1/Contents:/opt/grib2/wgrib2:/home/gempak/NAWIPS/os/linux64/bin:/home/ldm/bin:
 export GAVERSION=2.0.1.oga.1
 source /home/gempak/NAWIPS/Gemenviron.profile
 export LAPS_DATA_ROOT=/wxrnd/lapsdata
 export LAPS_SRC_ROOT=/opt/laps-0-50-19
 export LAPSINSTALLROOT=/opt/laps-0-50-19

Machine Specific paths

This is probably the “best practices”; way and I would recommend doing this from now instead of .bashrc. As it allows anyone who logs into the machine to have the same PATH settings.

$ touch /etc/profile.d/postgresl.sh
$ vim /etc/profile.d/postgresql.sh;  # make sure file ends with .sh or it will not be read by filesystem
 export PATH=/usr/pgsql-9.3/bin:$PATH
 export MANPATH=$MANPATH:/usr/pgsql-9.3/share/man
# save, log out and log back in
$ env | grep PATH;
# and you should now see the new PATH settings

Repeat this for any/all source built files that have non-standard paths or settings.

Change Hostname

The following files are what you need to edit to change the hostname of a machine

$ vim /etc/hostname;
# edit to newhostname.example.com
$ vim /etc/hosts; 
# add/edit "newhostname.example.com newhostname" to beginning of both lines in file
$ hostname newhostname.example.com
$ export HOSTNAME=newhostname.example.com;

What is Kaizen?

The Japanese words “kai-” which means “change” and “-zen” which means “good.”

Kaizen is a process created by the Toyota company to focus on continual improvement of their manufacturing processes. Possibly creating the granddaddy and literal analogue to what we in the tech industry understand as the general concept of Agile Software development. Kaizen as a strategy requires that there is involvement from all levels of a company. It doesn’t work as effectively if only one business unit attempts to use this method, and that is because for an organization of any size, the change comes from all places. The planning comes from identifying inefficient processes and then rapidly iterating on solutions until a proper one is found. For that to be most effective, you’ll need more than just a single team.

Read more on Kaizen: here, here, and here

How do I use Kaizen?

During new years it is customary in many parts of the world, to set new years resolutions. These are often big monumental changes, and as often joked about, falter after a few weeks. The focus of Kaizen is on small incremental improvements that can be done now, get those all complete, then re-evaluate, the next small incremental improvement to get closer to the overall goal.

Kaizen applied to real life

I gained too much weight over quarantine and need to lose 30 pounds.

If you immediately run to the gym, jump on a treadmill, and begin running, good for you. Keep at it. but what happens to most of us, is life. Simply said, things get in our way, we skip a day, and the next time you have a chance to blink, that goal is three months behind and maybe you’ve gone in the opposite direction and need to lose more weight than when you began.

Instead of setting one goal that is far away and hard to imagine how many days, how many miles, how many meals are between it and you now. Set a weekly goal. I’ll reduce calories and workout enough that I lose 2 lbs a week.

This way at the end of each week, you have mental check-ins and can adapt much more often to changes in your schedule, time, stress, and diet.

At no part am I saying, don’t set a big goal, those are great and we all have them. Just don’t forget to map out a realistic path to achieving that, with check-ins at regular intervals along the way to ensure you remain on track.

How can a DevOps Team use Kaizen?

An example of this, is one I actually witnessed at a previous employer. This company had 2-dozen separate business units (tons of merges and acquisitions) each with their own SaaS product, with entirely different legacy of development and technical debt.

In an effort to standardize processes, tech stacks, and stream line communication, all DevOps and SysAdmin teams became a company wide SRE organization. Suddenly we were in charge of release cadence, creating Service Level Indicators, and overall worshiping the Google SRE handbook, in addition to everything else we did previously.

Each group had to define Service Level Objectives that were better than our Service Level Agreements with our customers. We also had to call out areas that were in danger of keeping us from not meeting those goals, whether those were single points of failure in our tech stack or if they were toil issues, like endless support tickets that could and should be automated.

Once we had this all defined and agreed upon with key stakeholders, we used this live document to re-asses at monthly intervals how well we were doing.

The key point is that we didn’t just have utopian and mostly empty goals like 4 nines of uptime. We had to prove with real data that we were meeting our Service Level Objectives. If we started to dip below that, then we were within our rights to hit the breaks on all development. Circle the wagons, and swarm issues until they were fully resolved.

A few problems with the way we did this (telling you this, because these things are difficult to implement): 1) There wasn’t a clear understanding of what the Kaizen process was. 2) It took too long. We spent a good 2-3 months just documenting things, most of that was due to poor buy in from different departments, so it took weeks to get questions answered. By the time we had everything together, other stakeholders were tired of waiting and their attention fizzled out. 3) It can’t just be a DevOps/SRE thing, it has to include everyone; finance, development, project management, sales, etc

Kaizen and Agile

If everything I’ve described now sounds a lot like Agile, good, it should. But the thing is, Agile is mostly seen as just a way to manage software development. Kaizen is much bigger than that, focusing on the entire company.

The core focus of both Agile and Kaizen rest on continually having periods that one critically evaluates goals, performance, and the gap between them to make the company or product successful. In Agile this is expressed through sprint retrospectives, sprint planning, and having well known definitions of done.

If a lot of your time is stuck with handling issues and not developing new infrastructure or features, then maybe it is time to talk cross-departments. Get out of your silo and figure out a solution. Challenge yourself, your team, and your company. I’m positive the company will be better off if you solve these critical but time consuming bugs. Of course, that makes Kaizen all the harder to pull off, as you need buy in from more than just your team. The flip side of this, is that it gives your department a more holistic view of the problems on the other side. Maybe the ticket management workflow is too cumbersome for that other group, and so they keep personally contacting you to help with a problem. Maybe if you had a week and 3 other dedicated hands on keyboards, you could knock out several critical known problems that keep you awake at night.

Kaizen Events

A Kaizen event (also known as a burst or blitz), can be a great way to kickstart this process in your company. Here’s a few steps to do that:

1. Plan: Gather a lot of people (virtually now), from different cross-functional teams. Create a list of top 5-10 improvement ideas. Talk this through, make sure everyone involved sees the value of these improvements and has a part in achieving it.
2. Execute: During the event 80% of the action items should be achievable. The other 20% should be completable within 30 days of the event.
3. Check: Have a post event meeting where the state of improvements are discussed and documented - did they work, were they effective?
4. Act: Keep these newly adapted processes or fixes; then start a new cycle

Back at Toyota, they made these week long events - five days and one night. So they are intense focused action, all hands on deck. And because they are that way, they should only be conducted a few times a year to avoid over use and burn out.

Be Kind

The main solution that I found to be the most painless to setup was from a small and somewhat newer project called Kind. There are others like MiniKub or Microk8s . All of them provide about the same thing, but watching the Con chatter it seemed a lot more folks were excited to share Kind(ness) as the main getting started tool.

The big difference is that Kind does not use a Virtual machine, but instead uses docker containers, so it feels much more integrated with the cloud native super project Kubernetes, which is afterall, a container orchestrator.

https://kind.sigs.k8s.io/

Getting Started

These instructions are going to be based around Windows specifically, because that is what I use for my daily computer. Because it’s windows, there are a few extra hoops to jump through, but have no worries, because we will step through those together.

1. Prerequisites

The biggest pre-req to make this all work seamlessly, is to have a version of windows (OS build 20211 or higher) that support WSL version 2. If you do not have a version of windows that is compatible with WSLv2, then the other projects like MiniKub or MicroK8S may work better for your use case.

Verify your system is up to date with V2 by following the Microsoft WSLv2 docs

WSL v2 is the Windows Subsystem for Linux, but what is special about V2, is that docker for windows can utilize WSL for managing near-natural (cgroups, shared kernel) docker containers within WSL instead of creating a headless Virtual Machine, which tends to be slow and overall a less than pleasing experience. Another improvement with WSLv2, is the integration with VSCode is much nicer. Through VSCode one can open a session in WSLv2 with whatever flavor of Linux you choose ( I went with Ubuntu 20.04) and have full Linux support/tooling, which often speeds things up as many documents and Cloud Native technologies are geared more towards the Linux/MacOS crowds.

Enable WSL2

With Admin Powershell

Enable-WindowsOptionalFeature -Online -FeatureName VirtualMachinePlatform, Microsoft-Windows-Subsystem-Linux

Reboot

wsl --set-default-version 2

Install a Linux distro from the Windows Store, I’ve gone with Ubuntu 20.04

Docker Desktop

After WSLv2, Docker for Windows needs to be installed and configured. Follow Docker’s instructions here: Docker Desktop

Once Docker Desktop is installed there is one setting to verify to make sure it is using the WSLv2 backend instead of a Virtual Machine.

1. Find the whale in your system tool bar (usually on the bottom right)
2. Right click on Whale, select Settings
3. Within General, make sure Use the WSL2 Based Engine is selected
4. Apply and Restart

Go

Now we are read to install the final Prerequisite, GO. Since we are using WSL for everything it must be a Linux compatible version of Golang and must be greater than version 1.11. Which of course does not come in the default (apt install golang) installation path. To install a recent version of go follow these instructions

# All of these steps are to be ran on Ubuntu 20.04 within WSLv2
wget https://golang.org/dl/go1.15.5.linux-amd64.tar.gz
sudo tar -C /usr/local -xzf go1.15.5.linux-amd64.tar.gz
export PATH=$PATH:/usr/local/go/bin
# to save the path for every start up do the following
sudo touch /etc/profile.d/go.sh
sudo chmod +x /etc/profile.d/go.sh
echo "export PATH=$PATH:/usr/local/go/bin" | sudo tee -a /etc/profile.d/go.sh

Then every time you login the path will be added to the default and magic can begin

Setup Kind

Lucky for us, although the prerequisites are a bit of jumble to get up and running, the install of Kind is a single one line

GO111MODULE="on" sudo go get sigs.k8s.io/kind@v0.9.0

install kubectl

The main and official tool to manage a kubernetes cluster is known as kubectl. Now is the time to install that, and double bonus the wonderful folks at Google, have distilled all of the complexities down to a single one liner.

curl -LO "https://storage.googleapis.com/kubernetes-release/release/$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)/bin/linux/amd64/kubectl"

From here you can start a kubernetes cluster on your machine

sudo kind create cluster
sudo kubectl create deployment nginx --image=nginx --port=80

…and you’re done…well sort of

Windows Funkiness

Since we’re a few layers deep in abstractions and subsystems and dockers, there’s a special way to deploy kind so that it can forward traffic from windows to a deployment which enables you to bring up the frontend deployment in a web browser or suing a cli tool like curl.

Start over

Assuming you started the nginx deployment from the above instructions:

sudo kubectl delete deployment nginx
sudo kind delete cluster

create a file in wsl2, cluster-config.yml

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
  extraPortMappings:
  - containerPort: 30000
    hostPort: 30000
    protocol: TCP

The above YAML once applied will give our windows machine access to a service running within Kubernetes on port 30000.

sudo kind create cluster --config=cluster-config.yml
sudo kubectl create deployment nginx --image=nginx --port=80
sudo kubectl create service nodeport nginx --tcp=80:80 --node-port=30000
curl http://localhost:30000

The above and more information on all of the WSL2 considerations for using Kind can be found on the official docs here: https://kind.sigs.k8s.io/docs/user/using-wsl2/

Summary

If you have Windows 10 OS build 20211 or higher, than this guide can take you from 0 kubernetes to up and running in about a half an hour, with most of that waiting for a few things to download.

Getting started with Kubernetes on Windows is getting easier and more accessible all the time, and thanks to tools like WSLv2 and Kind, now is a great opportunity to get started without a high barrier to entry.

As a lazy soul and DevOps engineer, I’ve gotten tired of manually building and syncing the changes for this website to the S3 bucket which hosts the content you are viewing. My goal was to automate the hugo build and upload to S3 for this very website. A perfect opportunity and chance to throw together some automation to handle these tasks for me. For me I needed 3 features or tasks in my pipeline:

1. A place to push code to/A git repo
2. A way to build the website/Continous Integration
3. A way to deploy the rendered output to an S3 bucket/Continous Deployment

Earlier this week, as the title suggests, I decided to take a gander at AWS’s offerings which are comprised of 4 maybe 5 services - AWS CodeCommit, AWS CodeBuild, AWS CodeDeploy, AWS CodePipeline, and AWS CodeStar. Each service does what it sounds, but much of the cookie cutter templates would prove to not work for me and hugo so AWS CodeStar was out immediately.

AWS CodeCommit

AWS CodeCommit proved to be the most straight forward thing to setup. If you’ve ever started a repo on github or any hosted git solution, they all work the same and this one is no different. HTTPS and SSH authentication/keys are all managed with your IAM user account, which is nice to not have a separate place to manage access and authentication.

The Service claims to be free for 5 or less users so a perfect tool for very small teams or personal projects, even if all you are looking for is a code backup solution.

My own negative for CodeCommit is that there’s no issue tracker, seems like they could setup a CodeTracker service and I’d think AWS could at least compete with Azure DevOps, but as it stands it’s just not a fully realized suite of tools yet. A great tool for personal projects, but not for teams.

AWS CodeBuild

AWS CodeBuild allows you, through a series of click through menus, to define an environment which can run a build process on a container or EC2 instance and output a file or folder to S3 or some other destination for storage.

This took the most time to get right, mostly because I was used to the way Azure DevOps handles Build tasks. In AZDO you can define everything; environment, server type (linux vs windows), steps, tasks, jobs, and integrate with tons of plugins for different services ALL in a single YAML file. What you can define in the buildspec.yml file for CodeBuild is only the task steps that will happen inside the build agent/runner.

Below is the buildspec.yml file for building a site with Hugo

version: 0.2

phases:
  install:
    commands:
      - echo Install Start
      - apt-get -qq update && apt-get -qq install curl
      - apt-get -qq install asciidoctor
      - curl -s -L https://github.com/gohugoio/hugo/releases/download/v0.67.1/hugo_0.67.1_Linux-64bit.deb -o hugo.deb
      - dpkg -i hugo.deb
    finally:
      - echo Install Finished
  build:
    commands:
      - echo Build Start
      - cd $CODEBUILD_SRC_DIR
      - rm -f buildspec.yml && rm -rf .git && rm -f README.md
      - hugo --quiet
    finally: 
      - echo Build Finished
artifacts:
  files:
    - '**/*'
  base-directory: $CODEBUILD_SRC_DIR/public/
  discard-paths: no

The Publish/Artifact Section gave me a lot of trouble. The S3 bucket which hosts the website has all the content at / and luckily there was some miscellaneous forum posts which steered me in the right direction.

• Name needs to be /
• Disable artifact encryption needs to be checked for the files to be readable to folks like you.

As I got that solved, I had everything I thought I needed, the git repo, the build process, and the file delivery to S3. A short lived victory, until I discovered that there is no way to auto-trigger CodeBuild from CodeCommit, you can run them manually, but that’s not the DevOps way. Nay, the only way to automagically start a build process is through CodePipeline.

AWS CodePipeline

The essential use of CodePipeline is to unify all the other CodeBlah services into one final grouping and be the missing automation part that so many of us seek.

• Source = Match to correct repo in AWS CodeCommit
• Build = Match to the correct Build process in AWS CodeBuild
• Deploy = ??? Re-define the Deploy to S3 but more confusingly

The question marks are from me, in my Build process it was delivering the output to my desired bucket. Apparently when ran through Pipelines, the Pipeline ignores those settings and will upload the artifact as a zipped up package to a secondary S3 bucket. Then you have the joy of setting up a third Deploy step of the pipeline to get the Build Artifact and push the output into the intended final destination.

If you are confused, because it seems like we already did that step in the Build process and now you have to set it up again, you’d be correct in that feeling.

CodePipeline also charges $1/per month for an active (ran at least once) pipeline.

Oh My

Overall I think there is good value to gain from having a semi-unified suite of tools to handle CI/CD. Far better than a spattering of separate Git service, Jenkins for CI/CD, and JIRA or Trello for project tasks/issue tracking. I do wish AWS would streamline a few of these features and not make them separate services, maybe for larger projects the complexity is justified, but for my experiment it felt overkill. Also if they dug in and made a project management/issue tracking service (even if it’s a github clone) that integrates with all of them, I suspect more teams would be willing to overcome some of the rougher edges of these services in order to have everything in AWS vs using another third party service be that Azure DevOps or the Atlassian suite of tools for the same.

The old version of my creative website ran marvelously with wordpress on the smallest available instance size inside google cloud. That worked fine, but it was always just a tiny bit slow and the more I thought about it, the more I realized I didn’t need many features. The unfortunate truth is that because wordpress is so popular for blogging, is that it creates a wide target for bots and hackers to attack. In short, all the management features that made wordpress a very nice and easy to use platform, was also creating this insecure mess that I felt I needed to extract myself from. Why does a very simple, mostly static, website require a database backend and bunch of miscellanies ways to log into the “backend”? All this website should do is present content, words, and images, nothing more.

The creative website is now compiled by Hugo, a static site generator, then uploaded to Amazon’s S3 Storage solution, then using tricks with their CDN called Cloudfront, you are served a website with no backend to manage, just content, words, and images. Plus it is far…far…far…less expensive. like ~$1.50/month vs ~$12.

The hard part, that took many months, was finding a way to convert the old website content from html to markdown syntax, fixing ALL the links, and then working with the new system/layout to get everything perfect.

I also created my own hosting/backend for playing music. I was tired of services like SoundCloud or Fanburst retiring or limiting what I could do, so I rolled my own.

Crossing the Streams

In general, I try to keep my creative works separate from my professional work, but today I thought I’d break that rule if only because I’m so proud with how the website turned out.

So yes, I amateurishly write books and music as way to let off steam and because I like creating and exploring a non-technical side of my nature.

For one post only, below is a condensed view of what all I’ve made and feel proud to share with all of you.

Among The Clouds

A Wild Girl, A Toxic Mist, A Rebel Princess && The City Among The Clouds

Jai is a huntress warrior for her tribe. When her village and mother falls sick from a toxic mist, she sets her target on the floating city that abandoned the surface centuries ago.

On her journey to find a cure, she encounters allies, violent arcade games, rebel gangs, kings and queens, and a weapon so dangerous that it could rewrite human history. With the help of Alice, the disgraced princess, Jai stumbles and battles through the maze of skyscrapers, love, and advanced technology aboard the floating city. Fighting to return home before time runs out on her loved ones.

Among The Clouds is a female-fronted, fast-paced, YA novel, set in a backdrop of cyberpunk and science fiction aesthetics. Among The Clouds is Stew Stunes third full-length novel which exists within his shared narrative “Universe of Chaos.”

Read More Here: Among The Clouds

The King’s Challenge

My second full-length novel is available on Amazon! A female-fronted dark fantasy adventure set in the same narrative universe as The Writers of the Universe, this story takes place in the world of Aura 400 years after the events in my first book.

The once prosperous nation of Redren fought back against the darkness. A darkness that entered their world through an impossible event and promised to deliver every living soul to its master, Death. The king and magici of an age long past found a way to defeat the darkness and wrote The Knowledge, a historical account that should never be forgotten. Within The Knowledge, they created The King’s Challenge a once in a generation event to determine the king‘s worthiness and readiness should the darkness return. As family and friends fall victim to a faceless shadow who has haunted Khymn throughout her life, she must decide on the ultimate choice: Save the world and turn her back on everything her people hold dear or be the one who allowed permanent darkness to fall over the entire world of Aura.

Read More Here: The King’s Challenge

The Writers of the Universe

The ending has changed! Through their words written in stories and books, The Writers of the Universe control the lives of us all. A struggling human author and quite possibly the most unlucky person in existence is about to leap from the page and journey to realms unknown in order to bring back chaos and free will to the universe. For without uncertainty, without fear of the unknown, there is no point in going through life if its only purpose is to get to the end of the book.

Read More Here: The Writers of the Universe

This document was written by Myself for a former employer, with contributions from Michael G in Feb 2019. This project was never fully deployed, and it is still to this day, one of my greatest professional disappointments - not being able to get this solution to replace the old solution. The failure was not of a technical nature but a loss of development talent in our project group and a near total refusal to invest in these legacy solutions. I think many companies with a major SFTP type workflow are stuck with something similar to what we had to deal with. It is my hope that companies left with a legacy solution one which worked 20 years ago, but in modern times falls far short on redundancy, security, and even cost. That there are companies, that care enough to invest in a robust solution like the one detailed here, will be able to take this document and build from this proposal.

A project to improve redundancy and security issues related to the current configuration of the upload and processing servers and ssh/sftp access allowed to clients into that server

Original Scope/Idea

Make it so that all upload and import functions are not dependent on a single server in a single AZ.

Move all customer logins to an LDAP/AD solution/get away from local Unix accounts:

Move all data to S3 backend and design an sftp → log in via AD -> S3 directly.

Then on the back-end mount s3 to main and have no direct access to main from outside

What ended up happening

Our original concept for this project was to join AWS’s Transfer for SFTP to AWS’ managed Active Directory in order to send uploads to S3 instead of the main server. What ended up happening through the discovery and Proof of Concept phase of this project is something much different but, in my opinion, an overall better solution. One that scales from dozens of logins to thousands or millions, is reliable, is secure and encrypted, is easy to manage and works for our team, development, and most of all our customers.

AWS Transfer for SFTP does offer its own user management but requires SSH Key authentication only. It does not support password authentication, and it was thought that it would be overly burdensome to have all our thousands of customers understand public/private keys and reimplement every customer to support this change. We want this to be as invisible to the customer experience as possible. This is why it was pertinent to stick with a custom authentication method in order to have success with this project.

We began with an attempt to set up AWS’ Managed Active Directory solution, this was a complete fail for us because you needed more Microsoft Tools, like a Certificate Authority, to provision it properly with SSL.

We then set up an OpenLDAP server, that we ourselves built and would have to manage, a compromise for sure, okay, but not great.

We shifted gears to understand the AWS Transfer for SFTP service AWS offered and even setting up a single test proved to show that we needed far more layers than originally thought. Originally thinking all we needed was some type of directory service plus the Transfer Server. At this point I thought we were doomed, instead, we learned that AWS SFTP needed to connect to AWS API Gateway to connect to AWS Lambda to run code that could verify an account in LDAP to enable access to a single S3 bucket/Directory. What we once thought as 3 new pieces of technology quickly grew to 5 independent pieces.

Lucky to us, AWS had provided a sample CloudFormation template, so we could set up a sample Lambda + API Gateway to see these pieces in action.

From there, the project started to take shape, and we made the discovery that we didn’t need LDAP/AD. It would be much faster and reduce a lot of upkeep cost to store a hash of the login information on DynamoDB and use that to verify accounts.

DynamoDB is a NoSQL solution that allows multi-region, multi-master tables, which we could leverage to make this a much more resilient and smarter application. Users could theoretically connect to the geographically closest SFTP Server and no matter which region their account/S3 bucket lives in have seemingly much faster connection to the service.

Since we had no experience with the API Gateway, all we could do is use the example one and recreate it manually, in order to understand all the pieces of the service. With the help of advanced logging to CloudWatch, we were set up to create our own API Gateway and pull data via the linked Lambda function.

We then built a new Lambda function in python to be able to query DynamoDB and return to the SFTP service the user’s bucket and home directory, along with other data required for the service.

Hooking a new Transfer server to the self-built API Gateway and Lambda function, then working through all the small misunderstandings between the documentation and our own implementation, we were eventually able to get a successful login and upload of data. From there we have continued to optimize the process by modifying the sample CloudFormation template to automatically build our custom Lambda Function, API gateway, S3 buckets, and IAM Roles in each region. At this time, CloudFormation has no hooks to the transfer service so we are unable to automate it using this method.

With the data being available in encrypted S3 buckets, Development can begin to shift how we do imports. Instead of shuffling data around from a manager process to an agent server, the agent service could pull directly from S3 and save a lot of processing and I/O. Furthermore, we think we can hook in additional AWS tools like RedShift and AWS Glue to make powerful Extract, Transform, Load Jobs/techniques and possibly make this whole import workflow completely serverless (!) and much closer to error-free. There’s a lot of work to get there, but the future is not that hard to imagine if we can make sure this new method works with datalink integration and other technical debt items that may inhibit a leap forward like this solution.

With other teams developing ways for client organizations to have a single account for all products, I believe using DynamoDB as the backend ‘Data-lake’ solution for all this account information would be the perfect use case. The way it can scale in both data housing, replication to multiple regions, and handle high volumes of transactional activity makes it a great candidate and one we would be remiss from not considering.

New Concept

• Every Link between services is done over encrypted traffic be that sftp or https, all data stored is encrypted at rest

Old Concept

• Main is a Single Point of Failure in the current design. 
• Allowing direct SSH Access to one of our most important machines is in our opinion a security and reliability threat to this product.

Images Created With https://www.draw.io/

Cost

Estimated Cost per month for all these new services

I believe with the new method what we would save in administrative and work time (although I do not have a way to quantify) and the future opportunities this solution provides, far outweigh the slight uptick in cost for this service compared with our traditional method.

Estimated Cost per month for traditional way

If all we did was set up a second main server in a failover auto-scaling group within each cluster to take over should the primary fail, here is the cost structure of that.

As we add more clusters this cost will continue to expand whereas the cost of the new method would stay fairly static and grow only with consumption of resources while adding new volume to the existing traffic 

To do

• Have Dev and QA review this solution and understand these changes, give feedback to DevOps for any changes
• Test this solution under “production” levels of use
• Integrate with Provisioning App
• If approved for production, the creation of a migration plan/user outreach/education on why these improvements are necessary and better

Important findings with the new solution

This section is for the lessons, caveats, and findings that were made during testing of this new method that may impact our ability to deploy or things we need to consider as we update datalink or inform our clients.

• The new solution only supports the SFTP protocol. It does not support scp, ssh or anything else other than sftp.
• I think this is especially important for Datalink client, as I know it supports both sftp and scp but not sure what the default is

SCP blocked

$ scp attendance.txt my1sftp@99-uploads.xx.net:
my1sftp@99-uploads.xx.net's password:
Could not chdir to home directory /dev/null: Not a directory
/bin/false: No such file or directory
lost connection

• S3 filesystem does not appear to understand the SETSTAT command, which clients like WinSCP use to preserve timestamps
• If you do not make these changes a warning pops up and looks kind of scary, selecting ‘Skip all’ on error will allow the file to upload as normal \

• To disable in WinSCP, Edit Site
• Advanced drop-down arrow, select Transfer Setting Rule
• Unselect “Preserve Timestamps”
• Preset Description - no stat
• Save
• Data stored on S3 are eventually consistent due to data replication
• https://docs.aws.amazon.com/AmazonS3/latest/dev/Introduction.html - Amazon S3 Data Consistency Model

Service Limitations

Demonstration

For testing this solution, we created a new org, called ‘Test SFTP 1’

We then have a script to copy the pertinent account data from the MySQL Database to DyanmoDB.

We have another script to run on main to “migrate” the /home/##### Directory to be on the mounted S3 filesystem and a sym-link to make the change invisible to the application.

Showcase of Functionality

sftp upload

$ sftp my1sftp@99-uploads.xx.net
my1sftp@99-uploads.xx.net's password:
Connected to 99-uploads.xx.net.
sftp>

# Demo of Upload/PUT Speed
sftp> put attendance.txt
Uploading attendance.txt to /99-5-uploads/175781/attendance.txt
attendance.txt                                                                                                  100% 5660KB   5.2MB/s   00:01
sftp> put organization\ full.txt
Uploading organization full.txt to /99-5-uploads/175781/organization full.txt
organization full.txt                                                                                           100%   99KB   1.4MB/s   00:00
sftp> put data_test.txt
Uploading student full.txt to /99-5-uploads/175781/data_test.txt
data_test.txt                                                                                                100%   28MB   7.7MB/s   00:03

# Demo of Download/GET Speed
sftp> get student\ full.txt
Fetching /99-5-uploads/175781/data_test.txt to data_test.txt
/99-5-uploads/175781/data_test.txt                                                                      100%   28MB   5.6MB/s   00:05

# Demonstration of security enhancements - no access to other users' files on the same S3 bucket
sftp> ls
attendance.txt          get-pip.py              organization full.txt   99-5-import.csv       data_test.txt
sftp> ls ../
Couldn't read directory: Permission denied
sftp> exit
Connection reset by 18.224.78.82 port 22

Results

I was then able to go to the website for this org and configure an import using the files uploaded to S3

I was also able to run the import and it appears to have worked without error

Import Success

Test Sftp 1 Import Log
STUDENTS - (IMPORT 3822 AT TEST SFTP 1)
STARTED:         2019-02-08 10:18:33 am EST
COMPLETED:  2019-02-08 10:18:43 am EST
RESULTS:
      Import handed to i3.localdomain IMPORTAGENT
      2019-02-08 10:18:36 am EST Data import started on host i3
      2019-02-08 10:18:36 am EST Data import using database importDB_2000002664
      2019-02-08 10:18:36 am EST Extracting data files.
      Import Files:
      /home/175781/data_test.txt (2019-02-08 15:13:55.00)
      Creating Database for import of org 2000002664
      2019-02-08 10:18:43 am EST Data import completed.

The Initial functionality test shows that the world continues to spin and to the application the change is invisible. woohoo!

¡MAJOR WIN!

Technical Breakdown of Technologies

I am going to structure each piece of this tech stack from backend to the front, so we can understand and further develop these technologies

The ‘99’ designation in names indicates that this is Dev/Testing

CloudFormation

AWS CloudFormation is a service that helps you model and set up your Amazon Web Services resources so that you can spend less time managing those resources and more time focusing on your applications that run in AWS. You create a template that describes all the AWS resources that you want (like Amazon EC2 instances or Amazon RDS DB instances), and AWS CloudFormation takes care of provisioning and configuring those resources for you.

Docs: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/Welcome.html

Amazon provided a sample CFN template that helped us tremendously with getting the API and Lambda portions setup. Of course, we have adapted thier CFN template to work with our own needs, but without that showing us the way this project would have taken much longer to put together.

Original CFN Template: https://da02pmi3nq7t1.cloudfront.net/aws-transfer-apig-lambda.cfn.yml 

One of our Templates: [REDACTED]

The templates that we have written setup S3 buckets, IAM Roles, Lambda Function, and API Gateway for each region. 

There are no CFN hooks into AWS Transfer/SFTP service at this time, so we cannot automate this the same way. We also leave DynamoDB out of CFN since the service has to be there before the rest of the stack

After deploying with CFN for the first time, you must do the following

1. Get Prod API URL for API Gateway
2. Setup Transfer Server with URL from step #1
3. Grab the Server ID, ex.:  s-36d2dbb2bb8941f59
4. Go to lambda and on line 33 set the new serverId
5. Still, in Lambda, Add new Trigger with API Gateway to Prod API Gateway. DO NOT REMOVE the one already there
6. At this point, all should be configured and testing can begin

DynamoDB

_Amazon DynamoDB is a key-value and document database that delivers single-digit millisecond performance at any scale. It's a fully managed, multiregion, multimaster database with built-in security, backup and restore, and in-memory caching for internet-scale applications. DynamoDB can handle more than 10 trillion requests per day and support peaks of more than 20 million requests per second._

Set up a Table

1. Create Table
2. Set Table Name - dydb-99-uploads
3. Primary key - orgid - Number
4. Uncheck Default Settings
5. Add Secondary Index
   1. username - String value
6. Provisioning: On Demand
7. Encryption: Default
8. Select Create

Set up Multi-Region, Replication

There can be no data in the tables to setup Streaming Replication, so you need to do this at the beginning right after setup, before you add any items to the table.

1. Select Global Tables Tab
2. Add region button
3. Select Region from menu, Continue
4. Repeat if necessary, for another region

The items that do not get replicated are settings related to backups or Tags, we have Point-in-Time Recovery backups of tables enabled in 1 region only, and you have to region hop to apply the same Tags to each table.

We’ve set our tables to replicate with 3 regions: us-west-2, us-east-1, us-east-2

AWS Lambda

AWS Lambda is a compute service that lets you run code without provisioning or managing servers. AWS Lambda executes your code only when needed and scales automatically, from a few requests per day to thousands per second.

Docs: https://docs.aws.amazon.com/lambda/latest/dg/welcome.html 

In each region that we have a table configured, we would like to have a lambda function that has Read Only access to the DynamoDB table in the same region.

Set up a Function

• Function Name: lambda-99-upload-us-west-2
• Runtime: Python 3.7
• Role: 99-upload-us-west-2-LambdaExecutionRole-XXXXXXX 
• lambda_function: link to git code
• Memory: Minimum 128 MB
• Timeout: 3 Sec

lambda_function.py

#!/usr/bin/env python
"""
Author: Tom Stewart 
About: Lambda function to handle SFTP Login for API Gateway/SFTp Transfer solution
       Hooks into DynamoDB where we store SSH224 of user creds


Exec: - AWS Lambda

Notes: -
	python3.7
"""
import json
import boto3
from boto3.dynamodb.conditions import Key, Attr
from hashlib import sha224
from base64 import b64encode

def _dyn_lookup_user(event):
    dynclient = boto3.resource('dynamodb')
    # dydb-uploads = prod, dydb-99-uploads = test
    dyntable = dynclient.Table('dydb-99-uploads')
    dynresponse = dyntable.query(
    IndexName='username-index',
         KeyConditionExpression=Key('username').eq(event['username'])
         )
    for d in dynresponse['Items']:
        return d

def _main(event):
    # Replace serverID with newly created one after Transfer setup
    if event['serverId'] != 's-36d2dbb2bb8941f59':
        return {'statusCode':401,
            'body': json.dumps({'statusMessage': 'Unauthorixed sftpserver'})
            }


    # once the test was online, we got thousands of bots trying to login with default creds, so easier to stop it asap then let it go further with processing
    bad_name_list = ['admin', 'support', 'root', 'ubuntu', 'tomcat', 'mysql', 
                    'solr',					'postgres', 'test', 'user', 'user1', 'centos', 
                    'ec2_user']
    if event['username'] in bad_name_list:
       return {'statusCode':401,
               'body': json.dumps({'statusMessage': 'Unauthorized user'})
              }


    authdyn = _dyn_lookup_user(event)
    ctx = sha224(event['password'].encode('utf-8'))
    phash = b64encode(ctx.digest())

    if authdyn is None:
        return {'statusCode':404,
                'body': json.dumps({'statusMessage': 'Username Not Found'})
               }


    if event['username'] == authdyn['username'] AND
       phash ==  authdyn['password'].encode('utf-8'):
       return {
          	'statusCode': 200,
          	'body': json.dumps({'statusMessage': 'Success'}),
          'Role': 'arn:aws:iam::024643489849:role/sftp-transfer',
          'Policy': '{"Version":"2012-10-17","Statement":[{"Sid":"VisualEditor0","Effect":"Allow","Action":"s3:ListBucket","Resource":"arn:aws:s3:::${transfer:HomeBucket}","Condition":{"StringLike":{"s3:prefix":["${transfer:HomeFolder}/*","${transfer:HomeFolder}"]}}},{"Sid":"VisualEditor1","Effect":"Allow","Action":["s3:ListAllMyBuckets","s3:GetBucketLocation"],"Resource":"*"},{"Sid":"VisualEditor2","Effect":"Allow","Action":["s3:PutObject","s3:GetObject","s3:DeleteObjectVersion","s3:DeleteObject","s3:GetObjectVersion"],"Resource":"arn:aws:s3:::${transfer:HomeDirectory}*"}]}',
         'HomeDirectory': authdyn['HomeDirectory'],
          }
    else:
         return {'statusCode':403,
                 'body': json.dumps({'statusMessage': 'Incorrect Password'})
		}

def lambda_handler(event, context):
    status = _main(event)
    return status

The execution role needs to have IAM access to DynamoDB tables and is executed by API Gateway.

Cloud formation configures all of this and downloads the python code from S3 buckets in the same region. S3 bucket: lambda-pl-upload-${region} Object: pl-upload.zip (prod) pl-upload-99.zip (Test)

Test data for lambda function:

test

{
  "serverId": "s-36d2dbb2bb8941f59",
  "username": "test99",
  "password": "fawrawt45245"
}

API Gateway

_Amazon API Gateway is an AWS service that enables you to create, publish, maintain, monitor, and secure your own [REST](https://en.wikipedia.org/wiki/Representational_state_transfer) and [WebSocket](https://tools.ietf.org/html/rfc6455) APIs at any scale._

Docs: https://docs.aws.amazon.com/apigateway/latest/developerguide/welcome.html 

Honestly, the only way I understand this part is to create the sample API Gateway provided by the sample CFN template and recreate it manually to find all the settings and items to configure. That being said I will try my best to outline all the things to setup.

Set up an API

1. • Create API

2. Protocol REST

3. Name: api-99-upload-us-west-2

4. Endpoint Type: RegionalResources

5. 1. Select /, Actions Create Resource- Name new resource /servers
   2. Select /servers, Actions Create Resource- Name new resource /{serverId}
   3. Select /servers/{serverId}, Actions Create Resource- Name new resource /users
   4. Select /servers/{serverId}/users, Actions Create Resource- Name new resource /{username}
   5. Select /servers/{serverId}/users/{username}, Actions Create Resource- Name new resource /{config
   6. Select /servers/{serverId}/users/{username}/{config, Actions Create Method- Name GET

6. GET Method

   1. Authorization AWS_IAM
   2. Request Paths: serverId and username
   3. HTTP Reguest Headers: Password
   4. All other settings leave as defaults

7. Integration Request

   1. Integration Type: Lambda Function
      1. Unselect Use Lambda Proxy Integration  - we will make our own custom mapping in a later step
      2. Lambda Function: lambda-99-upload-us-west-2
   2. Mapping Template

      1. Select “When no template matchets the reguest Content-Type header

      2. • mapping template

      3. application/json

      4. Insert this object

         Mapping Template

         {
           "username": "$input.params('username')",
           "password": "$input.params('Password')",
           "serverId": "$input.params('serverId')"
         }
         

      5. Save

8. Integration Response

   1. No changes to be made here

9. Method Response

   1. Before we can do this step we have to create Models for the response data

   2. On left-hand bar find Models

   3. Create

   4. Model Name: 200ResponseModel

      200ResponseModel

      {"$schema":"http://json-schema.org/draft-04/schema#","title":"200Response","type":"object","properties":{"Policy":{"type":"string"},"Role":{"type":"string"},"HomeDirectory":{"type":"string"}}}
      

   5. Repeat to Create a 4XX Response model:

      4XXResponseModel

      {"$schema":"http://json-schema.org/draft-04/schema#","title":"4xxResponse","type":"object","properties":{"statusMessage":{"type":"string"}}}
      

10. Return to Resources - GET - Method Response

    1. Add Response
    2. HTTP Status: 200
    3. Response Body
    4. Content type: Application/json
    5. Models: 200ResponseModel
    6. Repeat for 401, 403, 404 - Model should be 4XXRespnseModel

11. Actions

    1. Deploy API
    2. Deploy to Prod stage
       1. Any changes made to the api will have to be Deployed to Prod

12. Use Test interface to test that the API + Lambda can communicate. 

13. Go back to the Lambda Function and add a new trigger for API Gateway

    1. Link to api-99-upload-us-west-2
    2. Prod Stage
    3. Save Lambda Function

14. Get Prod url for Transfer service

    1. Back on API Gateway, navigate to Stages
    2. Select ‘Prod’
    3. Invoke URL: right click copy URL: ← this is the custom identity provider URL that Transfer needs
       1. Example: https://redacted.execute-api.us-west-2.amazonaws.com/prod

AWS Transfer for SFTP

_AWS Transfer for SFTP (AWS SFTP) is a fully managed AWS service that enables you to transfer files over Secure File Transfer Protocol (SFTP), into and out of Amazon Simple Storage Service (Amazon S3) storage._

Docs: https://docs.aws.amazon.com/transfer/latest/userguide/what-is-aws-transfer-for-sftp.html

For Test, we have 2 Transfer Servers setup in us-west-2 and us-east-2 regions

For Prod, we imagine having 1 in each region for a total of 3

Set up a Transfer Server

1. Create Server
2. Custom hostname: 
   1. None is fine, for Test we have Amazon Route53 DNS alias: 99-upload-us-west-2.pt-xx.net
3. Identity Provider: 2. Custom 3. Paste the link from Prod URL of API Gateway
4. Invocation role:  4. 99-upload-${region}-TransferidentityProviderRole-XXXXXXX  5. This is created by CFN template and is unique for every Transfer Service
5. Logging Role;  6. can be blank, but if you need cloudwatch logs, select sftp-log~~-~~role
6. Enter Tags
7. Create

The Transfer server takes about 10-15 minutes to set up and become “Online”

You can immediately test with SFTP (so long as you have a set username/password in the DynamoDB table or one can use the very effective Test_transfer python script

test_transfer.py

#!/usr/bin/python
import boto3
client = boto3.client('transfer', 'us-west-2')
response = client.test_identity_provider(
    ServerId='s-36d2dbb2bb8941f59',
    UserName='test99',
    UserPassword='fawrawt45245'
    )
print (response)

Route53 / GeoDNS

Amazon Route 53 is a highly available and scalable Domain Name System (DNS) web service.

Docs: https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/Welcome.html

Set Up a GeoDNS Record

Steps to set up a GeoDNS record for one address so that users are provided with the fastest response time based on their location

1. Hosted zones, xx.net
2. Create Record Set
3. Name: 99-uploads.xx.net
4. Type: CNAME
5. Alias Yes
6. Alias target: Select 99-uploads-us-west-2.xx.net.
7. Routing Policy: Latency
8. Region: us-west-2
9. Save Record Set
10. Repeat steps with same Name & Settings, but different Alias Target/Region

After discussing GeoDNS with the team at large we determined not to go forward with using this feature and instead just having regional endpoints, this is because we worried about the cost of intra-regional data transfers, something I hadn’t thought of when developing the solution. This is a strong case for over-engineering a project beyond the scope of what’s necessary.

IAM Policies and Roles

CloudFormation generates the Roles and Policies that are generally needed for each service. However, it does not add permissions to the existing sftp-transfer Role for the S3 buckets that is attached via the Lambda function if there is a successful authentication attempt

sftp-transfer: arn:aws:iam::111111111111:role/sftp-transfer

When production is created, we will have to attach the policy arn:aws:iam::024643489849:policy/pl-pt-uploads-sftp-transfer to sftp-transfer in order to allow uploads and downloads from/to the S3 buckets

The Lambda Function passes the following Role and Policy. The Scope-down policy enables a chroot like action so that the user can only view files in their directory.

Role and Policy

'Role': 'arn:aws:iam::111111111111:role/sftp-transfer',
'Policy': '{"Version":"2012-10-17","Statement":[{"Sid":"VisualEditor0","Effect":"Allow","Action":"s3:ListBucket","Resource":"arn:aws:s3:::${transfer:HomeBucket}","Condition":{"StringLike":{"s3:prefix":["${transfer:HomeFolder}/*","${transfer:HomeFolder}"]}}},{"Sid":"VisualEditor1","Effect":"Allow","Action":["s3:ListAllMyBuckets","s3:GetBucketLocation"],"Resource":"*"},{"Sid":"VisualEditor2","Effect":"Allow","Action":["s3:PutObject","s3:GetObject","s3:DeleteObjectVersion","s3:DeleteObject","s3:GetObjectVersion"],"Resource":"arn:aws:s3:::${transfer:HomeDirectory}*"}]}',

The special Policy cannot be attached with a traditional ARN and thus requires posting the entire JSON object back to the user at login time. https://docs.aws.amazon.com/transfer/latest/userguide/users-policies-scope-down.html 

• A Funny aside, as we were starting to evaluate this service, the original Scope Down policy did not have the ${transfer:HomeFolder} variable and thus required the userName to match the Directory in the bucket. The result was that we had a lot of trouble getting the chroot like scope-down policy to apply correctly. We had a mysterious case where with one S3 bucket the chroot worked, but not with a different bucket. Never could figure out the difference. Lucky for us they updated the ability almost overnight while I was experimenting with this.

S3 Buckets

CloudFormation handles all of the bucket creation for the “stacks”. Each bucket is created in the same region as the main server it should be configured to handle data for.

Production S3 buckets are not yet provisioned as we still need to test S3→ Main Solutions

Mounting S3 buckets to a Filesystem

To Mount S3 to the main server, we leveraged an opensource project called s3fs-fuse. It is well regarded through the community and is used in many production systems.

It is in the standard or EPEL repositories for both CentOS 7 and Ubuntu servers.

s3fs-fuse github

Set up of S3 Filesystem

It happened on 99-5, on main, i1 and h1, for testing purposes and proof-of-concept. All servers are CentOS 7.x. All actions as root.

Install and Mount S3 filesystem

$ yum update (to 7.6)
$ yum install s3fs-fuse
$ reboot
$ echo [access keys in .aws/credentials in format ID:KEY]  > /etc/passwd-s3fs
$ chmod 600 /etc/passwd-s3fs
$ mkdir /sftp
$ s3fs 99-5-uploads /sftp -o passwd_file=/etc/passwd-s3fs

It is important that each upload action (i.e. copy to s3) starts with first locally copying what needs to be uploaded (transparently by s3fs) and then upload it. That means that there must be enough space in the local filesystem to support any file, which will uploaded to s3. The place where the intermediate temporary file is created, can be set with the use_cache directive.

In order to permanently set mount at boot, in case we mount the bucket 99-5-uploads to /sftp as root and temp. files in /tmp/s3-cache, we will add to fstab as follows.

s3fs#99-5-uploads  /sftp   fuse   allow_other,use_cache=/tmp/s3-cache,rw,nosuid,nodev,uid=0,gid=0   0 0

Note: For the tests, we disabled caching (which works only for reading from S3, i.e. GET) , the setting was as : use_cache="" .

Git File Tracking 

Since s3fs uses the Fuse sub-system and not a true filesystem, there is no way for inotify to notice changes to s3 and notify the filewatcher.py process.

I’m flagging this as a process that will need to be updated to harness S3 Object Version History features in order to replicate the git features we’ve set up. 

When testing on 99-5 server, we had to move the path because the symbolic links caused issues with another internal tool that leverages inotify and local git repo checkins for file tracking (which does work with writing files from the system, just not via sftp/s3)

Size tests

Credit to Michael G, a former colleague of mine who led the performance testing of this solution

The S3 buckets, at the time of writing (Feb. 2019) can support up to 5TB, with files up to 5GB. However, the upload process may happen in two ways: As a single stream for files up to 2GB and multipart, which allows up to 5GB.

The s3fs, by default support multipart upload. However, it chooses the single stream way, if the file is up to 20MB. In case of streaming-like commands (e.g. dd to the s3) the s3fs sees it as less than 20MB and uses single stream, with the limit of 2GB.

For the size tests (and speed of single uploads) we used two types of files, text and binary, and various sizes, 1MB, 5MB, 10MB, 50MB, 100MB, 200MB, 500MB, 1GB, 2GB and 5GB. The purpose of that is not to simulate the actual workload, instead to stress the s3fs and know its limitations. The sizes were chosen to compare up/download times and investigate any speed differentiation in relation to size. The text files are close to real text, i.e. created by arbitrary words of the american-english dictionary, not just letters. This way, any compressing capability will exhibit its merits.

Speed of transfer from outside

1. Uploading to the sftp server was found practically the same with uploading via sftp to the master server. The result is expected since the AWS SFTP is nothing more but an instance that reads/writes files from/to S3.
2. The upload/download speed was found quite stable and consistent, with about 5MBytes/sec from Pennsylvania, USA and about 3.3MBytes/sec from Brno, Europe.
   This measurement, although not exact, is indicative for the expected upload speed from most, if not all, of the customers.
3. Since the traffic from/to the outside world is more or less limited by the Internet-AWS connections, it is important to have a picture of the actual abilities of the S3 mount. This can be measured with copies in both directions, i.e. uploads and downloads. The test host is i1. All results are saved in an worksheet.
   1. The tests consisted of repetitions of uploads and dowloads, overwriting or writing new files, repeating same file and whole set twice, with or without cache. Sizes where from 1MB to 5GB.
   2. The most prevalent conclusion is that the results are highly inconsistent, with speed varying between 5 MB/s and 30 MB/s for upload and 60 MB/s and 200 MB/s for downloads. An exemplar case is the not-cached download of 50MB, which saw speeds between 28 MB/s and 66 MB/s.
   3. Binary and text files offer similar speeds. In some cases they seemed to be different, but that was widely inconsistent: In other cases the difference was reversed.
   4. The results show that cache has no effect in writing (i.e. uploading).
   5. Caching in downloading, though, offers an improvement of 1.8x to 7x comparing to not-cached. The highest benefit is in smaller size files, below 50MB.
   6. The speed for uploads reaches highest values from the 50MB.
   7. Download speed changes differently, with highest values for sizes between 5MB and 1GB, but lowering for 2GB and 5GB. That applies to both cached and not cached tests.

Performance tests (stress)

The test consist of multiple reading and writing to S3, mounted to various numbers of systems. The reason is to understand and document the abilities of the S3 as mounted filesystem (s3fs).
Based on the outcome of this test, it can be decided whether agent servers can directly access files on S3.

Describing the setup

The number of agent servers can be up to 10, therefore this is the max number of test hosts.
For better understanding of the scalability of S3 as a file system, the tests run various mixes of reading and writing, from one host, single thread, to ten hosts with 10 threads each.

As target objects (i.e. files) are used the files created in the size tests, only the text version, since there is no usable difference, while those files resemble more to the actual load.

The hosts used are T3.medium, which -according to documentation and various sites - offer a good performance.   It is important to note that the AWS instances offer very different throughput in networking. A simple T2.nano can reach 100Mbps, while the chosen t3.medium offers quite stable 250 Mbps, with bursts that reach 10 Gbps.

There is no scalability in the various instance types, only those two numbers, so there is no reason to try different instances.

Preliminary results

These tests help identify the right scaling mix of reads and writes and a basis for comparison. It also helped fine-tune the tests.

Info from: https://cloudonaut.io/ec2-network-performance-cheat-sheet/ , https://aws.amazon.com/ec2/instance-types/, https://aws.amazon.com/ec2/faqs/#Which_instance_types_support_Enhanced_Networking, https://docs.aws.amazon.com/AmazonS3/latest/dev/request-rate-perf-considerations.html

1. The time spent reading from the local filesystem is hardly measurable, being consistently under 0.09 sec for 200MB and under 0.004 sec for 10MB. Compared to the actual time spent within s3fs, the local read is negligible. It does not participate in the outcome, since it is lost by rounding the measurements.
2. Writing to the local filesystem is quite higher (around 1.5 sec), still much smaller than the s3fs part. However, that part is skipped by sending the read data to /dev/null. This way, the time measured for get actions is cleaned from local filesystem overhead.
3. As seen from the results, the effect of threaded multiple copies is more prevalent with bigger size of files. The speed degradation is (averagely) linear (the increase in performance with 200MB files is likely statistical “noise”, possibly some arbitrary improved performance in S3-side or other AWS-infrastructure change).
4. The results show that it is very important to run the tests from different hosts, to identify the load: Is it S3-based or host’s network throughput ?
5. Also, the results show that there is no need to run all the threaded tests. However, for validity and confirmation, there could be a set limited to 1 up to 8 threads.

	Size (in MB)
Th-count	1	5	10	50	100	200
1	3,9	9,1	13,7	22,8	22,2	21,5
2	3,7	10,5	13,0	20,0	20,5	20,4
3	3,9	10,1	11,5	17,9	17,5	18,8
4	3,2	8,4	12,0	16,7	17,4	18,7
5	3,1	9,0	10,1	18,1	17,2	16,5
6	2,5	7,4	9,2	17,4	18,3	14,8
7	3,0	7,4	9,7	15,9	15,1	13,1
8	3,0	8,4	9,4	15,3	15,1	11,5
9	2,9	7,6	9,4	13,5	15,3	10,5
10	2,7	7,2	9,3	11,7	15,1	12,7
AVG	1,74	8,51	10,73	16,92	17,37	15,84
MIN	2,46	7,20	9,21	11,66	15,06	10,52
MAX	3,92	10,53	13,66	22,77	22,18	21,46

Results

The final tests comprises of the following (code attached as the_tests.sh):

• Threaded copies from the host to the S3 (i.e. PUT or upload)
• Threaded copies to the host from the S3 (i.e. GET or download)
• Threaded copies between the host and the S3 in both directions.

The tests are repeated 30 times (i.e. 30 iterations of the same copy command), on each thread.
Threads run from 1 to 15, i.e. 15 parallel copies from a single host.

The hosts were finally as follows:

• a single t3.medium
• a single m4.xlarge

Unfortunately, this is how far the setup-tests could go. There were continues issues with running the tests. Running 15 repetitions and up to 10 threads, or running the test with only 1MB files in place of each size, it caused no errors. However, when the full-scal setup ran, it never made it to the end. Sometimes, the cp command stuck, while others the whole host faced problems, like an unreachable /var/TEST directory, while in tests. In most cases, the only solution was to reboot.

The configuration of s3fs (i.e. mount options) was the aforementioned for m4.xlarge and t3.medium first run. In t3.medium, another try was done with different configuration (described later), which indeed alleviated the issues.

Tests need to finalise and re-run, at some point.

Caveats

There may be some issues with stability. There were cases that the connection was dropped and, in any action, appears: “Transport endpoint is not connected”.
It may be solved with various limit, timeout and retries parameters. The mixture tried now is: stat_cache_expire=10,retries=9,readwrite_timeout=30,multireq_max=50,connect_timeout=20 .

There are symptoms with using the S3 that might stem from the test or the hosts used. If such symptoms persist, they may critically impede application on production.
The symptoms are explained in current (2019-03-13) results section: The m4.xlarge host did not make it, with the full scale tests.

only put 50 1_15_14 6.12 7.25 3.20 1.80 7.82 
cp: error writing '/sftp/uploads/test.50m.bin.1_15_14': No space left on device 
cp: failed to extend '/sftp/uploads/test.50m.bin.1_15_14': No space left on device 
         2.39 7.54 8.01 1.61 5.02 
cp: error writing '/sftp/uploads/test.50m.bin.1_15_14': No space left on device 
[...]
only put 5 1_15_14 1.08 0.89 1.69 1.28 0.82 0.52 0.85 0.75 0.51 1.71 0.73 cp: cannot create regular file '/sftp/uploads/test.5m.bin.1_15_14': Input/output error 1.05 cp: cannot create regular file '/sftp/uploads/test.5m.bin.1_15_14': Input/output error 2.03 cp: cannot create regular file '/sftp/uploads/test.5m.bin.1_15_14': Input/output error 

Source of info:

• https://github.com/s3fs-fuse/s3fs-fuse/wiki/Fuse-Over-Amazon
• https://code.google.com/archive/p/s3fs/issues/314
• https://github.com/s3fs-fuse/s3fs-fuse/issues/254
• https://github.com/s3fs-fuse/s3fs-fuse/issues/506#issuecomment-454331760
• https://github.com/s3fs-fuse/s3fs-fuse/issues/340
• https://github.com/s3fs-fuse/s3fs-fuse/issues/152#issuecomment-278894805
• https://github.com/s3fs-fuse/s3fs-fuse/issues/748

What does a script or program or binary executable sound like? It may seem like a silly question, but this is actually one I have an answer for. Did you know that with some free open source tools you can transform any type of data into a digital representation of what a computer thinks it should sound like, and thus create “music” in ways completely unintended by most anyone.

Data Bending & Glitch Art

No, this isn’t the fifth element of the next Avatar series. This is a technique that came from the before time; when things were analog, like cameras, recorders, tape reels, VCRs, and really any medium one can imprint on, would wear and tear over time and use. In some cases, these pieces of analog equipment could be altered in a way that produced extra effect on the final image or recording. Like adding a physical and permanent instagram filter to ones camera, or an unintentional echo from a lousy microphone. Full hipster points, if your gear did this by accident, but as all cool things go, the more people notice them, the more people want to replicate it and try it themselves.

So here we are, far past the age of tape drives and dial up, where we don’t have to wait for our gear to break down for us to be able to play and experiment. The emergent term for this kind of fun is called, Glitch Art or Data Bending

I was able to achive the above effect by, saving the image as a .TIFF. Using Audacity, import as RAW uncompressed “U-Law” encoding, Adding an invert effect at the middle of the sound profile, then exporting it as .TIFF with RAW uncompress U-LAW encoding.

Of course, this post is not about visual but auditory data bending.

Bring the Noise

Using audacity, we can import ANY file and represent that as sound. Now you can solve the question for what Git, Google Chrome, or even the exported image from above sound like.

As the good man, Chuck D, once said….YEEAAAAA BOY…bring the noise

The fun part ( if you are okay with getting lots of duds of harsh white noise ) is exploring this very strange avenue to produce music never before heard by human kind.

The Steps to do this

1. Download/Open Audacity
2. File -> Import -> Raw Data
3. Select ANY file, anything above a few dozen KB will have enough “time” to get meaningful sounds out of.
4. Encoding: U-LAW or A-LAW - U LAW seems to have the best results
5. Byte Order: Little Endian
6. Import and Modify to your hearts desire
7. Once happy with your experiment, Export As MP3 or WAV

The Final Result

Shameless cross promotion aside, here is a live example of taking sound from different binary data, slapping a beat on top of it, and calling it music.

Hurts doesn’t it…that’s intentional, if you read the book it makes sense narratively for The Firth World. A world created to be punished by it’s authors/creators.

In High School, my music teacher once told me, “Music is just organized noise."
With this method, we’re testing that theory. Sure this is no 4:33, avant-garde probably would roll it’s eyes at calling this music. And maybe not in it’s raw form, but part of my love for music comes from breaking rules and boundaries. This is forcing terrible harsh noise, kicking and screeching, into some semblance of a tune that maybe with the right composer and audience could call music.

My aunt once told me after listening to some of my early guitar recordings, “What’s the point if it’s just noise? No one want’s to hear that."
Yeah they were bad, but I was having fun. So I made a small album of literal machine noise using these techniques. The Punk in me rides on, I guess.

Final words: don’t be afraid to bend your files and data…but like don’t break your data. Although, audacity makes a temp copy of the file during editing, don’t like overwrite the source file. That will break things…have fun.

I’ve made the decision to split my digital resume from my digital blog. The resume is basically a few pages that infrequently change, and this is in conflict with the nature of a blog. So I’m splitting the work. The resume site will remain as is, with but a single link change to forward folks to this new blog.

To make both my historical posts and new content faster to produce, I’ve since learned about these neat little technologies called static site generators. In essence, after the initial setup, make it much easier to add content and publish quicker without worrying about links and formatting, just words and content.

• This blog is created with Hugo, a static site generator written in go
• The Theme is slightly adapted from hello-friend-ng
• This blog site is hosted in AWS S3, and distributed via AWS Cloudfront
• This list has gotten too long

Memory of a Postmortem

I am writing about a past experience, one that never ceases to make me laugh and shudder at how crazy and simple the solution became once we had it figured out. I do apologize for lack of screen shots, this all occurred over 3 years ago, but I think the lessons learned are still relevant.

Background

One of the products I was gifted with administering was a very large content management solution, that on this single product when you added up each individual site that we hosted on this platform, was in the millions of views a day. This was all co-hosted in a datacenter, a Webscreen hardware solution to block malicious traffic, running a series of Cisco firewalls, some F5 Load Balancers, 20 some VCenter clusters, and five or so storage back ends. The app itself was a very bloated java app, that required no less than 28GB of RAM per running process, we generally ran 2 processes on each VM, with northward of 60 VMs in total to handle the traffic. Totally ready for cloud scale, am I right?

All of that + not a single solution for centralized logging and because this product had changed hands so many times through acquisitions and mergers, we had not a single developer resource available. Just keep it alive, it's still making money, but let's not do anything to improve Quality of Life. Being someone whose job it is to fix things, I pushed back as hard as I could on this.

Fun side note -- when I joined, the team had migrated monitoring to New Relic, but the old Nagios instance was still online and was emitting over 700 critical notifications per minute. The old team had built a lot of custom monitoring to alert when things were going south with the app, the new team pretty much lost all that insight when trying to migrate to New Relic. Two months of side work between sprints, I was able to clean out the cruft and turn Nagios back into it's former glory, and it became our first place to look for issues presenting.

Where Things Went Wrong

Blamoo! DOS attack! everything is on fire, and people are losing their minds...about once a month we'd get a huge surge in traffic, our Webscreen solution would fail to respond, all the traffic came in, slammed our very heavy java apps, and 20 severs or so would fall over, and it would take us an hour or so to mitigate and get things back in order. It was very disruptive to our clients, and these events felt entirely invisible to me, the webscreen interface was overly clicky, cisco logs were too fast to read and isolate patterns.

...and eventually it got worse...like every week we were down for hours, the time between events getting shorter, our managers and clients getting redder in the face each time. Jobs were threatened, it got so bad, that our firewalls were even crashing from the overwhelming amount of traffic. They would fail over, die, fail over again, and to make matters worse, our secondary was not in full sync with our primary, so things like certs would fail and have to be manually added up to the secondary. What do we do? It's chaos.

Where Things Went Right

We are blind, I decried to our manager. We've got to build something that can show us what is happening. Out of choices, and having a good track record in the past, my manager let me off the reins. I abandoned my ongoing sprint, and created a new epic to do this.

I knew we had to be able to visualize the traffic, then narrow down to exact IPs. Which sounded more and more like a centralized logging solution.

At the time, the 2 most popular open-source logging solutions were ELK(Elasticsearch, Logstash, Kibana) stack and graylog2. I had small experience with each, but wanted something that could handle billions of events from our firewalls. I didn't think I needed app metrics, or data from the F5s. We just needed to see the traffic, then we could build out black lists and anything else. This solution would be highly targeted to this need only, so as to not waste time with extra configuration or use cases.

I ended up going with ELK stack. I had tried graylog2 for another solution and while some things appeared easier, without the ability to transform logs(Logstash), it gave me a lot of trouble in my testing. So I moved on to ELK, I configured Kibana and Logstash to be on one VM, with 3 nodes of 2 TB volumes, for elasticsearch.

We were able to very quickly build visualizations and dashboards, the next DDos hit and we were prepared within about 3 days of time from Okay to Ready.

This is the part I wish I still had screen shots for -- It became immediately obvious that what the firewall was seeing and what the Webscreen was showing were opposites. The webscreen sits in front of the firewalls, so the patterns should be the exact same, with some reduced amount of traffic on the firewalls if the webscreen was activated. Well, we got the opposite, The webscreen saying I don't see a huge spike of incoming traffic, our firewalls logs jumping from thousands to millions per second. Something was very very off about the whole situation. I talked with other experts around our team and the only thing we could agree on was that it looked like the webscreen was wrong about everything.

Digging into the logs further, we realized that a large majority of the IPs in the flood were our own! It turned out that the webscreen had a software button that could flip which "side" was being monitoring. Ingress or Outgress, somehow through all the months of continued Dos, we had missed that someone had flipped the configuration to monitor our Outgress traffic between the webscreen and the firewalls. We were defending the internet from ourselves (probably a wise choice jk).

After pressing the single button, we pretty much did not experience a Denial of Service event that took us down ever again. Although any proof had long since gone away, the sheer amount (this was not the only incident) of misconfigurations and poor practices lead me to believe that there were some intentional bombs left behind by old teams. But with the right attitude and tools, you too can root out those gremlins and defeat them. Turn an old legacy behemoth, into something that can float as long as the company demands.

The ELK stack also showed that over 75% of our internal traffic was coming/going to a dead graphite server. It had locked up so bad, it had rebooted into read only single user mode, but the agents on every server didn't care and kept sending, failing that, and resending, on and on.

In Summary

When big problems present, have a way to comprehend what is happening, whether this is centralized logging with visualizations, or good and deep monitoring. Which technology you choose to accomplish that goal, is much less important than getting something quickly stood up and immediately provide value to your team.

After all those lessons learned, and some more company shuffling we had a new product (the one above finally End of Life'd), once again without centralized logging and wouldn't you believe it management would not approve me to work on a solution like the above. Sighting that since we were in the cloud it would be too expensive. I argue that the expense from downtime and disruptive work is far more than the cost of storing and computing that data, but could not get them to see it that way.

One part of my head is firmly planted in technology, while another is constantly working on my side-projects such as fiction writing. Sometimes those things cross into each other - I like to imagine stories and one of these sessions was about a person working a menial job and slowly realizing that they were a small piece in a Neural Network, they keep trying to solve the same problem in their life, each time trying something different, and as they begin to solve it and become aware to their reality, the AI controlling everything begins to crumble. A bit like The Matrix meets The Net.

OR something like that...

The other key ingredient in this novel idea would be to have an actual AI write parts of the book and see if people could tell. The beginning would be written entirely by myself, with the AI slowly fading in like intrusive thoughts in between the narrative, and finally taking over the whole story.

What I didn't know was if that was technically possible or if I was seeing too much fiction in my science fiction.

And so the following is an account of my adventures trying to harness the current state of Machine Learning and make use of for my project.

GPT-2

In early 2019, the company OpenAI released a new Machine Learning model called GPT-2. Claiming that it was so good at text generation that they had to reduce the size of the model offered to the public for safety and ethical reasons. Right away this one felt like the real deal, I had found a half dozen of random student projects on github, trying to DIY different style of GANs or RNNs for text generation, but none of them felt like they would be powerful enough. In addition to the hype around gpt-2, there were a number of helpful articles and walk-throughs on how to get started using it, and or training it on new data. This is what I really wanted, the ability to train the model on previous and highly rated fiction novels to see if it could compete.

Data Wrangling

To train any ML model you need data, and lots of it + plus time and a hearty GPU. After the side quest of acquiring the proper hardware and getting CUDA/tensorflow setup, I was ready to tackle the Data part. Over the years I've collected a hefty amount of ebooks. For this project, I needed more, so I began seeking out alternative sources until I had over 1000 different books from Fantasy and SciFI.

All of these books came in either epub format, .lit (?) or a flat PDF, so the task was to convert all of those to text, which then could be imported into a binary blob for training. I wrote a script, ( eboook.py ) to handle this task, but what I learned is that no 2 ebooks are the same, with different headings, section names, etc. It's almost infeasible to select just "Chapter 1" and output that, as in most cases that could be section 002, or 004, or any number really. First pass was to output the full book, into individual text files.

I want to pause here and give a big shout out to Gwern, the creator of thiswaifudoesnotexist.net and the gpt-2-poetry project. His work allowed me and many others to follow in his footsteps and use this awesome technology. User nsheppard's Git repo seems to be the best for training GPT-2 on new data.

$ ls -l ../data_origs/originals/txt_proc/*.txt | wc -l
1019
$ PYTHONPATH=src python3 encode.py ../data_origs/originals/txt_proc/ novels.npz
# takes like 15 minutes with the large dataset
$ PYTHONPATH=src python3 train.py --model_name 117M --dataset novels.npz --save_every 1000 --sample_every 100 --run_name novel1
Loading dataset...
100%
dataset has 38779104 tokens
Training...
[16 | 6.98] loss=3.34 avg=3.34
[17 | 8.45] loss=3.25 avg=3.30
[18 | 9.93] loss=3.29 avg=3.29
[19 | 11.42] loss=3.27 avg=3.29
[20 | 12.90] loss=3.22 avg=3.27
[21 | 14.38] loss=3.20 avg=3.26
...
99 | 131.98] loss=3.21 avg=3.22
======== SAMPLE 1 ========
ok to them, he thought.  
They turned around—and the girl turned, and she fell.  
Hitting a hard wall, he pulled a gun from her waist, and fired at her.  
She did not dare go to ground.  
The girl was unconscious.  
Her breathing was heavy, and she did not speak.  
She saw the light of a flash in the sky.  
A massive star would have exploded that day.The next day she and Alia were 
working with the man who lived in a forest,and in the morning she saw that 
it had exploded and the town had fallen.  
She recognized it as the village where her brothers lived.  
It was small and quiet here, but she could tell that men had been there 
before the town's fall.  
She had a hard time telling the difference between what might have been the 
town and the village.  
They were very far away, near the old town, but they were near what she 
called the 'gates.  
She had her head above the ground as well, and she did not want to go in 
and shoot anyone.  
She had left her weapon behind, and she went to the town gate.  
She looked out of the gate, and she saw that there were two men hiding 
beneath a stone-thick fence.  
She took off her gloves, and saw that they were carrying the two men.  
She stood before them.  
Her eyes were dark and wide, and she did not dare go in.  
She saw him running toward the fence, and she did not dare go in on him.
She made her way through a man's door, and saw that he was carrying an iron.  
With her pistol, she fired at him.  
With the gun and the iron gun that was with her, she shot him in a single 
blow, and the man who had shot him stood 
up and ran toward her.And the same thing happened to her brothers and 
sisters—where are you?She went out to the woods, knowing she had to.  
With a knife and a saw, she started into the woods, aiming toward the 
hills below.  
Some day she and Alia had got back down the road, and her brothers and 
sisters saw her go.And she stood in the open, and she thought, I am going 
to take you back down the road.  
It was going to be a long day, and she knew what it was.  

...
# and between 24 and 48 hours later it's been properly cooked.

For all trained models, I used the same couple of statements to see what results I would get.

“What am I missing?” He asked to the machine before him. The machine was a metal contraption with a sphere in the center. If his machine could keep that sphere in the center without external energy sources, he would achieve the improbable. Infinite Energy.

$ python3 src/interactive_conditional_samples.py --model_name novel1 --temperature 0.9 --top_k 30
model prompt >>> “What am I missing?” He asked to the machine before him. The machine was a metal contraption with a 
sphere in the center. If his machine could keep that sphere in the center without external energy sources, he would 
achieve the improbable. Infinite Energy.
======================================== SAMPLE 1 ========================================
The infinite.

He had no other answer; why was there no other thing to get him to understand? All he knew now about the existence
of the infinite was — could NOT be.

He looked around, saw it, saw the other world outside, and saw it, and it held him and him, and he saw, and felt it,
and yet it did not release him to the world.

This was a machine.

This was a machine belonging to the agency of others, of all the universes, and all the entities that comprised it.
“We are not brothers,” the machine said.

“We are not like other men.

We are not as we were.”The man had no words for “brother” but he had said that, and the machine responded by clicking.

The machine had to know the world.

Could he guess what it would make of his friends? They were holding him in a circle of three, not two.

The circle of three was larger than the one at which he was holding.

“Your friends are not the only entities in existence who are connected to you,” the machine told him.

“They have been here before and they are still here.

They are the Earths, the Great Sphere, that we know of you.

They will be all the way to Earth.

They will stand in the clearing before you.”“It is a prophecy of mine,” the man said, feeling the machine’s presence.

“That we must be ready.” His hand flapped like the hand of a lover.“All in good time,” the machine said.

“For the Balance.”He was, of course, not a great man, but his spirit was tender.

He knew that the Balance was behind everything.

His world was not balanced on a single step.

He’d never been any great in any person, but it seemed that the great Balance had always been balanced on a single step,
before they’d cut that one off.

He’d never known that he had ever been born a human, but he felt that balance with a mortal might be impossible

I'll state here again, this is where I wasn't sure if I was expecting too much of the technology or if I should be happy with the current results and learn to love the quirks.

Phase 2...let's try all the things

As any real or pretend scientist will tell you, it can be hard to find the point at which enough is enough and to let the results be as they are, and so I continued to break down my data into different sizes and shapes, sometimes only using the first few chapters instead of the entire book, or outputting the books into one mega text file, instead of individual files. A few attempts using only a single author but their entire catalogue. Over the course of a few weeks, I would setup the data in the evening, start the training, and in the morning decide if it was doing well enough to continue or stop and try something else. But with all that effort and fun, I never really got better results than the first training. I do think I need better formatted data, but at this point I'm not sure how to achieve that. But I will keep trying, and eventually I do believe I'll have something I can work with to make this sci-fi daydream become a real novel.

fantasy1

python3 src/interactive_conditional_samples.py --model_name fantasy1 --temperature 0.9 --top_k 30
Model prompt >>> “What am I missing?” He asked to the machine before him. The machine was a metal contraption with a 
sphere in the center. If his machine could keep that sphere in the center without external energy sources, he would 
achieve the improbable. Infinite Energy.
======================================== SAMPLE 1 ========================================
The object itself was a tiny cube; it was not large enough to reach into the center of the sphere to cause it to 
collapse. As if it were an egg.
The contraption was the first I encountered. “How do they do it?” He asked as he moved over it.
“The best I can do is to just do it without being able to do anything else with the machine.” As he pointed toward
the object, the object began to slide outward.
The contraption was a large, white, circular sphere that had two sides. They were the corners of the ballista. It was
a large, circular ballista. The ballista moved with its axis, and this time it was not moving with the other side.
The two halves of the sphere were at the same time, and the machine seemed to be moving along some sort of horizontal
axis.
When I stood there, I wondered what the purpose of the contraption was, but it was obvious. The object was being moved,
as if through a series of concentric circles. It felt as if the machine was being moved. And then, just as the
contraption began to move, the object suddenly vanished.
“I see. I see.” I said to the machine as I moved over the ballista.
“Oh. That . . .” The object vanished.
“Why don’t you go to the other side?” I asked him.
“I know it would be a waste to bring that sphere to the other side.”
“But why?”
“Because, just because something is gone, it doesn’t mean nothing. I can do something that I can do without touching
anything.”
“Well, we need to go to it,” I said.
“Then go to it,” he said back.
“We need to go to that side.” I pointed over the machine.
“We are in a room,” he said.
“The right side is where the ballista was moving,” I said.
“The left,” he said.
“I’m sure.”
“The machine can do everything,” he said.

fantasy2

$ python3 src/interactive_conditional_samples.py --model_name fantasy2 --temperature 0.9 --top_k 30
What am I missing?” He asked to the machine before him. The machine was a metal contraption with a sphere in 
the center. If his machine could keep that sphere in the center without external energy sources, he would achieve 
the improbable. Infinite Energy.
======================================== SAMPLE 1 ========================================
 He’d built it himself.
His arm grew numb, then it stopped, and he felt a chill and the feel of something cold, something as solid as the earth.
His heart stopped, and his lungs were already failing. He tried to turn his head to look at the ceiling—it was a very
tall thing, about forty feet tall. It had a dome, like the dome in the cavern outside the city itself, where air rose
above it. It would have been easy to take off a piece of it just then, and take up two or three of its sides just to
reach the dome. He felt the weight of the thing falling off in the air like a falling horse. Just then something slammed
into it, a rock like a rock thrown back a few feet. It slammed against his arm and, somehow, he fell.
He felt the cold air, and he felt its bite. It was a cold, heavy, strong thing. He had to pull himself up. He was
nearly at the top of the dome when the rock hit. He didn’t have to pull back, not even to the point where he didn’t
feel the whole thing hit. He was suddenly falling again. It stopped, and he let out a small cry.
“No! No!”
The air was so cold, he was almost choking.
“Aaaaahhhh!”
His hands were freezing so badly that he couldn’t get up. He was holding his head against the dome, and his lungs
were going to shut down. As he started to get up, something hit him again. It was a big rock, and it smashed out at
him, one big chunk of stone falling on top of him and hitting the floor, hitting the ceiling, hitting the window of
the cellar, and hitting all of them down, right up to the very top and leaving him on his face. He struggled up and
found his arm and began to struggle.
A big stone hit him in the face, hard, and he was not sure why—the thing hit his face in a terrible collision. It hit
his chest and nose, and his mouth opened. He started to breathe hard, and his chest started growing larger and bigger,
and his mouth came out of him, and he tried to open his mouth, to tell the thing what to do. It wouldn’t, but something
else hit him in the throat.

Steven King Books

$ python3 src/interactive_conditional_samples.py --model_name steve1 --temperature 0.9 --top_k 30
Model prompt >>> "What am I missing?” He asked to the machine before him. The machine was a metal contraption with a 
sphere in the center. If his machine could keep that sphere in the center without external energy sources, he would 
achieve the improbable. Infinite Energy.
======================================== SAMPLE 1 ========================================
“Thank you, Mr. Potter.”
The machine was a little too small to put the cylinder up on its own, but it did what it was supposed to do, which was
to put the cylinder in place on the machine. Because the machine could move so fast, it could not be kept moving by
some weird mechanical device. And as far more than three other machines had ever been put into the center of the machine,
it had managed to keep it moving on its own. That was the point, and as of now it was the only one left to do so.
“You’re right, and it did its job, but my machine is not a machine.”
“Why do you have to worry about that?”
“I have no need for that.”
The machine seemed to shake and twitching slightly as it did so. It was a little too small to make the cylinder move up
and down, but that didn’t stop it from moving.
It held the cylinder in place and started moving. That was the only thing it could do, and it was not a much bigger
machine.
“I want to say a few words,”
The machine seemed to move and to start moving as soon as it was moved. It didn’t stop moving, but it changed. It had
changed its head, got its hands on its legs and began to pull them at the same time to make it move. It was moving at
three hundred miles a second on its own.
“I want to say a few words,” the robot said at long last.
It did so with the same sudden and sudden speed of its original motion, but it could have stopped now, and the machine
would still have been moving. The machine would have moved faster than it was now, but no more.
“What do I do now?”
“I want to say a few words,” the robot asked.
It did so with a very small smile. The robot had a little piece of the machine in its hands and had started to pick it
up, putting it on its shoulders and making it fly up.
“I want to say a few words,” the robot said. It did so, and with a small smile.
The robot turned as fast as it could.

117M - The original gpt-2

Here are results with the stock gpt-2 model 117M

Model prompt >>> "What am I missing?” He asked to the machine before him. The machine was a metal contraption with a 
sphere in the center. If his machine could keep that sphere in the center without external energy sources, he would 
achieve the improbable. Infinite Energy.
======================================== SAMPLE 1 ========================================
‡
When she looked at him, his face had turned dark. It was dark and cold, but she kept the smile on her smile.
“I’ve seen you before, don’t get me wrong. I’ve met you before. But I am not like you. I am not as tall as you.”
A smile began to appear on the white-robed woman’s face, and then he let out a soft sigh. She smiled back, she smiled
back… but then she stopped.
“I’ve become like you. I’ve lost my sense of right and wrong.” He said in a voice very similar to her smile that he
had been following. He said that he would go back to that situation if he did not think that she was trying to do
him harm, but that she did not understand him at all and was simply trying to kill him. The girl looked even more
pale, not like she had taken on a human form. At this point, she started crying out in frustration and crying out
in pain.
But you were just too strong…"
A voice from the machine spoke, and the girl froze, her eyes widened. She was holding him by the hand, and he was
trembling, his shoulders sagging, his face trembling. However, the words were not a threat, but something to her.
“I’m sorry about that.”
She nodded, and the machine slowly took on a voice, and the girl cried out again. However, no one spoke, and the
voice was silent for a few short seconds.
“I know you want to kill me, and I know you want to kill me, and I know you want to kill me. But I’m sorry for
that. I’m sorry we didn’t live in peace, for the past ten years, and for now. I want to see you grow up and find
your purpose, and I want to see you understand that I’m not a robot, who’s just a little girl who wanted a place
to live. I want to see you understand that you need to find happiness, and I want to see you see the joy that you
will have with humans, and that you will live with everyone else who lives.”
The machine slowly began to work on his hands.
“And now you want to kill me?”
“No, I want my life to be worth anything, even if it means the end of your life.” The girl shook her head

And that is the story so far on my adventure with Machine Learning for fiction writing. So as one can see, at least the way I've got it setup, it isn't ready for prime time, but given more time and understanding on my part, I think it still has a lot of potential. Something I will keep experimenting with and building until I have a good platform.