poij poij

Year: 2018

  • WeeChat Setup

    So you decided to try WeeChat?

    What options do you need to set? what plugins? what scripts?

    I’ll go over some of the most essential of these, and share my full configs.

    Options

    • logger.level.irc
      The default is 9, which includes joins and parts. In most cases you can set this to 3, which only includes messages.
    • weechat.look.buffer_notify_default
      The default here is all, which will add joins and parts to your hotlist. set it to message.
    • weechat.look.confirm_quit
      Set this to on. You’ll thank me when you type /quit and mean /close.
    • weechat.look.highlight
      Add a comma-separated list of names/terms you’d like to trigger a highlight here
    • weechat.look.prefix_align_max
      Set this to something between 10 and 20. Otherwise, long nicks will crush your available screen real estate.
    • buflist.format.indent
      Adjusts the display of your channel list. I use ${color:237}${if:${buffer.next_buffer.local_variables.type}=~^(channel|private)$?├:└} to print some nice unicode characters to draw nice guides.
    • buflist.format.number
      If you want to skip the . or space after the number, set it to ${color:green}${number}
    • irc.look.color_nicks_in_names
      Set this to on
    • irc.look.color_nicks_in_nicklist
      Set this to on
    • irc.look.server_buffer
      Set this to independent to prevent automatic merges with the core WeeChat buffer. Especially useful if you plan on using autosort.py
    • irc.server_default.autoconnect
      Set this to on so you don’t have to set it for every new network you add.

    Scripts

    These scripts can be managed with the built in /script tool. Press i, then enter on the selected script to install it.

    • highmon.pl
      Set aside a buffer to list the places your nick has been mentioned.
    • colorize_nicks.py
      Show nicks in chat with colors.
    • go.py
      Fuzzy quick jump by buffer number or channel name.
    • autojoin.py
      Use /autojoin --run to save all the channels you’re currently in to be autojoined the next time you start WeeChat.
    • autosort.py
      Use this script in tandem with irc.look.server_buffer=independent to keep your channel and server list in order.
    • colorize_lines.pl
      I use this script to highlight the entire line of messages I’ve been mentioned in. Check the options in the source or with /help colorize_lines.
    • grep.py
      Quickly search history and buffers with /grep.

    The rest of my configs

    You can find the rest of my configs here. Be sure to keep the WeeChat Documentation handy as well.

    If you have any questions, feel free to ping me on IRC. I’m ben on libera and tilde.chat.

    Screenshot

    Here’s a screenshot of my current configs

    Bonus

    If you have an existing setup, you can check the config changes you’ve made with /set diff.

    Additionally, feel free to use my .gitignore, add your ~/.weechat to source control, and compare.

    Hope you’ve enjoyed customizing your WeeChat!

    EDIT: s/freenode/libera/g

  • Proactive (reactive?) redundancy

    After the fiasco earlier this week, I’ve been taking steps to minimize the impact if tilde.team were to go down. It’s still a large SPOF (single-point-of-failure), but I’m reasonably certain that at least the IRC net will remain up and functional in the event of another outage.

    The first thing that I set up was a handful of additional IRCd nodes: see the wiki for a full list. slash.tilde.chat is on my personal vps, and bsd.tilde.chat is hosted on the bsd vps that I set up for tilde.team.

    I added the ipv4 addresses for these machines, along with the IP for yourtilde.com as A records for tilde.chat, creating a DNS round-robin. host tilde.chat will return all four. Requesting the DNS record will return any one of them, rotating them in a semi-random fashion. This means that when connecting to tilde.chat on 6697 for IRC, you might end up on any of {your,team,bsd,slash}.tilde.chat.

    This creates the additional problem that visiting the tilde.chat site will end up at any of those 4 machines in much the same way. For the moment, the site is deployed on all of the boxes, making site setup issues hard to debug. the solution to this problem is to use a subdomain as the round-robin host, as other networks like Libera.Chat do (see host irc.libera.chat for the list of servers).

    I’m not sure how to make any of the other services more resilient. It’s something that I have been and will continue to research moving forward.

    The other main step that I have taken to prevent the same issue from happening again was to configure the firewall to drop outgoing requests to the subnets as defined in rfc 1918.

    I’d like to consider at least this risk to be mitigated.

    Thanks for reading,

    ~ben

    Update: the round robin host is now irc.tilde.chat, which resolves the site issues that we were having, due to the duplicated deployments.

  • November 13 post mortem

    We had something of an outage on november 13, 2018 on tilde.team.

    I awoke, not suspecting anything to be amiss. As soon as I logged in to check my email and IRC mentions, it became clear.

    tilde.team was at the least inaccessible, and at the worst, down completely. According to the message in my inbox, there had been an attempted “attack” from my IP.

    We have indications that there was an attack from your server. Please take all necessary measures to avoid this in the future and to solve the issue.

    At this point, I have no idea what could have happened overnight while I was sleeping. The timestamp shows that it arrived only 30 minutes after I’d turned in for the night.

    When I finally log on in the morning to check mails and IRC mentions, I find that I’m unable to connect to tilde.team… strange, but ok; time to troubleshoot. I refresh the webmail to see what I’m missing. It ends up failing to find the server. Even stranger! I’d better get the mails off my phone if they’re on my @tilde.team mail!

    Here, i launch in to full debugging mode: what command was it? who ran it?

    Searching ~/.bash_history per user was not very successful. Nothing I could find was related to net or map. I had checked sudo grep nmap /home/*/.bash_history and many other commands.

    At this point, I had connected with other ~teammates across other IRC nets (#!, ~town, etc). Among suggestions to check /var/log/syslog, /var/log/kern.log, and dmesg, I finally decided to check ps. ps -ef | grep nmap yielded nmap on an obscured uid and gid, which I shortly established to belong to a LXD container I had provisioned for ~fosslinux.

    I’m not considering methods of policing access to any site over port 80 and port 443. This is crazy. How do you police nmap when it isn’t scanning on every port?

    After a bit of shit-talking and reassurance from other sysadmins, I reexamined and realized that ~fosslinux had only run nmap for addresses in the 10.0.0.0/8 space. The 10/8 address space is intended to not be addressable outside the local space. How could Hetzner have found out about a localhost network probe!?

    Finally, after speaking with more people than I expected to speak with in one day, I ended up sending three different support emails to Hetzner support, which finally resulted in them unlocking the IP.

    It’s definitely time to research redundancy options!

  • Italy

    I just got back from a 10-day backpacking trip to Italy and I’d like to share some of the photos I took!

    The travel plan was Rome > Venice > Florence > Naples > Pompei/Vesuvius > Capri > Amalfi / Positano.

    This is the Roman Forum (with Colosseum in the background) as seen from the Palatine.

    Here’s me standing next to the same piece of ancient Rome as I did the last time I visited in 2015. See my github profile for the previous one.

    The inside of St. Pete’s basilica. We climbed the 551 steps to the top of it for a crazy view of Rome.

    Venice was the next stop. This is the grand canal.

    We visited all the main museums in Firenze (aka Florence, but Firenze is far cooler). This is the view from the roof of our hostel.

    After arriving in Naples, we headed out to see Vesuvius and the ruins of Pompei. I’m not sure why Pompei has an extra ‘i’ in English, but it only has one in modern-day Italian. Look at the ruts left from the ancient chariots in the basalt stone!

    Here see Vesuvius from Pompei’s main square.

    We did a tour around the island of Capri. (think Capri pants, Caprese salads, etc). We also took a chairlift to the top of the island for the stunning view. Photos just don’t show the scale and sheer drop from the cliff.

    This is the view from our airbnb in Amalfi. It was quite a pain to get to… but it was worth it!

    Here’s me and ~supercock789 somewhere along the path of the gods.

    What a whirlwind! I’m pretty sure we visited all the major tourist attractions in the cities that we visited. And in quite a short time!

    There are a few extra photos here

    Thanks for stopping by!

  • Utteranc.es

    I somehow stumbled upon utterances today at lunch. (I think someone had it forked on their github page).

    No matter how I found it, I still decided to add it to my blog here with bashblog. Utterances is a commenting system that leverages github issues. For example, a comment on a post shows up on github like this.

    Now we just need to figure out if it can be pointed at a gitea instance like tildegit. Might be time for a PR!

  • No more google

    I figured I should probably get some notes down about moving off google.

    To start, I’ll get a list of the things i was able to easily replace:

    I’m still using:

    • gplay music/youtube
    • google maps (open streetmap isn’t good enough to replace it)
    • google photos – but this is going to be replaced long-term with syncthing

    2024 Update: No update. Still using maps, photos, and music.

  • DNS shenanigans post-mortem

    Let’s start by saying I probably should have done a bit more research before diving head-first into this endeavor.

    I’ve been thinking about transferring my domains off google domains for some time now, as part of my personal goal to self host and limit my dependence on google and other large third-party monstrosities. Along that line, I asked for registrar recommendations. ~tomasino responded with namesilo. I found that they had $3.99 registrations for .team and .zone domains, which is 1/10th the cost of the $40 registration on google domains.

    I started out by getting the list of domains from the google console. 2 or 3 of them had been registered within the last 60 days, so I wasn’t able to transfer those just yet. I grabbed all the domain unlock codes and dropped them into namesilo. I failed to realize that the DNS panel on google domains would disappear as soon as it went through, but more importantly that the nameservers would be left pointing to the old defunct google domains ones.

    I updated the nameservers as soon as I realized this error from the namesilo panel. Some of the domains propagated quickly. Others, not so much. tilde.team was still in a state of flux between the old and new nameservers.

    In a rush to get the DNS problem fixed, and under recommendation from several people on IRC, I decided to switch the nameservers for tilde.team and tilde.zone to cloudflare, leaving another layer of flux for the DNS to be stuck in…

    Of the five domains that I moved to cloudflare, 3 returned with a DNSSEC error, claiming that I needed to remove the DS record from that zone. D’oh!

    I removed the DNSSEC from those affected domains, so we should be good to go as soon as it all propagates through the fickle beast that is DNS.

  • LXD networking and additional IPs

    Now that tilde.team is on a fancy-shmancy new dedi server, I’ve tried to get a secondary IP address assigned to a LXD container (which I plan to use for my personal stuff). LXD shows that the secondary IP is being picked up by that container, but I’m still seeing the host machine’s IP as the external address.

    I’m not sure how I’ll need to configure the network settings on the host machine (now that we’re running ubuntu 18.04 and it uses netplan for configs and not /etc/network/interfaces). Another confusing thing is that the main config in /etc/netplan says that the network config is handled by systemd-networkd…

    At least i have through the end of the year when my current VPS runs out to get this up and running.

    Ping me on IRC or email if you have experience with this.

  • Dotfiles

    Finally got around to updating my dotfiles to use gnu stow. I adapted ~tomasino’s makefile for use with the configs that I’m keeping with it.

    Now I just need to figure out why my ssh config doesn’t copy/symlink my config to ~/.ssh when it already exists.

  • bashblog and your gopherhole

    I’ve created a repo for the tilde.team customizations to bashblog.

    It will now make sure that your ~/public_gopher exists and symlink your blog into it with a nice gophermap to list all the markdown styled posts.

    Try it out and let me know if there are any problems!