Month: November 2018

  • WeeChat Setup

    So you decided to try WeeChat?

    What options do you need to set? what plugins? what scripts?

    I’ll go over some of the most essential of these, and share my full configs.

    Options

    • logger.level.irc
      The default is 9, which includes joins and parts. In most cases you can set this to 3, which only includes messages.
    • weechat.look.buffer_notify_default
      The default here is all, which will add joins and parts to your hotlist. set it to message.
    • weechat.look.confirm_quit
      Set this to on. You’ll thank me when you type /quit and mean /close.
    • weechat.look.highlight
      Add a comma-separated list of names/terms you’d like to trigger a highlight here
    • weechat.look.prefix_align_max
      Set this to something between 10 and 20. Otherwise, long nicks will crush your available screen real estate.
    • buflist.format.indent
      Adjusts the display of your channel list. I use ${color:237}${if:${buffer.next_buffer.local_variables.type}=~^(channel|private)$?├:└} to print some nice unicode characters to draw nice guides.
    • buflist.format.number
      If you want to skip the . or space after the number, set it to ${color:green}${number}
    • irc.look.color_nicks_in_names
      Set this to on
    • irc.look.color_nicks_in_nicklist
      Set this to on
    • irc.look.server_buffer
      Set this to independent to prevent automatic merges with the core WeeChat buffer. Especially useful if you plan on using autosort.py
    • irc.server_default.autoconnect
      Set this to on so you don’t have to set it for every new network you add.

    Scripts

    These scripts can be managed with the built in /script tool. Press i, then enter on the selected script to install it.

    • highmon.pl
      Set aside a buffer to list the places your nick has been mentioned.
    • colorize_nicks.py
      Show nicks in chat with colors.
    • go.py
      Fuzzy quick jump by buffer number or channel name.
    • autojoin.py
      Use /autojoin --run to save all the channels you’re currently in to be autojoined the next time you start WeeChat.
    • autosort.py
      Use this script in tandem with irc.look.server_buffer=independent to keep your channel and server list in order.
    • colorize_lines.pl
      I use this script to highlight the entire line of messages I’ve been mentioned in. Check the options in the source or with /help colorize_lines.
    • grep.py
      Quickly search history and buffers with /grep.

    The rest of my configs

    You can find the rest of my configs here. Be sure to keep the WeeChat Documentation handy as well.

    If you have any questions, feel free to ping me on IRC. I’m ben on libera and tilde.chat.

    Screenshot

    Here’s a screenshot of my current configs

    Bonus

    If you have an existing setup, you can check the config changes you’ve made with /set diff.

    Additionally, feel free to use my .gitignore, add your ~/.weechat to source control, and compare.

    Hope you’ve enjoyed customizing your WeeChat!

    EDIT: s/freenode/libera/g

  • Proactive (reactive?) redundancy

    After the fiasco earlier this week, I’ve been taking steps to minimize the impact if tilde.team were to go down. It’s still a large SPOF (single-point-of-failure), but I’m reasonably certain that at least the IRC net will remain up and functional in the event of another outage.

    The first thing that I set up was a handful of additional IRCd nodes: see the wiki for a full list. slash.tilde.chat is on my personal vps, and bsd.tilde.chat is hosted on the bsd vps that I set up for tilde.team.

    I added the ipv4 addresses for these machines, along with the IP for yourtilde.com as A records for tilde.chat, creating a DNS round-robin. host tilde.chat will return all four. Requesting the DNS record will return any one of them, rotating them in a semi-random fashion. This means that when connecting to tilde.chat on 6697 for IRC, you might end up on any of {your,team,bsd,slash}.tilde.chat.

    This creates the additional problem that visiting the tilde.chat site will end up at any of those 4 machines in much the same way. For the moment, the site is deployed on all of the boxes, making site setup issues hard to debug. the solution to this problem is to use a subdomain as the round-robin host, as other networks like Libera.Chat do (see host irc.libera.chat for the list of servers).

    I’m not sure how to make any of the other services more resilient. It’s something that I have been and will continue to research moving forward.

    The other main step that I have taken to prevent the same issue from happening again was to configure the firewall to drop outgoing requests to the subnets as defined in rfc 1918.

    I’d like to consider at least this risk to be mitigated.

    Thanks for reading,

    ~ben

    Update: the round robin host is now irc.tilde.chat, which resolves the site issues that we were having, due to the duplicated deployments.

  • November 13 post mortem

    We had something of an outage on november 13, 2018 on tilde.team.

    I awoke, not suspecting anything to be amiss. As soon as I logged in to check my email and IRC mentions, it became clear.

    tilde.team was at the least inaccessible, and at the worst, down completely. According to the message in my inbox, there had been an attempted “attack” from my IP.

    We have indications that there was an attack from your server. Please take all necessary measures to avoid this in the future and to solve the issue.

    At this point, I have no idea what could have happened overnight while I was sleeping. The timestamp shows that it arrived only 30 minutes after I’d turned in for the night.

    When I finally log on in the morning to check mails and IRC mentions, I find that I’m unable to connect to tilde.team… strange, but ok; time to troubleshoot. I refresh the webmail to see what I’m missing. It ends up failing to find the server. Even stranger! I’d better get the mails off my phone if they’re on my @tilde.team mail!

    Here, i launch in to full debugging mode: what command was it? who ran it?

    Searching ~/.bash_history per user was not very successful. Nothing I could find was related to net or map. I had checked sudo grep nmap /home/*/.bash_history and many other commands.

    At this point, I had connected with other ~teammates across other IRC nets (#!, ~town, etc). Among suggestions to check /var/log/syslog, /var/log/kern.log, and dmesg, I finally decided to check ps. ps -ef | grep nmap yielded nmap on an obscured uid and gid, which I shortly established to belong to a LXD container I had provisioned for ~fosslinux.

    I’m not considering methods of policing access to any site over port 80 and port 443. This is crazy. How do you police nmap when it isn’t scanning on every port?

    After a bit of shit-talking and reassurance from other sysadmins, I reexamined and realized that ~fosslinux had only run nmap for addresses in the 10.0.0.0/8 space. The 10/8 address space is intended to not be addressable outside the local space. How could Hetzner have found out about a localhost network probe!?

    Finally, after speaking with more people than I expected to speak with in one day, I ended up sending three different support emails to Hetzner support, which finally resulted in them unlocking the IP.

    It’s definitely time to research redundancy options!