This is the home of the social.coop tech group! If you are new to the group, please see the new user guide.
- Entry points:
- Other places to look for information:
- https://anagora.org/twg contains pointers.
The tech group is responsible for operating and maintaining the following services:
|Social.coop mastodon instance||admin panel|
|wiki.social.coop public wiki||https://wiki.social.coop|
|registrant||organization: Xarxa integral de professionals i usuaries|
|DNS / DDOS protection||cloudflare|
|Mailgun for emails sent by mastodon||mailgun|
|@social.coop email aliases||webarch.mail|
|Object store for backups and digital assets||digital ocean spaces|
|Monitoring / metrics||datadog|
Our git.coop repositories
|tech gitlab group||list of all repos|
|sauce||docker config and some systemd services (to be migrated)|
|pass||encrypted password store for shared passwords|
|wiki||code for the public metalsmith wiki|
- tech governance
- server access (to vote for giving people access to server)
Our fediverse instance is the raison d`etre of the social.coop coop. This is what the community signs up for and our primary responsibility. The primary points of administration are:
- the admin panel
- the datadog dashboard
- ssh cli access Access via ssh on port 2022 e.g.
ssh email@example.com -p 2022.
Mastodon is running under Docker-compose.
Compose is a tool for defining and running multi-container Docker applications. With Compose, you use a YAML file to configure your application’s services. Then, with a single command, you create and start all the services from your configuration. We are using Systemd as the init system for docker-compose.
|social.coop-mastodon||a service to control the mastodon installation via docker-compose|
|social.coop-remove-media||runs the media cleanup command to remove remote media >7 days old via a .timer|
|certbot||runs the renewals via .timer|
|systemctl list-timers||lists timers!|
|journalctl -f||tail ALL system logs|
|docker-compose logs -f web||view and tail web logs (when in
|docker-compose logs -f db||view and tail db logs (you get the pattern?)|
|journalctl -f -u certbot||see when certbot was run|
|journalctl -f -u social.coop-mastodon||see the output from the docker-compose commands ran with systemctl, but not the docker container logs themselves|
|journalctl -f -u social.coop-remove-media||see what the remove media command is up to|
All of these commands must be run on runko.social.coop in the
||List all Docker containers|
||Stop a service|
||Start a service|
||scale a service, in this case create 5 sidekiq default queue workers|
||Redeploy (only changed things)|
If you want to run commands make sure to use the
--rm argument, or the containers will hang around.
docker-compose run --rm web rails console (to get a rails console)
Configure by the pg-dump-to-s3 ansible role.
Location of Postgres database files:
Mastodon upgrade notes
- it is recommended to run upgrades inside of
tmuxin case you are disconnected from the server.
- get current version from social.coop
- find next version from mastodon github releases
- check upgrade notes
- check whether there are DB migrations
- make backup?
systemctl start pg-dump-to-s3.service
- takes 15 mins or so?
- separate command to see backup progress
- make merge request on git.coop sauce repo to bump version in a couple of places in docker-compose.yaml
git diff v3.1.2..v3.1.3 -- docker-compose.ymlin mastodon repo after pulling to check whether there were any changes we should consider mirroring to our docker-compose file
- could be cool to make these merge requests in advance
- write a toot announcing upgrade and boost on admin account
- touch file on server to activate maintenance mode
- actually do the upgrade
- migration command creates a fresh web container and runs the migration command and then deletes that new container
- turn maintenance mode off
- we copy static assets outside of the container so they can be served by nginx
- there's a command for this which moves stuff into a temporary dir in nginx and pulls assets out of docker container into that folder in docker container
- ssh forwarding is nice, then with
sudo -E -syou have ssh access to stuff you do from host machine(?)
Mastodon and Wiki run on runko.social.coop, a dedicated server at Hetzner:
- 32GB RAM (4x RAM 8192 MB DDR3)
- i7-4770 CPU @ 3.40GHz).
- 2x 250 GB disks (SSD)
- Ubuntu 18.04
We are using LVM.
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:0 0 223.6G 0 disk ├─sda1 8:1 0 1G 0 part /boot └─sda2 8:2 0 222.6G 0 part ├─vg0-root1 253:0 0 25G 0 lvm / ├─vg0-root2 253:1 0 25G 0 lvm └─vg0-opt 253:2 0 396.1G 0 lvm /opt sdb 8:16 0 223.6G 0 disk └─sdb1 8:17 0 223.6G 0 part └─vg0-opt 253:2 0 396.1G 0 lvm /opt
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert opt vg0 -wi-ao---- 396.13g root1 vg0 -wi-ao---- 25.00g root2 vg0 -wi-a----- 25.00g
opt is mounted at
wiki.social.coop has two main purposes:
- a public-facing site with information about social.coop
- the registration system for new users
It's configured/deployed via ansible using the wiki.social.coop role and the
wiki tag, so
ansible-playbook server.playbook.yml --tags wiki will set it up.
The configuration secrets are stored in the pass repo at:
deployment/wiki/gitlab_token deployment/wiki/gitlab_username deployment/wiki/mailgun_password deployment/wiki/webhook_secret
On the server it lives at:
/opt/social.coop/wiki.social.coop/ and is running as the systemd service