website/content/post/2019-01-12-PaulBSD-Infrastructure.md

+++
categories = ["Infra"]
date = "2019-01-13"
menu = ""
tags = ["paulbsd","infra"]
title = "PaulBSD Infrastructure"
+++

# PaulBSD Infrastructure

PaulBSD.com is a mixed infrastructure made of on-premise part based in Normandy, and cloud instances made upon scaleway.com in Amsterdam / object storage in Paris.

PaulBSD infrastructure main regions are :
- Caen (Legacy production)
- Saint-Lô (Backups)
- Amsterdam (Production)
- Paris (Object storage)

Up to 80% of the applications are managed by SaltStack, some house-made applications / scripts / tools written in Python 3 and Golang for back services, and HTML/CSS/JS on front services. All PaulBSD infrastructure is open-sourced

## Overview

Front machines are cloud instances with mixed architectures (amd64 and arm64). Back machines are mainly legacy. Front machines are Linux based instances based on Ubuntu 18.04 (20.04 migration is work-in-progress). NetBSD 8.1 is used for legacy, front and back services, and is going to be replaced.

Some services :

- Mail relaying is located on front instances, using postfix. Security and mail signing is based upon opendkim. IMAP is served using dovecot with a PostgreSQL instance for authentication and accounting.
- Nftables is used for firewall on Linux and NPF on NetBSD
- Monitoring tools are sensu-go (with mail and Telegram notifications), and metrics are centralized on a stack based on Telegraf, InfluxDB, Grafana
- Internal docs are based on dokuwiki
- Geographical data replication is made using Syncthing (locations are Caen and Saint-Lô. Amsterdam will replace Caen in the future). rsync is used for some scripts
- RDBMS are MariaDB and PostgreSQL
- Git repositories for house-made applications and a mirrored copy of GitHub useful projects are stored by Gitea (git.paulbsd.com)
- Burp / Bareos do the backups of code and data
- NextCloud serves the personal cloud storage / Caldav shares
- IPAM based on Netbox
- ...

Total data space (free + used) across all regions : <= 3,5 To

Clients are Ubuntu (managed by SaltStack) and Android devices

I will make a schema in the future to show how services are organized

## Internal applications

All projects described below are house-made applications / tools, open-licensed, for many usages


### qrz

qrz (qrz.paulbsd.com) is based on bootstrap and datatables for front application. Datatables is using the server-side processing, which is querying on a MariaDB database. A scrapper has been written in Golang that fetch data in an html page on the internet. This scrapper is available at git.paulbsd.com/paulbsd/qrz and is using an embedded MySQL connector


### g2g

g2g is a golang executable that fetchs starred project on github and POST mirror copies of these repositories on a gitea / gogs instance. It is mainly a script that call REST webservices of github and gitea


### fuelprices

fuelprices is a little golang script that fetch fuel prices on the opendata of the french governement, and insert it on an influxdb time-series


### dip / dipc

dip is a webservice based in golang that return public ip addresses like ipinfo.io or showmyip.com (instance available at ip.paulbsd.com). HTML or JSON output are possible

dipc is the client for dip (as a portable executable file)


Have fun !
added article 2020-01-12 22:01:13 +01:00			`+++`
			`categories = ["Infra"]`
updated paulbsd infrastructure article 2020-01-13 20:45:09 +01:00			`date = "2019-01-13"`
added article 2020-01-12 22:01:13 +01:00			`menu = ""`
			`tags = ["paulbsd","infra"]`
			`title = "PaulBSD Infrastructure"`
			`+++`

updated paulbsd infrastructure article 2020-01-13 20:45:09 +01:00			`# PaulBSD Infrastructure`
added article 2020-01-12 22:01:13 +01:00
update article 2020-01-12 22:31:17 +01:00			`PaulBSD.com is a mixed infrastructure made of on-premise part based in Normandy, and cloud instances made upon scaleway.com in Amsterdam / object storage in Paris.`
added article 2020-01-12 22:01:13 +01:00
update article 2020-01-12 22:31:17 +01:00			`PaulBSD infrastructure main regions are :`
			`- Caen (Legacy production)`
			`- Saint-Lô (Backups)`
			`- Amsterdam (Production)`
			`- Paris (Object storage)`
added article 2020-01-12 22:01:13 +01:00
update article 2020-01-12 22:31:17 +01:00			`Up to 80% of the applications are managed by SaltStack, some house-made applications / scripts / tools written in Python 3 and Golang for back services, and HTML/CSS/JS on front services. All PaulBSD infrastructure is open-sourced`
added article 2020-01-12 22:01:13 +01:00
updated paulbsd infrastructure article 2020-01-13 20:45:09 +01:00			`## Overview`
update article 2020-01-12 22:31:17 +01:00
updated paulbsd infrastructure article 2020-01-13 20:45:09 +01:00			`Front machines are cloud instances with mixed architectures (amd64 and arm64). Back machines are mainly legacy. Front machines are Linux based instances based on Ubuntu 18.04 (20.04 migration is work-in-progress). NetBSD 8.1 is used for legacy, front and back services, and is going to be replaced.`
update article 2020-01-12 22:31:17 +01:00
			`Some services :`

updated paulbsd infrastructure article 2020-01-13 20:45:09 +01:00			`- Mail relaying is located on front instances, using postfix. Security and mail signing is based upon opendkim. IMAP is served using dovecot with a PostgreSQL instance for authentication and accounting.`
update article 2020-01-12 22:31:17 +01:00			`- Nftables is used for firewall on Linux and NPF on NetBSD`
			`- Monitoring tools are sensu-go (with mail and Telegram notifications), and metrics are centralized on a stack based on Telegraf, InfluxDB, Grafana`
			`- Internal docs are based on dokuwiki`
			`- Geographical data replication is made using Syncthing (locations are Caen and Saint-Lô. Amsterdam will replace Caen in the future). rsync is used for some scripts`
			`- RDBMS are MariaDB and PostgreSQL`
			`- Git repositories for house-made applications and a mirrored copy of GitHub useful projects are stored by Gitea (git.paulbsd.com)`
			`- Burp / Bareos do the backups of code and data`
updated paulbsd infrastructure article 2020-01-13 20:45:09 +01:00			`- NextCloud serves the personal cloud storage / Caldav shares`
update article 2020-01-12 22:31:17 +01:00			`- IPAM based on Netbox`
			`- ...`

			`Total data space (free + used) across all regions : <= 3,5 To`

updated paulbsd infrastructure article 2020-01-13 20:45:09 +01:00			`Clients are Ubuntu (managed by SaltStack) and Android devices`
update article 2020-01-12 22:31:17 +01:00
			`I will make a schema in the future to show how services are organized`

updated paulbsd infrastructure article 2020-01-13 20:45:09 +01:00			`## Internal applications`

			`All projects described below are house-made applications / tools, open-licensed, for many usages`


			`### qrz`

			`qrz (qrz.paulbsd.com) is based on bootstrap and datatables for front application. Datatables is using the server-side processing, which is querying on a MariaDB database. A scrapper has been written in Golang that fetch data in an html page on the internet. This scrapper is available at git.paulbsd.com/paulbsd/qrz and is using an embedded MySQL connector`


			`### g2g`

updated article 2020-01-17 00:05:12 +01:00			`g2g is a golang executable that fetchs starred project on github and POST mirror copies of these repositories on a gitea / gogs instance. It is mainly a script that call REST webservices of github and gitea`
updated paulbsd infrastructure article 2020-01-13 20:45:09 +01:00

			`### fuelprices`

updated infrastructure article 2020-01-22 08:17:35 +01:00			`fuelprices is a little golang script that fetch fuel prices on the opendata of the french governement, and insert it on an influxdb time-series`
updated paulbsd infrastructure article 2020-01-13 20:45:09 +01:00

			`### dip / dipc`

updated article content 2020-01-17 06:00:39 +01:00			`dip is a webservice based in golang that return public ip addresses like ipinfo.io or showmyip.com (instance available at ip.paulbsd.com). HTML or JSON output are possible`
updated paulbsd infrastructure article 2020-01-13 20:45:09 +01:00
			`dipc is the client for dip (as a portable executable file)`


updated article 2020-01-17 00:05:12 +01:00			`Have fun !`