diff --git a/INSTALL.md b/INSTALL.md new file mode 100644 index 0000000..2f87b6a --- /dev/null +++ b/INSTALL.md @@ -0,0 +1,83 @@ +# Installation + +The deployment script is largely based on the helm chart [jitsi-helm](https://github.com/jitsi-contrib/jitsi-helm/). The dependencies include [k3s](https://k3s.io/), [traefik](https://traefik.io/) and [argocd](https://argoproj.github.io/). It also uses [Let's Encrypt](https://letsencrypt.org/) for signing TLS certificates. + +## Prerequisites + + * A GNU/Linux host with Debian/Ubuntu installed and root privileges + * The host has a public IPv4 address + * Allow these traffic through firewall: 80/TCP, 443/TCP, 5222/TCP (necessary only for external jvb), 30000/UDP (30001/UDP for test) + * An email inbox address for receiving ACME notification mails + * A domain name that has a DNS A record pointing to + * (optional) Another domain name (for test deployment purposes) that has a DNS A record pointing to + * (optional) Yet another domain name (for accessing the ArgoCD web UI) that has a DNS A record pointing to + +## Install/Upgrade + +The initial installation needs to be run from command line. But afterwards, ArgoCD web UI can be used instead to fulfill the subsequent (re)install/upgrade/uninstall needs. **All the shell commands need to be run with root user.** There are two installation modes, prod and test. By default, prod is installed. It can be switched to test by setting environment variable `TEST_INSTALL`. + +### Install/Upgrade from command line + +Run the following shell command by providing the 2 mandatory arguments: fully-qualified domain name for accessing jitsi web, and an email address for receiving Let's Enrypt's ACME mails. + +```bash +curl -sL https://raw.githubusercontent.com/shanghailug/jitsi-deploy/main/deploy_jitsi.sh | + bash -s - +``` + +Alternatively, an additional environment variable `ARGOCD_FQDN` can be provided to enable ArgoCD web server's ingress, so that it can be accessed post installation, for future operations: + +```bash +curl -sL https://raw.githubusercontent.com/shanghailug/jitsi-deploy/main/deploy_jitsi.sh | + ARGOCD_FQDN= bash -s - +``` + +Before committing to a prod installation, the whole setup can be tested by using a test hostname, only requesting certificates from staging instance of Let's Encrypt, and installing into `test` k8s namespace. This can be done by setting `TEST_INSTALL` and `STAGING_CERT` environment variable and giving test hostname as command argument, like this: + +```bash +curl -sL https://raw.githubusercontent.com/shanghailug/jitsi-deploy/main/deploy_jitsi.sh | + TEST_INSTALL=1 STAGING_CERT=1 ARGOCD_FQDN= bash -s - +``` + +The installed applications can then be updated/upgraded by rerunning exactly the same command, when the git repo is updated or it's desirable to enable ArgoCD web after initial installation is done. The already installed components will usually be kept as-is if their versions match, or be upgraded otherwise. If k3s needs to be upgraded, however, it's probably a better idea to [tear down](#tear-down) the whole setup before-hand. + +### Install/Upgrade from ArgoCD web UI + +If the initial installation enabled ArgoCD web UI's ingress by providing the environment variable `ARGOCD_FQDN`, then the ArgoCD web server can be accessed via `https://${ARGOCD_FQDN}/`. +Please refer to [ArgoCD docs](https://argo-cd.readthedocs.io/en/stable/getting_started/#6-create-an-application-from-a-git-repository) for more details about how to create/update applications using helm charts from a git repo. The login's name is `admin` and the login's password can be retrieved after initial installation, by running the following command on the host: + +```bash +kubectl -n argocd get secret/argocd-initial-admin-secret -o jsonpath='{ .data.password }' | base64 -d +``` + +## Settings + +The following list of environment variables can be used to customize or alter the installation. + +Environment Variable | Description | Default Value | Default behaviour +--- | --- | --- | --- +`ARGOCD_FQDN` | fully-qualified hostname for accessing ArgoCD web UI | "" | don't enable web ingress for ArgoCD server +`ARGOCD_VERSION` | argocd release to install | "v2.3.3" | +`DEPLOY_GIT_REPO` | the git repo url for retrieving artifacts | `https://github.com/shanghailug/jitsi-deploy.git` | +`DEPLOY_GIT_VERSION` | the revision of artifacts to checkout and use from the repo | "" | use the default branch when git repo is cloned locally +`EXCLUDE_JVB` | Exclude built-in jvb component (so that an external one can be registered for use) | "" | include jvb +`K3S_VERSION` | k3s release to install | "v1.23.6+k3s1" | +`TEST_INSTALL` | when set to non-empty, install an app called `jitsitest` into `test` k8s namespace | "" | install an app called `jitsi` into `prod` k8s namespace + + +## Uninstall + +### Uninstall the applications + +The applications can be uninstalled either from ArgoCD web UI or by running something like the following commands: + +```bash +argocd app delete jitsi +argocd app delete jitsitest +``` + +### Tear down + +```bash +/usr/local/bin/k3s-uninstall.sh +``` diff --git a/argocd/cmd-params-cm.yaml b/argocd/cmd-params-cm.yaml new file mode 100644 index 0000000..9016764 --- /dev/null +++ b/argocd/cmd-params-cm.yaml @@ -0,0 +1,10 @@ +apiVersion: v1 +kind: ConfigMap +metadata: + name: argocd-cmd-params-cm + namespace: argocd + labels: + app.kubernetes.io/name: argocd-cmd-params-cm + app.kubernetes.io/part-of: argocd +data: + server.insecure: "true" diff --git a/argocd/ingressroute-server.yaml.sh b/argocd/ingressroute-server.yaml.sh new file mode 100755 index 0000000..2eeaaff --- /dev/null +++ b/argocd/ingressroute-server.yaml.sh @@ -0,0 +1,26 @@ +cat < /dev/null && test -f /etc/rancher/k3s/k3s.yaml) || \ - curl -sfL https://get.k3s.io | sh -s - --write-kubeconfig-mode 644 - -export KUBECONFIG=/etc/rancher/k3s/k3s.yaml -kubectl get namespace jitsi &> /dev/null || \ - sudo -E kubectl create namespace jitsi - -sudo -E kubectl config set-context --current --namespace=jitsi - -# helm -which helm &> /dev/null || \ - curl -sfL https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash -s - - -# jitsi -sudo -E helm repo add jitsi https://jitsi-contrib.github.io/jitsi-helm/ -sudo -E helm install shlug-jitsi jitsi/jitsi-meet -f values.yml -n jitsi - diff --git a/deploy_jitsi.sh b/deploy_jitsi.sh new file mode 100755 index 0000000..5577cb9 --- /dev/null +++ b/deploy_jitsi.sh @@ -0,0 +1,225 @@ +#!/usr/bin/env bash + +function err { + echo $1 1>&2 + exit 1 +} + +# check usage +if [ $# -ne 2 ]; then + err "usage: $0 " +fi + +# check sudo +if [ $EUID -ne 0 ]; then + err "sudo?" +fi + +# host OS packages +apt update && apt -y install grep bind9-dnsutils iproute2 curl wget git + +# parameters +export FQDN=$1 +export ACME_EMAIL=$2 +export PUBLIC_IP=$(nslookup ${FQDN} | grep -A1 Name: | grep Address: | cut -d' ' -f2) +if [ -z "${PUBLIC_IP}" ]; then + err "can't resolve hostname: ${FQDN}" +else + echo "resolved hostname '${FQDN}' to ip address ${PUBLIC_IP}" +fi +if ! (curl -s https://ipinfo.io/ip | grep -q ${PUBLIC_IP}); then + err "the host doesn't have such public ip: ${PUBLIC_IP}" +fi + +if [ -n "${TEST_INSTALL}" ]; then + export HELM_NAME=jitsitest + export NAMESPACE=test + export JVB_PORT=30001 +else + export HELM_NAME=jitsi + export NAMESPACE=prod + export JVB_PORT=30000 +fi + +# versions +K3S_VERSION=${K3S_VERSION:-"v1.23.6+k3s1"} +HELM_VERSION=${HELM_VERSION:-"v3.8.2"} +ARGOCD_VERSION=${ARGOCD_VERSION:-"v2.3.3"} +HELM_ARCHIVE="helm-${HELM_VERSION}-linux-amd64.tar.gz" +DEPLOY_GIT_REPO=${DEPLOY_GIT_REPO:-"https://github.com/shanghailug/jitsi-deploy.git"} + +# workspace +WS_DIR=${HOME}/deploy/$(date +"%Y%m%d_%H%M%S") +SRC_DIR=${WS_DIR}/jitsi-deploy +mkdir -p ${WS_DIR} + +function get_helm { + if ! which helm || ! ( helm version | grep -q ${HELM_VERSION} ); then + cd ${WS_DIR}/ + wget https://get.helm.sh/${HELM_ARCHIVE} + tar -zxvf ${HELM_ARCHIVE} + mv $(find -type f -name helm) /usr/local/bin/ + fi +} + +function get_src { + cd ${WS_DIR}/ + git clone ${DEPLOY_GIT_REPO} + cd $SRC_DIR/ + if [ -n "${DEPLOY_GIT_VERSION}" ]; then + git checkout ${DEPLOY_GIT_VERSION} + fi +} + +function do_k3s { + INSTALL_K3S= + # nuke + if [ -n "${NUKE_K3S}" ] && [ -f /usr/local/bin/k3s-uninstall.sh ]; then + /usr/local/bin/k3s-uninstall.sh + INSTALL_K3S=1 + elif ! which k3s; then + INSTALL_K3S=1 + fi + + # install k3s + if [ -n "${INSTALL_K3S}" ]; then + curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION=${K3S_VERSION} INSTALL_K3S_EXEC="--tls-san ${PUBLIC_IP}" sh - + fi + + echo -n "waiting for k3s server node to become ready ." + while ! (kubectl get node | grep -q -w Ready); do + echo -n "." + sleep 1 + done + echo "ready." + export KUBECONFIG=/etc/rancher/k3s/k3s.yaml + kubectl get node -o wide +} + +function get_argocd { + if ! which argocd || ! (argocd -n argocd version --client | grep ^argocd: | grep -q ${ARGOCD_VERSION}); then + cd ${WS_DIR}/ + wget https://github.com/argoproj/argo-cd/releases/download/${ARGOCD_VERSION}/argocd-linux-amd64 + chmod a+x argocd-linux-amd64 + mv argocd-linux-amd64 /usr/local/bin/argocd + fi +} + +function do_traefik { + cd ${SRC_DIR}/ + ./traefik-config.yaml.sh | kubectl apply -f - + + echo -n "waiting for helm-install-traefik to become ready ." + while [ $(kubectl -n kube-system get job | grep helm-install-traefik | grep -c '1/1') -ne 2 ]; do + echo -n "." + sleep 1 + done + echo "ready." + kubectl -n kube-system get job -o wide +} + +function do_argocd { + cd ${SRC_DIR}/ + + kubectl create ns argocd --dry-run=client -o yaml | kubectl apply -f - + kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml + if [ -n "${ARGOCD_FQDN}" ]; then + export ARGOCD_FQDN + kubectl apply -f argocd/cmd-params-cm.yaml + kubectl -n argocd rollout restart deploy/argocd-server + argocd/ingressroute-server.yaml.sh | kubectl apply -f - + # ARGOCD_PASSWD=$(kubectl -n argocd get secret/argocd-initial-admin-secret -o jsonpath='{.data.password}' | base64 -d) + fi + + echo -n "waiting for argocd to become ready ." + while [ $(kubectl -n argocd get pods | grep -c '1/1') -ne 7 ]; do + echo -n "." + sleep 1 + done + echo "ready." + kubectl -n argocd get all +} + +function do_chart { + cd ${SRC_DIR}/jitsi + + if [ -n "${EXCLUDE_JVB}" ]; then + EXCLUDE_JVB_VALUES_FILE="-f values-jvb-off.yaml" + fi + + if [ -n "${STAGING_CERT}" ]; then + CERT_RESOLVER="le-staging" + else + CERT_RESOLVER="le-prod" + fi + + helm -n ${NAMESPACE} upgrade -i --create-namespace ${HELM_NAME} . \ + -f values.yaml \ + $EXCLUDE_JVB_VALUES_FILE \ + --set certResolver=${CERT_RESOLVER} \ + --set fqdn=${FQDN} \ + --set jitsi-meet.publicURL=https://${FQDN} \ + --set jitsi-meet.jvb.publicIP=${PUBLIC_IP} \ + --set jitsi-meet.jvb.UDPPort=${JVB_PORT} +} + +function do_app { + cd ${WS_DIR}/ + + if [ -n "${DEPLOY_GIT_VERSION}" ]; then + SET_GIT_REVISION="--revision ${DEPLOY_GIT_VERSION}" + fi + + if [ -n "${EXCLUDE_JVB}" ]; then + EXCLUDE_JVB_VALUES_FILE="--values values-jvb-off.yaml" + fi + + if [ -n "${STAGING_CERT}" ]; then + CERT_RESOLVER="le-staging" + else + CERT_RESOLVER="le-prod" + fi + + argocd login --core + ORIG_NAMESPACE=$(kubectl config view --minify -o jsonpath='{..namespace}') + kubectl config set-context --current --namespace=argocd + + kubectl create ns ${NAMESPACE} --dry-run=client -o yaml | kubectl apply -f - + argocd app create ${HELM_NAME} \ + --upsert \ + --repo ${DEPLOY_GIT_REPO} \ + --path jitsi \ + ${SET_GIT_REVISION} \ + --dest-server https://kubernetes.default.svc \ + --dest-namespace ${NAMESPACE} \ + --values values.yaml \ + ${EXCLUDE_JVB_VALUES_FILE} \ + --helm-set certResolver=${CERT_RESOLVER} \ + --helm-set fqdn=${FQDN} \ + --helm-set jitsi-meet.publicURL=https://${FQDN} \ + --helm-set jitsi-meet.jvb.publicIP=${PUBLIC_IP} \ + --helm-set jitsi-meet.jvb.UDPPort=${JVB_PORT} + + sleep 5 # there is a race if sync happens too quickly, so that it becomes a partial sync + argocd app sync ${HELM_NAME} + kubectl config set-context --current --namespace=${ORIG_NAMESPACE} +} + +# installation starts from here +( + get_helm + + get_src + + do_k3s + + get_argocd # 'argocd version' depends on k3s setup + + do_traefik + + do_argocd + + do_app + +# installation ends here +) 2>&1 | tee ${WS_DIR}/deploy.log diff --git a/jitsi/Chart.lock b/jitsi/Chart.lock new file mode 100644 index 0000000..f582b6c --- /dev/null +++ b/jitsi/Chart.lock @@ -0,0 +1,6 @@ +dependencies: +- name: jitsi-meet + repository: https://jitsi-contrib.github.io/jitsi-helm + version: 1.2.2 +digest: sha256:165664c1a23bc9760177e63740a861360eee007b432d9044ea449e77fba95d94 +generated: "2022-05-02T17:15:02.132446+08:00" diff --git a/jitsi/Chart.yaml b/jitsi/Chart.yaml new file mode 100644 index 0000000..09309a9 --- /dev/null +++ b/jitsi/Chart.yaml @@ -0,0 +1,7 @@ +apiVersion: v2 +name: jitsi-deploy +version: 0.1.0 +dependencies: +- name: jitsi-meet + version: 1.2.2 + repository: "https://jitsi-contrib.github.io/jitsi-helm" diff --git a/jitsi/charts/jitsi-meet-1.2.2.tgz b/jitsi/charts/jitsi-meet-1.2.2.tgz new file mode 100644 index 0000000..29703ec Binary files /dev/null and b/jitsi/charts/jitsi-meet-1.2.2.tgz differ diff --git a/jitsi/templates/ingressroute-web.yaml b/jitsi/templates/ingressroute-web.yaml new file mode 100644 index 0000000..4d001d1 --- /dev/null +++ b/jitsi/templates/ingressroute-web.yaml @@ -0,0 +1,16 @@ +apiVersion: traefik.containo.us/v1alpha1 +kind: IngressRoute +metadata: + name: jitsi-web + namespace: {{ .Release.Namespace }} +spec: + entryPoints: + - websecure + routes: + - kind: Rule + match: Host(`{{ .Values.fqdn }}`) + services: + - name: {{ .Release.Name }}-jitsi-meet-web + port: 80 + tls: + certResolver: {{ .Values.certResolver }} diff --git a/jitsi/templates/ingressroutetcp-prosody.yaml b/jitsi/templates/ingressroutetcp-prosody.yaml new file mode 100644 index 0000000..e88f82d --- /dev/null +++ b/jitsi/templates/ingressroutetcp-prosody.yaml @@ -0,0 +1,12 @@ +apiVersion: traefik.containo.us/v1alpha1 +kind: IngressRouteTCP +metadata: + name: {{ .Release.Name }}-ingressroutetcp-prosody +spec: + entryPoints: + - xmpp-{{ .Release.Namespace }} + routes: + - match: HostSNI(`*`) + services: + - name: {{ .Release.Name }}-prosody + port: 5222 diff --git a/jitsi/values-jvb-off.yaml b/jitsi/values-jvb-off.yaml new file mode 100644 index 0000000..1e6e6a8 --- /dev/null +++ b/jitsi/values-jvb-off.yaml @@ -0,0 +1,5 @@ +jitsi-meet: + jvb: + replicaCount: 0 + service: + enabled: false diff --git a/jitsi/values.yaml b/jitsi/values.yaml new file mode 100755 index 0000000..7ff6811 --- /dev/null +++ b/jitsi/values.yaml @@ -0,0 +1,41 @@ + +certResolver: le-staging + +fqdn: "" + +jitsi-meet: + publicURL: "" + + tz: Asia/Shanghai + + web: + ingress: + enabled: false + + jicofo: + livenessProbe: + failureThreshold: 30 + periodSeconds: 10 + readinessProbe: + failureThreshold: 30 + periodSeconds: 10 + + jvb: + service: + # enabled: true + type: NodePort + # It may be required to change the default port to a value allowed by Kubernetes (30000-32768) + UDPPort: 30000 + + livenessProbe: + failureThreshold: 30 + periodSeconds: 10 + readinessProbe: + failureThreshold: 30 + periodSeconds: 10 + + websockets: + enabled: true + + # Use public IP of one of your node, or the public IP of a loadbalancer in front of the nodes + publicIP: "" diff --git a/traefik-config.yaml.sh b/traefik-config.yaml.sh new file mode 100755 index 0000000..4b8a215 --- /dev/null +++ b/traefik-config.yaml.sh @@ -0,0 +1,36 @@ +cat <