Administering AO

This article is for the people who run AO for your organization. It covers what Admins do in AO, the settings Terradepth support changes for you, and the admin tool for running your own AO server.

Who does what

Role	Who	What they manage
Admin	People in your organization	Users, roles, groups and project access, in AO
Super Admin	Terradepth support	Organizations, Org Settings and System Settings for the whole AO system
IT administrator	Your own staff, on your own AO server only	The server itself, with `ao-mgr.sh`: setup, email, HTTPS, organizations, backups and upgrades

On AO in the cloud and AO for government, Terradepth runs the system. On your own AO server (self-hosted or AO OnBoard), the accounts ao-mgr.sh creates with init, org and admin are Admins, and Terradepth support makes the changes that need a Super Admin.

What Admins manage in AO

Click your initial in the top right corner and choose Users or Groups.

Task	Where	Read
Add, edit and remove users, and set their roles	Users	User Management
Send a new invitation or a password reset	Users, then open the user	Signing in and passwords
Create groups and give them access to projects	Groups, and a project's Group Access tab	User Management
Attach a project to one of your organization's cloud mounts	The project's Details tab, under File Source	Cloud mounts

Admins see every project in the organization, private ones included, and can edit them and share them with groups. Professionals share only the projects they created, with the groups they created. Public projects open read-only.

Settings Terradepth support changes

To add or change one of these, contact support@terradepth.com with your organization's name and which AO you use. Each article says what to prepare.

Setting	Read
Single sign-on with Microsoft, Google or your own identity provider	Set up single sign-on
Cloud mounts to your own storage, such as S3, Azure or a file server	Add a cloud mount
External layers for everyone in an organization, or on your whole AO server	External layers
Features turned on per organization, such as Remote Sync	Features turned on per organization
Sign-in records, and changes to single sign-on providers	Security in AO

If you run your own AO (self-hosted or AO OnBoard)

New server? Start with Set up your AO server: checklist.

Your IT administrator manages the AO server with ao-mgr.sh, the admin tool in the folder each AO release comes in. Run it on the AO server, from that folder, with the same Linux account each time; ./ao-mgr.sh help lists every command. Some servers have ao_manager instead, with the same setup, email, organization, user and backup commands: run it as sudo /opt/terradepth/ao_manager <command>.

Task	Command
First-time setup	`./ao-mgr.sh init`
Change email (SMTP) settings	`./ao-mgr.sh smtp`
Change the Cesium license key	`./ao-mgr.sh cesium`
Choose how people reach AO: HTTPS with a certificate, or HTTP only	`./ao-mgr.sh tls setup`
Check AO's HTTPS certificate and its expiry date	`./ao-mgr.sh tls status`
Create another organization and its first Admin	`./ao-mgr.sh org`
Add an Admin to an organization	`./ao-mgr.sh admin`
Add a user with the role you choose (Admin, Professional, User or Guest) and a password you set	`./ao-mgr.sh create_user`
Check that AO's services are running	`./ao-mgr.sh status`
Back up AO's database, and list backups	`./ao-mgr.sh backup`, `./ao-mgr.sh list_backups`
After an upgrade, delete what's kept for going back to the previous release	`./ao-mgr.sh delete-rollback`

Older releases don't have the tls and delete-rollback commands: ./ao-mgr.sh help shows what yours has.

When init, org, admin or create_user asks for an email address, type it in lower case: an account created with capital letters can't sign in with a password.

Use org, admin and create_user only for new email addresses: for an email that already has an account, they reactivate it and replace its role, name, organization and password. To change an existing person, use Users in AO. Give org a name no organization has used: with an existing name, even a removed organization's, it turns that organization back on and replaces its Cesium token. These commands don't check AO's password rules and show the password as you type it, so choose one that meets the rules in Signing in and passwords and type it where no one can see.

Terradepth support signs in to your server with its own Super Admin account, AO Support, which init sets up. To turn that account off, run ./ao-mgr.sh disable_support_user; to let support sign in again, run ./ao-mgr.sh enable_support_user. While it's off, support can't make the changes above.

First-time setup

init asks for your mail server and sending account, your company's contact details, the Cesium license key, your organization's name and Cesium token (required: if you don't have one, ask Terradepth support), the first Admin's details, and last, how people will reach AO (see HTTPS, below; older releases skip this). For that last step it asks for your sudo password. If it stops after creating the organization, finish with ./ao-mgr.sh admin rather than running init again. If only the last step fails, run ./ao-mgr.sh tls setup.

For the email prompts, and what to do on a network without email, see Set up email for invitations and password resets.

HTTPS

AO serves HTTP on port 8888. To serve it over HTTPS on port 443, run ./ao-mgr.sh tls setup on the AO server (init asks the same question); it asks for your sudo password. Enter the address people will use, with no path, such as https://ao.example.com. Then choose how AO gets its certificate:

Choice	Use it when	What to know
Self-signed certificate	Testing, or a handful of computers	Browsers warn until each computer trusts this certificate. AO replaces it each year, and each computer must then trust the new one
Appliance CA	Your network has no internet access	AO creates its own certificate authority and issues its certificate offline. Install the authority's root certificate on each computer once
Customer-provided certificate and private key	Your organization issues certificates	PEM files on the AO server, with the private key unencrypted. Supply a new certificate before this one expires
Tailscale	The AO server is on your tailnet	Use the server's full `.ts.net` name, with MagicDNS and HTTPS certificates turned on for your tailnet. Needs internet access. The name appears in public certificate transparency logs
Let's Encrypt	AO has a public DNS name	The name must point to the AO server, and port 80 must be open to it from the internet, with nothing else using it. Install Certbot on the server first. AO asks for an email address and for you to accept Let's Encrypt's subscriber agreement. The name appears in public certificate transparency logs
HTTP only	You don't want HTTPS	Turns off HTTPS that `ao-mgr.sh` set up

HTTP stays on. Port 8888 keeps working, with no redirect to HTTPS. Give people the HTTPS address.
Names and trust are yours to set up. AO doesn't create DNS records, change the server's network or firewall, or make computers trust its certificate. ./ao-mgr.sh tls dns-record --hostname <name> --address <IP address> prints the DNS record for your network team. People must use the name on the certificate.
Appliance CA root certificate: ./ao-mgr.sh tls export-ca > ao-root.crt saves it, and openssl x509 -in ao-root.crt -noout -fingerprint -sha256 shows its SHA-256 fingerprint. Check the fingerprint with whoever installs it, over a channel you trust. Install it only on computers that use this AO, and keep the AO server and its backups secure: those computers trust every certificate this authority issues.
Windows computers: internal\setup-https-client.ps1 in the release folder installs the root certificate, adds AO's name to the computer's hosts file with the AO server's IP address, and puts an Absolute Ocean shortcut on the desktop. Give the AO server a fixed address first, such as a DHCP reservation. Copy the script and ao-root.crt to a folder on the Windows computer, open PowerShell as an administrator in that folder, and run powershell -ExecutionPolicy Bypass -File .\setup-https-client.ps1 -Hostname <AO name> -Address <AO server's IP address> -CaCertificate .\ao-root.crt -CaSha256 <fingerprint>. Type the fingerprint as its 64 letters and digits, without the colons.
Renewal: a daily job on the server renews AO's certificate before it expires. A customer-provided certificate is the exception: replace it yourself. ./ao-mgr.sh tls status shows the certificate's names, fingerprint and expiry date, and warns within 30 days of expiry.
Backups: AO's database backup doesn't include its certificates or the Appliance CA, so back up the whole VM. If the Appliance CA is lost, every computer needs a new root certificate.
Your own TLS proxy in front of port 8888 also works.

If ./ao-mgr.sh help doesn't list tls (older releases), put AO behind your own TLS proxy (see Network and firewall requirements), or ask support@terradepth.com to set up HTTPS. CUSTOMER_INSTALL_GUIDE.md in the release folder has the full HTTPS options.

With single sign-on, register AO's HTTPS address with your identity provider: see Set up single sign-on.

Backups

./ao-mgr.sh backup saves a copy of AO's database on the server; ./ao-mgr.sh storage-info shows where AO keeps files and backups. A database backup doesn't include uploaded files, so also back up AO's file storage or the whole VM, and keep a copy off the server. Backups aren't encrypted: store them, and delete old ones, as your data policy requires. Restoring replaces AO's database and can't be undone: contact support@terradepth.com first.

Upgrades

Each release arrives as a complete folder, so the server needs no access to a container registry. Copy it to a new folder on the server and keep the previous one. Schedule downtime, stop people from making changes, and back up the VM along with any AO storage kept outside it. Then follow the guide in the folder, CUSTOMER_INSTALL_GUIDE.md. In short:

Install: run ./install-upgrade-ao.sh from the new release folder. If AO uses HTTPS set up with ao-mgr.sh tls, run sudo ./install-upgrade-ao.sh instead. It asks you to confirm, then upgrades AO and checks its database and services.
Wait for Container-based installation completed successfully! and Host system setup complete!, then run ./ao-mgr.sh status.
Sign in, open existing projects and visualizations, download a file, and check single sign-on, email and other connections, before people start making changes again.
When you're sure you won't go back to the previous release, run ./ao-mgr.sh delete-rollback. It lists what it will delete and asks you to confirm; pressing Enter keeps everything.

If the install stops, run the same command again from the same folder: it picks up where it stopped. Don't delete AO's containers, volumes or data, or files in the release folder, to get past an error: contact support@terradepth.com with the error.

Until you run delete-rollback, an upgrade that moved AO's database to a new major version can be undone with ./ao-mgr.sh rollback-stack --accept-new-write-loss. The flag confirms that changes made in AO after the upgrade aren't carried back. For other upgrades, the way back is the VM backup you made first: contact support@terradepth.com before restoring it.

The installer works on a new server, or on one that already runs AO installed from a release folder. On other AO servers it stops before changing anything: contact support@terradepth.com to plan the move.

Older release folders, with install-containerized-ao-pd.sh instead: run ./ao-mgr.sh e2e-seed from the new release folder while the current version runs, install with ./install-containerized-ao-pd.sh --runtime docker (after a stop, run it again with --resume), then run ./ao-mgr.sh e2e-verify, ./ao-mgr.sh status and ./ao-mgr.sh e2e-cleanup. e2e-verify saves a full, unencrypted copy of AO's database under ~/.local/state/ao-upgrade-e2e/ in the home folder of the account that ran it, and e2e-cleanup keeps it, as do later upgrades: store or delete it as your data policy requires.

Questions? Contact support@terradepth.com.