TT - TeamTalk Server Box Administration Utility
Doug Lee
Last Revised April, 2024

tt is a utility designed to help those who host several TeamTalk servers on a single box or virtual machine (VM). It may also be used to facilitate testing of TeamTalk server versions and configurations in a local Windows/WSL or virtual-machine environment, though not all features may be supported there. This document describes this utility's usage and capabilities.

This is a command-line utility and is intended for use within a command terminal, such as an ssh session or a Linux or WSL console session. As such, invocation is typically from a shell and consists of a command, such as tt list or tt rlist -l |more.

This utility is meant to be run on machines where TeamTalk (TT) servers are physically running, or to be run through ssh connections with such machines. This is not a utility of use to TeamTalk users who do not run a server or have direct access to running servers for administrative assistance. It is in fact most likely to be useful to those who run many TeamTalk servers at once. See the "System Requirements" section below for further information.

tt is distributed under the BSD license. Run tt license for details, or examine the tt code below the line starting with "def do_license" to see the license without running the program.

This Python-based utility was originally inspired by a set of shell and systemd scripts written and maintained by Daniel Nash (2021-22). The rlist command is a full absorption of a never formally released TTMan project by Doug Lee started in 2018. Similarly, the logsum command is a full absorption of a stand-alone utility written by Doug Lee in 2010 and maintained occasionally since.

Table of Contents

System Requirements

General Requirements

In order to function properly, tt requires the following system configuration on the box where it is to be used:

Note that the rlist command is one of the few commands that may prove useful on boxes that do not organize their TeamTalk servers as just described. Also, when tt is run through ssh connections, the above requirements apply to the ssh targets more than to the local machine.

Once operational, this utility will also need, or will set up as necessary, the following:

Notes For WSL

The Windows Subsystem for Linux (WSL) does not, at least by default, support use of the systemd utility for managing services and startup-time activities. The use of tt under WSL will therefore likely be limited to TeamTalk server binary installation and updating, and TeamTalk server creation and configuration. The starting and stopping of TeamTalk servers must be managed in some other way. If systemd is enabled under WSL, other features of this utility should begin to function as expected.

Notes For MacOS

There are numerous limitations to this utility's ability to function effectively on MacOS:

Installation and Setup Instructions

To install and configure this utility on a new system:

  1. Download and unpack tt.tgz into /usr/bin or another suitable folder that is included in the PATH environment variable for root. This file contains only the tt file, without any containing folder.
  2. To verify that tt will run on your system, type tt without arguments as a command while running as root. This will print a basic help page that includes a list of all available commands, grouped into categories. This is the same page produced by typing tt help. tt help followed by a command or category name will provide more detailed help related to that command or category.
  3. If you have unusual configuration needs, address those now by creating a tt.conf file in the same folder where tt itself resides. This is normally not necessary, however. Run tt TTConf for details on what can be placed in this file. In summary, this file can alter the following:
  4. To install the TeamTalk server itself, run tt binupdate as root. If you are managing professional TeamTalk servers, instead run tt binupdate -p. Add -b if you want to install a beta TeamTalk server version, or add a version number like 5.11 if you want to select a specific server version. When you do one of these, tt will set up other required infrastructure - systemd files, tt user and group, etc., if necessary.

TeamTalk Server Creation and Maintenance

There are two ways to create a new server:

If you intend that file transfers be allowed by a TeamTalk server, no matter which way it was created, make sure that the full path for the file upload folder is correct, and make the final component of that path take the form of files, a dot, and the exact server name. For instance, if all TeamTalk server files are in /home/tt and the new server is called mine, type /home/tt/files.mine for the user file upload path when prompted by the TeamTalk server setup wizard.

If you need to reconfigure a server later, use tt reconfigure followed by the server name. To remove a server, run tt remove followed by the server name.

To make a server start automatically at system boot, run tt enable followed by the server name. tt disable will remove this server from the set started automatically at boot time.

A server can be started manually with tt start and stopped with tt stop, each followed by the server name. A server can also be restarted (stopped and immediately started again) with tt restart followed by the server name. Finally, tt reload followed by a server name will reload the server configuration without restarting the server. Note that this does not reload everything; run tt help reload for details.

Other commands commonly used over the lifetime of a set of TeamTalk servers:

Miscellaneous Commands

tt help provides help for all commands supported by the utility. Typing tt without arguments produces the same information. This information also includes the revision number of tt itself. Run tt help followed by a command name for more help on that command. This help will often contain more detail than is found in this document.

tt man opens this document from its home on the Internet. tt rel opens the same document but jumps to the Revision History section (release notes). Both of these commands accept a -b option to use the beta tt release's documentation instead of the regular one.

tt license prints the license that covers the utility.

tt version prints the version of the installed TeamTalk server binary.

tt init will initialize a system for use with this utility but without downloading the TeamTalk server binary. This command is normally not needed since it is automatically run as part of many commands if necessary.

tt list will list servers configured on this box. See also The RList Command for a way to list actually running servers.

tt analyze analyzes the server configurations on this box and reports any of the following as appropriate:

By default, this command scans all servers that would be found by a tt list command; i.e., all servers defined on the box. As an alternative, file and/or folder names may be specified on the command line, in which case they are scanned instead. Folders appearing on the command line are searched recursively Files whose names do not end in .xml and folders whose names begin with a dot are quietly skipped.

tt log and tt logcat will print system logs for a server. These commands require the Linux journalctl command. See also The LogSum Command for a way to summarize TeamTalk server log files. This command does not require journalctl.

tt selfupdate will update this utility in place from its home website. Including -b on the command line will update to the latest beta version instead of the latest official release. Including -y will force the update without a prompt for permission. Including -c or -n will instead just check for an update and report the situation without actually updating the utility. If -y is given along with -c or -n, it will be ignored and no update will be performed.

If you want to remove this tt utility, use tt uninstall This will remove as many of the following as necessary and possible, in this order, but will prompt for permission before removing each in turn:

The RList Command

The rlist command can perform the following tasks:

Syntax: tt rlist [-i | -l] [-<count>[d|i|t]] [filters...] [-k | -r]

Without options or with -i, this command lists all servers on the machine that are currently running, in a tabular format. If filters are given, only servers matching all filters are listed. Filters are character strings that may appear in a server's name, config or log file path, or working directory path. A port number may also appear to limit matches to servers running on that TCP or UDP port.

The list can also be filtered with an option consisting of a dash (-), a number 0 or higher, and optionally a letter indicating which count to check. Counts are i for IP count, d for duplicate count, and t (the default type if none is given) for total of the two. A common option here is -1 to ignore all servers with only one active connection, which effectively causes servers to be ignored if they currently don't have enough tonnections to permit an active conversation. -0 is also useful for removing servers from the list that have no connections at all.

If no formatting options are given, the output consists of the following information for each server listed:

If -i is given, an extra line is printed for each server indicating the IP addresses and ports to which clients are connected.

when -l is given, the output instead consists of a multiline output block listing fields of information about the server by field name. The first line for each server appears flush left, and subsequent lines of information about that server are indented beneath it. The first line indicates the server's process id (pid) and full executable path. Subsequent lines are meant to be self-explanatory, but the following notes may be useful:

Regardless of output format, the output ends with a single line that indicates the following:

If -k or -r is given, each listed server is killed (shut down) or reloaded, respectively, after the server list is displayed. (Run help reload for information on what reloading a server does.) Filters work here just as for the rlist command without -k or -r. It is very wise to run a rlist command without -k or -r first to make sure the correct servers will be shut down or reloaded.

The LogSum Command

This command scans through one or more TeamTalk server log files and produces a summary of contents. This command is based on a 2010 utility that this author wrote before TeamTalk 5 was released and has maintained occasionally since. As a result, this command is capable of understanding both TeamTalk 4 and TeamTalk 5 log files. Support for TeamTalk 4 logs is subject to removal.

Note: Server log inspection is of course only possible for servers that actually write server logs. Additionally, TeamTalk 5.9 introduced the ability for server administrators to alter which events are recorded in such files. Changes in these settings will naturally affect the results of this command. Finally, the specific versions of TeamTalk servers that generated the files will affect what is stored in them; e.g., SSL errors only appear in logs for professional servers, subscription and intercept changes only appear starting in TeamTalk server version 5.11, and log line formats may change from time to time.

Specify one or more file and/or folder names on the command line. Specifying "-" causes stdin to be processed as a TeamTalk server log file. Explicitly named files should be TeamTalk server logs and will be processed. Named folders will be recursively scanned for logs, and each will be processed. Such a scan will consider any file a TeamTalk server log if it meets all of these criteria:

Any file not matching these rules is quietly skipped.

An explicitly listed file may also be a diff output file, in which case only added log lines from any represented actual log file will be examined. In this mode, which is started automatically per file by a recognition of the format, both old-style and unified-style diff files are handled. This also works for "-" to make stdin process as diff output. This in combination with a version control system provides means to get periodic stats from a log file without rotating log files.

Diagnostic output goes to stderr during log processing. The name of each processed file is printed as the file's processing begins. If more than one million lines of data are processed, a line indicating how many million have been processed so far is printed, so that very long-running analyses don't look stalled. (This feature was prompted by a 32 gig, 349 million line analysis run in 2010.) Note that lines are counted across files, not started at one for each file.

At the conclusion of a run, statistics will be sent to stderr indicating number of files and lines processed, number of seconds used, and number of lines processed per second.

Result output is sent to stdout and consists of line categories and counts found of each, after a copy of the final statistics just described that are also sent to stderr.

For each of several line types, the name and number of occurrences is shown. Types with no matches are omitted. Any line not matching a known type will appear in a dynamic category. The category will consist of the line with commonly unique elements replaced, and with a possible prefix indicating things like "no timestamp." "Commonly unique elements" include IP/TCP/UPD addresses, nicknames, etc. This all means that unrecognized lines are grouped sensibly by what they say. Dynamic entries will appear below known types, sorted by decreasing frequency.

WARNING: Although there was much effort to prevent this, dynamic entries may occasionally include IP addresses, channel numbers, etc. They should not include user names, nicknames, channel names, etc. Nevertheless, be sure to scan any results for unwanted information before releasing statistics for public examination.

A few terms in log line type names deserve explanation:

Server starts and stops
These represent actual log entries for server starts and stops. They may differ, however, when the TeamTalk server's machine or VM is restarted without a proper shutdown, the TeamTalk server is killed by a signal, etc.
Connects and disconnects
These are connections to the server before login, and disconnections from the server. Connections to TeamTalk servers are often made by non-TeamTalk clients that are scanning for available services on machines; so these numbers may be significantly higher than the numbers of logins and logouts. Connect counts may exceed disconnect counts for the same reasons server starts may exceed server stops.
IPV4on6
This term may occur in login and connect counts. These represent connections from an IPV4 client to an IPV6 socket on the TeamTalk server. These addresses begin with "::ffff:"
Logins from text clients
Text clients do not set up UDP connections, so their UDP address appears as ":::0" (the IPV6 version of an empty address).
Solo transmitter updates
TeamTalk 5.2 introduced single-transmitter (no interruption) channels, and TeamTalk 5.9 began logging events when the transmitter in such a channel is changed automatically or by a user. This utility provides counts for both of these event types.
Subscriptions that include intercepting
In a log entry for a user's subscription changes, intercept flags are pairs of capitalized letters, whereas normal subscriptions are lower-case letter pairs. TeamTalk records all set subscription and intercept bits any time it logs a change in them for a user. As a result, the count of log entries that include set subscription bits may not reflect the number of intercepted users or the number of times users were intercepted. However, whereas normal subscription changes can occur automatically at login due to user's default subscription settings, entries containing intercept bits represent user-initiated actions.
kicks from server for login replacement
When a server disallows multiple simultaneous logins by one user, and that user tries to log in while already logged in, this log entry is generated as the older login is kicked to let the new one in.
user creates/updates
Unfortunately, there is no difference in server logs between user account creations and updates to existing accounts.
Total failed assertions
An assertion is a statement that the code author expected to be true at all times. A failed assertion is an occasion where this expectation proved to be false. TeamTalk servers sometimes generate these as an indication of an unexpected (by the TeamTalk author) condition. These may sometimes be useful to report to TeamTalk forums, Github, etc. The specific errors encountered are counted separately later in the dynamic-entry section of the results. To report any of these to the TeamTalk author, first find examples of the actual entries in log files, in order to include exact source code file and line numbers in your submission. Include the TeamTalk server version number in your submission as well.
Total SSL errors
These occur on professional TeamTalk servers, sometimes quite often. The specific errors encountered are counted separately later in the dynamic-entry section of the results. A few things are commonly seen:

The fln Command For Saving Disk Space

For box administrators who allow TeamTalk users to upload files to TeamTalk servers, the fln command may save significant disk space depending on user upload patterns. This command will look for duplicate files among all files uploaded to all TeamTalk servers managed by this utility and can convert sets of duplicates into links to a single instance of the file.

If run without option arguments, fln will first report what can be done. The output consists of statistical information on what was found, as well as the specific names of files to be linked. This may cause significant output on the first tt fln run on a particular box. tt will then prompt the user for permission to make the suggested links.

The above behavior can be altered with any one of the following options:

-n
Print what is found and could be done, but do not make any changes or prompt for permission to do so.
-y
Make the links without requiring confirmation.
-q (lower case)
Make the links but print only what was done, not summaries of file states before action. The output will not include individual file/link names.
-Q (upper case)
Make the links but print nothing.
-b
Make backups of every file replaced by a link, by appending ".bk" to its original name. This option is not intended for regular use but was used during development. It has the side effect of negating all space savings until the backup files are deleted.

tt fln also accepts one or more folder paths on the command line. By default, the target box's TeamTalk file tree (/home/tt unless altered by tt.conf) is scanned. If one or more folders are given, tt fln scans the listed folders instead. This is useful for boxes that do not use the normally expected file structure for their TeamTalk server data.

The command tt fln -q is a decent candidate for a cron job to be run as the TeamTalk server user (usually tt). There probably won't be a need to run this very often though, unless TeamTalk users are in the habit of uploading unusually often and duplicating many of their downloads across servers and channels..

Important notes and caveats:

Trivia: ln is the name of the Linux (and other flavors) command for making hard file links. The letters "fln" do not in fact stand for "file links, now!" although given the distinct possibility of tt fln -y being used often enough to rescue a file system with no remaining disk space, this tt author is happy to let people think so.

One final very fine bit of print on how tt fln determines how to link files: If a user upload exists in five places (channels and/or TeamTalk servers on the same box), one instance must be selected as the one to which all others will point. This will be called the target instance. The following rules apply, in order:

  1. If any of the instances has an external link (one pointing outside the TeamTalk data area), the first of these is used as the target instance. This can prevent such a link from reducing the space savings, since deleting a file with a hard link only deletes the link, not the file itself. tt fln will warn of such external links on finding them. If two different instances of the same user upload have external links, tt fln will also indicate this as "held space," meaning space that is held by external links despite tt fln's efforts.
  2. If no user upload has an external link, then the oldest one according to operating system last-modification time is used as the target instance.

Remote Usage

Note: For reasons that will shortly be explained, Many commands are likely not to work as intended when run in this way. Commands especially well-suited to this usage are noted as such later in this section.

tt supports running itself on a remote computer via an ssh connection. The syntax for this is the same as for local usage except for the addition of a host specification between tt and the rest of the command being run. Running tt in this way is no more or less secure than using scp to copy tt to the remote host and then running it inside an ssh session remotely; but this single-command approach can definitely be more convenient, especially for remote hosts that do not also have tt installations.

A host specification may be of the form user@hostname, as for ssh itself, or @host if it is not necessary to specify a user name (because of a ~/.ssh/config entry or because the remote system's user name is the same as the local one). Note the initial @ sign that distinguishes a host name from a command word.

It is actually acceptable to include more than one host specification. In that case, each is interpreted in turn from the standpoint of the previous host in the list. The local tt instance is sent to the first host, then run on the first host to send itself to the second, etc., until it is run on the final host to return results all the way back to the local one. The multi-command equivalent to this would be to scp tt to the first host, log into it and scp to the second, etc., then to log in to the final host and run the utility, deleting the various copies afterward. This author expects this scenario to occur infrequently and only to two levels of hosts.

Example commands, each of which list TeamTalk servers on the final host listed in the command:

tt @host1 list
tt @host1 @host2 rlist -l

Note that remote usage of this type causes the local tt to be run on the remote host(s). Hosts need not also have tt installed for this to work.

Important caveats:

Especially as a result of the last two items above, the following commands are most likely to be useful when run remotely, some with the given caveats; whereas other commands are likely not to work well remotely:

Special Uses

This section documents special uses of this utility that are outside the normal every-day circumstances.

Fixing Logs After a File System Overflow

When a file system runs out of space, TeamTalk logging is forced to stop for lack of storage. Logging will not resume without intervention. Additionally, a TeamTalk configuration .xml file can become empty if the running TeamTalk server tries to update it when there is no room to write the update.

The following procedure should both avoid lost configuration data and resume logging when possible. The order of these steps is essential:

  1. Fix the out-of-space disk condition: Make sure there is sufficient disk space for all server activities before going on to the next step.
  2. Save configuration: As an administrator in TeamTalk, save the current configuration. This will fix the .xml if its size was 0 by writing to disk what should have been there already. If you are fixing more than one TeamTalk server on the same box, be sure to make each server individually save its configuration before moving on.
  3. Verify that the above step worked: Make sure the .xml file or files contain appropriate data. If the previous step failed and you execute the next step without noticing, you will lose your TeamTalk configuration information in the next step.
  4. Restart logging: Using this tt utility, make the TeamTalk server reload its data from the .xml file, which will also make it restart logging. The command for this is tt reload followed by the server's name (the .xml file's name without the .xml extension). If you have saved the configurations for all running servers on the box, the single command tt reload all will restart logging on all servers at once.

Known Issues

Issues shown in this section may be fixed in future tt revisions but are not addressed as of this writing.

The rlist command may show 0 connections to all servers if the user id under which it is run does not grant lsof sufficient permissions to provide adequate information. This situation does not produce an error message.

The requirements and operation of the rlist command is notably different from other commands, which may confuse new tt users.

Why I Wrote This Utility

Short answer: I wrote it for myself and a few friends, then decided to publicize the result. The rest of this section is for anyone more curious and includes more information about Daniel Nash's original shell script and this author (Doug)'s 2018 TTMan program, on both of which this Python utility is based.

When I began my excursions into TeamTalk in 2011 or so, there were comparatively few people setting up their own servers. By now (2022), this has changed significantly as more and more people have begun experimenting with Linux variants, both in the cloud and on home equipment. Now there are not only many people running their own servers but also a fair number of people running TeamTalk servers for others, sometimes for free and sometimes for nominal fees.

One of my trademarks is liking to automate just about anything that is a sufficient mix of annoying and repetitive. As I have often helped people with Linux, TeamTalk server setup and administration, and the like; I wound up helping several people at once in 2021 and 2022 with the administration of what I started to call "TeamTalk server farms," meaning boxes or virtual machines that hosted a sometimes significant number of individual TeamTalk servers. I had already been doing this for one person for years and had, starting in 2018, written the TTMan utility for listing information about running servers on a box.

In 2021, I became aware of Daniel Nash's shell script that sought to ease the management of such multi-server boxes. I considered the idea quite useful but found it hard to implement sufficient error checking strictly within shell scripting. Versions of Daniel's work also varied from machine to machine; and some used zsh while others used Bash. This complicated sharing updates.

I thus started working on a parallel Python version and told him so, showing him code now and then as the project progressed. Many commands in this Python-based tt utility first appeared in Daniel's shell script. Daniel is also the author of the systemd service file that is included within this Python utility. For reference, I have saved a June 2, 2022 copy of Daniel's distribution on my site. This was the latest version of his work that I referenced while writing this utility.

At the time of writing for this section (September, 2022), I am using this utility to help manage approximately 25 boxes that each run several TeamTalk servers; and I thought I might share the utility for anyone else out there doing a similar thing.

Revision History

Here is the revision history of this utility, most recent revision first:

Revision 172, released April 8, 2024

Revision 160, released November 11, 2023

Revision 149, released July 12, 2023

Revision 140, released May 29, 2023

5 older revisions back through September 4, 2022

Revision 137, released March 27, 2023

Revision 117, released November 25, 2022

Revision 101, released November 13, 2022

Revision 99, released October 3, 2022

Revision 95, released September 4, 2022