No description
This repository has been archived on 2026-11-17. You can view files and clone it, but you cannot make any changes to its state, such as pushing and creating new issues, pull requests or comments.
  • Go 97.5%
  • HTML 0.9%
  • JavaScript 0.5%
  • Shell 0.4%
  • Dockerfile 0.4%
  • Other 0.3%
Find a file
2022-02-21 13:24:15 +01:00
api bugfix 2021-05-10 16:06:32 +02:00
archival Initial creation of a common export interface 2020-08-21 22:51:08 -06:00
assets Update oEmbed providers 2021-03-12 20:21:06 -07:00
cmd Add structured logging support 2021-04-15 22:17:35 -06:00
common add DomainName option to specify different public domain 2021-05-09 16:59:56 +02:00
controllers remove origin override 2021-05-09 17:55:49 +02:00
dev get dev nginx working with latest docker 2021-05-09 16:43:39 +02:00
docker Support enough to get Complement to run the tests 2020-07-24 22:00:10 -06:00
docs Move even more documentation to docs.t2bot.io 2020-08-15 15:41:06 -06:00
internal_cache Deprecate in-memory cache, preferring redis 2021-04-25 14:54:58 -06:00
ipfs_proxy Add optional Sentry support for error reporting 2021-03-17 21:00:42 -06:00
matrix Fix handling of guest accounts (MSC3069) 2021-03-21 21:29:33 -06:00
metrics Add optional Sentry support for error reporting 2021-03-17 21:00:42 -06:00
migrations Revert "Early gzip compression work" 2021-05-03 19:38:07 -06:00
pipline/upload_pipeline WIP on new upload pipeline 2020-07-19 20:35:18 -06:00
plugins Add antispam plugin support + OCR example 2021-01-02 23:59:00 -07:00
quota Add support for quotas 2020-08-02 00:12:11 -06:00
redis_cache Add experimental Redis caching 2020-07-19 18:57:44 -06:00
storage Revert "Early gzip compression work" 2021-05-03 19:38:07 -06:00
synapse Pull URL preview images for Synapse from the url cache 2020-08-17 08:47:43 -06:00
tasks Add optional Sentry support for error reporting 2021-03-17 21:00:42 -06:00
templates Handle the bulk of export operations 2019-10-27 20:34:35 -06:00
templating Track the user ID for exports/imports 2019-12-27 12:04:00 -07:00
tests/performance Bump lodash from 4.17.15 to 4.17.19 in /tests/performance 2020-07-17 10:42:34 +00:00
thumbnailing Fix memory exhaustion from crafted images 2021-04-15 20:19:05 -06:00
types Revert "Early gzip compression work" 2021-05-03 19:38:07 -06:00
util Revert "Early gzip compression work" 2021-05-03 19:38:07 -06:00
.dockerignore Make IPFS daemon persistent 2020-03-06 21:15:43 -07:00
.gitignore Update dev environment 2021-04-15 16:17:20 -06:00
.gitlab-ci.yml download image from proxy and use buildx for build 2022-02-21 13:24:15 +01:00
build-dist.sh Condense dist build script for ease of change 2020-07-20 20:58:42 -06:00
build.sh Fix build scripts 2020-01-04 16:34:07 -07:00
CHANGELOG.md Revert "Early gzip compression work" 2021-05-03 19:38:07 -06:00
ci-complement.sh Update complement CI 2020-09-06 22:29:50 -06:00
Complement.Dockerfile Try to fix permission errors on complement scripts 2020-09-06 22:49:31 -06:00
config.sample.yaml Revert "Early gzip compression work" 2021-05-03 19:38:07 -06:00
CONTRIBUTING.md Remove signoff requirement 2020-07-20 21:25:28 -06:00
Dockerfile download image from proxy and use buildx for build 2022-02-21 13:24:15 +01:00
go.mod Add optional Sentry support for error reporting 2021-03-17 21:00:42 -06:00
go.sum Add optional Sentry support for error reporting 2021-03-17 21:00:42 -06:00
LICENSE Initial commit 2017-11-10 22:41:51 -07:00
README.md Update dev environment 2021-04-15 16:17:20 -06:00

matrix-media-repo

Build status

matrix-media-repo is a highly customizable multi-domain media repository for Matrix. Intended for medium to large environments consisting of several homeservers, this media repo de-duplicates media (including remote media) while being fully compliant with the specification.

Smaller/individual homeservers can still make use of this project's features, though it may be difficult to set up or have higher than expected resource consumption - please do your research before deploying this as this project may not be useful for your environment.

For help and support, visit #mediarepo:t2bot.io. Administrator documentation can be found on docs.t2bot.io.

Installing

For installation instructions, see docs.t2bot.io.

Deployment

For deployment information, see docs.t2bot.io.

Developers

To properly run the media repo in a development setting, it must be compiled manually once to ensure the assets are set up correctly: follow the compilation steps posted on docs.t2bot.io.

If you'd like to use a regular Matrix client to test the media repo, docker-compose -f dev/docker-compose.yaml up will give you a Conduit homeserver behind an nginx reverse proxy which routes media requests to http://host.docker.internal:8001. To test accurately, it is recommended to add the following homeserver configuration to your media repo config:

name: "localhost"
csApi: "http://localhost:8008" # This is exposed by the nginx container

Federated media requests should function normally with this setup, though the homeserver itself will be unable to federate. For convenience, an element-web instance is also hosted at the same port from the root.

A postgresql server is also created by the docker stack for ease of use. To use it, add the following to your configuration:

database:
  postgres: "postgres://postgres:test1234@127.0.0.1:5432/postgres?sslmode=disable"
  pool:
    maxConnections: 10
    maxIdleConnections: 10

Note that the postgresql image is insecure and not recommended for production use. It also does not follow best practices for database management - use at your own risk.

Importing media from synapse

Media is imported by connecting to your synapse database and downloading all the content from the homeserver. This is so you have a backup of the media repository still with synapse. Do not point traffic at the media repo until after the import is complete.

Note: the database options provided on the command line are for the Synapse database. The media repo will use the connection string in the media-repo.yaml config when trying to store the Synapse media.

Note: the import script is not available to the Docker container. Binaries of the script are included with every release though if you want to avoid building it yourself.

  1. Build the media repo (as stated above)
  2. Edit/setup media-repo.yaml per the install instructions above
  3. Run bin/import_synapse. The usage is below.
    Usage of import_synapse:
      -baseUrl string
            The base URL to access your homeserver with (default "http://localhost:8008")
      -config string
            The path to the media repo configuration (with the database section completed) (default "media-repo.yaml")
      -dbHost string
            The PostgresSQL hostname for your Synapse database (default "localhost")
      -dbName string
            The name of your Synapse database (default "synapse")
      -dbPassword string
            The password for your Synapse's PostgreSQL database. Can be omitted to be prompted when run
      -dbPort int
            The port for your Synapse's PostgreSQL database (default 5432)
      -dbUsername string
            The username for your Synapse's PostgreSQL database (default "synapse")
      -migrations string
            The absolute path the media repo's migrations folder (default "./migrations")
      -serverName string
            The name of your homeserver (eg: matrix.org) (default "localhost")
      -workers int
            The number of workers to use when downloading media. Using multiple workers risks deduplication not working as efficiently. (default 1)
    
    Assuming the media repository, postgres database, and synapse are all on the same host, the command to run would look something like: bin/import_synapse -serverName myserver.com -dbUsername my_database_user -dbName synapse
  4. Wait for the import to complete. The script will automatically deduplicate media.
  5. Point traffic to the media repository.

Export and import user data

The admin API for this is specified in docs/admin.md, though they can be difficult to use for scripts. The bin/gdpr_export and bin/gdpr_import binaries do the process for you, and do so in memory but against the real media repo database and datastores - this moves the resource intensiveness to the binary you're running instead of the media repo instance, but still makes reads and writes to your database and datastores. For example, when exporting a user's data the binary will pull all the data locally and write it to disk for you, but during that process the user's export is accessible via the main media repo too. The export is deleted if the binary is successful at exporting the data.

Note: Imports done through this method can affect other homeservers! For example, a user's data export could contain an entry for a homeserver other than their own, which the media repo will happily import. Always validate the manifest of an import before running it!

Ensuring you have your media repo config available, here's the help for each binary:

Usage of gdpr_export:
  -config string
        The path to the configuration (default "media-repo.yaml")
  -destination string
        The directory for where export files should be placed (default "./gdpr-data")
  -entity string
        The user ID or server name to export
  -migrations string
        The absolute path for the migrations folder (default "./migrations")
  -templates string
        The absolute path for the templates folder (default "./templates")
Usage of gdpr_import:
  -config string
        The path to the configuration (default "media-repo.yaml")
  -directory string
        The directory for where the entity's exported files are (default "./gdpr-data")
  -migrations string
        The absolute path for the migrations folder (default "./migrations")
  -verify
        If set, no media will be imported and instead be tested to see if they've been imported already