• Tildes Docs
• Development

General Development Info

Note: This page is fairly scattered overall and should be split up into multiple pages in the future. The goal is mostly to have a lot of information that developers might find useful so that they, at least, have a starting point to find out more about how something works.

If you haven't already, follow the instructions on the Development Setup page to get a development version running on your local machine.

Vagrant

In general, you shouldn't need to do very much with Vagrant directly, but there are a few commands you should know:

• vagrant up - Run this to boot the VM (or create a new one). It will probably take a few minutes to finish.
• vagrant ssh - Open an SSH session into the VM. You will, probably, want to have one of these open almost all the time while developing to be able to run checks, look at the database, etc.
• vagrant provision - If you make any changes to the Salt states, run this to re-apply everything to the VM and make sure its state matches the expected one
• vagrant halt - Shut down the VM.
• vagrant destroy - This will completely destroy the VM, and next time you run vagrant up it will re-build it from scratch. If your dev environment ever ends up broken, it's usually simplest to just destroy/recreate like this.

Changing the Vagrant box's resources

This generally shouldn't be necessary, but it's possible to change the resources the Vagrant box has access to by editing the Vagrantfile in the base directory. By default, the box is set up with 4GB of RAM and 4 CPUs, but you can adjust these by editing these two lines near the bottom of the file:

vb.memory = "4096"
vb.cpus = "4"

Important - if you edit those lines:

• Do not set the memory lower than 2048 (2GB). The VM will almost certainly end up swapping itself to death if it has less memory than that.
• If you change the values, run git update-index --assume-unchanged Vagrantfile so that git ignores those changes to the file and won't always want to commit them along with any actual updates you make to the Tildes code.

SSHing into the VM

You can open an SSH session into the VM by running vagrant ssh. Once inside, the application's virtual environment (venv) should automatically be activated for you, and you should be in the base application directory (/opt/tildes/).

Unless otherwise stated, all commands mentioned in this page should be assumed as needing to be run from inside the VM, from the directory /opt/tildes/, with the venv activated.

Shared folders with the VM

Vagrant maintains a few shared folders with the VM, which means that any changes you make to files in these folders from outside will be available inside the VM immediately. This allows you to edit the code from your main OS and have those changes reflected by your dev site that is being served from the VM. Warning: This sharing is bi-directional, so any changes made in these folders inside the VM will also affect the folders outside. Take care not to delete or change files in the shared folders inside the VM unless you want those changes to affect the "real" version of the file outside.

The following folders are shared (defined in the Vagrantfile):

Keeping your dev version up-to-date

First of all, it's definitely easiest if you keep your master branch "clean" by always working on all changes in their own branch. Assuming you're working that way and have the official repo set up as a remote named "upstream" (similar to the setup in this guide), you can update to the latest code by running (from your own machine, NOT the Vagrant VM):

git checkout master
git pull upstream master

After the pull completes, you may need to run additional commands depending on what exactly was updated. These commands are safe, so it should always be fine to just run them anyway if you're not sure:

• If there were any changes to files in the salt/ directory, you should re-apply the Salt states to your Vagrant VM by running (from your own machine, while the Vagrant VM is running) vagrant provision.
• If there are any new files in the tildes/alembic/versions/ directory, you will need to run (from inside the VM) alembic upgrade head.

As always, if something goes wrong and your dev VM ends up in a broken state, you can just destroy and recreate it and everything should be functional again. All your code changes, branches, etc. will be safe, as long you didn't make any changes directly inside the VM (which you usually shouldn't be doing anyway).

You should also try to update your branch(es) to use the latest code as soon as possible (and definitely before submitting a merge request), which should usually be as simple as, for each branch (again, on your own machine):

git checkout my-branch-name
git rebase master

Overall language/framework/server

The application is written in Python 3.8, using the Pyramid web framework.

It is served by Gunicorn behind an nginx reverse-proxy.

General development/debugging

The development version is set up to automatically restart when any changes are detected to the code, so it should not be necessary to do anything manually to have changes take effect.

It is running in debug mode, which will mean that any Python errors encountered should be displayed in the browser including a full traceback, ability to open an interactive shell in-browser and inspect variables, and so on. The Pyramid debug toolbar is enabled and accessible on any page by clicking the red Pyramid icon on the right edge. The debug toolbar interface allows you to see various information about each request, such as examining properties of the request, and checking which database queries were executed (and how long they each took).

If you introduce a "lower-level" error such as a Python syntax error, the application will fail when it attempts to restart and you won't be able to access the usual debugging methods. This usually appears as a "502 Bad Gateway" error in the browser. To see the details of the error preventing Gunicorn from starting, you will need to run sudo journalctl -u gunicorn.service -f, which should display the traceback. Gunicorn is set up to automatically try restarting every 30 seconds, so after fixing the error you shouldn't need to do anything except wait (up to) 30 seconds for it to restart. However, you can always restart it immediately by running sudo systemctl restart gunicorn.socket.

Interactive shell

You can open an interactive IPython shell inside the app environment by running invoke shell. This will include a request object that you can use to make database changes via request.db_session, just as inside app code. Note that any database changes will need to be manually committed by calling request.tm.commit().

Adding new Python packages/libraries

If you need to add a new Python dependency, you need to update a few files that define which dependencies the app installs. The list of the packages that need to be installed to run Tildes is kept in requirements.in, while the list of packages needed for a development version is in requirements-dev.in (which includes requirements.in). These files are used to generate requirements.txt and requirements-dev.txt, which are snapshots of the individual packages (including all dependencies) and the versions that are currently being used.

To install a new package, you should follow this process:

1. Add the package's name to requirements.in (or requirements-dev.in if it's only needed in the development version) in the correct (alphabetical) location.
2. Run both of these commands inside the VM, regardless of which .in file you added it to:
   • pip-compile --no-header --no-annotate requirements.in
   • pip-compile --no-header --no-annotate requirements-dev.in
3. From your machine, run vagrant provision. This will run the Salt states, which includes installing packages from requirements-dev.txt, so it will install the new one and any dependencies.
4. Make sure to include all of the modified requirements .in and .txt files in your commit.

HTML templates and CSS

HTML templates are written in the Jinja2 templating language.

CSS is written with Sass (SCSS syntax), and uses Spectre.css as a base. The SCSS is organized in a manner similar to the SMACSS style. Changes to the SCSS files (in the tildes/scss/ directory) will automatically be detected and compiled for you by a service running inside the Vagrant VM. If your changes are failing to compile due to an error, you can run sudo journalctl -u boussole.service -f to view that service's logs and see the error.

In the event that the service has failed to start, you may need to restart boussole.service, webassets.service, and gunicorn.service in order to get the full pipeline working.

Javascript, AJAX and the "web API"

Javascript usage is being kept minimal. Intercooler is being used for all AJAX, which allows defining AJAX actions easily using HTML data attributes (note that it is configured so that a data- prefix is required on attributes, so the attributes will be similar to data-ic-post-to instead of just ic-post-to).

When a new endpoint is needed for Intercooler, it should be grouped in the "web API", which are all located inside /api/web/. These endpoints are intended solely for "internal" usage by the site and can be changed as needed for any functionality. For Intercooler's purposes, they should generally return HTML fragments that are intended to be swapped into the page, but can also do things like use Intercooler response headers to trigger actions.

Intercooler includes jQuery 3 as a dependency, so jQuery is available as well.

All other Javascript behavior should be implemented using the RSJS guidelines, which generally means that the code for each behavior is defined inside a file in static/js/behaviors/ and must be attached to a "component" on the site by using an HTML data attribute with the data-js- prefix.

All the files in behaviors are automatically merged into the site's javascript file. However, if you add a new behavior file, then it won't be picked up until you run sudo systemctl restart webassets.service.

Database

The primary database being used is PostgreSQL 12.

You can access the command-line interface to the database by running psql -U tildes tildes. This can be useful to check exactly how things are stored or make sure that changes are applying correctly, but modifying data directly may cause inconsistencies that are usually managed by the app code, so it's often not the right approach.

Interacting with the database in the app

Database interactions inside the app are through the SQLAlchemy library. These interactions should almost always go through the per-request SQLAlchemy session available as request.db_session.

Each request is wrapped in a transaction by default, and it is not usually necessary to commit or rollback changes manually. The pyramid_tm library automatically handles most transaction tasks, including committing the transaction at the end of each request (or rolling it back if an error occurred).

If you really do need to manually control the transaction, you can call functions on the transaction manager directly such as request.tm.commit(). If you need to execute a rollback, it is almost always correct to use a savepoint and roll back to it, instead of rolling back the entire transaction (which could unexpectedly interfere with other database operations).

Making changes to the database tables

If you need to make any changes to the tables or columns (including adding new ones), you must create a revision using the Alembic migration tool which will allow all other instances of the site to easily apply the same changes to their database.

The general process for creating an Alembic migration is:

1. Make the changes to the table definitions inside the app code (not by altering the database directly).
2. If you are adding a new table, edit tildes/tildes/database_models.py and import the new model class along with the others already included there. Skip this step if you're only modifying existing tables.
3. Run alembic revision --autogenerate -m "Short description of the changes". You will see some output including the name of a new file that was generated inside alembic/versions/.

4. Open that new file inside your editor and review/edit all the code inside the upgrade() and downgrade() functions. Ensure that the code inside upgrade() includes all steps necessary to make the changes you need. Alembic isn't able to detect everything, so there may be some omissions. Similarly, ensure that downgrade() contains all steps to reverse those changes. Delete the automatic comments inside both these functions.

   Note: the Alembic version file should never import any constants, read text in from files, etc. It should have all that sort of information hardcoded so that the effects of the migration aren't modified if those external resources change in the future.

5. Run alembic upgrade head to apply your upgrade and make sure there are no errors. If the upgrade applies successfully, you should also connect to the database with psql -U tildes tildes and verify that all your changes were applied correctly.

6. If the upgrade looks correct, run alembic downgrade -1 to apply your downgrade and make sure there are no errors. As above, you should connect to the database to verify that all your changes were removed and the database is back to the state it was previously in.
7. If you run into any errors or missing changes during upgrade/downgrade, you may need to edit your script more and test more downgrades/upgrades until everything is correct. Once you're finished fixing any issues, run one final alembic upgrade head to make sure everything has been applied.
8. Make sure to include your Alembic file inside the commit that requires these database changes.

Data validation/sanitization

Most data validation and sanitization is being performed by Marshmallow schemas, which are located inside the schemas/ directory. In general, any data that's intended to go into the database should be first filtered through a schema to ensure it fits the restrictions or intended format. This should mostly be handled automatically, since SQLAlchemy is set up to do automatic validation using the associated schema for a model.

These Marshmallow schemas are also used by the webargs library, which takes incoming data from various sources in HTTP requests (POST/form data, url, query string, etc.) and passes it through the schemas. This allows validation/sanitization and some other niceties like automatic 422 responses when the data is invalid.

Instrumentation and metrics

Prometheus is being used for application and system metrics. Your development version's metrics are accessible and queryable at http://localhost:9090.

You can add new metrics in the application using the Prometheus Python client, but make sure that you understand the different metric types available and use an appropriate one, and try to include labels so that the metrics can be filtered in useful ways if that's applicable.

Running checks on your code

Once you start making changes to the code that you want to contribute to the project, you should ensure that the following commands all do not return errors (any merge requests that fail these checks will not be accepted).

• invoke test --full - Runs the full set of tests.
• invoke type-check - Runs the mypy static type checks.
• invoke code-style-check --full - Runs the full set of code style checks, including the Black Python code-formatter (errors from Black don't need to be fixed manually, just run black . to reformat your code).

If you use the included Git hooks, then a quick subset of these checks will automatically be run whenever you commit (and prevent you from committing if they fail), and all checks will be run before you can push your code to a remote repo.

Testing

Tests are written using the pytest framework and are located inside the tests/ directory. pytest will only detect new tests if they are inside a file whose name starts with test_, and with a function name also starting with test_ (other files/functions can be used for setup, fixtures, and so on).

Tests are able to (and should) use all database functions that the app itself can use. They operate inside their own PostgreSQL database, which uses nested transactions and transactional DDL to create tables that exist only inside the overall testing session's transaction (which is never committed). This ensures that no database changes persist between sessions even if the tests crash, and each testing session is always working with a clean database.

Tests also have access to their own temporary Redis instance that can be used when testing all functions involving Redis, such as rate-limiting.

The pytest-mock library is available for mocking/patching, as well as the freezegun library for tests that need control of time/date.

Functional testing can be done via the webtest library, which allows "higher-level" testing by sending requests to the app's URLs, examining responses, and so on. All tests using webtest should be kept inside the webtests/ subdirectory, and will not be run by default with invoke test, but can be included by adding the --webtests flag.

Type-checking with mypy

All function definitions must include type annotations that define the types of each argument and the return value. These annotations have no effect at runtime, but mypy is able to statically analyze the code using them to detect potential errors when functions are inadvertently called with arguments of the wrong type.

The type annotations are, generally, very straightforward and you can probably easily understand how to write them by just looking at existing functions, but the following pages in the mypy docs might be useful if you run into issues or need more information:

• mypy syntax cheat sheet (Python 3)
• Built-in types
• Kinds of types
• Common issues