General Development Info
- SSHing into the VM
- Shared folders with the VM
- Keeping your dev version up-to-date
- Overall language/framework/server
- General development/debugging
- Interactive shell
- Adding new Python packages/libraries
- HTML templates and CSS
- Interacting with the database in the app
- Making changes to the database tables
- Data validation/sanitization
- Instrumentation and metrics
- Running checks on your code
- Type-checking with mypy
Note: This page is fairly scattered overall and should be split up into multiple pages in the future. The goal is mostly to have a lot of information that developers might find useful so that they, at least, have a starting point to find out more about how something works.
If you haven't already, follow the instructions on the Development Setup page to get a development version running on your local machine.
In general, you shouldn't need to do very much with Vagrant directly, but there are a few commands you should know:
vagrant up- Run this to boot the VM (or create a new one). It will probably take a few minutes to finish.
vagrant ssh- Open an SSH session into the VM. You will, probably, want to have one of these open almost all the time while developing to be able to run checks, look at the database, etc.
vagrant provision- If you make any changes to the Salt states, run this to re-apply everything to the VM and make sure its state matches the expected one
vagrant halt- Shut down the VM.
vagrant destroy- This will completely destroy the VM, and next time you run
vagrant upit will re-build it from scratch. If your dev environment ever ends up broken, it's usually simplest to just destroy/recreate like this.
Changing the Vagrant box's resources
This generally shouldn't be necessary, but it's possible to change the resources the Vagrant box has access to by editing the
Vagrantfile in the base directory. By default, the box is set up with 4GB of RAM and 4 CPUs, but you can adjust these by editing these two lines near the bottom of the file:
vb.memory = "4096" vb.cpus = "4"
Important - if you edit those lines:
- Do not set the memory lower than
2048(2GB). The VM will almost certainly end up swapping itself to death if it has less memory than that.
- If you change the values, run
git update-index --assume-unchanged Vagrantfileso that git ignores those changes to the file and won't always want to commit them along with any actual updates you make to the Tildes code.
SSHing into the VM
You can open an SSH session into the VM by running
vagrant ssh. Once inside, the application's virtual environment (venv) should automatically be activated for you, and you should be in the base application directory (
Unless otherwise stated, all commands mentioned in this page should be assumed as needing to be run from inside the VM, from the directory
/opt/tildes/, with the venv activated.
Shared folders with the VM
Vagrant maintains a few shared folders with the VM, which means that any changes you make to files in these folders from outside will be available inside the VM immediately. This allows you to edit the code from your main OS and have those changes reflected by your dev site that is being served from the VM. Warning: This sharing is bi-directional, so any changes made in these folders inside the VM will also affect the folders outside. Take care not to delete or change files in the shared folders inside the VM unless you want those changes to affect the "real" version of the file outside.
The following folders are shared (defined in the Vagrantfile):
|Outside folder||Path inside VM|
Keeping your dev version up-to-date
First of all, it's definitely easiest if you keep your
master branch "clean" by always working on all changes in their own branch. Assuming you're working that way and have the official repo set up as a remote named "upstream" (similar to the setup in this guide), you can update to the latest code by running (from your own machine, NOT the Vagrant VM):
git checkout master git pull upstream master
After the pull completes, you may need to run additional commands depending on what exactly was updated. These commands are safe, so it should always be fine to just run them anyway if you're not sure:
- If there were any changes to files in the
salt/directory, you should re-apply the Salt states to your Vagrant VM by running (from your own machine, while the Vagrant VM is running)
- If there are any new files in the
tildes/alembic/versions/directory, you will need to run (from inside the VM)
alembic upgrade head.
As always, if something goes wrong and your dev VM ends up in a broken state, you can just destroy and recreate it and everything should be functional again. All your code changes, branches, etc. will be safe, as long you didn't make any changes directly inside the VM (which you usually shouldn't be doing anyway).
You should also try to update your branch(es) to use the latest code as soon as possible (and definitely before submitting a merge request), which should usually be as simple as, for each branch (again, on your own machine):
git checkout my-branch-name git rebase master
The application is written in Python 3.8, using the Pyramid web framework.
It is served by Gunicorn behind an nginx reverse-proxy.
The development version is set up to automatically restart when any changes are detected to the code, so it should not be necessary to do anything manually to have changes take effect.
It is running in debug mode, which will mean that any Python errors encountered should be displayed in the browser including a full traceback, ability to open an interactive shell in-browser and inspect variables, and so on. The Pyramid debug toolbar is enabled and accessible on any page by clicking the red Pyramid icon on the right edge. The debug toolbar interface allows you to see various information about each request, such as examining properties of the request, and checking which database queries were executed (and how long they each took).
If you introduce a "lower-level" error such as a Python syntax error, the application will fail when it attempts to restart and you won't be able to access the usual debugging methods. This usually appears as a "502 Bad Gateway" error in the browser. To see the details of the error preventing Gunicorn from starting, you will need to run
sudo journalctl -u gunicorn.service -f, which should display the traceback. Gunicorn is set up to automatically try restarting every 30 seconds, so after fixing the error you shouldn't need to do anything except wait (up to) 30 seconds for it to restart. However, you can always restart it immediately by running
sudo systemctl restart gunicorn.socket.
You can open an interactive IPython shell inside the app environment by running
pshell development.ini. This will include a
request object that you can use to make database changes via
request.db_session, just as inside app code. Note that any database changes will need to be manually committed by calling
Adding new Python packages/libraries
If you need to add a new Python dependency, you need to update the
requirements*.txt files that track which dependencies the app installs during the VM initialization. There are two of these files, and they function differently:
requirements-to-freeze.txt is a list of all the packages that need to be installed to run Tildes, where
requirements.txt is a snapshot of the individual packages (including all dependencies) and the versions that are currently being used.
To install a new package, you should follow this process:
- Add the package's name to
requirements-to-freeze.txtin the correct (alphabetical) location, and add it at the bottom of
requirements.txtwithout a version number specified.
- From your machine, run
vagrant provision. This will run the Salt states, which includes installing packages from
requirements.txt, so it will install the new one and any dependencies.
- From inside the VM, run
pip freeze | grep -v tildes > requirements.txt. This will overwrite
requirements.txtwith a list of all packages installed and their versions, so there will be lines added for any new packages that were installed.
- Make sure to include both
requirements-to-freeze.txtin your commit.
HTML templates and CSS
HTML templates are written in the Jinja2 templating language.
CSS is written with Sass (SCSS syntax), and uses Spectre.css as a base. Changes to the SCSS files (in the
tildes/scss/ directory) will automatically be detected and compiled for you by a service running inside the Vagrant VM. If your changes are failing to compile due to an error, you can run
sudo journalctl -u boussole.service -f to view that service's logs and see the error.
The SCSS is organized in a manner similar to the SMACSS style. One specific thing to note is that the
_themes.scss file contains style definitions that differ between the site themes. All theme-specific CSS should be kept exclusively in that file.
data- prefix is required on attributes, so the attributes will be similar to
data-ic-post-to instead of just
When a new endpoint is needed for Intercooler, it should be grouped in the "web API", which are all located inside
/api/web/. These endpoints are intended solely for "internal" usage by the site and can be changed as needed for any functionality. For Intercooler's purposes, they should generally return HTML fragments that are intended to be swapped into the page, but can also do things like use Intercooler response headers to trigger actions.
Intercooler includes jQuery 3 as a dependency, so jQuery is available as well.
static/js/behaviors/ and must be attached to a "component" on the site by using an HTML data attribute with the
All the files in
sudo systemctl restart webassets.service.
The primary database being used is PostgreSQL 12.
You can access the command-line interface to the database by running
psql -U tildes tildes. This can be useful to check exactly how things are stored or make sure that changes are applying correctly, but modifying data directly may cause inconsistencies that are usually managed by the app code, so it's often not the right approach.
Interacting with the database in the app
Database interactions inside the app are through the SQLAlchemy library. These interactions should almost always go through the per-request SQLAlchemy session available as
Each request is wrapped in a transaction by default, and it is not usually necessary to commit or rollback changes manually. The pyramid_tm library automatically handles most transaction tasks, including committing the transaction at the end of each request (or rolling it back if an error occurred).
If you really do need to manually control the transaction, you can call functions on the transaction manager directly such as
request.tm.commit(). If you need to execute a rollback, it is almost always correct to use a savepoint and roll back to it, instead of rolling back the entire transaction (which could unexpectedly interfere with other database operations).
Making changes to the database tables
If you need to make any changes to the tables or columns (including adding new ones), you must create a revision using the Alembic migration tool which will allow all other instances of the site to easily apply the same changes to their database.
The general process for creating an Alembic migration is:
- Make the changes to the table definitions inside the app code (not by altering the database directly).
- If you are adding a new table, edit
tildes/tildes/database_models.pyand import the new model class along with the others already included there. Skip this step if you're only modifying existing tables.
alembic revision --autogenerate -m "Short description of the changes". You will see some output including the name of a new file that was generated inside
Open that new file inside your editor and review/edit all the code inside the
downgrade()functions. Ensure that the code inside
upgrade()includes all steps necessary to make the changes you need. Alembic isn't able to detect everything, so there may be some omissions. Similarly, ensure that
downgrade()contains all steps to reverse those changes. Delete the automatic comments inside both these functions.
Note: the Alembic version file should never import any constants, read text in from files, etc. It should have all that sort of information hardcoded so that the effects of the migration aren't modified if those external resources change in the future.
alembic upgrade headto apply your upgrade and make sure there are no errors. If the upgrade applies successfully, you should also connect to the database with
psql -U tildes tildesand verify that all your changes were applied correctly.
- If the upgrade looks correct, run
alembic downgrade -1to apply your downgrade and make sure there are no errors. As above, you should connect to the database to verify that all your changes were removed and the database is back to the state it was previously in.
- If you run into any errors or missing changes during upgrade/downgrade, you may need to edit your script more and test more downgrades/upgrades until everything is correct. Once you're finished fixing any issues, run one final
alembic upgrade headto make sure everything has been applied.
- Make sure to include your Alembic file inside the commit that requires these database changes.
Most data validation and sanitization is being performed by Marshmallow schemas, which are located inside the
schemas/ directory. In general, any data that's intended to go into the database should be first filtered through a schema to ensure it fits the restrictions or intended format. This should mostly be handled automatically, since SQLAlchemy is set up to do automatic validation using the associated schema for a model.
These Marshmallow schemas are also used by the webargs library, which takes incoming data from various sources in HTTP requests (POST/form data, url, query string, etc.) and passes it through the schemas. This allows validation/sanitization and some other niceties like automatic 422 responses when the data is invalid.
Instrumentation and metrics
Prometheus is being used for application and system metrics. Your development version's metrics are accessible and queryable at http://localhost:9090.
You can add new metrics in the application using the Prometheus Python client, but make sure that you understand the different metric types available and use an appropriate one, and try to include labels so that the metrics can be filtered in useful ways if that's applicable.
Running checks on your code
Once you start making changes to the code that you want to contribute to the project, you should ensure that the following commands all do not return errors (any merge requests that fail these checks will not be accepted).
pytest- Runs the full set of tests.
mypy .- Runs the mypy static type checks.
black --check .- Runs the Black code-formatter checks (issues don't need to be fixed manually, just run
black .to reformat your code).
prospector- Runs the code style checks.
If you use the included Git hooks, then the tests, mypy, and Black will automatically be run whenever you commit (and prevent you from committing if they fail), and all checks will be run before you can push your code to a remote repo.
Tests are written using the pytest framework and are located inside the
tests/ directory. pytest will only detect new tests if they are inside a file whose name starts with
test_, and with a function name also starting with
test_ (other files/functions can be used for setup, fixtures, and so on).
Tests are able to (and should) use all database functions that the app itself can use. They operate inside their own PostgreSQL database, which uses nested transactions and transactional DDL to create tables that exist only inside the overall testing session's transaction (which is never committed). This ensures that no database changes persist between sessions even if the tests crash, and each testing session is always working with a clean database.
Tests also have access to their own temporary Redis instance that can be used when testing all functions involving Redis, such as rate-limiting.
The pytest-mock library is available for mocking/patching, as well as the freezegun library for tests that need control of time/date.
Functional testing can be done via the webtest library, which allows "higher-level" testing by sending requests to the app's URLs, examining responses, and so on. All tests using webtest should be kept inside the
webtests/ subdirectory, and can be excluded from a test run by running
pytest -k "not webtests".
Type-checking with mypy
All function definitions must include type annotations that define the types of each argument and the return value. These annotations have no effect at runtime, but mypy is able to statically analyze the code using them to detect potential errors when functions are inadvertently called with arguments of the wrong type.
The type annotations are, generally, very straightforward and you can probably easily understand how to write them by just looking at existing functions, but the following pages in the mypy docs might be useful if you run into issues or need more information:
You can propose changes to this page by editing the copy of it available in the wiki for the ~tildes.official group on Tildes itself.