skywatch.blue
A tool for parsing traffic on the jetstream and applying a moderation workstream based on regexp based rules
Merge pull request #21 from skywatch-bsky/16
Closes #16
authored by
Scarnecchia
and committed by
GitHub
9d252306
f3dbfc14
+91
-29
+7
-7
.env.example
+7
-7
.env.example
···
1
-
MOD_DID=did:plc:xxx
2
-
OZONE_URL=ozone.example.xyz
3
-
OZONE_PDS=mushroom.us-region.host.bsky.network
4
-
BSKY_HANDLE=mod.example.xyz
5
-
BSKY_PASSWORD=xxxx
1
+
MOD_DID=
2
+
OZONE_URL=
3
+
OZONE_PDS=
4
+
BSKY_HANDLE=
5
+
BSKY_PASSWORD=
6
6
HOST=127.0.0.1
7
-
PORT=4000 // Not currently used
7
+
PORT=4000
8
8
METRICS_PORT=4001
9
-
FIREHOSE_URL=wss://jetstream1.us-east.bsky.network/subscribe
9
+
FIREHOSE_URL=
10
10
CURSOR_UPDATE_INTERVAL=10000
11
11
LABEL_LIMIT=2900 * 1000
12
12
LABEL_LIMIT_WAIT=300 * 1000
+83
-21
README.md
+83
-21
README.md
···
1
-
# skywatch-tools
1
+
# Skywatch Automod
2
2
3
-
This is a rewrite of the original skywatch-tools project in TypeScript. The original project was written in Bash. The purpose of this project is to automate the moderation by the Bluesky independent labeler skywatch.blue
3
+
This project provides tools for automating moderation of the Bluesky social network. It listens to the Bluesky firehose stream, analyzes various types of content against user-defined rules, and performs moderation actions such as applying labels, reporting content, or leaving comments.
4
4
5
-
## Installation and Setup
5
+
## Features
6
6
7
-
To install dependencies:
7
+
- **Real-time Moderation:** Monitors the Bluesky firehose in real-time.
8
+
- **Content-Aware Analysis:** Analyzes posts, user profiles (display names, descriptions), and handles
9
+
- **Flexible Rule Engine:** Uses regular expressions for defining moderation checks.
10
+
- **Variety of Actions:** Can apply labels, create reports (for posts or accounts), and post comments on accounts.
11
+
- **Configurable:** Highly configurable through environment variables and a central constants file.
12
+
- **Allowlisting:** Supports allowlisting for DIDs and text patterns to reduce false positives.
13
+
- **URL Unshortening:** Automatically resolves shortened URLs in posts before checking them.
14
+
- **Monitoring:** Exposes a Prometheus metrics endpoint to monitor its activity. (untested)
15
+
- **Resilient:** Persists the firehose cursor to gracefully handle restarts without missing events.
16
+
17
+
## How It Works
18
+
19
+
The application connects to the Bluesky firehose and subscribes to a set of collections (e.g., posts, profiles). When a new event is received, it is passed through a series of checks defined in `src/constants.ts`. These checks are categorized by content type:
20
+
21
+
- `POST_CHECKS`: For post content and links.
22
+
- `HANDLE_CHECKS`: For user handles.
23
+
- `PROFILE_CHECKS`: For user display names and descriptions.
24
+
25
+
If the content matches a check's criteria (and is not excluded by an allowlist), a corresponding moderation action is triggered. These actions (labeling, reporting, etc.) are performed using the Bluesky API.
26
+
27
+
## Getting Started
28
+
29
+
### Prerequisites
30
+
31
+
- Node.js (v20 or higher recommended)
32
+
- `bun` package manager
33
+
- A Bluesky account for the bot.
34
+
- A Bluesky labeler account
35
+
36
+
### 1. Installation
37
+
38
+
Clone the repository and install the dependencies:
8
39
9
40
```bash
10
-
bun i
41
+
git clone <repository-url>
42
+
cd skywatch-automod-public
43
+
bun install
11
44
```
12
45
13
-
Modify .env.example with your own values and rename it to .env
46
+
### Configuration
47
+
48
+
There are two main configuration files you need to set up:
49
+
50
+
- **Checks (`src/constants.ts`):**
51
+
This file defines the rules for your automod. You need to create it by copying the example file:
52
+
53
+
```bash
54
+
cp src/constants.ts.example src/constants.ts
55
+
```
56
+
57
+
Then, edit `src/constants.ts` to define your own checks. For detailed instructions on how to create checks, please see [developing_checks.md](./src/developing_checks.md).
58
+
59
+
- **Environment Variables (`.env`):**
60
+
This file contains credentials and other runtime configuration. You will need to create a `.env` file and populate it with your specific values. You can use `.env.example` as a reference if it exists in the
61
+
62
+
### 3. Running the Application
63
+
64
+
Once configured, you can start the automod with:
14
65
15
66
```bash
16
67
bun run start
17
68
```
18
69
19
-
To run in docker:
70
+
### 4. Running with Docker
71
+
72
+
You can also build and run the application as a Docker container.
20
73
21
74
```bash
22
-
docker build -pull -t skywatch-automod .
23
-
docker run -d -p 4101:4101 skywatch-automod
75
+
docker compose up --build
24
76
```
25
77
26
-
## Brief overview
78
+
To run it in the background, add the `-d` flag:
27
79
28
-
Currently this tooling does one thing. It monitors the bluesky firehose and analyzes content for phrases which fit Skywatch's criteria for moderation. If the criteria is met, it can automatically label the content with the appropriate label.
29
-
30
-
In certain cases, where regexp will create too many false positives, it will flag content as a report against related to the account, so that it can be reviewed later.
80
+
```bash
81
+
docker compose up --build -d
82
+
```
31
83
32
-
For information on how to set-up your own checks, please see the [developing_checks.md](./src/developing_checks.md) file.
84
+
Make sure your `.env` file is present when building the Docker image, as it will be copied into the container. We recommend creating an empty `cursor.txt` file in the application root directory as well.
33
85
34
-
_TODO_:
86
+
#### Configuration Variables
35
87
36
-
- [ ] Remove unused types
37
-
- [ ] Update the types needed to be more specific to the checks rather than bluesky content types
38
-
- [ ] Consider how to write directly to OzonePDS database rather than using the API. May require running the same instance as Ozone to allow for direct database access.
39
-
- [ ] Add compose.yaml for easy deployment
40
-
- [ ] Make the metrics server work (or remove it)
88
+
The following environment variables are used for configuration:
41
89
42
-
Create a seperate program to watch OZONE_PDS firehose labels, and update the lists as needed. This will remove dependency on broken ruby tools created by aegis.
90
+
| Variable | Description | Default |
91
+
| ------------------------ | ---------------------------------------------------------------- | ----------------------------------------- |
92
+
| `DID` | The DID of your moderation service for atproto-proxy headers. | `""` |
93
+
| `OZONE_URL` | The URL of the Ozone service. | `""` |
94
+
| `OZONE_PDS` | The Public Downstream Service for Ozone. | `""` |
95
+
| `BSKY_HANDLE` | The handle (username) of the bot's Bluesky account. | `""` |
96
+
| `BSKY_PASSWORD` | The app password for the bot's Bluesky account. | `""` |
97
+
| `HOST` | The host on which the server runs. | `127.0.0.1` |
98
+
| `PORT` | The port for the main application (currently unused). | `4100` |
99
+
| `METRICS_PORT` | The port for the Prometheus metrics server. | `4101` |
100
+
| `FIREHOSE_URL` | The WebSocket URL for the Bluesky firehose. | `FIREHOSE_URL=wss://jetstream1.us-east.bsky.network/subscribe` |
101
+
| `CURSOR_UPDATE_INTERVAL` | How often to save the firehose cursor to disk (in milliseconds). | `60000` |
102
+
| `LABEL_LIMIT` | (Optional) API call limit for labeling. | `undefined` |
103
+
| `LABEL_LIMIT_WAIT` | (Optional) Wait time when label limit is hit. | `undefined` |
104
+
| `LOG_LEVEL` | The logging level. | `info` |
+1
-1
src/config.ts
+1
-1
src/config.ts
···
11
11
? Number(process.env.METRICS_PORT)
12
12
: 4101; // Left this intact from the code I adapted this from
13
13
export const FIREHOSE_URL =
14
-
process.env.FIREHOSE_URL ?? "wss://jetstream.atproto.tools/subscribe";
14
+
process.env.FIREHOSE_URL ?? "wss://jetstream1.us-east.bsky.network/subscribe";
15
15
export const WANTED_COLLECTION = [
16
16
"app.bsky.feed.post",
17
17
"app.bsky.actor.defs",