Add Combining-Blocklists

Combining-Blocklists.md

@@ -0,0 +1,59 @@
+# Combining blocklists
+
+`dnscrypt-proxy` includes a tool to build block lists from local and remote lists in common formats.
+
+That tool:
+
+- Converts 3rd party lists in common formats (such as HOSTS-file lists) into entries suitable for `dnscrypt-proxy`
+- Removes junk/typos/entries that don't represent host names
+- Removes duplicates and merges overlapping entries in order to make lists drastically smaller and faster
+- Measures how 3rd party lists overlap
+- Can prevent time-restricted entries from being included
+
+Named `generate-domains-blocklist.py`, the tool requires a Python interpreter. Python is available for virtually all operating systems. Any Python 3.x version should work. The script may also work with Python 2.x but this is not a supported configuration since Python 2.x has reached end of life.
+
+The script is included in the `dnscrypt-proxy` source code in the `utils` directory. It can also be downloaded directly here:
+
+- [`generate-domains-blocklist.py`](https://raw.githubusercontent.com/DNSCrypt/dnscrypt-proxy/master/utils/generate-domains-blocklists/generate-domains-blocklist.py)
+- [example configuration files](https://github.com/DNSCrypt/dnscrypt-proxy/blob/master/utils/generate-domains-blocklists/generate-domains-blocklist.py)
+
+It can be run with the `python3 generate-domains-blocklist.py` command, followed by relevant parameters.
+
+```text
+usage: generate-domains-blocklist.py [-h] [-c CONFIG]
+                                     [-a ALLOWLIST] [-r TIME_RESTRICTED] [-i]
+                                     [-o OUTPUT_FILE] [-t TIMEOUT]
+
+Create a unified blocklist from a set of local and remote files
+
+optional arguments:
+  -h, --help            show this help message and exit
+  -c CONFIG, --config CONFIG
+                        file containing blocklist sources
+  -a ALLOWLIST, --allowlist ALLOWLIST
+                        file containing a set of names to exclude from the
+                        blocklist
+  -r TIME_RESTRICTED, --time-restricted TIME_RESTRICTED
+                        file containing a set of names to be time restricted
+  -i, --ignore-retrieval-failure
+                        generate list even if some urls couldn't be retrieved
+  -o OUTPUT_FILE, --output-file OUTPUT_FILE
+                        save generated blocklist to a text file with the
+                        provided file name
+```
+
+The most common usage pattern is simply:
+
+```sh
+python3 generate-domains-blocklist.py -o blocklist.txt
+```
+
+which will load its configuration from a [`domains-blocklist.conf`](https://raw.githubusercontent.com/DNSCrypt/dnscrypt-proxy/master/utils/generate-domains-blocklists/domains-blocklist.conf) configuration file.
+
+Edit that configuration file to include a combination of local (starting with `file:`) files and remote (starting with `https://`) lists.
+
+The example configuration already includes some popular sources, that you can comment or uncomment according to your needs.
+
+Be aware that all 3rd party lists include false positives and obsolete entries. Using a few high-quality, well-maintained lists is always preferable to trying to create the biggest possible list.
+
+If an external source is unreachable or returns temporary errors, an existing output list will not be overwritten, so that the previous version can still be used. That behavior can be changed with the `--ignore-retrieval-failure` options.