Important information for pyfunceble >= 4.0.0

When you upgrade PyFunceble from <4.x to any version 4.x there will be:

  • a SQL conversion if you use the --database-type mysql or mariadb.

  • a JSON to CSV conversion if one of those files is found in your filesystem:

    • inactive_db.json

    • whois_db.json

  • The conversion of both SQL and json to CSV will take a “bit” of time as, it is done in a single process mode, to avoid any hick-ups instead of loading the entire file into memory. Loading the entire *.json file into memory can have severe consequences depending on the size of the source file.

    Note

    Once the job is done the json2csv shouldn’t appear again for this or later 4.x releases.

  • A workaround for waiting on the rather slow json2csv, you can delete

    • inactive_db.json

    • whois_db.json (Very Very Very bad idea in the long run…)

    However you will probably not benefit for by deleting the whois_db.json as this is the definitive slowest lookup process in the test flow, do to the limitation in available API call you can do to WHOIS servers before getting banned. Therefore we CAN NOT recommend deleting this file, rather than waiting for the conversion to finish.

  • The output directory structure have been altered to work with the ability to test more than one source at the time. Prior to version 4.0.0.ax the output hierarchy looked like output/domains/ACTIVE/list. In Pyfunceble version >=4.x this have been altered to include the source name and append to the folder structure. From this version it will therefor looks like output/{{ input_source_name }}/domains/ACTIVE list.

Note

As consequence of the time consuming conversion, we will advise you to run a simple pyfunceble command like:

pyfunceble -d mypdns.org

pyfunceble --database-type mariadb -d mypdns.org

How long time does it take

A few numbers to help you schedule your upgrade process.

We have tested the SQL conversion with the following specifications

  • 2 Xeon CPU x86_64 (8 cores) 2 GHz

  • 48 GB ram.

  • 1 SSD Kingston KC-600

  • Mariadb 10.5, default config

  • Non dedicated

The database contains
  • Roughly 265.000 records in the test tables

  • Approximately 1.000.000 records within the pyfunceble_whois table.

This process toke about 10 hours to complete.