Bookworm

Creating a Bookworm with the required files.

The central creation for a Bookworm is handled by a python script installed with the pip module.

If you have already successfully installed Bookworm, this should be relatively easy. Navigate in the command line to the folder where you have placed the input.txt, jsoncatalog.txt, and field_descriptions.txt files.

Initialize the folder

First, type

bookworm init

to set up your folder. It will prompt you for the name of the bookworm name, username, and password to build. The username and password are gleaned from your local settings: they'll only need to be changed for advanced users. (The username and password are not your personal username and password. They are, instead, a default username and password that the web site will use. This is for an extra layer of security; all Bookworm queries are performed by a user with no privileges to change the database. In general, the right place to store this is at your system-level my.cnf file. (It may be located somewhere like /etc/my.cnf on OS X, or /etc/mysql/my.cnf on Ubuntu.) These must be the same as the username and password that Apache will use when executing a CGI script.)

The bookworm name is a string to identify the bookworm internally for things like API queries. By default, it pulls the name of the folder that you're in. I recommend that it be a single word in lowercase; it can't have any spaces. Note that any existing mysql databases with the name you choose will be deleted. So if you have a Wordpress install, for instance, don't name your bookworm "wordpress" or "wp!"

Build the bookworm

Second, type

bookworm build all

Bookworm will run through all the data and build the Bookworm in place.

Test it out

If you want to see if your bookworm has worked, you can try testing it out.

bookworm serve

will launch a test webserver through python accessible locally. If you have a time variable, read the console to see the URL for the test. You can also explore it through the D3 plugin; but note that the default queries may not immediately display anything there.

Install the linechart GUI to your global web directory.

For a more permanent webserver, you can type

bookworm build linechartGUI

which will install the linechart GUI to /var/www. If that's not where you store your web file, you may need to move the created folder around.

Understanding the Workflow.

All jobs are dispatched through python, and some further ones are handled through a Makefile bundled in the script--if you can read through the dependency chain to see how it's put together, you'll understand all the elements.

For reference, the general workflow of the Makefile is the following:

  1. Build the directory structure in .bookworm/texts/.
  2. Derive .bookworm/metadata/field_descriptions_derived.json from files/metadata/field_descriptions.json.
  3. Derive .bookworm/metadata/jsoncatalog_derived.txt from files/metadata/jsoncatalog.txt.
  4. Create metadata catalog files in .bookworm/metadata/.
  5. Create, if not pre-defined, a file at .bookworm/wordlist/wordlist.txt that defines the tokens that will be counted (the million most common tokens).
  6. Encode unigrams and bigrams from the binaries into .bookworm/encoded.
  7. Load wordcounts into MySQL database.
  8. Load metadata into MySQL database.
  9. Create temporary MySQL table and .json file that will be used by the web app.

At any point, you can backtrack part of the way by clearing out files from .bookworm/targets.