G5 Organisation

Data transformations

• Auxiliary data
  Not directly used by the program - Useful as a reference, to check if g5 has not introduced errors.
  Contain copies of original documents, like scans or files.
  Versioned in another repository: github.com/tig12/g5-aux
• Raw data
  Input of g5.
  Contain usable version of auxiliary data, like files converted to UTF8
  or scans transformed to lists through OCR and human corrections.
  Conversion between auxiliary to raw data is done by humans, not by program.
  Versioned with g5 code, in data/raw
• Intermediate data
  • Temporary data
    Raw data are sanitized, corrected, standardized and stored in temporary CSV files.
    These CSV files are stored by default in directory data/tmp.
    Not versioned
  • Human corrections and additions
    The conversion between raw and tmp uses human corrections stored in YAML files.
    Versioned with g5, in data/db.
• G5 database
  data/tmp are then loaded and merged in a postgresql database.
• Exports

gauquelin5
    └── data
        ├── auxiliary   # can be absent or removed
        ├── raw         # location imposed (because versioned with g5)
        ├── tmp         # location set in config.yml
        ├── db          # location imposed (because versioned with g5)
        └── output      # location set in config.yml