Graph Database Builder (graphdb_builder)

• Ontology Databases
  • Ontologies Parsers
    • efoParser.py
    • icdParser.py
    • oboParser.py
    • reflectParser.py
    • snomedParser.py
  • ontologies_controller.py
• Biomedical Databases
  • Biomedical Databases Parsers
    • cancerGenomeInterpreterParser.py
    • corumParser.py
    • disgenetParser.py
    • drugBankParser.py
    • drugGeneInteractionDBParser.py
    • exposomeParser.py
    • foodbParser.py
    • goaParser.py
    • gwasCatalogParser.py
    • hgncParser.py
    • hmdbParser.py
    • hpaParser.py
    • intactParser.py
    • jensenlabParser.py
    • mutationDsParser.py
    • oncokbParser.py
    • pathwayCommonsParser.py
    • pfamParser.py
    • pspParser.py
    • reactomeParser.py
    • refseqParser.py
    • siderParser.py
    • signorParser.py
    • smpdbParser.py
    • stringParser.py
    • textminingParser.py
    • uniprotParser.py
  • databases_controller.py
• Experimental Data
  • Experimental Data Parsers
    • clinicalParser.py
    • proteomicsParser.py
    • wesParser.py
  • experiments_controller.py
• User Creation
  • users_controller.py
• CKG Builder
  • create_user.py
  • importer.py
  • loader.py
  • builder.py

builder_utils.py

readDataset(uri)

readDataFromCSV(uri)

Read the data from csv file

readDataFromTXT(uri)

Read the data from tsv or txt file

readDataFromExcel(uri)

Read the data from Excel file

get_files_by_pattern(regex_path)

get_extra_pairs(directory, extra_file)

parse_contents(contents, filename)

Reads binary string files and returns a Pandas DataFrame.

export_contents(data, dataDir, filename)

Export Pandas DataFrame to file, with UTF-8 endocing.

write_relationships(relationships, header, outputfile)

Reads a set of relationships and saves them to a file.

Parameters

• relationships (set) – set of tuples with relationship data: source node, target node, relationship type, source and other attributes.

• header (list) – list of column names.

• outputfile (str) – path to file to be saved (including filename and extention).

write_entities(entities, header, outputfile)

Reads a set of entities and saves them to a file.

Parameters

• entities (set) – set of tuples with entities data: identifier, label, name and other attributes.

• header (list) – list of column names.

• outputfile (str) – path to file to be saved (including filename and extention).

get_config(config_name, data_type='databases')

Reads YAML configuration file and converts it into a Python dictionary.

Parameters

• config_name (str) – name of the configuration YAML file.

• data_type (str) – configuration type (‘databases’ or ‘ontologies’).

Returns

Dictionary.

Note

Use this function to obtain configuration for individual database/ontology parsers.

expand_cols(data, col, sep=';')

Expands the rows of a dataframe by splitting the specified column

Parameters

• data – dataframe to be expanded

• col (str) – column that contains string to be expanded (i.e. ‘P02788;E7EQB2;E7ER44;P02788-2;C9JCF5’)

• sep (str) – separator (i.e. ‘;’)

Returns

expanded pandas dataframe

setup_config(data_type='databases')

Reads YAML configuration file and converts it into a Python dictionary.

Parameters

data_type – configuration type (‘databases’, ‘ontologies’, ‘experiments’ or ‘builder’).

Returns

Dictionary.

Note

This function should be used to obtain the configuration for databases_controller.py, ontologies_controller.py, experiments_controller.py and builder.py.

get_full_path_directories()

Reads Builder YAML configuration file and returns the full path of all directories. :return: Dictionary.

list_ftp_directory(ftp_url, user='', password='')

Lists all files present in folder from FTP server.

Parameters

• ftp_url (str) – link to access ftp server.

• user (str) – username to access ftp server if required.

• password (str) – password to access ftp server if required.

Returns

List of files contained in ftp server folder provided with ftp_url.

setup_logging(path='log.config', key=None)

Setup logging configuration.

Parameters

• path (str) – path to file containing configuration for logging file.

• key (str) – name of the logger.

Returns

Logger with the specified name from ‘key’. If key is None, returns a logger which is the root logger of the hierarchy.

download_from_ftp(ftp_url, user, password, to, file_name)

download_PRIDE_data(pxd_id, file_name, to='.', user='', password='', date_field='publicationDate')

This function downloads a project file from the PRIDE repository

Parameters

• pxd_id (str) – PRIDE project identifier (id. PXD013599).

• file_name (str) – name of the file to dowload

• to (str) – local directory where the file should be downloaded

• user (str) – username to access biomedical database server if required.

• password (str) – password to access biomedical database server if required.

• date_field (str) – projects deposited in PRIDE are search based on date, either submissionData or publicationDate (default)

downloadDB(databaseURL, directory=None, file_name=None, user='', password='', avoid_wget=False)

This function downloads the raw files from a biomedical database server when a link is provided.

Parameters

• databaseURL (str) – link to access biomedical database server.

• directory (str or None) –

• file_name (str or None) – name of the file to dowload. If None, ‘databaseURL’ must contain filename after the last ‘/’.

• user (str) – username to access biomedical database server if required.

• password (str) – password to access biomedical database server if required.

searchPubmed(searchFields, sortby='relevance', num='10', resultsFormat='json')

Searches PubMed database for MeSH terms and other additional fields (‘searchFields’), sorts them by relevance and returns the top ‘num’.

Parameters

• searchFields (list) – list of search fields to query for.

• sortby (str) – parameter to use for sorting.

• num (str) – number of PubMed identifiers to return.

• resultsFormat (str) – format of the PubMed result.

Returns

Dictionary with total number of PubMed ids, and top ‘num’ ids.

is_number(s)

This function checks if given input is a float and returns True if so, and False if it is not.

Parameters

s – input

Returns

Boolean.

getMedlineAbstracts(idList)

This function accesses NCBI over the WWWW and returns Medline data as a handle object, which is parsed and converted to a Pandas DataFrame.

Parameters

idList (str or list) – single identifier or comma-delimited list of identifiers. All the identifiers must be from the database PubMed.

Returns

Pandas DataFrame with columns: ‘title’, ‘authors’, ‘journal’, ‘keywords’, ‘abstract’, ‘PMID’ and ‘url’.

remove_directory(directory)

listDirectoryFiles(directory)

Lists all files in a specified directory.

Parameters

directory (str) – path to folder.

Returns

List of file names.

listDirectoryFolders(directory)

Lists all directories in a specified directory.

Parameters

directory (str) – path to folder.

Returns

List of folder names.

listDirectoryFoldersNotEmpty(directory)

Lists all directories in a specified directory.

Parameters

directory (str) – path to folder.

Returns

List of folder names.

checkDirectory(directory)

Checks if given directory exists and if not, creates it.

Parameters

directory (str) – path to folder.

flatten(t)

Code from: https://gist.github.com/shaxbee/0ada767debf9eefbdb6e Acknowledgements: Zbigniew Mandziejewicz (shaxbee) Generator flattening the structure

>>> list(flatten([2, [2, (4, 5, [7], [2, [6, 2, 6, [6], 4]], 6)]]))
[2, 2, 4, 5, 7, 2, 6, 2, 6, 6, 4, 6]

pretty_print(data)

This function provides a capability to “pretty-print” arbitrary Python data structures in a forma that can be used as input to the interpreter. For more information visit https://docs.python.org/2/library/pprint.html.

Parameters

data – python object.

convertOBOtoNet(ontologyFile)

Takes an .obo file and returns a NetworkX graph representation of the ontology, that holds multiple edges between two nodes.

Parameters

ontologyFile (str) – path to ontology file.

Returns

NetworkX graph.

getCurrentTime()

Returns current date (Year-Month-Day) and time (Hour-Minute-Second).

Returns

Two strings: date and time.

convert_bytes(num)

This function will convert bytes to MB…. GB… etc.

Parameters

num – float, integer or pandas.Series.

copytree(src, dst, symlinks=False, ignore=None)

file_size(file_path)

This function returns the file size.

Parameters

file_path (str) – path to file.

Returns

Size in bytes of a plain file.

Return type

str

buildStats(count, otype, name, dataset, filename, updated_on=None)

Returns a tuple with all the information needed to build a stats file.

Parameters

• count (int) – number of entities/relationships.

• otype (str) – ‘entity’ or ‘relationsgips’.

• name (str) – entity/relationship label.

• dataset (str) – database/ontology.

• filename (str) – path to file where entities/relationships are stored.

Returns

Tuple with date, time, database name, file where entities/relationships are stored, file size, number of entities/relationships imported, type and label.

unrar(filepath, to)

Decompress RAR file :param str filepath: path to rar file :param str to: where to extract all files

compress_directory(folder_to_backup, dest_folder, file_name)

Compresses folder to .tar.gz to create data backup archive file.

Parameters

• folder_to_backup (str) – path to folder to compress and backup.

• dest_folder (str) – path where to save compressed folder.

• file_name (str) – name of the compressed file.

read_gzipped_file(filepath)

Opens an underlying process to access a gzip file through the creation of a new pipe to the child.

Parameters

filepath (str) – path to gzip file.

Returns

A bytes sequence that specifies the standard output.

parse_fasta(file_handler)

Using BioPython to read fasta file as SeqIO objects

Parameters

file_handler (file_handler) – opened fasta file

Return iterator records

iterator of sequence objects

batch_iterator(iterator, batch_size)

Returns lists of length batch_size.

This can be used on any iterator, for example to batch up SeqRecord objects from Bio.SeqIO.parse(…), or to batch Alignment objects from Bio.AlignIO.parse(…), or simply lines from a file handle.

This is a generator function, and it returns lists of the entries from the supplied iterator. Each list will have batch_size entries, although the final list may be shorter.

Parameters

• iterator (iterator) – batch to be extracted

• batch_size (integer) – size of the batch

Return list batch

list with the batch elements of size batch_size

source: https://biopython.org/wiki/Split_large_file

mapping.py

reset_mapping(entity)

Checks if mapping.tsv file exists and removes it.

Parameters

entity (str) – entity label as defined in databases_config.yml

mark_complete_mapping(entity)

Checks if mapping.tsv file exists and renames it to complete_mapping.tsv.

Parameters

entity (str) – entity label as defined in databases_config.yml

getMappingFromOntology(ontology, source=None)

Converts .tsv file with complete list of ontology identifiers and aliases, to dictionary with aliases as keys and ontology identifiers as values.

Parameters

• ontology (str) – ontology label as defined in ontologies_config.yml.

• source (str or None) – name of the source database for selecting aliases.

Returns

Dictionary of aliases (keys) and ontology identifiers (values).

getMappingFromDatabase(id_list, node, attribute_from='id', attribute_to='name')

getMappingForEntity(entity)

Converts .tsv file with complete list of entity identifiers and aliases, to dictionary with aliases as keys and entity identifiers as values.

Parameters

entity (str) – entity label as defined in databases_config.yml.

Returns

Dictionary of aliases (keys) and entity identifiers (value).

getMultipleMappingForEntity(entity)

Converts .tsv file with complete list of entity identifiers and aliases, to dictionary with aliases to other databases as keys and entity identifiers as values.

Parameters

entity (str) – entity label as defined in databases_config.yml.

Returns

Dictionary of aliases (keys) and set of unique entity identifiers (values).

get_STRING_mapping_url(db='STRING')

Get the url for downloading the mapping file from either STRING or STITCH

Parameters

db (str) – Which database to get the url from: STRING or STITCH

Returns

url where to download the mapping file

getSTRINGMapping(source='BLAST_UniProt_AC', download=True, db='STRING')

Parses database (db) and extracts relationships between identifiers to order databases (source).

Parameters

• url (str) – link to download database raw file.

• source (str) – name of the source database for selecting aliases.

• download (bool) – wether to download the file or not.

• db (str) – name of the database to be parsed.

Returns

Dictionary of database identifers (keys) and set of unique aliases to other databases (values).

buildMappingFromOBO(oboFile, ontology)

Parses and extracts ontology idnetifiers, names and synonyms from raw file, and writes all the information to a .tsv file. :param str oboFile: path to ontology raw file. :param str ontology: ontology database acronym as defined in ontologies_config.yml.

map_experiment_files(project_id, datasetPath, mapping)

map_experimental_data(data, mapping)

get_mapping_analytical_samples(project_id)