From f82ecd6fc79cc43625475e09b763cfbbfa5d9d8e Mon Sep 17 00:00:00 2001 From: Vincent Hellendoorn <1426353+VHellendoorn@users.noreply.github.com> Date: Wed, 9 Mar 2022 12:55:58 -0500 Subject: [PATCH] Update data collection details --- Data/README.md | 28 +++++++++++++++++++++++ {Mining => Data}/clone_repo.sh | 0 {Mining => Data}/code-merges.txt | 0 {Mining => Data}/code-vocab.json | 0 {Mining => Data}/collect_data.sh | 0 {Mining => Data}/deduplicate.py | 0 {Mining => Data}/extract_code.py | 0 {Mining => Data}/gh_crawler.py | 0 {Mining => Data}/requirements.txt | 0 {Mining => Data}/yield_from_code_files.py | 0 Mining/README.md | 16 ------------- 11 files changed, 28 insertions(+), 16 deletions(-) create mode 100644 Data/README.md rename {Mining => Data}/clone_repo.sh (100%) rename {Mining => Data}/code-merges.txt (100%) rename {Mining => Data}/code-vocab.json (100%) rename {Mining => Data}/collect_data.sh (100%) rename {Mining => Data}/deduplicate.py (100%) rename {Mining => Data}/extract_code.py (100%) rename {Mining => Data}/gh_crawler.py (100%) rename {Mining => Data}/requirements.txt (100%) rename {Mining => Data}/yield_from_code_files.py (100%) delete mode 100644 Mining/README.md diff --git a/Data/README.md b/Data/README.md new file mode 100644 index 0000000..27fa151 --- /dev/null +++ b/Data/README.md @@ -0,0 +1,28 @@ +# Code Data Collection +This directory contains scripts to [mine](#mining) a dataset of code similar to the one used to train [PolyCoder](https://arxiv.org/pdf/2202.13169.pdf), as well as details of that dataset. Note that because of the nature of the GH API, the exact results of each query will be different, so this will not precisely replicate the training data + +## Mining +Update `gh_crawler.py` by adding your GH API token (line 6). Then, run `collect_data.sh`, which invokes the GitHub API crawler (`gh_crawler.py`), followed by a repo cloning script (`clone_repo.sh`, in parallel), which uses `extract_code.py` to extract all source code files in the corresponding language (and filter very long/short files), and finally `deduplicate.py` to remove duplicate files. + +Once this is completed, you can use [gpt-neox](https://github.com/EleutherAI/gpt-neox)'s `preprocess_data.py` (currently in `tools/`) to tokenize this dataset for the model, using a either the pretrained code vocabularies by providing the `code-vocab.json` and `code-merges.txt` files, or producing a new one. + +At the time of this writing*, the following command processes the entire `Code/` directory to a new directory named `Preprocessed/` using the pretrained vocabularies across 16 parallel workers (assuming that `gpt-neox` is checked out in the current directory): +``` +mkdir Preprocessed +sudo python3 gpt-neox/tools/preprocess_data.py --input Code --tokenizer-type GPT2BPETokenizer --vocab-file code-vocab.json --merge-file code-merges.txt --output-prefix Preprocessed/code --workers 16 +``` +And that's it! Just modify the `local_setup.yml` config in the gpt-neox toolkit to point it to the new vocab & merges file and data directory and it should be able to train. + +*I did have to modify the `yield_from_files` function to recursively yield all (shuffled) files from a directory; the default version uses `lm_dataformat`, which balks at code file extensions. The updated function can be found in `yield_from_code_files.py`. + +## PolyCoder Data +The approach above was used to collect 249GB of multi-lingual training data to train [PolyCoder](https://arxiv.org/pdf/2202.13169.pdf) -- see the paper and top-level directory for details. Because of the ever-changing nature of repos on GitHub, running the above won't get you back the exact data, which is quite fine for most purposes (our training run didn't even use all of it), but it's naturally useful to know what data we used. We therefore release a list of all files used for training and their SHA-256 hash in [this file](https://zenodo.org/record/6341643/files/index.zip) (warning: zipped, still large), formatted as `{language}__{organization}__{project}__{full__file__path}\tSHA` (using double underscores instead of slashes in the file path). + +To check whether a file was used during training, I strongly encourage considering not just its path but also its hashed contents. Files are often duplicated verbatim across and within projects. The following Python code was used to create the hash values in the file above, which allows fast deduplication of a new file against the set of all hashes used in our training data: + +```python +import hashlib +with open(file_path, 'rb') as f: + bytes = f.read() + hash = hashlib.sha256(bytes).hexdigest(); +``` diff --git a/Mining/clone_repo.sh b/Data/clone_repo.sh similarity index 100% rename from Mining/clone_repo.sh rename to Data/clone_repo.sh diff --git a/Mining/code-merges.txt b/Data/code-merges.txt similarity index 100% rename from Mining/code-merges.txt rename to Data/code-merges.txt diff --git a/Mining/code-vocab.json b/Data/code-vocab.json similarity index 100% rename from Mining/code-vocab.json rename to Data/code-vocab.json diff --git a/Mining/collect_data.sh b/Data/collect_data.sh similarity index 100% rename from Mining/collect_data.sh rename to Data/collect_data.sh diff --git a/Mining/deduplicate.py b/Data/deduplicate.py similarity index 100% rename from Mining/deduplicate.py rename to Data/deduplicate.py diff --git a/Mining/extract_code.py b/Data/extract_code.py similarity index 100% rename from Mining/extract_code.py rename to Data/extract_code.py diff --git a/Mining/gh_crawler.py b/Data/gh_crawler.py similarity index 100% rename from Mining/gh_crawler.py rename to Data/gh_crawler.py diff --git a/Mining/requirements.txt b/Data/requirements.txt similarity index 100% rename from Mining/requirements.txt rename to Data/requirements.txt diff --git a/Mining/yield_from_code_files.py b/Data/yield_from_code_files.py similarity index 100% rename from Mining/yield_from_code_files.py rename to Data/yield_from_code_files.py diff --git a/Mining/README.md b/Mining/README.md deleted file mode 100644 index 3933d5a..0000000 --- a/Mining/README.md +++ /dev/null @@ -1,16 +0,0 @@ -## Purpose -Scripts to construct a dataset of code in a similar way to the ones used to train the released models. Note that because of the nature of the GH API, the exact results of each query will be different, so this will not precisely replicate the training data. - -## Usage -Update `gh_crawler.py` by adding your GH API token (line 6). Then, run `collect_data.sh`, which invokes the GitHub API crawler (`gh_crawler.py`), followed by a repo cloning script (`clone_repo.sh`, in parallel), which uses `extract_code.py` to extract all source code files in the corresponding language (and filter very long/short files), and finally `deduplicate.py` to remove duplicate files. - -Once this is completed, you can use [gpt-neox](https://github.com/EleutherAI/gpt-neox)'s `preprocess_data.py` (currently in `tools/`) to tokenize this dataset for the model, using a either the pretrained code vocabularies by providing the `code-vocab.json` and `code-merges.txt` files, or producing a new one. - -At the time of this writing*, the following command processes the entire `Code/` directory to a new directory named `Preprocessed/` using the pretrained vocabularies across 16 parallel workers (assuming that `gpt-neox` is checked out in the current directory): -``` -mkdir Preprocessed -sudo python3 gpt-neox/tools/preprocess_data.py --input Code --tokenizer-type GPT2BPETokenizer --vocab-file code-vocab.json --merge-file code-merges.txt --output-prefix Preprocessed/code --workers 16 -``` -And that's it! Just modify the `local_setup.yml` config in the gpt-neox toolkit to point it to the new vocab & merges file and data directory and it should be able to train. - -*I did have to modify the `yield_from_files` function to recursively yield all (shuffled) files from a directory; the default version uses `lm_dataformat`, which balks at code file extensions. The updated function can be found in `yield_from_code_files.py`.