sixmonth-redact / SOURCES.md
joehachem's picture
update sources
50dffa6 verified
|
Raw History Blame Contribute Delete
3.36 kB

Sources

The public word lists and registries the model's word knowledge was derived from, with their licences.

Source What it gave the model Licence
US Census Bureau, 2010 surnames, https://www2.census.gov/topics/genealogy/2010surnames/names.zip Surnames with frequencies Public domain
US Social Security Administration, national baby names 1880 to 2017, https://www.ssa.gov/oact/babynames/names.zip Given names with frequencies Public domain
INSEE, Fichier des prénoms 1900 to 2021, https://www.insee.fr/fr/statistiques/2540004 French given names Licence Ouverte (Etalab)
Wikidata given-name items, https://query.wikidata.org Given names CC0
Wikidata people of Lebanon, Syria, Palestine, Jordan, Armenia, Iraq and Egypt (English and French labels), https://query.wikidata.org Levantine name tokens CC0
GeoNames, cities1000, LB and admin1 dumps, https://download.geonames.org/export/dump/ Place names, Lebanese villages and quarters, states and provinces CC BY 4.0
USPS Publication 28, appendices C1 and C2, https://pe.usps.com/text/pub28/ Street suffixes and secondary unit designators Public domain
Wikipedia, "Street suffix" (UK and Australia sections), https://en.wikipedia.org/wiki/Street_suffix Street suffixes CC BY-SA 4.0
US FDA National Drug Code directory, https://www.accessdata.fda.gov/cder/ndctext.zip Drug names Public domain
US FDA 510(k) clearances, https://www.accessdata.fda.gov/premarket/ftparea/pmn96cur.zip Medical and dental device names and makers (contact person column not used) Public domain
French public drug database (BDPM), https://base-donnees-publique.medicaments.gouv.fr French drug names Licence Ouverte (Etalab)
NLM MeSH 2025 descriptors, https://nlmpubs.nlm.nih.gov/projects/mesh/2025/ Medical terms NLM terms (free use with attribution)
glutanimate/wordlist-medicalterms-en (from OpenMedSpel and Raj&Co Medical Spell Checker), https://github.com/glutanimate/wordlist-medicalterms-en Medical terms GPL-3.0
Wiktionary, dentistry, teeth and medicine categories (en, fr, es), https://www.wiktionary.org Dental and medical terms CC BY-SA
Peter Norvig, most frequent English words from the Google Web Trillion Word Corpus, https://norvig.com/ngrams/count_1w.txt English word frequencies MIT
SCOWL 2020.12.07 English word lists, Kevin Atkinson, http://wordlist.aspell.net English words by size level SCOWL licence (permissive)
Webster's Second International word list (/usr/share/dict/words) English words Public domain
Lexique 3.83, http://www.lexique.org French word forms and frequencies CC BY-SA 4.0
OpenSubtitles 2018 word frequencies (hermitdave FrequencyWords), es, fr, de, it, pt, nl, da, sv, https://github.com/hermitdave/FrequencyWords Common words in eight languages CC BY-SA 4.0
Sixmonth Clinical, dental and calendar words that are never names Released with this model (GPL-3.0)

Because the weights embed GPL-3.0 material (the medical terms list) and CC BY-SA 4.0 material (Lexique, OpenSubtitles frequency lists, Wikipedia, Wiktionary), the model is released under GPL-3.0; CC BY-SA 4.0 material may be adapted under GPLv3. Attribution for CC BY sources (GeoNames) and Licence Ouverte sources (INSEE, BDPM) is given above.