{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Real public exports: test run\n", "\n", "Bibliometric analysis generated by Pho (pho.chat) with pho_biblio 1.1. Run the cells in order:\n", "they install two packages, ask for your export files, re-run the full analysis and open the\n", "tables and figures. Every number comes from the code in this notebook; edit it freely.\n", "\n", "Files used: scopus_litstudy.csv, scopus_mk.csv, savedrecs.txt" ] }, { "cell_type": "code", "metadata": {}, "execution_count": null, "outputs": [], "source": [ "!pip -q install python-docx openpyxl" ] }, { "cell_type": "code", "metadata": {}, "execution_count": null, "outputs": [], "source": [ "# Upload the same exports (skip if they are already next to this notebook).\n", "import os\n", "FILES = [\"scopus_litstudy.csv\", \"scopus_mk.csv\", \"savedrecs.txt\"]\n", "missing = [f for f in FILES if not os.path.exists(f)]\n", "if missing:\n", " try:\n", " from google.colab import files as colab_files\n", " colab_files.upload()\n", " except ImportError:\n", " print(\"Put these files next to the notebook:\", missing)" ] }, { "cell_type": "code", "metadata": {}, "execution_count": null, "outputs": [], "source": [ "%%writefile pho_biblio.py\n", "\"\"\"pho_biblio: a bibliometric report from Scopus and Web of Science exports.\n", "\n", "Generated by Pho (pho.chat). Runs anywhere with Python 3.9+, numpy, pandas,\n", "matplotlib, python-docx and openpyxl (Google Colab has the first three; add\n", "the last two with: pip install python-docx openpyxl).\n", "\n", "Covers the main Biblioshiny analyses: main information, annual production and\n", "average citations per year, most relevant and most cited sources, authors,\n", "countries (SCP/MCP) and documents, source and author local impact (h, g, m),\n", "Bradford's law, Lotka's law, trend topics, thematic map, three-field plot,\n", "and co-authorship, keyword and country collaboration networks.\n", "\n", " import pho_biblio\n", " pho_biblio.report([\"scopus.csv\", \"savedrecs.txt\"], title=\"Urban heat research\")\n", "\n", "Reads, in any mix: Scopus CSV (preferred, it carries citations, keywords and\n", "affiliations), Web of Science \"Plain text\" and \"Tab delimited\" files, and RIS\n", "from either database.\n", "\n", "It counts. It never asks a model anything, never touches the network, and\n", "reports which fields each file was missing instead of guessing them.\n", "\"\"\"\n", "import csv\n", "import io\n", "import json\n", "import math\n", "import os\n", "import re\n", "import shutil\n", "import sys\n", "from collections import Counter, defaultdict\n", "from datetime import date\n", "\n", "import numpy as np\n", "\n", "VERSION = \"1.1\"\n", "TOP_N = 15\n", "NETWORK_NODES = 50\n", "MIN_EDGE = 2\n", "\n", "csv.field_size_limit(min(sys.maxsize, 2 ** 31 - 1))\n", "\n", "LABELS = {\n", " \"en\": {\n", " \"report\": \"Bibliometric report\",\n", " \"generated\": \"Generated by Pho (pho.chat) on %s with pho_biblio %s.\",\n", " \"data\": \"Data\",\n", " \"main\": \"Main information\",\n", " \"figures\": \"Figures\",\n", " \"methods\": \"Methods\",\n", " \"limits\": \"Limitations\",\n", " \"rerun\": \"Re-run this analysis\",\n", " \"file\": \"File\",\n", " \"database\": \"Database\",\n", " \"records\": \"Records\",\n", " \"metric\": \"Metric\",\n", " \"value\": \"Value\",\n", " \"coverage\": \"Field coverage after merging (share of records that carry the field)\",\n", " \"field\": \"Field\",\n", " \"production\": \"Annual production and citations\",\n", " \"sources\": \"Sources\",\n", " \"source_impact\": \"Source local impact (h, g and m index from the citations in these exports).\",\n", " \"bradford\": \"Bradford's law: %d of %d sources form the core zone (zone 1).\",\n", " \"authors\": \"Authors and countries\",\n", " \"author_impact\": \"Author local impact (h, g and m index from the citations in these exports).\",\n", " \"lotka\": \"Lotka's law on %d authors: fitted beta = %.3f, C = %.4f, R squared = %.3f (Lotka's law: beta = 2, C = 0.6079).\",\n", " \"documents\": \"Most cited documents\",\n", " \"conceptual\": \"Conceptual structure\",\n", " \"social\": \"Social structure\",\n", " \"rerun_text\": \"Open bibliometric_analysis.ipynb in Google Colab (File > Upload notebook), run the cells and upload the same exports when asked: it re-runs this analysis and opens every table and figure, ready to extend. Without Colab: pip install numpy pandas matplotlib python-docx openpyxl, then import pho_biblio; pho_biblio.report([\\\"scopus.csv\\\", \\\"savedrecs.txt\\\"]). The *_nodes and *_edges sheets of bibliometric_tables.xlsx import into VOSviewer (map + network files) and Gephi (edge table).\",\n", " },\n", " \"vi\": {\n", " \"report\": \"Báo cáo trắc lượng thư mục\",\n", " \"generated\": \"Tạo bởi Phở (pho.chat) ngày %s, pho_biblio %s.\",\n", " \"data\": \"Dữ liệu\",\n", " \"main\": \"Thông tin chính\",\n", " \"figures\": \"Biểu đồ\",\n", " \"methods\": \"Phương pháp\",\n", " \"limits\": \"Giới hạn\",\n", " \"rerun\": \"Chạy lại phân tích này\",\n", " \"file\": \"Tệp\",\n", " \"database\": \"Cơ sở dữ liệu\",\n", " \"records\": \"Số bản ghi\",\n", " \"metric\": \"Chỉ số\",\n", " \"value\": \"Giá trị\",\n", " \"coverage\": \"Độ phủ trường dữ liệu sau khi gộp (tỉ lệ bản ghi có trường đó)\",\n", " \"field\": \"Trường\",\n", " \"production\": \"Sản lượng và trích dẫn theo năm\",\n", " \"sources\": \"Nguồn công bố\",\n", " \"source_impact\": \"Tác động cục bộ của nguồn (chỉ số h, g, m tính từ số trích dẫn trong các tệp xuất).\",\n", " \"bradford\": \"Định luật Bradford: %d trên %d nguồn thuộc vùng lõi (vùng 1).\",\n", " \"authors\": \"Tác giả và quốc gia\",\n", " \"author_impact\": \"Tác động cục bộ của tác giả (chỉ số h, g, m tính từ số trích dẫn trong các tệp xuất).\",\n", " \"lotka\": \"Định luật Lotka trên %d tác giả: beta ước lượng = %.3f, C = %.4f, R bình phương = %.3f (định luật Lotka: beta = 2, C = 0,6079).\",\n", " \"documents\": \"Tài liệu được trích dẫn nhiều nhất\",\n", " \"conceptual\": \"Cấu trúc khái niệm\",\n", " \"social\": \"Cấu trúc xã hội\",\n", " \"rerun_text\": \"Mở bibliometric_analysis.ipynb trong Google Colab (File > Upload notebook), chạy lần lượt các ô và tải lên đúng các tệp xuất khi được hỏi: notebook chạy lại toàn bộ phân tích và mở mọi bảng, biểu đồ để làm tiếp. Không dùng Colab: pip install numpy pandas matplotlib python-docx openpyxl, rồi import pho_biblio; pho_biblio.report([\\\"scopus.csv\\\", \\\"savedrecs.txt\\\"]). Các sheet *_nodes và *_edges trong bibliometric_tables.xlsx nhập được vào VOSviewer và Gephi.\",\n", " },\n", "}\n", "\n", "# ─── reading ──────────────────────────────────────────────────────────────\n", "\n", "WOS_LIST_TAGS = {\"AU\", \"AF\", \"C1\", \"C3\", \"CR\", \"RI\", \"OI\", \"BE\", \"CA\"}\n", "\n", "\n", "def _read_text(path):\n", " with open(path, \"rb\") as fh:\n", " raw = fh.read()\n", " for enc in (\"utf-8-sig\", \"utf-16\", \"latin-1\"):\n", " try:\n", " text = raw.decode(enc)\n", " if enc == \"utf-16\" and \"\\x00\" in text:\n", " continue\n", " return text\n", " except UnicodeDecodeError:\n", " continue\n", " return raw.decode(\"utf-8\", \"replace\")\n", "\n", "\n", "def detect(text):\n", " \"\"\"(database, format) for a Scopus or Web of Science export, else (None, None).\"\"\"\n", " head = text[:20000]\n", " first = head.splitlines()[0] if head else \"\"\n", " if re.match(r\"^FN\\s+.*Web of Science\", first, re.I) or (\n", " re.search(r\"^PT [A-Z]\\s*$\", head, re.M) and re.search(r\"^UT (WOS|[A-Z]+):\", head, re.M)\n", " ):\n", " return \"Web of Science\", \"wos-plain\"\n", " if first.startswith(\"PT\\tAU\\t\"):\n", " return \"Web of Science\", \"wos-tab\"\n", " if \",\" in first:\n", " cols = [c.strip().lower() for c in next(csv.reader([first]))]\n", " if \"eid\" in cols and \"source title\" in cols and \"title\" in cols:\n", " return \"Scopus\", \"csv\"\n", " if re.search(r\"^TY\\s{1,2}-\", head, re.M):\n", " if re.search(r\"^DB\\s{1,2}-\\s*Scopus\", head, re.M | re.I) or re.search(\n", " r\"^UR\\s{1,2}-\\s*https?://www\\.scopus\\.com/\", head, re.M | re.I\n", " ):\n", " return \"Scopus\", \"ris\"\n", " if re.search(r\"^AN\\s{1,2}-\\s*WOS:\", head, re.M | re.I) or re.search(\n", " r\"^DB\\s{1,2}-\\s*Web of Science\", head, re.M | re.I\n", " ):\n", " return \"Web of Science\", \"ris\"\n", " return None, \"ris\"\n", " return None, None\n", "\n", "\n", "def _split(value, sep=\";\"):\n", " return [v.strip() for v in (value or \"\").split(sep) if v.strip()]\n", "\n", "\n", "def _int(value):\n", " m = re.search(r\"\\d+\", str(value or \"\"))\n", " return int(m.group(0)) if m else None\n", "\n", "\n", "def _year(value):\n", " m = re.search(r\"\\b(1[89]\\d{2}|20\\d{2})\\b\", str(value or \"\"))\n", " return int(m.group(1)) if m else None\n", "\n", "\n", "def _record(database, **kw):\n", " rec = {\n", " \"database\": database,\n", " \"title\": \"\",\n", " \"authors\": [],\n", " \"year\": None,\n", " \"source\": \"\",\n", " \"doi\": \"\",\n", " \"author_keywords\": [],\n", " \"index_keywords\": [],\n", " \"affiliations\": [],\n", " \"cited_by\": None,\n", " \"doc_type\": \"\",\n", " \"references\": None,\n", " }\n", " rec.update({k: v for k, v in kw.items() if v not in (None, \"\", [])})\n", " rec[\"title\"] = re.sub(r\"\\s+\", \" \", rec[\"title\"]).strip()\n", " rec[\"doi\"] = re.sub(r\"^https?://(dx\\.)?doi\\.org/\", \"\", rec[\"doi\"].strip(), flags=re.I).lower()\n", " return rec\n", "\n", "\n", "def _scopus_csv(text):\n", " reader = csv.reader(io.StringIO(text))\n", " header = [f.strip().lower().lstrip(\"\\ufeff\") for f in next(reader, [])]\n", " # Scopus names one \"Funding Text 1\" column but writes one cell per funding\n", " # text, so later columns shift right; read the columns after it from the end.\n", " funding = [i for i, c in enumerate(header) if c.startswith(\"funding text\")]\n", " last_funding = funding[-1] if funding else -1\n", " out = []\n", " for cells in reader:\n", " extra = len(cells) - len(header)\n", " if extra > 0 and last_funding >= 0:\n", " cells = cells[: last_funding + 1] + cells[last_funding + 1 + extra :]\n", " row = dict(zip(header, cells))\n", " g = lambda k: (row.get(k) or \"\").strip()\n", " title = g(\"title\")\n", " if not title:\n", " continue\n", " authors = g(\"authors\")\n", " if authors.lower() == \"[no author name available]\":\n", " authors = \"\"\n", " names = _split(authors) if \";\" in authors else [a.strip() for a in re.split(r\"(?<=\\.),\\s+\", authors) if a.strip()]\n", " affil = g(\"affiliations\")\n", " if not affil and g(\"authors with affiliations\"):\n", " # \"Surname I., Univ X, City, Country; ...\" keep the address part.\n", " affil = \";\".join(p.split(\".,\", 1)[-1] for p in _split(g(\"authors with affiliations\")))\n", " abstract = g(\"abstract\")\n", " out.append(_record(\n", " \"Scopus\",\n", " title=title,\n", " authors=[re.sub(r\"\\s*\\(\\d+\\)$\", \"\", n) for n in names],\n", " year=_year(g(\"year\")),\n", " source=g(\"source title\"),\n", " doi=g(\"doi\"),\n", " author_keywords=_split(g(\"author keywords\")),\n", " index_keywords=_split(g(\"index keywords\")),\n", " affiliations=_split(affil),\n", " cited_by=_int(g(\"cited by\")) if g(\"cited by\") else 0 if \"cited by\" in header else None,\n", " doc_type=g(\"document type\"),\n", " references=len(_split(g(\"references\"))) if g(\"references\") else None,\n", " abstract=None if abstract.lower() == \"[no abstract available]\" else abstract,\n", " ))\n", " return out\n", "\n", "\n", "def _wos_record(fields):\n", " one = lambda t: re.sub(r\"\\s+\", \" \", \" \".join(fields.get(t, []))).strip()\n", " title = one(\"TI\")\n", " if not title:\n", " return None\n", " names = fields.get(\"AF\") or fields.get(\"AU\") or []\n", " affil = []\n", " for line in fields.get(\"C1\", []):\n", " affil.append(re.sub(r\"^\\[[^\\]]*\\]\\s*\", \"\", line).strip())\n", " if not affil and fields.get(\"RP\"):\n", " affil = [re.sub(r\"^.*?\\(corresponding author\\),\\s*\", \"\", one(\"RP\"))]\n", " tc = one(\"TC\") or one(\"Z9\")\n", " return _record(\n", " \"Web of Science\",\n", " title=title,\n", " authors=[n.strip() for n in names if n.strip()],\n", " year=_year(one(\"PY\") or one(\"EY\") or one(\"EA\")),\n", " source=one(\"SO\"),\n", " doi=one(\"DI\"),\n", " author_keywords=_split(one(\"DE\")),\n", " index_keywords=_split(one(\"ID\")),\n", " affiliations=[a for a in affil if a],\n", " cited_by=_int(tc) if tc else None,\n", " doc_type=one(\"DT\"),\n", " references=_int(one(\"NR\")),\n", " abstract=one(\"AB\"),\n", " )\n", "\n", "\n", "def _wos_plain(text):\n", " out, fields, last, open_ = [], {}, None, False\n", " for line in text.splitlines():\n", " if not line.strip():\n", " continue\n", " m = re.match(r\"^([A-Z][A-Z0-9])(?: (.*))?$\", line)\n", " if m:\n", " tag, value = m.group(1), (m.group(2) or \"\").strip()\n", " if tag == \"ER\":\n", " rec = _wos_record(fields) if open_ else None\n", " if rec:\n", " out.append(rec)\n", " fields, last, open_ = {}, None, False\n", " continue\n", " if tag == \"EF\":\n", " break\n", " if tag in (\"FN\", \"VR\"):\n", " continue\n", " if tag == \"PT\" and open_:\n", " rec = _wos_record(fields)\n", " if rec:\n", " out.append(rec)\n", " fields = {}\n", " open_ = True\n", " fields.setdefault(tag, []).append(value)\n", " last = tag\n", " elif last and re.match(r\"^\\s+\\S\", line):\n", " if last in WOS_LIST_TAGS:\n", " fields[last].append(line.strip())\n", " else:\n", " fields[last][-1] = (fields[last][-1] + \" \" + line.strip()).strip()\n", " if open_:\n", " rec = _wos_record(fields)\n", " if rec:\n", " out.append(rec)\n", " return out\n", "\n", "\n", "def _wos_tab(text):\n", " out = []\n", " reader = csv.reader(io.StringIO(text), delimiter=\"\\t\", quoting=csv.QUOTE_NONE)\n", " header = None\n", " for row in reader:\n", " if not any(c.strip() for c in row):\n", " continue\n", " if header is None:\n", " header = [h.strip() for h in row]\n", " continue\n", " fields = {}\n", " for tag, value in zip(header, row):\n", " value = value.strip()\n", " if tag and value:\n", " fields[tag] = _split(value) if tag in WOS_LIST_TAGS else [value]\n", " rec = _wos_record(fields)\n", " if rec:\n", " out.append(rec)\n", " return out\n", "\n", "\n", "def _ris(text, database):\n", " out, fields = [], {}\n", " for line in text.splitlines():\n", " m = re.match(r\"^([A-Z][A-Z0-9])\\s{1,2}-\\s?(.*)$\", line)\n", " if not m:\n", " continue\n", " tag, value = m.group(1), m.group(2).strip()\n", " if tag == \"ER\":\n", " title = \" \".join(fields.get(\"TI\", []) or fields.get(\"T1\", []))\n", " if title:\n", " notes = \" \".join(fields.get(\"N1\", []))\n", " cited = re.search(r\"Cited By(?: \\(since \\d+\\))?:\\s*(\\d+)\", notes, re.I)\n", " out.append(_record(\n", " database or \"Imported\",\n", " title=title,\n", " authors=fields.get(\"AU\", []) or fields.get(\"A1\", []),\n", " year=_year(\" \".join(fields.get(\"PY\", []) or fields.get(\"Y1\", []))),\n", " source=\" \".join((fields.get(\"T2\") or fields.get(\"JO\") or fields.get(\"JF\") or [\"\"])[:1]),\n", " doi=\" \".join(fields.get(\"DO\", [])[:1]),\n", " author_keywords=fields.get(\"KW\", []),\n", " affiliations=[a for v in fields.get(\"AD\", []) for a in _split(v)],\n", " cited_by=int(cited.group(1)) if cited else None,\n", " doc_type=\" \".join(fields.get(\"TY\", [])[:1]),\n", " abstract=\" \".join(fields.get(\"AB\", []) or fields.get(\"N2\", [])),\n", " ))\n", " fields = {}\n", " continue\n", " if value:\n", " fields.setdefault(tag, []).append(value)\n", " return out\n", "\n", "\n", "def load(path):\n", " \"\"\"Records of one export file, each labelled with the database that wrote it.\"\"\"\n", " text = _read_text(path)\n", " database, fmt = detect(text)\n", " if fmt == \"csv\":\n", " return _scopus_csv(text)\n", " if fmt == \"wos-plain\":\n", " return _wos_plain(text)\n", " if fmt == \"wos-tab\":\n", " return _wos_tab(text)\n", " if fmt == \"ris\":\n", " return _ris(text, database)\n", " raise ValueError(\n", " \"%s is not a Scopus CSV, Web of Science plain-text/tab-delimited or RIS export\" % os.path.basename(path)\n", " )\n", "\n", "\n", "def _title_key(title):\n", " t = re.sub(r\"<[^>]+>\", \" \", title.lower())\n", " return re.sub(r\"[^\\w]+\", \"\", t)\n", "\n", "\n", "def merge(paths):\n", " \"\"\"Load every file and merge duplicates (same DOI, else same title and year).\n", "\n", " Returns (records, per_file) where per_file lists what each file contributed.\n", " \"\"\"\n", " records, by_key, per_file = [], {}, []\n", " for path in paths:\n", " recs = load(path)\n", " per_file.append({\"file\": os.path.basename(path), \"database\": recs[0][\"database\"] if recs else \"?\", \"records\": len(recs)})\n", " for rec in recs:\n", " keys = []\n", " if rec[\"doi\"]:\n", " keys.append(\"doi:\" + rec[\"doi\"])\n", " if rec[\"title\"]:\n", " keys.append(\"t:%s:%s\" % (_title_key(rec[\"title\"]), rec[\"year\"] or \"\"))\n", " hit = next((by_key[k] for k in keys if k in by_key), None)\n", " if hit is None:\n", " rec[\"databases\"] = [rec[\"database\"]]\n", " records.append(rec)\n", " hit = rec\n", " else:\n", " if rec[\"database\"] not in hit[\"databases\"]:\n", " hit[\"databases\"].append(rec[\"database\"])\n", " for field, value in rec.items():\n", " if field in (\"database\", \"databases\"):\n", " continue\n", " if not hit.get(field) and value:\n", " hit[field] = value\n", " if rec[\"cited_by\"] is not None:\n", " hit.setdefault(\"cited_by_\" + rec[\"database\"], rec[\"cited_by\"])\n", " for k in keys:\n", " by_key.setdefault(k, hit)\n", " return records, per_file\n", "\n", "\n", "# ─── counting ─────────────────────────────────────────────────────────────\n", "\n", "COUNTRY_ALIASES = {\n", " \"usa\": \"United States\", \"united states of america\": \"United States\", \"u.s.a.\": \"United States\",\n", " \"uk\": \"United Kingdom\", \"england\": \"United Kingdom\", \"scotland\": \"United Kingdom\", \"wales\": \"United Kingdom\",\n", " \"north ireland\": \"United Kingdom\", \"northern ireland\": \"United Kingdom\",\n", " \"peoples r china\": \"China\", \"people's republic of china\": \"China\", \"pr china\": \"China\",\n", " \"south korea\": \"South Korea\", \"republic of korea\": \"South Korea\", \"korea\": \"South Korea\",\n", " \"viet nam\": \"Vietnam\", \"russian federation\": \"Russia\", \"turkiye\": \"Turkey\", \"türkiye\": \"Turkey\",\n", " \"iran (islamic republic of)\": \"Iran\", \"taiwan, province of china\": \"Taiwan\",\n", " \"bosnia & herceg\": \"Bosnia and Herzegovina\", \"u arab emirates\": \"United Arab Emirates\",\n", " \"czech republic\": \"Czechia\", \"fed rep ger\": \"Germany\",\n", "}\n", "\n", "\n", "def country_of(affiliation):\n", " \"\"\"Country from the last part of an address; None when it does not look like one.\"\"\"\n", " part = affiliation.strip().rstrip(\".\").split(\",\")[-1].strip()\n", " if not part:\n", " return None\n", " if re.search(r\"\\bUSA$\", part):\n", " return \"United States\"\n", " low = part.lower()\n", " if low in COUNTRY_ALIASES:\n", " return COUNTRY_ALIASES[low]\n", " if re.search(r\"\\d\", part) or len(part) > 40:\n", " return None\n", " return part.title() if part.isupper() else part\n", "\n", "\n", "def _author_key(name):\n", " \"\"\"'Nguyen, Thi Hoa' / 'Nguyen T.H.' / 'Nguyen, TH' -> 'Nguyen T.H.' (family + initials).\"\"\"\n", " name = re.sub(r\"\\s+\", \" \", name).strip()\n", " if \",\" in name:\n", " family, given = [p.strip() for p in name.split(\",\", 1)]\n", " else:\n", " parts = name.split(\" \")\n", " cut = len(parts)\n", " while cut > 1 and re.fullmatch(r\"(?:[A-Z]\\.?-?){1,5}\", parts[cut - 1]):\n", " cut -= 1\n", " if cut == len(parts):\n", " family, given = parts[-1], \" \".join(parts[:-1])\n", " else:\n", " family, given = \" \".join(parts[:cut]), \" \".join(parts[cut:])\n", " if re.fullmatch(r\"[A-Z]{1,5}\", given.replace(\" \", \"\")):\n", " initials = \"\".join(c + \".\" for c in given.replace(\" \", \"\"))\n", " else:\n", " initials = \"\".join(w[0].upper() + \".\" for w in re.split(r\"[\\s.\\-]+\", given) if w)\n", " return (\"%s %s\" % (family.title() if family.isupper() else family, initials)).strip()\n", "\n", "\n", "def _source_name(source):\n", " s = re.sub(r\"\\s+\", \" \", source or \"\").strip()\n", " return s.title() if s.isupper() else s\n", "\n", "\n", "def _countries(rec):\n", " return sorted({c for c in (country_of(a) for a in rec[\"affiliations\"]) if c})\n", "\n", "\n", "# Indexer bookkeeping terms (Embase/EMTREE and MeSH check tags, publication types).\n", "# Scopus puts them in Index Keywords of nearly every biomedical record, where they\n", "# would otherwise top the keyword table and name every theme. Author keywords are\n", "# never filtered.\n", "INDEX_TERM_STOPLIST = {\n", " \"article\", \"review\", \"priority journal\", \"controlled study\", \"major clinical study\", \"clinical article\",\n", " \"human\", \"humans\", \"nonhuman\", \"animal\", \"animals\", \"male\", \"female\", \"adult\", \"aged\", \"middle aged\",\n", " \"middle age\", \"young adult\", \"adolescent\", \"child\", \"infant\", \"newborn\", \"aged, 80 and over\", \"child, preschool\",\n", " \"animal experiment\", \"animal model\", \"animal tissue\", \"human tissue\", \"human cell\", \"unclassified drug\",\n", " \"support, non-u.s. gov't\", \"support, u.s. gov't, p.h.s.\", \"support, u.s. gov't, non-p.h.s.\",\n", " \"research support, non-u.s. gov't\", \"research support, u.s. gov't, p.h.s.\",\n", " \"research support, n.i.h., extramural\", \"methodology\", \"procedures\", \"priority\", \"short survey\",\n", "}\n", "\n", "\n", "def _keywords(rec):\n", " if rec[\"author_keywords\"]:\n", " words = rec[\"author_keywords\"]\n", " else:\n", " words = [w for w in rec[\"index_keywords\"] if w.strip().lower() not in INDEX_TERM_STOPLIST]\n", " return sorted({w.strip().lower() for w in words if w.strip()})\n", "\n", "\n", "def summary(records):\n", " years = [r[\"year\"] for r in records if r[\"year\"]]\n", " cited = [r[\"cited_by\"] for r in records if r[\"cited_by\"] is not None]\n", " authors_per = [len(r[\"authors\"]) for r in records if r[\"authors\"]]\n", " countries_per = [len({c for c in (country_of(a) for a in r[\"affiliations\"]) if c}) for r in records if r[\"affiliations\"]]\n", " all_authors = {_author_key(a) for r in records for a in r[\"authors\"]}\n", " span = (min(years), max(years)) if years else (None, None)\n", " by_year = Counter(years)\n", " growth = None\n", " if years and span[1] > span[0] and by_year[span[0]] > 0:\n", " growth = (by_year[span[1]] / by_year[span[0]]) ** (1.0 / (span[1] - span[0])) - 1\n", " return {\n", " \"documents\": len(records),\n", " \"timespan\": \"%s-%s\" % span if years else \"unknown\",\n", " \"annual_growth_rate_pct\": round(growth * 100, 2) if growth is not None else None,\n", " \"sources\": len({r[\"source\"].lower() for r in records if r[\"source\"]}),\n", " \"authors\": len(all_authors),\n", " \"single_authored_documents\": sum(1 for n in authors_per if n == 1),\n", " \"authors_per_document\": round(float(np.mean(authors_per)), 2) if authors_per else None,\n", " \"international_coauthorship_pct\": round(100.0 * sum(1 for n in countries_per if n > 1) / len(countries_per), 2) if countries_per else None,\n", " \"mean_citations_per_document\": round(float(np.mean(cited)), 2) if cited else None,\n", " \"author_keywords\": len({k for r in records for k in (w.lower() for w in r[\"author_keywords\"])}),\n", " \"document_types\": dict(Counter(r[\"doc_type\"] or \"unknown\" for r in records).most_common()),\n", " }\n", "\n", "\n", "def coverage(records):\n", " n = max(len(records), 1)\n", " fields = {\n", " \"year\": lambda r: r[\"year\"],\n", " \"authors\": lambda r: r[\"authors\"],\n", " \"source\": lambda r: r[\"source\"],\n", " \"doi\": lambda r: r[\"doi\"],\n", " \"keywords\": lambda r: r[\"author_keywords\"] or r[\"index_keywords\"],\n", " \"affiliations\": lambda r: r[\"affiliations\"],\n", " \"cited_by\": lambda r: r[\"cited_by\"] is not None,\n", " \"abstract\": lambda r: r.get(\"abstract\"),\n", " }\n", " return {k: round(100.0 * sum(1 for r in records if f(r)) / n, 1) for k, f in fields.items()}\n", "\n", "\n", "def tables(records, top=TOP_N):\n", " import pandas as pd\n", "\n", " by_year = Counter(r[\"year\"] for r in records if r[\"year\"])\n", " cites_year = defaultdict(int)\n", " for r in records:\n", " if r[\"year\"] and r[\"cited_by\"]:\n", " cites_year[r[\"year\"]] += r[\"cited_by\"]\n", " this_year = date.today().year\n", " rows = []\n", " for y in (range(min(by_year), max(by_year) + 1) if by_year else []):\n", " n, c = by_year.get(y, 0), cites_year.get(y, 0)\n", " mean = c / n if n else 0.0\n", " citable = max(this_year - y + 1, 1)\n", " rows.append({\"year\": y, \"documents\": n, \"citations\": c, \"mean_citations_per_document\": round(mean, 2),\n", " \"citable_years\": citable, \"mean_citations_per_year\": round(mean / citable, 3)})\n", " annual = pd.DataFrame(rows, columns=[\"year\", \"documents\", \"citations\", \"mean_citations_per_document\",\n", " \"citable_years\", \"mean_citations_per_year\"])\n", "\n", " authors, author_cites = Counter(), Counter()\n", " for r in records:\n", " for a in {_author_key(a) for a in r[\"authors\"]}:\n", " authors[a] += 1\n", " author_cites[a] += r[\"cited_by\"] or 0\n", " t_authors = pd.DataFrame([{\"author\": a, \"documents\": n, \"citations\": author_cites[a]} for a, n in authors.most_common(top)])\n", "\n", " sources, source_cites = Counter(), Counter()\n", " for r in records:\n", " if r[\"source\"]:\n", " s = _source_name(r[\"source\"])\n", " sources[s] += 1\n", " source_cites[s] += r[\"cited_by\"] or 0\n", " t_sources = pd.DataFrame([{\"source\": s, \"documents\": n, \"citations\": source_cites[s]} for s, n in sources.most_common(top)])\n", "\n", " countries, country_cites, mcp = Counter(), Counter(), Counter()\n", " for r in records:\n", " cs = _countries(r)\n", " for c in cs:\n", " countries[c] += 1\n", " country_cites[c] += r[\"cited_by\"] or 0\n", " if len(cs) > 1:\n", " mcp[c] += 1\n", " t_countries = pd.DataFrame([{\n", " \"country\": c, \"documents\": n, \"single_country_publications\": n - mcp[c], \"multiple_country_publications\": mcp[c],\n", " \"mcp_ratio\": round(mcp[c] / n, 3), \"citations\": country_cites[c],\n", " \"mean_citations\": round(country_cites[c] / n, 2),\n", " } for c, n in countries.most_common(top)], columns=[\"country\", \"documents\", \"single_country_publications\",\n", " \"multiple_country_publications\", \"mcp_ratio\", \"citations\", \"mean_citations\"])\n", "\n", " kw = Counter(k for r in records for k in _keywords(r))\n", " t_keywords = pd.DataFrame([{\"keyword\": k, \"occurrences\": n} for k, n in kw.most_common(top)])\n", "\n", " cited = sorted((r for r in records if r[\"cited_by\"]), key=lambda r: -r[\"cited_by\"])[:top]\n", " t_cited = pd.DataFrame([{\n", " \"title\": r[\"title\"], \"first_author\": r[\"authors\"][0] if r[\"authors\"] else \"\", \"year\": r[\"year\"],\n", " \"source\": r[\"source\"], \"doi\": r[\"doi\"], \"cited_by\": r[\"cited_by\"], \"databases\": \", \".join(r[\"databases\"]),\n", " } for r in cited])\n", " return {\"annual\": annual, \"authors\": t_authors, \"sources\": t_sources, \"countries\": t_countries,\n", " \"keywords\": t_keywords, \"most_cited\": t_cited}\n", "\n", "\n", "def network(records, kind, nodes=NETWORK_NODES, min_edge=MIN_EDGE):\n", " \"\"\"Co-occurrence network: kind=\"authors\" (co-authorship) or \"keywords\".\n", "\n", " Full counting: each document adds 1 to every pair it contains. Keeps the\n", " nodes most frequent items and edges seen at least min_edge times.\n", " \"\"\"\n", " items = []\n", " for r in records:\n", " if kind == \"authors\":\n", " items.append(sorted({_author_key(a) for a in r[\"authors\"]}))\n", " elif kind == \"countries\":\n", " items.append(_countries(r))\n", " else:\n", " items.append(_keywords(r))\n", " freq = Counter(i for its in items for i in its)\n", " keep = [i for i, _ in freq.most_common(nodes)]\n", " index = {k: n for n, k in enumerate(keep)}\n", " weights = Counter()\n", " for its in items:\n", " its = [i for i in its if i in index]\n", " for a in range(len(its)):\n", " for b in range(a + 1, len(its)):\n", " weights[(its[a], its[b])] += 1\n", " edges = [(a, b, w) for (a, b), w in weights.items() if w >= min_edge]\n", " linked = {a for a, _, _ in edges} | {b for _, b, _ in edges}\n", " keep = [k for k in keep if k in linked]\n", " return {\"nodes\": [(k, freq[k]) for k in keep], \"edges\": sorted(edges, key=lambda e: -e[2])}\n", "\n", "\n", "def _clusters(net):\n", " \"\"\"Weighted label propagation, deterministic (fixed order, ties to the smallest label).\"\"\"\n", " names = [n for n, _ in net[\"nodes\"]]\n", " labels = {n: i for i, n in enumerate(names)}\n", " adj = defaultdict(list)\n", " for a, b, w in net[\"edges\"]:\n", " adj[a].append((b, w))\n", " adj[b].append((a, w))\n", " for _ in range(30):\n", " changed = False\n", " for n in names:\n", " if not adj[n]:\n", " continue\n", " score = Counter()\n", " for m, w in adj[n]:\n", " score[labels[m]] += w\n", " best = max(score.values())\n", " new = min(l for l, s in score.items() if s == best)\n", " if new != labels[n]:\n", " labels[n], changed = new, True\n", " if not changed:\n", " break\n", " order = {l: i for i, (l, _) in enumerate(Counter(labels.values()).most_common())}\n", " return {n: order[labels[n]] for n in names}\n", "\n", "\n", "def _layout(net, iterations=300):\n", " \"\"\"Fruchterman-Reingold with a fixed seed, so the same data draws the same map.\"\"\"\n", " names = [n for n, _ in net[\"nodes\"]]\n", " k = len(names)\n", " rng = np.random.default_rng(42)\n", " pos = rng.uniform(-1, 1, size=(k, 2))\n", " if k < 2:\n", " return {n: pos[i] for i, n in enumerate(names)}\n", " idx = {n: i for i, n in enumerate(names)}\n", " w = np.zeros((k, k))\n", " for a, b, wt in net[\"edges\"]:\n", " w[idx[a], idx[b]] = w[idx[b], idx[a]] = wt\n", " w = w / w.max() if w.max() > 0 else w\n", " opt = math.sqrt(4.0 / k)\n", " temp = 0.2\n", " for _ in range(iterations):\n", " delta = pos[:, None, :] - pos[None, :, :]\n", " dist = np.linalg.norm(delta, axis=-1) + 1e-9\n", " rep = (opt ** 2 / dist ** 2)[:, :, None] * delta\n", " att = -(dist / opt * w)[:, :, None] * delta\n", " disp = (rep + att).sum(axis=1)\n", " length = np.linalg.norm(disp, axis=1)[:, None] + 1e-9\n", " pos += disp / length * np.minimum(length, temp)\n", " pos -= pos.mean(axis=0)\n", " temp = max(temp * 0.985, 0.005)\n", " span = np.abs(pos).max() or 1.0\n", " return {n: pos[i] / span for i, n in enumerate(names)}\n", "\n", "\n", "# ─── Biblioshiny-style indicators ─────────────────────────────────────────\n", "# Names follow bibliometrix / Biblioshiny so a researcher who knows those tools\n", "# finds the same table under the same name.\n", "\n", "def _h_g(cites):\n", " \"\"\"h-index and g-index of a list of citation counts.\"\"\"\n", " c = sorted((x or 0 for x in cites), reverse=True)\n", " h = sum(1 for i, x in enumerate(c, 1) if x >= i)\n", " g, total = 0, 0\n", " for i, x in enumerate(c, 1):\n", " total += x\n", " if total >= i * i:\n", " g = i\n", " return h, g\n", "\n", "\n", "def impact(records, kind, top=TOP_N):\n", " \"\"\"Local impact (Biblioshiny \"Authors' / Sources' Local Impact\"): h, g and m index.\n", "\n", " kind is \"authors\" or \"sources\". m = h / years since the first document here\n", " (current year included). Citations are the counts the databases exported.\n", " \"\"\"\n", " import pandas as pd\n", "\n", " col = \"author\" if kind == \"authors\" else \"source\"\n", " groups = defaultdict(list)\n", " for r in records:\n", " if kind == \"authors\":\n", " keys = {_author_key(a) for a in r[\"authors\"]}\n", " else:\n", " keys = {_source_name(r[\"source\"])} if r[\"source\"] else set()\n", " for k in keys:\n", " groups[k].append(r)\n", " this_year = date.today().year\n", " rows = []\n", " for k, recs in groups.items():\n", " h, g = _h_g([r[\"cited_by\"] for r in recs])\n", " years = [r[\"year\"] for r in recs if r[\"year\"]]\n", " start = min(years) if years else None\n", " rows.append({col: k, \"h_index\": h, \"g_index\": g,\n", " \"m_index\": round(h / (this_year - start + 1), 3) if start else None,\n", " \"citations\": sum(r[\"cited_by\"] or 0 for r in recs), \"documents\": len(recs), \"first_year\": start})\n", " rows.sort(key=lambda x: (-x[\"h_index\"], -x[\"citations\"], -x[\"documents\"], x[col]))\n", " return pd.DataFrame(rows[:top], columns=[col, \"h_index\", \"g_index\", \"m_index\", \"citations\", \"documents\", \"first_year\"])\n", "\n", "\n", "def bradford(records):\n", " \"\"\"Bradford's law: sources ranked by documents, split into three zones of about a third of the documents each.\"\"\"\n", " import pandas as pd\n", "\n", " freq = Counter(_source_name(r[\"source\"]) for r in records if r[\"source\"])\n", " total = sum(freq.values())\n", " rows, cum = [], 0\n", " for rank, (src, n) in enumerate(sorted(freq.items(), key=lambda kv: (-kv[1], kv[0])), 1):\n", " cum += n\n", " zone = 1 if (rank == 1 or cum <= round(total / 3.0)) else 2 if cum <= round(2 * total / 3.0) else 3\n", " rows.append({\"rank\": rank, \"source\": src, \"documents\": n, \"cumulative_documents\": cum, \"zone\": zone})\n", " return pd.DataFrame(rows, columns=[\"rank\", \"source\", \"documents\", \"cumulative_documents\", \"zone\"])\n", "\n", "\n", "def lotka(records):\n", " \"\"\"Lotka's law: how many authors wrote 1, 2, 3... documents, and the fitted exponent.\n", "\n", " Fits log10(share of authors) = log10(C) - beta * log10(documents) by least\n", " squares, as bibliometrix does. Lotka's own law is beta = 2, C = 0.6079.\n", " \"\"\"\n", " import pandas as pd\n", "\n", " per_author = Counter(a for r in records for a in {_author_key(x) for x in r[\"authors\"]})\n", " dist = Counter(per_author.values())\n", " total = sum(dist.values())\n", " rows = [{\"documents_written\": n, \"authors\": dist[n], \"share_of_authors\": round(dist[n] / total, 4),\n", " \"lotka_expected_share\": round(0.6079 / n ** 2, 4)} for n in sorted(dist)]\n", " fit = {\"authors\": total, \"beta\": None, \"C\": None, \"r_squared\": None}\n", " if len(rows) >= 2:\n", " x = np.log10([r[\"documents_written\"] for r in rows])\n", " y = np.log10([r[\"authors\"] / total for r in rows])\n", " slope, intercept = np.polyfit(x, y, 1)\n", " pred = slope * x + intercept\n", " ss = float(((y - y.mean()) ** 2).sum())\n", " fit.update({\"beta\": round(float(-slope), 3), \"C\": round(float(10 ** intercept), 4),\n", " \"r_squared\": round(1 - float(((y - pred) ** 2).sum()) / ss, 3) if ss > 0 else None})\n", " return pd.DataFrame(rows, columns=[\"documents_written\", \"authors\", \"share_of_authors\", \"lotka_expected_share\"]), fit\n", "\n", "\n", "def trend_topics(records, min_freq=None, per_year=3):\n", " \"\"\"Trend topics: for each frequent keyword, the 1st quartile, median and 3rd quartile of its publication years.\n", "\n", " Keeps the per_year most frequent keywords for each median year. min_freq\n", " defaults to 5, lowered for small collections (2 below 100 documents).\n", " \"\"\"\n", " import pandas as pd\n", "\n", " if min_freq is None:\n", " min_freq = 5 if len(records) >= 100 else 2\n", " years = defaultdict(list)\n", " for r in records:\n", " if r[\"year\"]:\n", " for k in _keywords(r):\n", " years[k].append(r[\"year\"])\n", " rows = []\n", " for k, ys in years.items():\n", " if len(ys) >= min_freq:\n", " q1, med, q3 = np.percentile(ys, [25, 50, 75])\n", " rows.append({\"keyword\": k, \"frequency\": len(ys), \"year_q1\": int(round(q1)),\n", " \"year_median\": int(round(med)), \"year_q3\": int(round(q3))})\n", " by_year = defaultdict(list)\n", " for row in rows:\n", " by_year[row[\"year_median\"]].append(row)\n", " kept = []\n", " for y in sorted(by_year):\n", " kept.extend(sorted(by_year[y], key=lambda r: (-r[\"frequency\"], r[\"keyword\"]))[:per_year])\n", " return pd.DataFrame(kept, columns=[\"keyword\", \"frequency\", \"year_q1\", \"year_median\", \"year_q3\"])\n", "\n", "\n", "def _louvain(names, edges, passes=50):\n", " \"\"\"Modularity clustering, local-moving phase of Louvain, deterministic (fixed order, ties to the lowest id).\"\"\"\n", " adj = defaultdict(dict)\n", " for a, b, w in edges:\n", " adj[a][b] = adj[a].get(b, 0.0) + w\n", " adj[b][a] = adj[b].get(a, 0.0) + w\n", " deg = {n: sum(adj[n].values()) for n in names}\n", " m2 = sum(deg.values())\n", " comm = {n: i for i, n in enumerate(names)}\n", " if m2 == 0:\n", " return comm\n", " tot = defaultdict(float)\n", " for n in names:\n", " tot[comm[n]] += deg[n]\n", " for _ in range(passes):\n", " moved = False\n", " for n in names:\n", " c0 = comm[n]\n", " tot[c0] -= deg[n]\n", " links = defaultdict(float)\n", " for other, w in adj[n].items():\n", " links[comm[other]] += w\n", " best, best_gain = c0, links.get(c0, 0.0) - tot[c0] * deg[n] / m2\n", " for c in sorted(links):\n", " gain = links[c] - tot[c] * deg[n] / m2\n", " if gain > best_gain + 1e-12:\n", " best, best_gain = c, gain\n", " comm[n] = best\n", " tot[best] += deg[n]\n", " moved = moved or best != c0\n", " if not moved:\n", " break\n", " order = {c: i for i, (c, _) in enumerate(Counter(comm.values()).most_common())}\n", " return {n: order[comm[n]] for n in names}\n", "\n", "\n", "QUADRANTS = {\n", " (True, True): \"Motor themes\",\n", " (False, True): \"Niche themes\",\n", " (False, False): \"Emerging or declining themes\",\n", " (True, False): \"Basic themes\",\n", "}\n", "\n", "\n", "def thematic_map(records, words=250, min_cluster=3, labels=3):\n", " \"\"\"Thematic map (Cobo et al. 2011, as in Biblioshiny).\n", "\n", " Keyword co-occurrence on the \"words\" most frequent keywords, normalised by\n", " the equivalence index e_ij = c_ij^2 / (c_i c_j), clustered by modularity.\n", " Callon centrality = 10 x the sum of a cluster's external links (how much it\n", " connects to other themes); Callon density = 100 x the sum of its internal\n", " links / its number of keywords (how developed it is). Quadrants split at the\n", " mean centrality and mean density.\n", " \"\"\"\n", " import pandas as pd\n", "\n", " net = network(records, \"keywords\", nodes=words, min_edge=1)\n", " freq = dict(net[\"nodes\"])\n", " names = [n for n, _ in net[\"nodes\"]]\n", " eq = [(a, b, w * w / float(freq[a] * freq[b])) for a, b, w in net[\"edges\"]]\n", " cl = _louvain(names, eq)\n", " members = defaultdict(list)\n", " for n in names:\n", " members[cl[n]].append(n)\n", " rows = []\n", " for c, ws in members.items():\n", " if len(ws) < min_cluster:\n", " continue\n", " inside = set(ws)\n", " internal = sum(e for a, b, e in eq if a in inside and b in inside)\n", " external = sum(e for a, b, e in eq if (a in inside) != (b in inside))\n", " ranked = sorted(ws, key=lambda w: (-freq[w], w))\n", " rows.append({\"cluster\": 0, \"label\": \"; \".join(ranked[:labels]), \"keywords\": len(ws),\n", " \"frequency\": sum(freq[w] for w in ws), \"centrality\": round(10 * external, 3),\n", " \"density\": round(100 * internal / len(ws), 3), \"all_keywords\": \"; \".join(ranked)})\n", " rows.sort(key=lambda r: (-r[\"frequency\"], r[\"label\"]))\n", " if rows:\n", " mc = float(np.mean([r[\"centrality\"] for r in rows]))\n", " md = float(np.mean([r[\"density\"] for r in rows]))\n", " for i, r in enumerate(rows, 1):\n", " r[\"cluster\"] = i\n", " r[\"quadrant\"] = QUADRANTS[(r[\"centrality\"] >= mc, r[\"density\"] >= md)]\n", " return pd.DataFrame(rows, columns=[\"cluster\", \"label\", \"quadrant\", \"keywords\", \"frequency\", \"centrality\", \"density\", \"all_keywords\"])\n", "\n", "\n", "def three_fields(records, n=10):\n", " \"\"\"Three-field plot (Sankey): countries -> authors -> keywords, the n most frequent of each.\n", "\n", " A band's width is the number of documents that carry both ends.\n", " \"\"\"\n", " fields = [(\"countries\", _countries), (\"authors\", lambda r: sorted({_author_key(a) for a in r[\"authors\"]})),\n", " (\"keywords\", _keywords)]\n", " tops = []\n", " for _, get in fields:\n", " c = Counter(x for r in records for x in get(r))\n", " tops.append([k for k, _ in sorted(c.items(), key=lambda kv: (-kv[1], kv[0]))[:n]])\n", " flows = [Counter(), Counter()]\n", " for r in records:\n", " vals = [[x for x in get(r) if x in set(top)] for (_, get), top in zip(fields, tops)]\n", " for i in (0, 1):\n", " for a in vals[i]:\n", " for b in vals[i + 1]:\n", " flows[i][(a, b)] += 1\n", " return {\"fields\": [f for f, _ in fields], \"columns\": tops,\n", " \"flows\": [sorted(((a, b, w) for (a, b), w in f.items()), key=lambda e: (-e[2], e[0], e[1])) for f in flows]}\n", "\n", "\n", "# ─── drawing ──────────────────────────────────────────────────────────────\n", "\n", "PALETTE = [\"#0072B2\", \"#E69F00\", \"#009E73\", \"#CC79A7\", \"#56B4E9\", \"#D55E00\", \"#F0E442\", \"#999999\"]\n", "\n", "\n", "def _mpl():\n", " import matplotlib\n", " matplotlib.use(\"Agg\")\n", " import matplotlib.pyplot as plt\n", " plt.rcParams.update({\n", " \"font.family\": \"sans-serif\",\n", " \"font.sans-serif\": [\"Arial\", \"Helvetica\", \"Liberation Sans\", \"DejaVu Sans\"],\n", " \"font.size\": 9, \"axes.titlesize\": 10, \"axes.labelsize\": 9,\n", " \"axes.spines.top\": False, \"axes.spines.right\": False,\n", " \"figure.dpi\": 100, \"savefig.dpi\": 300, \"savefig.bbox\": \"tight\",\n", " })\n", " return plt\n", "\n", "\n", "def _bar(plt, df, label, value, title, path, xlabel):\n", " fig, ax = plt.subplots(figsize=(6.3, max(2.2, 0.28 * len(df) + 0.8)))\n", " d = df.iloc[::-1]\n", " ax.barh([str(x)[:60] for x in d[label]], d[value], color=PALETTE[0])\n", " ax.set_title(title, loc=\"left\")\n", " ax.set_xlabel(xlabel)\n", " ax.tick_params(axis=\"y\", length=0)\n", " fig.savefig(path)\n", " plt.close(fig)\n", "\n", "\n", "def _draw_network(plt, net, title, path):\n", " fig, ax = plt.subplots(figsize=(6.3, 5.2))\n", " ax.set_axis_off()\n", " ax.set_title(title, loc=\"left\")\n", " if not net[\"nodes\"]:\n", " ax.text(0.5, 0.5, \"Not enough co-occurrences to draw a network.\", ha=\"center\", va=\"center\", transform=ax.transAxes)\n", " fig.savefig(path)\n", " plt.close(fig)\n", " return {}\n", " pos = _layout(net)\n", " cl = _clusters(net)\n", " top_w = max(w for _, _, w in net[\"edges\"])\n", " for a, b, w in net[\"edges\"]:\n", " (x1, y1), (x2, y2) = pos[a], pos[b]\n", " ax.plot([x1, x2], [y1, y2], color=\"#BBBBBB\", lw=0.4 + 2.0 * w / top_w, alpha=0.6, zorder=1)\n", " top_f = max(f for _, f in net[\"nodes\"])\n", " for name, f in net[\"nodes\"]:\n", " x, y = pos[name]\n", " ax.scatter([x], [y], s=20 + 280 * f / top_f, color=PALETTE[min(cl[name], len(PALETTE) - 1)],\n", " edgecolor=\"white\", linewidth=0.5, zorder=2)\n", " labelled = sorted(net[\"nodes\"], key=lambda n: -n[1])[:25]\n", " for name, _ in labelled:\n", " x, y = pos[name]\n", " ax.annotate(name[:32], (x, y), fontsize=6.5, ha=\"center\", va=\"bottom\", xytext=(0, 4), textcoords=\"offset points\", zorder=3)\n", " fig.savefig(path)\n", " plt.close(fig)\n", " return cl\n", "\n", "\n", "def _empty(plt, title, path, message):\n", " fig, ax = plt.subplots(figsize=(6.3, 2.0))\n", " ax.set_axis_off()\n", " ax.set_title(title, loc=\"left\")\n", " ax.text(0.5, 0.5, message, ha=\"center\", va=\"center\", transform=ax.transAxes)\n", " fig.savefig(path)\n", " plt.close(fig)\n", "\n", "\n", "def _draw_bradford(plt, df, path):\n", " if not len(df):\n", " return _empty(plt, \"Core sources by Bradford's law\", path, \"No source titles in these records.\")\n", " fig, ax = plt.subplots(figsize=(6.3, 3.0))\n", " ax.plot(df[\"rank\"], df[\"documents\"], color=PALETTE[0], lw=1.2)\n", " core = df[df[\"zone\"] == 1]\n", " ax.axvspan(0.8, core[\"rank\"].max() + 0.2, color=PALETTE[2], alpha=0.15, lw=0)\n", " ax.set_xscale(\"log\")\n", " from matplotlib.ticker import NullLocator, ScalarFormatter\n", " ax.set_xticks([x for x in (1, 2, 5, 10, 20, 50, 100, 200, 500, 1000) if x <= max(df[\"rank\"].max(), 1)])\n", " ax.xaxis.set_major_formatter(ScalarFormatter())\n", " ax.xaxis.set_minor_locator(NullLocator())\n", " ax.set_xlabel(\"Source rank (log scale)\")\n", " ax.set_ylabel(\"Documents\")\n", " ax.set_title(\"Core sources by Bradford's law (shaded: zone 1)\", loc=\"left\")\n", " for _, row in core.head(8).iterrows():\n", " ax.annotate(str(row[\"source\"])[:40], (row[\"rank\"], row[\"documents\"]), fontsize=6.5, xytext=(4, 2), textcoords=\"offset points\")\n", " fig.savefig(path)\n", " plt.close(fig)\n", "\n", "\n", "def _draw_lotka(plt, df, fit, path):\n", " if not len(df):\n", " return _empty(plt, \"Author productivity (Lotka's law)\", path, \"No author names in these records.\")\n", " fig, ax = plt.subplots(figsize=(6.3, 3.0))\n", " ax.plot(df[\"documents_written\"], df[\"share_of_authors\"] * 100, \"o-\", color=PALETTE[0], lw=1.2, ms=4, label=\"Observed\")\n", " ax.plot(df[\"documents_written\"], df[\"lotka_expected_share\"] * 100, \"--\", color=PALETTE[5], lw=1.0, label=\"Lotka (beta = 2)\")\n", " ax.set_xlabel(\"Documents written\")\n", " ax.set_ylabel(\"% of authors\")\n", " t = \"Author productivity (Lotka's law)\"\n", " if fit[\"beta\"] is not None:\n", " t += \" fitted beta = %.2f, C = %.3f\" % (fit[\"beta\"], fit[\"C\"])\n", " ax.set_title(t, loc=\"left\")\n", " ax.xaxis.get_major_locator().set_params(integer=True)\n", " ax.legend(frameon=False, fontsize=7)\n", " fig.savefig(path)\n", " plt.close(fig)\n", "\n", "\n", "def _draw_countries(plt, df, path):\n", " if not len(df):\n", " return _empty(plt, \"Countries\", path, \"No affiliations in these records.\")\n", " fig, ax = plt.subplots(figsize=(6.3, max(2.2, 0.28 * len(df) + 0.8)))\n", " d = df.iloc[::-1]\n", " ax.barh(d[\"country\"], d[\"single_country_publications\"], color=PALETTE[0], label=\"Single-country (SCP)\")\n", " ax.barh(d[\"country\"], d[\"multiple_country_publications\"], left=d[\"single_country_publications\"], color=PALETTE[1], label=\"Multiple-country (MCP)\")\n", " ax.set_title(\"Countries: single- and multiple-country publications\", loc=\"left\")\n", " ax.set_xlabel(\"Documents\")\n", " ax.tick_params(axis=\"y\", length=0)\n", " ax.legend(frameon=False, fontsize=7, loc=\"lower right\")\n", " fig.savefig(path)\n", " plt.close(fig)\n", "\n", "\n", "def _draw_trends(plt, df, path):\n", " if not len(df):\n", " return _empty(plt, \"Trend topics\", path, \"Too few repeated keywords to show trends.\")\n", " fig, ax = plt.subplots(figsize=(6.3, max(2.4, 0.24 * len(df) + 0.8)))\n", " d = df.iloc[::-1].reset_index(drop=True)\n", " top = d[\"frequency\"].max()\n", " for i, row in d.iterrows():\n", " ax.plot([row[\"year_q1\"], row[\"year_q3\"]], [i, i], color=\"#BBBBBB\", lw=1.2, zorder=1)\n", " ax.scatter([row[\"year_median\"]], [i], s=15 + 120 * row[\"frequency\"] / top, color=PALETTE[0], zorder=2)\n", " ax.set_yticks(range(len(d)))\n", " ax.set_yticklabels([str(k)[:40] for k in d[\"keyword\"]])\n", " ax.tick_params(axis=\"y\", length=0)\n", " ax.xaxis.get_major_locator().set_params(integer=True)\n", " ax.set_xlabel(\"Year (dot = median, line = 1st to 3rd quartile; size = frequency)\")\n", " ax.set_title(\"Trend topics\", loc=\"left\")\n", " fig.savefig(path)\n", " plt.close(fig)\n", "\n", "\n", "def _draw_thematic(plt, df, path):\n", " if not len(df):\n", " return _empty(plt, \"Thematic map\", path, \"Too few keyword co-occurrences to form themes.\")\n", " fig, ax = plt.subplots(figsize=(6.3, 5.0))\n", " mc, md = df[\"centrality\"].mean(), df[\"density\"].mean()\n", " ax.axvline(mc, color=\"#999999\", lw=0.6, ls=\"--\")\n", " ax.axhline(md, color=\"#999999\", lw=0.6, ls=\"--\")\n", " top = df[\"frequency\"].max()\n", " for i, row in df.iterrows():\n", " ax.scatter([row[\"centrality\"]], [row[\"density\"]], s=60 + 900 * row[\"frequency\"] / top,\n", " color=PALETTE[i % len(PALETTE)], alpha=0.55, edgecolor=\"white\", zorder=2)\n", " ax.margins(0.18)\n", " (x0, x1), (y0, y1) = ax.get_xlim(), ax.get_ylim()\n", " placed = []\n", " # Largest themes keep all three words; the rest show their first keyword (the\n", " # full lists are in the Themes sheet). Labels that would overlap are nudged up\n", " # with a leader line, so a crowded quadrant stays readable.\n", " for rank, (_, row) in enumerate(df.sort_values(\"frequency\", ascending=False).iterrows()):\n", " words = row[\"label\"].split(\"; \")\n", " text = \"\\n\".join(words if rank < 6 else words[:1])\n", " lines = text.count(\"\\n\") + 1\n", " w, h = 0.011 * max(len(t) for t in text.split(\"\\n\")), 0.034 * lines\n", " px, py = (row[\"centrality\"] - x0) / (x1 - x0), (row[\"density\"] - y0) / (y1 - y0)\n", " ly = py\n", " for _ in range(40):\n", " if all(abs(px - qx) > (w + qw) / 2 or abs(ly - qy) > (h + qh) / 2 for qx, qy, qw, qh in placed):\n", " break\n", " ly += 0.02\n", " placed.append((px, ly, w, h))\n", " moved = abs(ly - py) > 1e-9\n", " ax.annotate(text, (row[\"centrality\"], row[\"density\"]), xytext=(px, ly), textcoords=\"axes fraction\",\n", " fontsize=6.5, ha=\"center\", va=\"center\", zorder=3,\n", " arrowprops=dict(arrowstyle=\"-\", color=\"#999999\", lw=0.5) if moved else None)\n", " ax.set_xlabel(\"Relevance degree (Callon centrality)\")\n", " ax.set_ylabel(\"Development degree (Callon density)\")\n", " ax.set_title(\"Thematic map\", loc=\"left\")\n", " for x, y, text, ha, va in ((0.99, 0.99, \"Motor themes\", \"right\", \"top\"), (0.01, 0.99, \"Niche themes\", \"left\", \"top\"),\n", " (0.01, 0.01, \"Emerging or\\ndeclining themes\", \"left\", \"bottom\"), (0.99, 0.01, \"Basic themes\", \"right\", \"bottom\")):\n", " ax.text(x, y, text, transform=ax.transAxes, ha=ha, va=va, fontsize=7.5, color=\"#666666\")\n", " ax.set_xlim(x0, x1)\n", " ax.set_ylim(y0, y1)\n", " fig.savefig(path)\n", " plt.close(fig)\n", "\n", "\n", "def _draw_three_fields(plt, tf, path):\n", " cols = tf[\"columns\"]\n", " if not all(cols) or not any(tf[\"flows\"]):\n", " return _empty(plt, \"Three-field plot\", path, \"Not enough linked countries, authors and keywords.\")\n", " fig, ax = plt.subplots(figsize=(6.3, 5.4))\n", " ax.set_axis_off()\n", " ax.set_title(\"Three-field plot: countries, authors, keywords\", loc=\"left\")\n", " xs = [0.0, 1.0, 2.0]\n", " width = 0.06\n", " size = []\n", " for i, items in enumerate(cols):\n", " s = {}\n", " for it in items:\n", " left = sum(w for a, b, w in tf[\"flows\"][i - 1] if b == it) if i > 0 else 0\n", " right = sum(w for a, b, w in tf[\"flows\"][i] if a == it) if i < 2 else 0\n", " s[it] = max(left, right)\n", " size.append({k: v for k, v in s.items() if v > 0})\n", " scale = 1.0 / max(sum(s.values()) for s in size)\n", " gap = 0.02\n", " pos = []\n", " for i, s in enumerate(size):\n", " y, p = 1.0, {}\n", " for it in cols[i]:\n", " if it not in s:\n", " continue\n", " h = s[it] * scale * (1 - gap * len(s))\n", " p[it] = [y - h, y]\n", " ax.add_patch(plt.Rectangle((xs[i] - width / 2, y - h), width, h, color=PALETTE[i], zorder=2))\n", " ha = \"right\" if i == 0 else \"left\"\n", " dx = -width if i == 0 else width\n", " ax.text(xs[i] + dx, y - h / 2, str(it)[:28], ha=ha, va=\"center\", fontsize=6.5, zorder=3)\n", " y -= h + gap\n", " pos.append(p)\n", " t = np.linspace(0, 1, 40)\n", " smooth = t * t * (3 - 2 * t)\n", " for i in (0, 1):\n", " out_cursor = {k: v[1] for k, v in pos[i].items()}\n", " in_cursor = {k: v[1] for k, v in pos[i + 1].items()}\n", " for a, b, w in tf[\"flows\"][i]:\n", " if a not in pos[i] or b not in pos[i + 1]:\n", " continue\n", " h_out = w * scale * (1 - gap * len(pos[i]))\n", " h_in = w * scale * (1 - gap * len(pos[i + 1]))\n", " y0t, y1t = out_cursor[a], in_cursor[b]\n", " out_cursor[a] -= h_out\n", " in_cursor[b] -= h_in\n", " top = y0t + (y1t - y0t) * smooth\n", " bot = (y0t - h_out) + ((y1t - h_in) - (y0t - h_out)) * smooth\n", " x = xs[i] + width / 2 + (xs[i + 1] - xs[i] - width) * t\n", " ax.fill_between(x, bot, top, color=PALETTE[i], alpha=0.25, lw=0, zorder=1)\n", " for i, name in enumerate(tf[\"fields\"]):\n", " ax.text(xs[i], 1.04, name.title(), ha=\"center\", va=\"bottom\", fontsize=8, fontweight=\"bold\")\n", " ax.set_xlim(-0.75, 2.75)\n", " ax.set_ylim(-0.02, 1.08)\n", " fig.savefig(path)\n", " plt.close(fig)\n", "\n", "\n", "# ─── report ───────────────────────────────────────────────────────────────\n", "\n", "# Figures also written loose in out/ (the rest are in figures.zip and the .docx).\n", "LOOSE_FIGURES = (\"fig01_annual_production.png\", \"fig10_thematic_map.png\", \"fig13_keyword_network.png\")\n", "\n", "\n", "def _source_code():\n", " try:\n", " with open(os.path.abspath(__file__), encoding=\"utf-8\") as f:\n", " return f.read()\n", " except (OSError, NameError):\n", " return None\n", "\n", "\n", "def notebook(paths, title, lang, source=None):\n", " \"\"\"A Colab / Jupyter notebook that re-runs this report and opens every table, ready to extend.\"\"\"\n", " files = [os.path.basename(p) for p in paths]\n", "\n", " def md(text):\n", " return {\"cell_type\": \"markdown\", \"metadata\": {}, \"source\": text.strip(\"\\n\").splitlines(True)}\n", "\n", " def code(text):\n", " return {\"cell_type\": \"code\", \"metadata\": {}, \"execution_count\": None, \"outputs\": [],\n", " \"source\": text.strip(\"\\n\").splitlines(True)}\n", "\n", " cells = [\n", " md(\"\"\"\n", "# %s\n", "\n", "Bibliometric analysis generated by Pho (pho.chat) with pho_biblio %s. Run the cells in order:\n", "they install two packages, ask for your export files, re-run the full analysis and open the\n", "tables and figures. Every number comes from the code in this notebook; edit it freely.\n", "\n", "Files used: %s\n", "\"\"\" % (title, VERSION, \", \".join(files))),\n", " code(\"!pip -q install python-docx openpyxl\"),\n", " code(\"\"\"\n", "# Upload the same exports (skip if they are already next to this notebook).\n", "import os\n", "FILES = %s\n", "missing = [f for f in FILES if not os.path.exists(f)]\n", "if missing:\n", " try:\n", " from google.colab import files as colab_files\n", " colab_files.upload()\n", " except ImportError:\n", " print(\"Put these files next to the notebook:\", missing)\n", "\"\"\" % json.dumps(files, ensure_ascii=False)),\n", " ]\n", " if source:\n", " cells.append(code(\"%%writefile pho_biblio.py\\n\" + source))\n", " else:\n", " cells.append(md(\"Upload **pho_biblio.py** (included with the report) next to this notebook before the next cell.\"))\n", " cells += [\n", " code(\"\"\"\n", "import importlib, pho_biblio\n", "importlib.reload(pho_biblio)\n", "result = pho_biblio.report(FILES, out=\"out\", title=%s, lang=%s)\n", "\"\"\" % (json.dumps(title, ensure_ascii=False), json.dumps(lang))),\n", " code(\"\"\"\n", "import pandas as pd\n", "tables = pd.read_excel(\"out/bibliometric_tables.xlsx\", sheet_name=None)\n", "for name in [\"summary\", \"annual\", \"source_impact\", \"bradford\", \"author_impact\", \"lotka\", \"countries\",\n", " \"keywords\", \"trend_topics\", \"thematic_map\", \"most_cited\"]:\n", " print(\"\\\\n==\", name)\n", " display(tables[name].head(20))\n", "\"\"\"),\n", " code(\"\"\"\n", "import glob, zipfile\n", "from IPython.display import Image\n", "zipfile.ZipFile(\"out/figures.zip\").extractall(\"out/figures\")\n", "for path in sorted(glob.glob(\"out/figures/*.png\")):\n", " print(os.path.basename(path))\n", " display(Image(path, width=700))\n", "\"\"\"),\n", " md(\"\"\"\n", "## Go further\n", "\n", "**merged_records.csv** holds one row per document after merging duplicates. Start your own\n", "statistics from it, for example citations by year:\n", "\"\"\"),\n", " code(\"\"\"\n", "df = pd.read_csv(\"out/merged_records.csv\")\n", "df.groupby(\"year\")[\"cited_by\"].describe()\n", "\"\"\"),\n", " md(\"\"\"\n", "The ***_nodes** and ***_edges** sheets of **bibliometric_tables.xlsx** open in VOSviewer and Gephi.\n", "\"\"\"),\n", " ]\n", " return {\"cells\": cells, \"metadata\": {\"kernelspec\": {\"display_name\": \"Python 3\", \"language\": \"python\", \"name\": \"python3\"},\n", " \"language_info\": {\"name\": \"python\"}}, \"nbformat\": 4, \"nbformat_minor\": 5}\n", "\n", "\n", "def _doc_table(doc, df, columns=None, rows=None, headers=None):\n", " columns = columns or list(df.columns)\n", " d = df[columns] if len(df) else df\n", " if rows:\n", " d = d.head(rows)\n", " tbl = doc.add_table(rows=1, cols=len(columns))\n", " tbl.style = \"Light Grid Accent 1\"\n", " for i, c in enumerate(columns):\n", " tbl.rows[0].cells[i].text = (headers or {}).get(c, c.replace(\"_\", \" \"))\n", " for _, row in d.iterrows():\n", " cells = tbl.add_row().cells\n", " for i, c in enumerate(columns):\n", " v = row[c]\n", " cells[i].text = \"\" if v is None or (isinstance(v, float) and math.isnan(v)) else str(v)[:80]\n", " return tbl\n", "\n", "\n", "def report(paths, out=\"out\", title=None, lang=\"en\", top=TOP_N, nodes=NETWORK_NODES, min_edge=MIN_EDGE):\n", " \"\"\"Read the exports, merge duplicates, compute Biblioshiny-style indicators, write the report.\n", "\n", " Writes 9 files into out: bibliometric_report.docx, bibliometric_tables.xlsx,\n", " merged_records.csv, figures.zip (14 figures), 3 key figures as PNG,\n", " bibliometric_analysis.ipynb (opens in Google Colab) and pho_biblio.py.\n", " Returns a dict of the key numbers and prints it, so the caller (or a model)\n", " only reports numbers this computed.\n", " \"\"\"\n", " import tempfile\n", " import zipfile\n", "\n", " import pandas as pd\n", "\n", " if isinstance(paths, str):\n", " paths = [paths]\n", " L = LABELS.get(lang, LABELS[\"en\"])\n", " title = title or L[\"report\"]\n", " os.makedirs(out, exist_ok=True)\n", " plt = _mpl()\n", "\n", " records, per_file = merge(paths)\n", " if not records:\n", " raise ValueError(\"No records with a title were found in %s\" % \", \".join(paths))\n", " read = sum(f[\"records\"] for f in per_file)\n", " s = summary(records)\n", " cov = coverage(records)\n", " t = tables(records, top)\n", " t[\"author_impact\"] = impact(records, \"authors\", top)\n", " t[\"source_impact\"] = impact(records, \"sources\", top)\n", " t[\"bradford\"] = bradford(records)\n", " t[\"lotka\"], lotka_fit = lotka(records)\n", " t[\"trend_topics\"] = trend_topics(records)\n", " t[\"thematic_map\"] = thematic_map(records)\n", " tf = three_fields(records)\n", " nets = {\"authors\": network(records, \"authors\", nodes, min_edge),\n", " \"keywords\": network(records, \"keywords\", nodes, min_edge),\n", " \"countries\": network(records, \"countries\", nodes, 1)}\n", " by_db = Counter(db for r in records for db in r[\"databases\"])\n", " both = sum(1 for r in records if len(r[\"databases\"]) > 1)\n", "\n", " work = tempfile.mkdtemp(prefix=\"pho_biblio_\")\n", " figs = {}\n", "\n", " def fig(name, caption):\n", " figs[name] = (os.path.join(work, name), caption)\n", " return os.path.join(work, name)\n", "\n", " a = t[\"annual\"]\n", " if len(a):\n", " f, ax = plt.subplots(figsize=(6.3, 2.8))\n", " ax.bar(a[\"year\"], a[\"documents\"], color=PALETTE[0])\n", " ax.set_title(\"Annual scientific production\", loc=\"left\")\n", " ax.set_ylabel(\"Documents\")\n", " ax.xaxis.get_major_locator().set_params(integer=True)\n", " f.savefig(fig(\"fig01_annual_production.png\", \"Annual scientific production (documents per publication year).\"))\n", " plt.close(f)\n", " f, ax = plt.subplots(figsize=(6.3, 2.8))\n", " ax.plot(a[\"year\"], a[\"mean_citations_per_year\"], \"o-\", color=PALETTE[0], lw=1.2, ms=3)\n", " ax.set_title(\"Average citations per year\", loc=\"left\")\n", " ax.set_ylabel(\"Mean citations per document per year\")\n", " ax.xaxis.get_major_locator().set_params(integer=True)\n", " f.savefig(fig(\"fig02_average_citations_per_year.png\", \"Average citations per document, divided by the years each publication year has been citable.\"))\n", " plt.close(f)\n", " if len(t[\"sources\"]):\n", " _bar(plt, t[\"sources\"], \"source\", \"documents\", \"Most relevant sources\", fig(\"fig03_top_sources.png\", \"Sources (journals, proceedings) with the most documents.\"), \"Documents\")\n", " _draw_bradford(plt, t[\"bradford\"], fig(\"fig04_bradford.png\", \"Bradford's law: sources ranked by documents; zone 1 (shaded) holds about a third of all documents.\"))\n", " if len(t[\"authors\"]):\n", " _bar(plt, t[\"authors\"], \"author\", \"documents\", \"Most relevant authors\", fig(\"fig05_top_authors.png\", \"Authors with the most documents (full counting; names matched by family name and initials).\"), \"Documents\")\n", " _draw_lotka(plt, t[\"lotka\"], lotka_fit, fig(\"fig06_lotka.png\", \"Author productivity: share of authors who wrote 1, 2, 3... documents, against Lotka's inverse-square law.\"))\n", " _draw_countries(plt, t[\"countries\"], fig(\"fig07_countries.png\", \"Countries by documents with at least one affiliation there, split into single-country (SCP) and multiple-country (MCP) publications.\"))\n", " if len(t[\"keywords\"]):\n", " _bar(plt, t[\"keywords\"], \"keyword\", \"occurrences\", \"Most frequent keywords\", fig(\"fig08_top_keywords.png\", \"Author keywords (Keywords Plus / index keywords where a record has none).\"), \"Occurrences\")\n", " _draw_trends(plt, t[\"trend_topics\"], fig(\"fig09_trend_topics.png\", \"Trend topics: publication years of each frequent keyword (median and interquartile range).\"))\n", " _draw_thematic(plt, t[\"thematic_map\"], fig(\"fig10_thematic_map.png\", \"Thematic map: keyword clusters placed by Callon centrality (relevance) and density (development); bubble size = keyword occurrences.\"))\n", " _draw_three_fields(plt, tf, fig(\"fig11_three_field_plot.png\", \"Three-field plot: the 10 most frequent countries, authors and keywords; band width = documents linking both ends.\"))\n", " cl_a = _draw_network(plt, nets[\"authors\"], \"Co-authorship network\", fig(\"fig12_coauthorship_network.png\",\n", " \"Co-authorship among the %d most productive connected authors; edges = documents written together (>= %d); colours = clusters (label propagation).\" % (len(nets[\"authors\"][\"nodes\"]), min_edge)))\n", " cl_k = _draw_network(plt, nets[\"keywords\"], \"Keyword co-occurrence network\", fig(\"fig13_keyword_network.png\",\n", " \"Keyword co-occurrence among the %d most frequent connected keywords; edges = documents using both (>= %d); colours = clusters.\" % (len(nets[\"keywords\"][\"nodes\"]), min_edge)))\n", " cl_c = _draw_network(plt, nets[\"countries\"], \"Country collaboration network\", fig(\"fig14_country_collaboration.png\",\n", " \"Country collaboration: edges = documents with affiliations in both countries; colours = clusters.\"))\n", "\n", " with zipfile.ZipFile(os.path.join(out, \"figures.zip\"), \"w\", zipfile.ZIP_DEFLATED) as z:\n", " for name, (path, _) in sorted(figs.items()):\n", " z.write(path, name)\n", " for name in LOOSE_FIGURES:\n", " if name in figs:\n", " shutil.copy(figs[name][0], os.path.join(out, name))\n", "\n", " # Workbook: every table, node and edge lists for VOSviewer/Gephi.\n", " main_rows = [(k, v if not isinstance(v, dict) else json.dumps(v, ensure_ascii=False)) for k, v in s.items()]\n", " main_rows += [(\"lotka_beta\", lotka_fit[\"beta\"]), (\"lotka_C\", lotka_fit[\"C\"]), (\"lotka_r_squared\", lotka_fit[\"r_squared\"]),\n", " (\"bradford_core_sources\", int((t[\"bradford\"][\"zone\"] == 1).sum()) if len(t[\"bradford\"]) else 0)]\n", " records_df = pd.DataFrame([{\n", " \"title\": r[\"title\"], \"authors\": \"; \".join(r[\"authors\"]), \"year\": r[\"year\"], \"source\": r[\"source\"],\n", " \"doi\": r[\"doi\"], \"document_type\": r[\"doc_type\"], \"cited_by\": r[\"cited_by\"],\n", " \"author_keywords\": \"; \".join(r[\"author_keywords\"]), \"index_keywords\": \"; \".join(r[\"index_keywords\"]),\n", " \"countries\": \"; \".join(_countries(r)),\n", " \"databases\": \"; \".join(r[\"databases\"]),\n", " \"cited_by_scopus\": r.get(\"cited_by_Scopus\"),\n", " \"cited_by_wos\": r.get(\"cited_by_Web of Science\"),\n", " } for r in records])\n", " tf_rows = [{\"from_field\": tf[\"fields\"][i], \"from\": a_, \"to_field\": tf[\"fields\"][i + 1], \"to\": b_, \"documents\": w}\n", " for i in (0, 1) for a_, b_, w in tf[\"flows\"][i]]\n", " with pd.ExcelWriter(os.path.join(out, \"bibliometric_tables.xlsx\")) as xw:\n", " pd.DataFrame(main_rows, columns=[\"metric\", \"value\"]).to_excel(xw, sheet_name=\"summary\", index=False)\n", " pd.DataFrame(per_file).to_excel(xw, sheet_name=\"files\", index=False)\n", " pd.DataFrame(list(cov.items()), columns=[\"field\", \"coverage_pct\"]).to_excel(xw, sheet_name=\"field_coverage\", index=False)\n", " for name in (\"annual\", \"sources\", \"source_impact\", \"bradford\", \"authors\", \"author_impact\", \"lotka\", \"countries\",\n", " \"keywords\", \"trend_topics\", \"thematic_map\", \"most_cited\"):\n", " t[name].to_excel(xw, sheet_name=name, index=False)\n", " pd.DataFrame(tf_rows, columns=[\"from_field\", \"from\", \"to_field\", \"to\", \"documents\"]).to_excel(xw, sheet_name=\"three_field_flows\", index=False)\n", " for kind, cl in ((\"authors\", cl_a), (\"keywords\", cl_k), (\"countries\", cl_c)):\n", " pd.DataFrame([{\"source\": x, \"target\": y, \"weight\": w} for x, y, w in nets[kind][\"edges\"]],\n", " columns=[\"source\", \"target\", \"weight\"]).to_excel(xw, sheet_name=kind + \"_edges\", index=False)\n", " pd.DataFrame([{\"id\": n, \"label\": n, \"weight\": f, \"cluster\": cl.get(n, 0) + 1} for n, f in nets[kind][\"nodes\"]],\n", " columns=[\"id\", \"label\", \"weight\", \"cluster\"]).to_excel(xw, sheet_name=kind + \"_nodes\", index=False)\n", " records_df.to_csv(os.path.join(out, \"merged_records.csv\"), index=False)\n", "\n", " # Word report, in the order Biblioshiny's menu uses.\n", " from docx import Document\n", " from docx.shared import Inches\n", "\n", " doc = Document()\n", " doc.add_heading(title, 0)\n", " doc.add_paragraph(L[\"generated\"] % (date.today().isoformat(), VERSION))\n", "\n", " def figure(name):\n", " if name in figure.done or name not in figs:\n", " return\n", " figure.done.append(name)\n", " doc.add_picture(figs[name][0], width=Inches(6.0))\n", " doc.add_paragraph(\"Figure %d. %s\" % (len(figure.done), figs[name][1]))\n", " figure.done = []\n", "\n", " doc.add_heading(L[\"data\"], 1)\n", " _doc_table(doc, pd.DataFrame(per_file), [\"file\", \"database\", \"records\"], headers={\"file\": L[\"file\"], \"database\": L[\"database\"], \"records\": L[\"records\"]})\n", " doc.add_paragraph(\n", " \"%d records read; %d duplicates merged (same DOI, else same normalised title and year); %d unique documents analysed. \"\n", " \"By database after merging: %s. Documents in more than one database: %d.\"\n", " % (read, read - len(records), len(records), \", \".join(\"%s %d\" % kv for kv in by_db.most_common()), both)\n", " )\n", "\n", " doc.add_heading(L[\"main\"], 1)\n", " main = pd.DataFrame([(k.replace(\"_\", \" \"), \"n/a\" if v is None else (\", \".join(\"%s %d\" % kv for kv in v.items()) if isinstance(v, dict) else v))\n", " for k, v in s.items()], columns=[\"metric\", \"value\"])\n", " _doc_table(doc, main, headers={\"metric\": L[\"metric\"], \"value\": L[\"value\"]})\n", " doc.add_paragraph(L[\"coverage\"])\n", " _doc_table(doc, pd.DataFrame(list(cov.items()), columns=[\"field\", \"%\"]), headers={\"field\": L[\"field\"]})\n", "\n", " doc.add_heading(L[\"production\"], 1)\n", " figure(\"fig01_annual_production.png\")\n", " figure(\"fig02_average_citations_per_year.png\")\n", " _doc_table(doc, a, [\"year\", \"documents\", \"citations\", \"mean_citations_per_document\", \"mean_citations_per_year\"])\n", "\n", " doc.add_heading(L[\"sources\"], 1)\n", " figure(\"fig03_top_sources.png\")\n", " doc.add_paragraph(L[\"source_impact\"])\n", " _doc_table(doc, t[\"source_impact\"], rows=10)\n", " figure(\"fig04_bradford.png\")\n", " core = t[\"bradford\"][t[\"bradford\"][\"zone\"] == 1] if len(t[\"bradford\"]) else t[\"bradford\"]\n", " doc.add_paragraph(L[\"bradford\"] % (len(core), len(t[\"bradford\"])))\n", " _doc_table(doc, core, [\"rank\", \"source\", \"documents\", \"cumulative_documents\"], rows=20)\n", "\n", " doc.add_heading(L[\"authors\"], 1)\n", " figure(\"fig05_top_authors.png\")\n", " doc.add_paragraph(L[\"author_impact\"])\n", " _doc_table(doc, t[\"author_impact\"], rows=10)\n", " figure(\"fig06_lotka.png\")\n", " if lotka_fit[\"beta\"] is not None:\n", " doc.add_paragraph(L[\"lotka\"] % (lotka_fit[\"authors\"], lotka_fit[\"beta\"], lotka_fit[\"C\"], lotka_fit[\"r_squared\"] if lotka_fit[\"r_squared\"] is not None else float(\"nan\")))\n", " _doc_table(doc, t[\"lotka\"], rows=10)\n", " figure(\"fig07_countries.png\")\n", " _doc_table(doc, t[\"countries\"], [\"country\", \"documents\", \"single_country_publications\", \"multiple_country_publications\", \"mcp_ratio\", \"citations\"], rows=10,\n", " headers={\"single_country_publications\": \"SCP\", \"multiple_country_publications\": \"MCP\", \"mcp_ratio\": \"MCP ratio\"})\n", " figure(\"fig14_country_collaboration.png\")\n", "\n", " doc.add_heading(L[\"documents\"], 1)\n", " _doc_table(doc, t[\"most_cited\"], [\"first_author\", \"year\", \"title\", \"source\", \"cited_by\"], rows=10)\n", "\n", " doc.add_heading(L[\"conceptual\"], 1)\n", " figure(\"fig08_top_keywords.png\")\n", " figure(\"fig09_trend_topics.png\")\n", " figure(\"fig10_thematic_map.png\")\n", " _doc_table(doc, t[\"thematic_map\"], [\"cluster\", \"label\", \"quadrant\", \"keywords\", \"frequency\", \"centrality\", \"density\"])\n", " figure(\"fig13_keyword_network.png\")\n", " figure(\"fig11_three_field_plot.png\")\n", "\n", " doc.add_heading(L[\"social\"], 1)\n", " figure(\"fig12_coauthorship_network.png\")\n", "\n", " doc.add_heading(L[\"methods\"], 1)\n", " doc.add_paragraph(\n", " \"Records were exported from %s and read with pho_biblio %s. Duplicates were merged on DOI, or on normalised title \"\n", " \"and publication year when a DOI was missing; a merged record keeps the first file's fields and fills gaps from the others. \"\n", " \"Counts use full counting (each document counts once for every author, country and keyword it carries). Authors are matched \"\n", " \"by family name and initials, without ORCID disambiguation. Countries are read from the last part of each affiliation; a \"\n", " \"multiple-country publication (MCP) has affiliations in more than one country. Keywords are author keywords, with index \"\n", " \"keywords (Scopus) or Keywords Plus (Web of Science) used only when a record has none; indexer check tags such as 'human', 'article' or 'priority journal' are left out of index keywords. Annual growth rate is the compound \"\n", " \"rate between the first and last publication year; average citations per year divide a year's mean citations by the years \"\n", " \"since publication (current year included).\"\n", " % (\" and \".join(sorted(by_db)), VERSION)\n", " )\n", " doc.add_paragraph(\n", " \"Local impact uses the citation counts in the exports: h-index (h documents with at least h citations each), g-index \"\n", " \"(the top g documents have at least g squared citations together) and m-index (h divided by years since the first document \"\n", " \"here). Bradford zones split the sources, ranked by documents, where the cumulative count passes one third and two thirds \"\n", " \"of all documents. Lotka's law is fitted by least squares on log10(share of authors) against log10(documents written). \"\n", " \"Trend topics keep keywords used in at least %d documents and show the quartiles of their publication years. The thematic \"\n", " \"map follows Cobo et al. (2011): co-occurrence of the 250 most frequent keywords, normalised by the equivalence index, \"\n", " \"clustered by modularity (Louvain local moving), with Callon centrality (10 x external links) and density (100 x internal \"\n", " \"links / keywords), quadrants split at the means. Networks keep the %d most frequent items and edges seen in at least %d \"\n", " \"documents, are laid out with the Fruchterman-Reingold algorithm (fixed seed) and clustered by weighted label propagation.\"\n", " % (5 if len(records) >= 100 else 2, nodes, min_edge)\n", " )\n", "\n", " doc.add_heading(L[\"limits\"], 1)\n", " gaps = [k for k, v in cov.items() if v < 80]\n", " doc.add_paragraph(\n", " \"Citation counts are those each database reported on the export date; Scopus and Web of Science count differently, so a merged \"\n", " \"record keeps the first count found and the workbook keeps both. h, g and m indexes are local to this collection, not an author's \"\n", " \"career indexes. Name matching can merge two people who share a family name and initials, or split one person written two ways. \"\n", " \"Results can differ slightly from bibliometrix/Biblioshiny where the two tools normalise names, countries or clusters differently. %s\"\n", " % (\"Fields under 80%% coverage, which weaken the related figures: %s.\" % \", \".join(gaps) if gaps else \"Every field used is present in at least 80% of records.\")\n", " )\n", "\n", " doc.add_heading(L[\"rerun\"], 1)\n", " doc.add_paragraph(L[\"rerun_text\"])\n", " doc.save(os.path.join(out, \"bibliometric_report.docx\"))\n", "\n", " src = _source_code()\n", " with open(os.path.join(out, \"bibliometric_analysis.ipynb\"), \"w\", encoding=\"utf-8\") as f:\n", " json.dump(notebook(paths, title, lang, src), f, ensure_ascii=False, indent=1)\n", " if src is not None:\n", " with open(os.path.join(out, \"pho_biblio.py\"), \"w\", encoding=\"utf-8\") as f:\n", " f.write(src)\n", " shutil.rmtree(work, ignore_errors=True)\n", "\n", " def head(df, n, cols=None):\n", " d = df[cols] if (cols and len(df)) else df\n", " return d.head(n).to_dict(\"records\")\n", "\n", " result = {\n", " \"files\": per_file,\n", " \"records_read\": read,\n", " \"duplicates_merged\": read - len(records),\n", " \"documents\": len(records),\n", " \"by_database\": dict(by_db),\n", " \"in_more_than_one_database\": both,\n", " \"summary\": s,\n", " \"field_coverage_pct\": cov,\n", " \"top_sources\": head(t[\"sources\"], 5),\n", " \"source_impact\": head(t[\"source_impact\"], 5),\n", " \"bradford_core_sources\": head(core, 10, [\"source\", \"documents\"]),\n", " \"top_authors\": head(t[\"authors\"], 5),\n", " \"author_impact\": head(t[\"author_impact\"], 5),\n", " \"lotka\": lotka_fit,\n", " \"top_countries\": head(t[\"countries\"], 5, [\"country\", \"documents\", \"multiple_country_publications\", \"mcp_ratio\", \"citations\"]),\n", " \"top_keywords\": head(t[\"keywords\"], 10),\n", " \"trend_topics\": head(t[\"trend_topics\"], 15, [\"keyword\", \"frequency\", \"year_median\"]),\n", " \"themes\": head(t[\"thematic_map\"], 10, [\"label\", \"quadrant\", \"frequency\", \"centrality\", \"density\"]),\n", " \"most_cited\": head(t[\"most_cited\"], 5, [\"first_author\", \"year\", \"title\", \"cited_by\"]),\n", " \"network_sizes\": {k: {\"nodes\": len(v[\"nodes\"]), \"edges\": len(v[\"edges\"])} for k, v in nets.items()},\n", " \"figures\": sorted(figs),\n", " \"written\": sorted(os.listdir(out)),\n", " }\n", " print(json.dumps(result, ensure_ascii=False, indent=1, default=str))\n", " return result" ] }, { "cell_type": "code", "metadata": {}, "execution_count": null, "outputs": [], "source": [ "import importlib, pho_biblio\n", "importlib.reload(pho_biblio)\n", "result = pho_biblio.report(FILES, out=\"out\", title=\"Real public exports: test run\", lang=\"en\")" ] }, { "cell_type": "code", "metadata": {}, "execution_count": null, "outputs": [], "source": [ "import pandas as pd\n", "tables = pd.read_excel(\"out/bibliometric_tables.xlsx\", sheet_name=None)\n", "for name in [\"summary\", \"annual\", \"source_impact\", \"bradford\", \"author_impact\", \"lotka\", \"countries\",\n", " \"keywords\", \"trend_topics\", \"thematic_map\", \"most_cited\"]:\n", " print(\"\\n==\", name)\n", " display(tables[name].head(20))" ] }, { "cell_type": "code", "metadata": {}, "execution_count": null, "outputs": [], "source": [ "import glob, zipfile\n", "from IPython.display import Image\n", "zipfile.ZipFile(\"out/figures.zip\").extractall(\"out/figures\")\n", "for path in sorted(glob.glob(\"out/figures/*.png\")):\n", " print(os.path.basename(path))\n", " display(Image(path, width=700))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Go further\n", "\n", "**merged_records.csv** holds one row per document after merging duplicates. Start your own\n", "statistics from it, for example citations by year:" ] }, { "cell_type": "code", "metadata": {}, "execution_count": null, "outputs": [], "source": [ "df = pd.read_csv(\"out/merged_records.csv\")\n", "df.groupby(\"year\")[\"cited_by\"].describe()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The ***_nodes** and ***_edges** sheets of **bibliometric_tables.xlsx** open in VOSviewer and Gephi." ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "name": "python" } }, "nbformat": 4, "nbformat_minor": 5 }