SemanText

SemanText is a free web-based toolkit for building and analyzing Indonesian-language corpora. It is designed for linguists, linguistics students, and lecturers who need a reliable way to assemble a corpus, inspect its vocabulary, and run basic quantitative analyses without installing software or paying for processing time.

All analysis runs in the browser. Your text never has to leave your machine, which makes SemanText suitable for working with sensitive or unpublished materials.

• Keyword ingestion: collect article URLs from news search and scrape them automatically.
• URL list ingestion: scrape a pasted list of article URLs.
• CSV ingestion: merge corpora exported by SemanText or produced elsewhere.
• Raw text ingestion: paste text or upload .txt files, one document per submission, up to 100,000 tokens.
• Word frequency: stopword-filtered frequency lists with charts.
• N-gram analysis: configurable multi-word frequency tables.
• Concordance (KWIC): keyword-in-context retrieval with a configurable window.
• Corpus overview and export: document, token, and source counts, with CSV export for every result.

SemanText is a refactor of the originalSemanText Streamlit prototype (2023)by MW Hidayat. The refactor replaces the Python server application with a static Astro site on Cloudflare Pages, ingests articles through Cloudflare Pages Functions, and keeps every analysis function in the browser.

The refactoring was developed with assistance from generative AI coding tools. The human author reviewed and tested all code and takes full responsibility for it, and development is validated by an automated test suite covering tokenization, frequency counts, concordance output, article extraction, and CSV round-trips (repository,npm test).

The project is open source under the MIT license and runs entirely on Cloudflare's free tier atsemantext.pages.dev.

• Scraping is best-effort. Bot-protected portals and JavaScript-rendered pages cannot be scraped from the free edge; such rows are marked blocked or empty rather than silently skipped.
• Dates are best-effort. Publication dates come from page metadata; absent metadata yields N/A.
• No POS tagging, lemmatization, or syntax-based collocations yet. The prototype's Stanza collocation feature was dropped because PyTorch models cannot run on the free Cloudflare or browser stack. Word frequency and n-grams are token-based only.
• Raw text is capped at 100,000 tokens per submission to keep the browser responsive.

The next phase of SemanText adds linguistically rich analysis. Neural models need compute that a free static site cannot provide, so these features are planned as optional, opt-in processors (most likely server-side add-ons) whose output is clearly disclosed to the user and never silently mixed with browser-local results.

1. 01

   Stanza integrationplanned
   Part-of-speech tagging, lemmatization, and dependency parsing for Indonesian via the Stanza pipeline, exposed as an optional processor for users who opt in to server-side analysis.

2. 02

   spaCy modelsplanned
   Indonesian spaCy pipelines for named-entity recognition, POS tagging, and morphological features, following the same opt-in processing model.

3. 03

   Other open-source modelsplanned
   Lightweight transformer models (for example via transformers.js) and statistical models for topic modeling, summarization, and sentiment analysis, running where possible in the browser and otherwise behind opt-in processing.

4. 04

   Statistical collocation measuresplanned
   Pointwise mutual information and related association measures for n-grams, restoring the statistical direction the original project planned but could not ship.

5. 05

   Corpus portabilityplanned
   Structured export of analysis outputs (TSV/CSV, optional CoNLL-U from Stanza runs) and import of existing corpus formats.

SemanText is released under theMIT license. Issues, feature requests, and research questions are welcome via the GitHub repository.