Australian-Text-Analytics-Platform/ldaca-wordflow: Wordflow 0.7.11

Abstract

Wordflow 0.7.11 lets you set the order of a category column's values and use it everywhere, including as a Trends axis; converts dates, numbers and text more reliably, checking the whole column first; adds several sheets of a spreadsheet at once; adds an Elapsed time type for transcript timestamps; shows a quick overview of a corpus in the Data Editor; makes Topic Modelling finish on large corpora and shows example texts of each topic; it also fixes Concordance's word-before column at the start of a text, sorts empty values to the bottom of every table, makes range filters behave the same for numbers and dates, and lets you add part of a folder to a Project in one step. Tokenisers Plain words for any language: the tokeniser formerly called "Plain words (English)" is now Plain words, marked Any language with spaces: it splits text at spaces and punctuation, so it is a baseline for French, Indonesian and any other language that separates words with spaces, and Wordflow now picks it automatically for them. Chinese, Japanese and Korean keep their own tokenisers. Concordance L1 at the start of a text: when a match is the first word of a text, L1 (the word just before it) is now empty, as R1 already was at the end of a text. Before, L1 showed the matched word itself, which also affected L1 counts and sorting. Text mode only; Tokens mode was right. L1 and R1 colouring in Tokens mode marks the whole word, not the same letters inside another word, for example the final A rather than the a in cat. Ignore punctuation from a word click too: opening Concordance by clicking a word in Frequency now has Ignore punctuation on, as a new Concordance tab does, so the context columns hold words and sorting is not split by punctuation. Preview says how much it searched: a Preview checks one page of documents. Its footer now says, for example, "after checking 20 of 26,163 documents" and that Run searches them all, so no matches in a Preview is not read as none in the corpus. Download the results directly: the download icon beside Add to Project saves the Concordance table as CSV or Excel, with the same column choices, in the order the table shows. It works the same in Quotation, and the Annotation tables and the Data Editor have a download icon too, so there is no need to add a table to the Project and export it. Category columns Set the order of a category: choosing Category for a column opens a window listing its values. Start from A to Z (numbers inside values count as numbers, so Q9 comes before Q10) or, for numbers and dates, Smallest first; or choose Z to A, Largest first, Most rows first or Fewest rows first, or drag the values into your own order (the keyboard works too). Empty values always come last. A column with many different values, or the document column, asks before Wordflow reads and lists them all, so a mistaken click can be cancelled. The window lists up to 300 values; any others follow in the chosen order. Choose Category again to change the order. The order is used everywhere: table sorting, Filter's value list, Group, Trends groups, and the Topic Map colour legend. Trends by category: a category column can be the Trends axis, with one bar per value in its order and empty values as the last bar. Stack combines category columns whose orders differ, and says which order the result uses. Editing a category, number or date column with Find & replace or Clean text into the same column makes it text; Wordflow now says so, and Undo restores it. Parquet files that mark a category as ordered: Wordflow can't read that order and now says so, so you can set it. Converting column types One conversion window: changing a column from text to a date, a date and time or a number, from numbers to a date, or from a date to text, opens a window that shows what the values become and checks the whole column first, for example 2,380 of 2,400 values convert; 20 would become empty, with the first values that don't convert. More date formats: Wordflow lists the formats that fit the column, including dates with times and time zones, month names, Twitter dates and email dates, and dates inside longer text such as the file name 2021_01_17_SmithInterview. Dates such as 03/01/2020 are read day first; Read month first switches with one click. No format codes needed: each part of a value (30, 01, 2020) has a menu saying what it is, such as Day, Month or Year, or Ignore for text around the date. The code is still there for anyone who prefers to type it. Two-digit years: for dates such as 30/01/20, you choose the century (1900s, 2000s, or a split year); Wordflow never guesses. Numbers to dates: Unix time (seconds, milliseconds, microseconds or nanoseconds) and Excel day numbers can become dates. Text to numbers: choose the decimal mark and thousands separator, and ignore currency signs and %, so $1,234.50 becomes 1234.5. Dates to text: choose how the dates are written, such as 30 January 2020, or build your own. Whole numbers round to the nearest: converting decimals to a whole number now rounds (2.5 becomes 3) instead of cutting off the decimals, and text such as 2.5 converts instead of becoming empty. Elapsed time for transcripts A new type, Elapsed time, for time into a recording, such as the start of each line of a transcript. It shows as 7:58.5, or 1:23:20.25 from an hour on. Spreadsheets: a column of times without a date (Excel cells formatted like mm:ss.0, which used to load as dates in December 1899) now loads as elapsed time, and Wordflow says so. Converting: text such as 7:58, 07:58.5, 1:23:20 or subtitle-style 00:07:58,542 becomes elapsed time (a time with two parts is minutes and seconds unless you choose hours and minutes); so do numbers of milliseconds, seconds, minutes, hours or days, and the time of day of a date and time. Elapsed time becomes text in several styles, or a number of seconds, minutes or hours. Using it: sort by it, filter it by typing times like 7:58, group it into blocks of minutes, total or average it per group (for example each speaker's speaking time), and plot it in Trends per second, minute, hour or a custom step. CSV exports write it as text, and Excel exports as Excel times. Topic Modelling Large corpora finish: a corpus of 26,000 news articles used to run for hours without finishing; it now takes about 12 minutes on a recent Mac, and a second run on the same text under a minute. Faster on every core: Topic Modelling now runs its embedding model on all processor cores on Mac and Windows, as it already did on Linux (Core ML on Mac was about 5 times slower for this model), and clustering now uses every core too. Small computers too: on a computer or virtual machine with few processors (for example 2), Topic Modelling now reads the text on all of them instead of one; in containers such as Binder it keeps to the processors it is allowed. Automatic segments pack short paragraphs: neighbouring paragraphs now share one segment while together they fit Max tokens, so a news article of many one-line paragraphs gives a few full segments instead of dozens of tiny ones (about 5 times fewer segments on that corpus). A paragraph is never cut to fill a segment; a paragraph too long for one segment is split into its sentences, packed the same way. Choose Paragraph to keep one paragraph per segment; a paragraph too long for one segment is now cut at the sentence boundary nearest its middle (and again if needed), instead of into one segment per sentence. Max topic size Auto judges by documents and explains itself: Auto now steps in when one topic is the main topic of more than half of the documents (before, more than half of the segments, which depends on the segmentation mode). The line under the settings says what it decided, for example that one topic was the main topic of 83% of documents and was split, or that splitting it would have left most of its text without a topic. After a run with Auto, the Max topic size field shows in grey the size Auto worked with, so you know which way to change it; Tab fills it in. Topic sampling: for very large corpora, a new checkbox finds the topics from a sample of segments picked with the Seed, then gives every other segment the topic of the sampled segment most similar to it, so every document still gets topics. It is off by default, and the note suggests it when a corpus has very many segments. Segments to sample shows a suggested size in grey (Tab fills it in to change it), and the note explains the trade-off: clustering time grows with the square of the sample, while a smaller sample can miss small topics. Topics on the right rows: with a Data Block made by Join, Add to Project could give a row the topics of another document, so a one-line text showed several topics. Join now keeps one row order, and Wordflow checks that the documents still match the run. To fix a Data Block made this way, choose Add to Project again from the same Topic Modelling result; there is no need to run Topic Modelling again. Progress you can follow: a long run now says which step it is on. In Tasks you see how far that step has got (for example "41,181 of 281,242 segments, about 5 min left") and how many processors it is using, so a slow computer no longer looks stuck at 10%. The Topic Modelling page shows one calm line with a link to the live progress in Tasks. Quotation and tokenising a large corpus report the same way. Example texts of each topic: click the eye at the right of a topic in the list, or right-click its bubble, to read the topic's texts beside the list. Most typical shows the segments closest to the centre of the topic first, Random a fixed random selection, and One per document (on by default) one segment per document. A badge such as Top 10% says how typical a segment is within its topic, and Open document shows the whole document, with the segment in bold and the topic's other segments highlighted. With Colour by on, the examples can