Archives of other dead forums

[Psephomancy]

In addition to hosting the archive of forum.electionscience.org, it would be good to host the archives of election-related Yahoo Groups (and maybe Google Groups, if CES group is going away?)

• [EM] Dumping election-methods Yahoo groups before the Dec 14 deadline
  Kristofer Munsterhjelm
• Reddit: All election-related Yahoo Groups will be deleted in 24 hours

I got some of the data from each of these, but I haven't gone through to see how complete they are:

• ApprovalVoting [Citizens For Approval Voting]
• AR-NewsWI ["Animal Right News- Wisconsin", not sure why listed]
• AVFA [Approval Voting Free Association]
• btpnc-talk [Libertarian Boston Tea Party Free Association, not sure why listed]
• Condorcet [Membership approved]
• electionmethods
• EMIG-Wikipedia [Wikipedia Election Methods Interest Group]
• instantrunoff-freewheeling
• InstantRunoffCA [Membership approved]
• InstantRunoffWI
• RangeVoting [Automatically rejected]
• stv-voting

Total size of my dumps are probably <1.5 GB with all the duplicated content removed.

Kristofer listed a few more groups that he downloaded. ArchiveTeam maybe got some that were not listed? Maybe others are floating around out there?

Unfortunately, I think some of these were lost forever, because they were accessible only to members, and no members archived them (unless they did so without posting about it).

[Jack Waugh]

Reconciling different formats and relational schemata and schemes of identification should be interesting, but probably possible.

The archive from Discourse embodies a strict hierarchy. At the root, it has "categories", and with a category, "topics", and within a topic, posts. There are no multiples of tags or categories associated to anything. What data schema would unify the archives coming from different origins?

What is the most human-friendly way to URL-encode arbitrary trees of constraints linked with logical operators for a query?

[Psephomancy]

@Jack-Waugh said in Archives of other dead forums:

What data schema would unify the archives coming from different origins?

Do they need to be in a unified format?

I've gone through all my files and removed duplicates, made a list of which contain what, etc. The dumps I have are in two different formats:

• Official dumps generated from the getmydata URL
• Scraped data using the yahoo-group-archiver tool

The getmydata dumps all contain messages, and most contain links and files as well. There are apparently bugs that make these dumps incomplete, however. (reddit, github)

Structure is:

• Messages
  • Series of .mbox files, each containing many messages
• Links
  • .url files and folders of .url files
• Files
  • Folders of uploaded files, usually one folder per user or project
  • Doesn't include attachments

The yahoo-group-archiver didn't do well in groups I wasn't a member of. The ApprovalVoting group contains nothing except the About description, for instance. In other groups it got more than the getmydata link did.

Structure is described on ArchiveTeam along with list of viewers:

• about
  • .json files and photo
• attachments
  • Folders of files, with .json metadata in each
• calendar
  • .json files
• databases
  • .json and .csv files
• email
  • .json files for metadata of many messages
  • .json files, one per message
• files
  • Same folder names and files as "Files" above, though each also contains .json metadata
• links
  • .json files
• members
  • .json files
• photos
  • Image files and .json metadata, don't seem to be included in "Files" of other format.
• polls
  • .json files
• topics
  • .json files for metadata of many messages
  • .json files, one per message
  • Folders containing attachments

These can be converted into .eml or .mbox files using Yahoo Groups Archive Tools

[Jack Waugh]

Sounds like a lot of work, if the messages are to be presented via a browser.

[Marylander]

@Jack-Waugh In that case we might want to figure out which of these forums are most worth the effort of archiving. (At least, if each forum would require separate work.)

If some of these forums weren't open to the public we might want to be careful about just publishing it though.

[Jack Waugh]

@Marylander They would require separate work iff they are in separate formats.

Maybe we need to ask stakeholders for a rating of the value of each type of archive, grouping them by format if any two of them have the same format.

[Psephomancy]

ArchiveTeam also got a few of them:

• AVFA
• electionmethods
• instantrunoff-freewheeling
• instantrunoffca-news
• InstantRunoffWI
• RangeVoting
• stv-voting

You can search for others at https://archive.org/details/archiveteam_yahoogroups

Some other groups have been converted to HTML for browsing on the web.

[Psephomancy]

Kristofer set up a browseable archive:

https://munsterhjelm.no/km/yahoo_lists_archive/

The forums I've archived have at least one message with at least one of
the terms "center squeeze", "Condorcet", "d'Hondt", "favorite betrayal",
"monotonicity", "Range voting", "Ranked Pairs", Sainte Lague", "Schulze
method" or "Score voting".

The browseable parts have the /web/ foldername:

https://munsterhjelm.no/km/yahoo_lists_archive/sd-2/web/2005-April/by-date.html

https://munsterhjelm.no/km/yahoo_lists_archive/sd-2/web/2005-April/msg00013.html