Logo

Z-i-V Scraper

Register Log In Back To Forums

Post #1 · Posted at 2016-12-25 12:25:39am 7.2 years ago

Offline AngledLuffa
AngledLuffa Avatar Member
111 Posts
United States
Reg. 2015-05-14


Last updated: 2016-12-25 12:25am
This forum seems mostly dead, but nevertheless it is the best fit for a project I've been working on.

I wrote a small program to scrape simfile categories from Z-i-V. Basically, you give it a category number and optionally a prefix to filter for, and it downloads the matching simfiles. It unzips the downloaded files in almost all cases (there are some cases where it can't figure out how). If you run it more than once in the same directory, it only downloads new simfiles.

The thing I use it for most is to download the current week from a simfile contest.

I don't see any robots.txt file that asks bots to stay away, but it's still worth asking if this is okay. If there's nothing wrong with using this kind of script, would it be suitable for a larger audience?

Also, is there any API on Z-i-V that returns the categories which are available or the songs available in a category? Right now it's getting the song list from a category by reading the webpage for that category. Similarly, I would like to add the ability to interactively choose which category to download, and that probably involves reading the simfiles homepage. If there's any way to get that information directly from Z-i-V, that would improve the reliability of the script and reduce the load on Z-i-V's servers.

The script is publicly available on github, and I'll be happy to share it if it's kosher.

It seems like there are no objections, so I added a cheap GUI to the script and posted it here:

https://github.com/AngledLuffa/stepmania-tools/tree/master/ziv

There's a text interface in scrape_category.py, and there's a GUI in scrape_interface.py. If there's anything missing from the documentation, please let me know so I can update the files.

Post #2 · Posted at 2016-12-25 12:33:11am 7.2 years ago

Offline Sigrev2
Sigrev2 Avatar Member+
4,159 Posts
United States
Reg. 2009-10-17

Nintendo Network ID: Sigrev2Nintendo Switch Friend Code: SW-2884-7660-37993DS Friend Code: 3883-7652-3160
"suffering from success"
>Z-i-V
c'mon, lad

Post #3 · Posted at 2016-12-25 12:40:52am 7.2 years ago

Offline GadgetJax
GadgetJax Avatar Member
757 Posts
United States
Reg. 2015-09-20

Quote: AnonyWolf
>Z-i-V
c'mon, lad

Did somebody mention me?

Post #4 · Posted at 2016-12-25 01:04:11am 7.2 years ago

Offline Loodee
Loodee Avatar Member+
275 Posts
Sweden
Reg. 2014-06-26

this thing's nice, gj
https://i.imgur.com/0PfGokW.pnghttps://i.imgur.com/d4gHmul.png

Post #5 · Posted at 2016-12-25 01:27:00am 7.2 years ago

Offline AngledLuffa
AngledLuffa Avatar Member
111 Posts
United States
Reg. 2015-05-14

Quote: Loodee
this thing's nice, gj

Glad to hear it's useful!

Quote: AnonyWolf
Quote: AngledLuffa
Z-i-V
c'mon, lad

Feel free to submit a pull request Smile
Register Log In Back To Forums

0 User(s) Viewing This Thread (Past 15 Minutes)

©2006-2024 Zenius -I- vanisher.com -5th style- IIPrivacy Policy
Web Server: 4% · Database: 4% · Server Time: 2024-03-29 14:02:14
This page took 0.006 seconds to execute.
Theme: starlight · Language: englishuk
Reset Theme & Language