Logo

Z-i-V Scraper

Register Log In Back To Forums

Post #1 · Posted at 2016-12-25 12:25:39am 2.9 years ago

Offline AngledLuffa
AngledLuffa Avatar Member
103 Posts
United States
Reg. 2015-05-14
Log in to send this user a private message

Last updated: 2016-12-25 12:25am
This forum seems mostly dead, but nevertheless it is the best fit for a project I've been working on.

I wrote a small program to scrape simfile categories from Z-i-V. Basically, you give it a category number and optionally a prefix to filter for, and it downloads the matching simfiles. It unzips the downloaded files in almost all cases (there are some cases where it can't figure out how). If you run it more than once in the same directory, it only downloads new simfiles.

The thing I use it for most is to download the current week from a simfile contest.

I don't see any robots.txt file that asks bots to stay away, but it's still worth asking if this is okay. If there's nothing wrong with using this kind of script, would it be suitable for a larger audience?

Also, is there any API on Z-i-V that returns the categories which are available or the songs available in a category? Right now it's getting the song list from a category by reading the webpage for that category. Similarly, I would like to add the ability to interactively choose which category to download, and that probably involves reading the simfiles homepage. If there's any way to get that information directly from Z-i-V, that would improve the reliability of the script and reduce the load on Z-i-V's servers.

The script is publicly available on github, and I'll be happy to share it if it's kosher.

It seems like there are no objections, so I added a cheap GUI to the script and posted it here:

https://github.com/AngledLuffa/stepmania-tools/tree/master/ziv

There's a text interface in scrape_category.py, and there's a GUI in scrape_interface.py. If there's anything missing from the documentation, please let me know so I can update the files.

Post #2 · Posted at 2016-12-25 12:33:11am 2.9 years ago

Offline AnonyWolf
AnonyWolf Avatar Member+
3,505 Posts
United States
Reg. 2009-10-17
Log in to send this user a private messageSkype: slimetimeplayerDiscord: DJ Vortivask #6276Twitter: Sigrev2SoundcloudTumblrYouTube
Xbox Gamer Tag: Sigrev2PlayStation Network: Sigrev2Nintendo Network ID: Sigrev2Nintendo Switch Friend Code: SW-2884-7660-3799Steam: Sigrev23DS Friend Code: 3883-7652-3160
"BEST TEETH IN THE GAME"
>Z-i-V
c'mon, lad

Post #3 · Posted at 2016-12-25 12:40:52am 2.9 years ago

Offline GadgetJax
GadgetJax Avatar Member
757 Posts
United States
Reg. 2015-09-20
Log in to send this user a private messageTwitter: https://twitter.com/JaxOfTheTCG
PlayStation Network: GadgetJaxNintendo Network ID: GadgetJaxSteam: GadgetJax 3DS Friend Code: 2466 5984 2925
"S O N I C W A V E"
Quote: AnonyWolf
>Z-i-V
c'mon, lad

Did somebody mention me?
DDR ACE RIVAL CODE: 5141-2365
------------------------
http://1.bp.blogspot.com/-x7RkmjQjVcA/VpcWcpPQAMI/AAAAAAAAAo4/id4UbxY5UpI/s1600/Clean_VOL4.png

Post #4 · Posted at 2016-12-25 01:04:11am 2.9 years ago

Offline Loodee
Loodee Avatar Member+
270 Posts
Sweden
Reg. 2014-06-26
Log in to send this user a private messageTwitter: ailoodeeYouTubeTwitch
Steam: loodee
this thing's nice, gj
http://i.imgur.com/aQNyGoh.png

Post #5 · Posted at 2016-12-25 01:27:00am 2.9 years ago

Offline AngledLuffa
AngledLuffa Avatar Member
103 Posts
United States
Reg. 2015-05-14
Log in to send this user a private message
Quote: Loodee
this thing's nice, gj

Glad to hear it's useful!

Quote: AnonyWolf
Quote: AngledLuffa
Z-i-V
c'mon, lad

Feel free to submit a pull request Smile
Register Log In Back To Forums

0 User(s) Viewing This Thread (Past 15 Minutes)

©2006-2019 Zenius -I- vanisher.com -5th style- IIPrivacy Policy
Web Server: 8% · Database: 4% · Server Time: 2019-12-07 20:02:27
This page took 0.002 seconds to execute.
Theme: starlight · Language: englishuk
Reset Theme & Language