Update: The Undermine Journal in Alpha Testing Still and Adding EU Realms

Updates on the development of The Undermine Journal from the developers:

Throughout this blog, and in fact, throughout the site, a realm is defined as the server and it’s faction. So when I say “realm”, think “Deathwing Horde” or “Spinebreaker Alliance.” So if you email asking for your realm to be added, include your faction! If you forget and I’m feeling nice, I might add both factions.. but with so many requests coming in, that’ll probably be rare.

To be able to scan the auction house of a realm via the Armory, one must have a character on that realm in the game. When I first started development, I created the characters on my existing World of Warcraft account. Since the crawlers need to login to battle.net automatically, I had to remove my authenticator from the account. I kinda missed using the authenticator while playing to protect my lootz, and I wanted some protection against my account getting banned for the scans. So I eventually started a new World of Warcraft subscription, on its own account, just for the crawlers.

The process of adding a realm is, basically, login to WoW, create the character on that realm, login to that character, log out, and wait ~6 hours for the Armory to see it. Add that character and realm to the list in The Undermine Journal’s database, and that’s all.

Each WoW subscription permits a maximum of 50 characters across all realms. Right now, I’m scanning approximately 47 different realms. (Neutral realms don’t count, as they can use existing characters on either faction.) So while I have a number of realm requests sitting in my inbox, it’ll be a few more days before I get another new account set up so I can add more characters and realms to the list.

Adding EU Realms: I live in the United States. It’s easiest for me to set up and pay for a US account. I have friends in the EU who can purchase an EU copy of WoW and game time cards when I’m ready to work over there. However, since this is a bit of a risky venture as far as Blizzard someday saying “no you” and shutting down scans, I don’t want to go through that hassle just yet. Maybe once the site is feature-complete and in beta testing. I have had a lot of requests for EU realms, so I know it’s anxiously awaited. I’m concerned about doing scans across the pond and how latency might affect the crawlers, but we’ll cross that bridge when we get to it. I am open to scanning EU realms, just not yet. (Source)

Questions/Answers on data collection:

There are a few small/cheap VPS on which I have installed crawler scripts. Each crawler polls TheUndermineJournal.com every few minutes, asking if a realm needs to be crawled (which happens about once an hour) and supplying current load information to assist in load balancing across crawlers. If a realm needs to be scanned, that realm is assigned to that crawler and battle.net credentials are supplied. The crawler then connects to the Armory via the same http web interface that’s open to the public and scans the auction house listings. Once the scan is complete, the crawler calculates the market price of all the items seen, and sends the raw auction listings along with the market prices back to the main site. The main site stores that data in a short-term file queue, and another process pops that data file off the queue and inserts it into the database.

Each cheap VPS has around 128MB of memory and can peak at about 10 realms being scanned at once. However, I’m trying to keep the number of running scans to about half that for failover compensation and other unexpected heavy demand (armory running slow that day, resume after a shutdown, etc). Also, I don’t want to flood the Armory with too many TCP connections from the same IP and risk that IP getting banned, so I doubt I’ll push that scan number higher even if I get some VPSes with more memory. Hard to say, though. (Source)

Submitting a request for your Realm:

Again, you can refer to this post about how to submit a request for your realm AND faction. In short, you send the developer an email with your specific realm, faction, and country. So for example Greymane-US, Horde. Along with the request, any suggestions and constructive feedback may increase your chances of having your server added. Here are some of our suggestions we sent over:

• For ore (or anything really) prices, maybe include stack prices or a toggle to see prices per stack.
• Show green and red arrows for increases and decreases in prices.
• Not sure if prices or averages or current ones at the time of the scan, but maybe have a toggle for various options.
• Track economy wide trends, as in the average price across all servers or low/medium/high population realms. Like an index (think S&P 500).
• List of top sellers in an economy, ranked by number of volume, prices, profit margin, ect.
• Add features that separate itself from auctioneer and other WoW addons. Maybe price predictions, indicating trends, and make recommendation to buy, hold, sell. (ex. Tracking battered hilt and predict or hint at its’ ability to retain a high value.
• Economy related blue posts, news, and other information on the front page.
• News about changes on item pages. An example would be the frozen orb page showing that patch 3.3.3 caused an increase in prices due to the new NPC orb exchange. And a graph would predict prices increasing over time.
• Possibly a replacement for LilSparkies workshop and other crafting spreadsheets. Where it could add up the cost of the materials and figure out what is the most possible craft. The gem cuts section has an advantage over LilSparkies. It let’s you know what gems are worth the most so people can buy the recipes if they don’t have them already.
• Cross-reference the data to WoW Popular.
• Anything that could remotely give us inside information or a slight edge in making a profit.

Any other suggestions or features you would like to see on The Undermine Journal?