Skip to content
Maintankadin

Theck's MATLAB thread - Cataclysm/4.x

tlitp
Tue Aug 10, 2010 2:12 pm

Re: Theck's MATLAB thread - Cataclysm/4.x

theckhd wrote:Importing a character's information would definitely be useful. All you'd need to do is come up with a list of the IID and enchant SID for each slot. (I)
You could include talents too, though that may be a little trickier on our end since I'm not sure what format it's in. (II)

I. It's a definitely more complex problem. For gear items we don't need IID entries, we need the stats on them (blame reforging). Furthermore, gems (easy one, as their IID is explicit) and socket bonuses - another pain in the arse.
Theck's current helm, for reference :
Code: Select all
<item displayInfoId="64692" durability="56" gem0Id="41380" gem1Id="40130" gem2Id="0" gemIcon0="inv_jewelcrafting_shadowspirit_02" gemIcon1="inv_jewelcrafting_gem_40" icon="inv_helmet_154" id="51266" level="277" maxDurability="100" name="Sanctified Lightsworn Faceguard" permanentEnchantIcon="ability_warrior_swordandboard" permanentEnchantItemId="44878" permanentenchant="3818" pickUp="PickUpLargeChain" putDown="PutDownLArgeChain" randomPropertiesId="0" rarity="4" seed="0" slot="0"/>

II. This part is fairly trivial.
Theck's current active tree :
Code: Select all
      <talentGroup active="1" group="1" icon="spell_holy_devotionaura" prim="Protection">

        <talentSpec treeOne="0" treeThree="18" treeTwo="53" value="000000000000000000000000000500513520310231133331232150230201200300000000000000"/>
        <glyphs>
          <glyph effect="Your Seal of Vengeance or Seal of Corruption also grants 10 expertise while active." icon="ui-glyph-rune-15" id="561" name="Glyph of Seal of Vengeance" type="major"/>
          <glyph effect="Reduces the cooldown of your Lay on Hands spell by 5 min." icon="ui-glyph-rune-11" id="455" name="Glyph of Lay on Hands" type="minor"/>
          <glyph effect="Reduces the mana cost of your Seal of Wisdom spell by 50%." icon="ui-glyph-rune-1" id="457" name="Glyph of the Wise" type="minor"/>
          <glyph effect="While Divine Plea is active, you take 3% reduced damage from all sources." icon="ui-glyph-rune-1" id="705" name="Glyph of Divine Plea" type="major"/>
          <glyph effect="Damage against Undead increased by 1% while your Sense Undead ability is active." icon="ui-glyph-rune-9" id="456" name="Glyph of Sense Undead" type="minor"/>
          <glyph effect="When you cast Hand of Salvation on yourself, it also reduces damage taken by 20%." icon="ui-glyph-rune-11" id="707" name="Glyph of Salvation" type="major"/>
        </glyphs>

      </talentGroup>
theckhd
Tue Aug 10, 2010 2:48 pm

Re: Theck's MATLAB thread - Cataclysm/4.x

tlitp wrote:I. It's a definitely more complex problem. For gear items we don't need IID entries, we need the stats on them (blame reforging). Furthermore, gems (easy one, as their IID is explicit) and socket bonuses - another pain in the arse.
Theck's current helm, for reference :
Code: Select all
<item displayInfoId="64692" durability="56" gem0Id="41380" gem1Id="40130" gem2Id="0" gemIcon0="inv_jewelcrafting_shadowspirit_02" gemIcon1="inv_jewelcrafting_gem_40" icon="inv_helmet_154" id="51266" level="277" maxDurability="100" name="Sanctified Lightsworn Faceguard" permanentEnchantIcon="ability_warrior_swordandboard" permanentEnchantItemId="44878" permanentenchant="3818" pickUp="PickUpLargeChain" putDown="PutDownLArgeChain" randomPropertiesId="0" rarity="4" seed="0" slot="0"/>

That contains most of the relevant information we need though:
IID (51266)
Gem IDs (41380 & 40130)
Enchant ID (44878, which is an IID).

I have no idea what the 3818 is, it doesn't seem to represent a SID.

Socket bonuses are trickier, I'll admit, but not a completely insurmountable one. We could always add socket colors and bonuses to the database and have equip() check for them.

Presumably once Reforging is implemented, that information will be returned by the XML as well. Probably as a "reforgeID" of some sort. We can worry about that once we have XML to parse.
Gaffer
Tue Aug 10, 2010 2:57 pm

Re: Theck's MATLAB thread - Cataclysm/4.x

theckhd wrote:Automatically generating the gear_db file would be cool, but would probably result in a bunch of wasted memory. Storing a bunch of Leather, Cloth, and Mail gear we don't care about, for example. It may still be of some use though.


I didn't realize you had planned on fully populating gear_db. I started putting this together because gear_db currently contains a minor subset of gear and I wanted to pull information on my gear in without hassle. What I had started doing was putting together something that queried a specific player's armory and then created a gear_db file containing only those pieces. If all you need is the id, that's obviously much simpler to retrieve and can either pull into a formatted structure for matlab or just write out the necessary equip()s directly to a downloaded file.

Currently, I was outputting something like this for each slot:
Code: Select all
%% Head
idb(51266).name='Sanctified Lightsworn Faceguard (Heroic)';
idb(51266).str=183;
idb(51266).sta=243;
idb(51266).ilvl=277;
idb(51266).def=96;
idb(51266).dodge=80;
idb(51266).parry=88;
idb(51266).armor=2239;



tlitp wrote:I. It's a definitely more complex problem. For gear items we don't need IID entries, we need the stats on them (blame reforging). Furthermore, gems (easy one, as their IID is explicit) and socket bonuses - another pain in the arse.


Retrieving item stats isn't a complex problem (you can get them from item-tooltip.xml) though it's difficult to imagine how reforging will play in and be represented via the armory. However, socket bonuses still cannot be obtained and some other data is difficult to retrieve (barmor vs earmor on a piece like the T10 chest). Querying from WoW Head instead of item-tooltip.xml makes most of this information readily available.

I would definitely not be surprised if the armory lags way behind in providing information for reforging.
theckhd
Tue Aug 10, 2010 3:27 pm

Re: Theck's MATLAB thread - Cataclysm/4.x

Yeah, my intent was for gear_db to contain all the stats for items we considered relevant. For example, a gear_db for Wrath would contain every tanking item from every instance from Naxx to Ruby Sanctum, as well as from badges. It would also contain DPS items we considered relevant, like slow DPS weapons, a few interesting trinkets, Libram of Valiance, and stuff like that. The only reason it's a small selection right now is that it's all we need for testing, and we don't have final item stats for much of anything in Cataclysm.

The original plan was to implement it by hand. That sounds like a lot of work (and it is), but it's not quite as bad as it sounds since we would only have to add one tier at a time, making the job incremental over a year or two. Also, gear_db entries have a very simple format, so I was hoping the community would pitch in. Even if they don't want SVN access, it would be easy enough to write up the code and post it in this thread. If one person said "I'll do all of the badge items from this tier," another took the tier items, another took off-set items, it would get done pretty quickly.

That said, if you want to write a utility that will pull all of that information from wowhead or the armory and format it properly as a gear_db entry, that'd be great too. It'll help populate gear_db that much faster.

Obviously if gear_db contains everything, then creating a gear set for a character is about as simple as querying for the IID/SID information and making a bunch of equip() calls, presuming we sort out how to handle sockets, socket bonuses, and reforging.
theckhd
Tue Aug 10, 2010 3:29 pm

Re: Theck's MATLAB thread - Cataclysm/4.x

As a side note, earmor and barmor won't be that hard to figure out. The base armor value of an item is fixed for a given ilvl. Given about a half an hour, I could either
A) plot barmor vs. ilvl and figure out what the functional dependence is, coming up with an analytical formula.
B) Write a simple lookup table

Alternatively, this info might already be on wowwiki - I haven't had time to check.
tlitp
Tue Aug 10, 2010 4:32 pm

Re: Theck's MATLAB thread - Cataclysm/4.x

Gaffer wrote:Retrieving item stats isn't a complex problem (you can get them from item-tooltip.xml) though it's difficult to imagine how reforging will play in and be represented via the armory. However, socket bonuses still cannot be obtained and some other data is difficult to retrieve (barmor vs earmor on a piece like the T10 chest). Querying from WoW Head instead of item-tooltip.xml makes most of this information readily available.

It is, as long as we have to deal with wowarmory's limitations. And we'll need wowarmory, as reforging and socket matching cannot be automated using any other resource. If Blizzard won't update their parsing system, we'll basically be forced to populate some entries manually.
Gaffer
Tue Aug 10, 2010 8:04 pm

Re: Theck's MATLAB thread - Cataclysm/4.x

theckhd wrote:As a side note, earmor and barmor won't be that hard to figure out. The base armor value of an item is fixed for a given ilvl. Given about a half an hour, I could either
A) plot barmor vs. ilvl and figure out what the functional dependence is, coming up with an analytical formula.
B) Write a simple lookup table

Alternatively, this info might already be on wowwiki - I haven't had time to check.


Information for previous expansions seems to be pretty readily available, This was the closest I could find for Wrath though. Just seemed odd that I couldn't find anything more specific this late in the expansion.

tlitp wrote:It is, as long as we have to deal with wowarmory's limitations. And we'll need wowarmory, as reforging and socket matching cannot be automated using any other resource. If Blizzard won't update their parsing system, we'll basically be forced to populate some entries manually.


I'm betting reforging information eventually makes it into character-sheet.xml, but experience shows us that armory updates tend to be a pretty low priority. Querying users for that reforging information and populating whatever adjustment file is still easier than having them manually adjust.
theckhd
Wed Aug 11, 2010 7:17 pm

Re: Theck's MATLAB thread - Cataclysm/4.x

Gaffer wrote:
theckhd wrote:As a side note, earmor and barmor won't be that hard to figure out. The base armor value of an item is fixed for a given ilvl. Given about a half an hour, I could either
A) plot barmor vs. ilvl and figure out what the functional dependence is, coming up with an analytical formula.
B) Write a simple lookup table

Alternatively, this info might already be on wowwiki - I haven't had time to check.


Information for previous expansions seems to be pretty readily available, This was the closest I could find for Wrath though. Just seemed odd that I couldn't find anything more specific this late in the expansion.


I spent a little time this afternoon playing around, and I think I've come to some reasonably good conclusions about how the armor model works. If you want to play along at home, here's the code.

Methodology: Take a bunch of (ilvl, armor) ordered pairs from both the current and cata wowhead databases, plot them and look for some sort of pattern. I've limited the scope of the investigation to chest pieces for simplicity.

First, let's consider the current (wolk) data:

Figure 1: Armor vs. Ilvl for plate chest items pre-Cataclysm
Image

The "model" traces use the wowwiki formula that gives a linear dependence on ilvl, with epic items having 25% more armor than their blue counterparts (in other words, epic = 1.25*blue).

The "epic" and "blue" traces are the actual data for current items, painstakingly copied over into matlab (by hand!) by yours truly. The "blue*1.25" and "blue*1.0345" traces are just scaled versions of the blue data set.

The first few things to note:
  1. The formula seems to do very well for classic and BC items, which means epics up through Sunwell gear (ilvl 159) and blues up to ilvl 115. The one exception is Carapace of Tuten'kash at ilvl 42, which we must assume has different itemization due to its low level and will ignore.
  2. The formula is blatantly wrong for Wrath items. The epic items fall far below the projected values from earlier expansions, and the blue items are far above the projected values, making the difference between a blue and an epic at a given ilvl much smaller.
  3. The "blue*1.25" curve seems to accurately reflect the relationship between epic and blue items in BC and clasic, but gives values that are too large for Wrath blues because of #2.
  4. Based on the ilvl 200 data point, an epic in Wrath only has about 3.45% more armor than a blue of equal ilvl. However, the region of overlap is small, and this relationship doesn't hold as we slide down the scale to the region of overlap between BC epics and early Wrath blues.

The implications here are fairly mundane. The developers obviously noted that the armor difference between blues and epics was going to grow dramatically with ilvl, and decided to change the formula to keep them more in line. Presumably, the function that converts ilvl to armor is piecewise, with the familiar linear relationship for items from BC and earlier, and a different function for Wrath items. The blue items in Wrath are defined by a function that gradually ramps them up to the level of early epics. Let's take a closer look at the Wrath epics, and this time, we'll include their Cataclysm armor values:

Figure 2: Armor vs. Ilvl for plate chest items pre- and post-Cataclysm, ilvl 200+
Image

It seems here that the Wrath epics have a piecewise linear dependence. The items from 200-232 follow one linear function, while the items from 239 on follow a second, different linear function. As surmised in the EJ thread, this is probably because of feedback about armor scaling beyond the 232 limit spurred on by the introduction of heroic modes.

These items all see a slight increase in armor in Cataclysm, but retain their different formulas. So we're not seeing a Grand Unified Theory of Armor-Ilvl interaction in Cataclysm, at any rate.

The functional forms spit out by cftool, for those interested, are:
Code: Select all
fit1a=1503+3.734*x
fit1b=306.7+8.842*x
fit2a=1961+1.882*x
fit2b=446.4+8.324*x


Now let's expand our view and look at the full ilvl range.

Figure 3: Armor vs. Ilvl for plate chest items pre- and post-Cataclysm, all ilvls
Image

Very interesting! It seems that the linear function for pre-Wrath epics has been changed. It appears to be a piecewise function with four different linear curves (early classic epics, late classic epics, early BC epics, late BC epics).

It's hard to see from this plot, but the classic/BC blues have almost the same functional dependence (divided by 1.25). This is more clearly seen on the next plot. The wrath blues also seem to have a piecewise dependence in Cataclysm, with the breaking point somewhere around ilvl 179.

Finally, let's look just at the data from the Cataclysm site:

Figure 4: Armor vs. Ilvl for plate chest items post-Cataclysm, all ilvls
Image

The red dots clearly show that the blues are simply following the same piecewise function as the epics, just smaller by a factor of 1.25. The teal dots show a similar pattern, but in this case the blues below ilvl 179 are about 11% behind (1.11), while the blues above 179 are about 3.45% behind (1.0345). The data point at ilvl 179 is a bit odd. If we assume it's 11% behind as I have in the teal curve, it's pretty linear (but much higher armor than the next few). If we assume it's 3.45% behind, it's a little too low. I'm not sure which I think is more likely - it's possible they made the piecewise cutoff 180 and didn't worry about the actual armor value since they never intended to have epics in that region and the blues would be normalized appropriately anyway.

Unfortunately, we only have three data points to work with for ilvl 270+ blues. However, I think it's reasonable to expect the same sort of dependence we see in Wrath. A sharp linear increase in armor to get us up to the next baseline, at which point it will change to a shallower linear dependence.

Finally, I decided to try and fit all of the sections of the piecewise function just for fun. Here's the result:
Figure 5: Armor vs. Ilvl for plate chest items post-Cataclysm, all ilvls
Image

The functions are:
Code: Select all
x1=40:58;
f1=58.87+11.9.*x1;
x2=59:87;
f2=499+4.162.*x2;
x3=88:108;
f3=-1300+24.86.*x3;
x4=109:130;
f4=1049+3.126.*x4;
x5=130:180;
f5=-959.2+18.58.*x5;
x6=170:235;
f6=1961+1.882.*x6;
x7=236:280;
f7=446.4+8.324.*x7;


The crossover points are only roughly estimated in most cases, and the 170-180 region is still a little bit of a mystery due to the 179 data point. Unfortunately, when I searched for blue plate chests with ilvls between 170 and 180, all it turned up was items at 179. An extra data point around 174 or so would really help. Perhaps we could extrapolate up from a green, but then we'd need to figure out whether the green->blue factor had changed at all across the domain, and I've already spent too much time on this today. :P
tlitp
Thu Aug 12, 2010 6:36 am

Re: Theck's MATLAB thread - Cataclysm/4.x

1. I suggest using MMOC, there are more beta entries.
2. Base armor seems to be independent of item quality : 1, 2.
3. Under the assumption that 2) holds true for (ilvl 300+) epics as well, fourier1 seems to be more accurate than poly1 (R^2 >0.998, RMSE <2) :
Code: Select all
barmor=a+b*cos(0.0138*ilvl)+c*sin(0.0138*ilvl)

slot      a     b       c
chest   3022  849.2  -209.6
head    2449  689.1  -178.4
should  2297  654.4  -130.6
waist   1703  479.9  -114.7
hands   1911  544.7  -112.6

No time for a more thorough parse, maybe someone else gets a better fit.

EDIT : The above results are valid on a data set composed of ilvl 272+ green/blue items (and ilvl 300+ epics where applicable).
theckhd
Thu Aug 12, 2010 1:06 pm

Re: Theck's MATLAB thread - Cataclysm/4.x

I've updated the m-file to include the beta entries from MMOC in their own array (axcat and aycat). I get the same sort of results you do - a fourier fit gives lower RMSE than a simple linear fit. In fact, you have to go to a 4th-order polynomial to beat the RMSE of the fourier fit.

I got slightly different coefficients out of cftool than you for the chests (a=3005 b=819.4 c=50.13), but yours gives a lower RMSE. I've left the fitting options set to whatever I was using last, which could be why.

I think we'll need more data points to draw any useful conclusion about Cataclysm items though. For one thing, we don't have direct evidence that epics will have the same armor as their green/blue counterparts, just that greens and blues won't have an armor difference. For another, the nonlinearity we're seeing is based on one or two data points out of six, so we could be fitting based on error (i.e. the armor formula changed since one or two items were implemented and the fix hasn't been properly rolled out to the old items yet, or the item hasn't been updated in MMOC's database).

The other reason I'm skeptical is that it seems odd to write such a complicated function for armor (4 parameter) and then choose parameters such that it looks that close to a linear dependence. We're talking about errors of ~2-8 armor in a value of ~3k.

Image
bentglasstube
Thu Aug 12, 2010 9:17 pm

Re: Theck's MATLAB thread - Cataclysm/4.x

I'm not sure if you guys figured out the strength/stamina thing or just gave up. I know that in WotLK at least (it looks like you were dealing with Cata calculations), the "white" and "green" components of stats are rounded separately, so the formula should look something like

Code: Select all
stat = floor((base + prof) * mod) + floor((gear + buff + cons) * mod)


For example, my paladin with Mark of the Wild (the armory appears to be back to secretly showing buffed stats if you log out with buffs):

Stamina
Code: Select all
base = 153
prof = 50 (mining)
gear = 3285
buff = 37
mod = 1.04 (sacred duty) * 1.06 (combat expertise) = 1.1024

sta = floor((153 + 50) * 1.1024) + floor((3285 + 37 + 0) * 1.1024)
    = 223 "white"                + 3662 "green"
    = 3885


Strength
Code: Select all
base = 151
gear = 1375
buff = 37
mod = 1.15 (divine strength)

str = floor(151 * 1.15) + floor((1375 + 37) * 1.15)
    = 173 "white"       + 1623 "green"
    = 1796


An additional benefit to this method is that you can check you character sheet to see if the white or green values you are getting are inconsistent and then adjust the formula for each component independently

Hopefully this is helpful.
theckhd
Fri Aug 13, 2010 1:31 am

Re: Theck's MATLAB thread - Cataclysm/4.x

Yeah, that's the formula we settled on. It still doesn't seem to be perfect, but it deviated the least (off by 1 at most) over a fairly large range.
tlitp
Fri Aug 13, 2010 12:03 pm

Re: Theck's MATLAB thread - Cataclysm/4.x

Another round of entries on MMOC, further strengthening the hypothesis of the lack of correlation between base armor and item quality. On the same topic, that "almost linear" fit is probably caused by working with integers. They use a floor/round somewhere and that skews the results.
theckhd
Fri Aug 13, 2010 2:02 pm

Re: Theck's MATLAB thread - Cataclysm/4.x

tlitp wrote:Another round of entries on MMOC, further strengthening the hypothesis of the lack of correlation between base armor and item quality. On the same topic, that "almost linear" fit is probably caused by working with integers. They use a floor/round somewhere and that skews the results.

I was thinking that, but I can't figure out exactly where. If the formula really is as simple as a*x+b, then round() would only give errors of +/- 2 or so. We're seeing differences in the 8-12 range, which would imply rounding at an earlier stage of a multi-stage formula, similar to our old stat formulas with multiple floors.

Given the old formula structure of (base+green*x)*blue*epic, I could believe a round(round(round(base+green*x)*blue)*epic), but that would be meaningless if the observed lack of correlation between armor and item quality is true, as blue=epic=1.
bentglasstube
Fri Aug 13, 2010 4:31 pm

Re: Theck's MATLAB thread - Cataclysm/4.x

theckhd wrote:Yeah, that's the formula we settled on. It still doesn't seem to be perfect, but it deviated the least (off by 1 at most) over a fairly large range.


Ah, my apologies for reposting it then. I tried to make sure it wasn't what was already there but I made the mistake of trying to read this whole thread in one sitting.

On another note, I do not know much of matlab, but I am a programmer by trade so if there is anything in particular I can do to help, I would be happy to do so.

Edit: I started perusing the code, and the talents specified in talents.m add up to 42 points.

Edit 2: More code reading.

I noticed in gear_db.m that you have Bile-Encrusted Medallion (Heroic) with the incorrect amount of earmor. It should have 756.
I was in the process of writing an item getter for wowhead (but now wowhead is timing out, maybe I got my IP banned) and I think it would be better to store the socket information in the idb array rather than adding the gems to the base item. Something like:

Code: Select all
idb(50682).name='Bile-Encrusted Medallion (Heroic)';
idb(50682).ilvl=277;
idb(50682).str=102;                                                             
idb(50682).sta=141;                                                             
idb(50682).dodge=46;                                                           
idb(50682).earmor=756;
idb(50682).rsock=1;
idb(50682).sb_stat='sta';
idb(50682).sb_val=6;


And then perhaps modify the equip function to take gems and determin final stats accordingly?

Code: Select all
egs(2) = equip('Bile-Encrusted Medallion', 'Solid Majestic Zircon');


Edit 3: I hate multiple consecutive posts so here we go.
I have completed an item fetcher that uses Sirgie (mmoc's db) to convert an item id into a gear_db.m format item. At the moment, it totally ignores sockets (and their bonuses) and the armor is liable to be incorrect as Sirgie's armor is shaky at best. I may have it try to grab armor values from wowhead, but the reason I chose to switch to Sirgie is that it updates more frequently so a lot of items there won't be on wowhead anyway. If this perl script would be useful to anyone I would be happy to share it.

An example run/output:
Code: Select all

$ perl getitem.pl 51266 51265 51267 51268 51269
idb(51266).name='Sanctified Lightsworn Faceguard (Heroic)';                     
idb(51266).ilvl=277;                                                           
idb(51266).str=183;                                                             
idb(51266).sta=243;                                                             
idb(51266).dodge=128;                                                           
idb(51266).parry=88;                                                           
idb(51266).barmor=2239;                                                         
                                                                               
idb(51265).name='Sanctified Lightsworn Chestguard (Heroic)';                   
idb(51265).ilvl=277;                                                           
idb(51265).str=183;                                                             
idb(51265).sta=251;                                                             
idb(51265).dodge=56;                                                           
idb(51265).parry=96;                                                           
idb(51265).barmor=2756;                                                         
idb(51265).earmor=1344;                                                         
                                                                               
idb(51267).name='Sanctified Lightsworn Handguards (Heroic)';                   
idb(51267).ilvl=277;                                                           
idb(51267).str=103;                                                             
idb(51267).sta=192;                                                             
idb(51267).hit=69;                                                             
idb(51267).dodge=68;                                                           
idb(51267).barmor=1723;                                                         
idb(51267).earmor=1148;                                                         
                                                                               
idb(51268).name='Sanctified Lightsworn Legguards (Heroic)';                     
idb(51268).ilvl=277;                                                           
idb(51268).str=139;                                                             
idb(51268).sta=251;                                                             
idb(51268).exp=93;                                                             
idb(51268).dodge=167;                                                           
idb(51268).barmor=2412;                                                         
                                                                               
idb(51269).name='Sanctified Lightsworn Shoulderguards (Heroic)';               
idb(51269).ilvl=277;                                                           
idb(51269).str=136;                                                             
idb(51269).sta=192;                                                             
idb(51269).dodge=99;                                                           
idb(51269).parry=71;                                                           
idb(51269).barmor=2067;