Drivers2 Ratings Test

[JoyfulPanda]

PART TWO - STREET CIRCUITS
Same experiment structure as before, only this time:
* Race distance was 40% instead of 25% since street races are shorter.
* All chassis, engines, and tires have been standardized across cars.

Ran 40 races (8 each) on Australia 95-08, Detroit 1999, Long Beach 2000, Toronto 1996, and Vancouver 1995.

Again thanks to Pavel for the work on these tracks - and Tjerk de Heer + Tony Krist for their work on Long Beach.


[b]CAR      STR   AVF    SD[/b]
Trac950     2    1.03   0.17

Trac850     5    1.87   0.34
Trac750     8    2.84   0.37
---
Pwr950     1    4.39   1.05
Trac650   11    4.89   1.20
Pwr850     4    5.59   1.16
Pwr750     7    7.26   1.79
Drag050    3    8.26   1.80
Pwr650    10    9.53   2.57
Drag150    6    9.87   4.01
Trac550    14  10.19   2.00
Drag250    9   11.61   1.92
Drag350   12  12.36   2.04
Pwr550    13  12.82   3.25
Drag450  15   14.47   2.58
Pwr450    16  15.00   2.99
Drag550   18  15.77   2.03
Drag650   21  17.16   2.73
Pwr350    19  18.64   2.75
Drag750   24  18.73   2.17
Drag850   27  19.66   2.04
Trac450    17  19.89   2.49
Drag950   30  20.03   2.06
Pwr250    22   21.64   2.21
Pwr150    25   23.25   1.25
Pwr050    28   24.57   1.29
Trac350    20  24.79   1.56
---
Trac250    23  26.22   1.18
Trac150   26   27.08   1.32
Trac050   29   28.16   1.22


As we can see, traction is still the most important value but not as overwhelmingly so as it is for speedways.

It looks like Power 900 ~= Traction 650, and Drag 50 ~= Power 700.


However, once again it looks like increase in drag (>500) has a relatively higher impact than a reduction in drag (<500). I'm starting to suspect that drag really is non-linear, but would need to design a test to validate this in some way

[samsepi0l]

However, once again it looks like increase in drag (>500) has a relatively higher impact than a reduction in drag (<500).  I'm starting to suspect that drag really is non-linear, but would need to design a test to validate this in some way

In the fluid dynamics engineering world, remember that drag is proportional to the square of velocity, so yes - it should be non-linear... =)

[JoyfulPanda]

PART THREE - SHORT OVALS

Same experiement as before.  This time ran 39 races total (13 each) on Milwaukee/Nazareth/Phoenix.

Results:

CAR      STR   AVF   SD
Trac590   9      1.47   1.16
Trac570   10    2.26   0.76
Trac550   12    3.16   1.57
---
Powr850   3    5.49    2.33
Trac530   14    5.76   2.98
Powr950   1    6.53    5.80
Powr750   5    8.46    3.44
Powr650   7    8.34    5.35
Trac510   15    9.74   3.67
Powr550  11   10.82  4.48
Drag050   2   12.79   2.38
Drag150   4   13.46   4.37
Drag250   6   12.74   4.42
Drag450  13  14.06   4.51
Drag350   8   14.38   4.51
Powr450  18  14.75   4.89
Drag550  20  15.97   4.33
Drag650  24  16.46   4.96
Drag850  28  16.53   4.30
Trac490  16   16.69   4.01
Drag950  30  16.70   4.59
Drag750  26  17.03   3.90
Powr350   23 18.65   4.02
Trac470  17   21.42   3.01
Powr250  25  22.00   3.39
Powr150  27  23.26   2.47
Powr050  29  24.27   2.38
Trac450  19   24.74   2.06
---
Trac430  21   26.72   1.69
Trac410  22   27.59   1.57


While Traction is still clearly the most powerful attribute on short ovals, compared to speedways it has a lesser impact, and we see an increase in the impact of power.

A change of 10 in traction is more impactful than a change of 50 in power, but less impactful than a change of 150 in power.

[JR82]

With these tests, with the "standard" attributes, i.e they all use same value, do they have any variation to mix the grid or is it all the same value (500-501 for example) with them starting in grid order based on which order they appear on drivers2.txt?

[JoyfulPanda]

I'm back doing some more experiments - this time with Aggression.

If you just want the key takeaways that I observed before I get into details here they are:
* Low aggression values (e.g., 1) will move through the field significantly faster than high aggression values (999). Even with identical stats elsewhere, a high agg value AI can end up lap(s) down to a low agg value AI on ovals as it struggles to move through lapped traffic.
* Aggression also affects defending, but less than passing. And its only noticeable on tracks where passing is difficult for the AI (e.g., street tracks).
* On the flip side, high aggression values (e.g., 999) can lead to cars bunching up in big packs as they struggle to pass slower cars. Ironically making passing more difficult even for faster cars with low aggression values (e.g., 1).
* In general the impact of aggression will vary significantly between tracks. I would love to give you a track-by-track breakdown of how strong the impact is, but for now just think of it as - the harder it is to pass at a circuit, the more aggression values will matter.
* I did not detect any difference in raw pace caused by differences in these values - but I did not try to test this directly. I only observed lap times of the leading car and noted no significant difference based on aggression values.
* From previous experiments, combined with this one, I have never detected a difference in DNFs based on aggression, including Accidents.



SETUP
To run this experiment I had a 22 car field. I set up the first 20 cars to be a realistic spread of pace, all with the same aggression values. They were a realistic spread but slower than normal cars - think like the generic cars that shipped with the base ICR2 game. Then I had two cars that were manipulated to always start on the back row of the grid, but with much faster race pace (accomplished by giving them high drag values). These cars had identical stats with no variance - but one had an Aggression setup of (1, 2), and the other had an Aggression setup of (998, 999).

For the rest of the post, Min Agg will refer to the car with (1, 2) Aggression, and Max Agg will refer to the car with (998, 999). To avoid any potential issues with Min Agg getting a headstart, Max Agg was on the inside of the back row (21st), and Min Agg was on the outside (22nd).

I simulated a series of races on 40% distance and 100% AI strength. Each time I ran the race once with opponents having (1, 2) Aggression, and once with (998, 999) Aggression.


SUPERSPEEDWAYS
(A) INDIANAPOLIS (Pavel's Indy 2000)
(With opponents at 998,999)
Min Agg knifed through the field - up to 9th place by end of lap 2, and into the lead by end of lap 7. Max Agg struggled. Was still in 15th place when Min Agg took the lead. Only up to 7th by lap 40. Did end up finishing 2nd but took almost the whole race to get there. Because it was struggling so much with traffic, it went a lap down to Min Agg on lap 25, and 2 laps down on lap 79.

(With opponents at 1,2)
Min again into lead by end of lap 7. But this time Max Agg was able to keep pace much better. Was 10th when Min Agg took lead. By lap 40 was into 4th place, and again finished 2nd. Was on the lead lap the whole time.

Again - the observation here is that because the opponents did not get stuck behind her as much, the faster AI cars were able to move through traffic much easier, even with poor aggression values themselves.


(B) FONTANA (Pavel's edit)
Compared to Indianapolis, it seemed like Aggression did not matter as much here.
(With opponents at 998,999)
Neither fast car had any trouble moving through the pack here. Max Agg actually took the lead first - on lap 10. However, Min Agg still ended up winning, and fast cars finished 1-2. From what I could tell, the difference was made in the fact there was still some difference in how quickly Min Agg could get through lapped traffic.

(With opponents at 1,2)
Min Agg was into the lead by lap 5, and Max Agg was into 2nd shortly afterwards. Again, notice that it was actually easier for the fast cars to get through the field here - and from observation it is again the "clumping" issue that is less present when the slower cars have low aggression values.




STREET
(A) AUSTRALIA (Pavel's excellent 95-07 circuit)
(Opponents at 998, 999)
Min Agg was already up to 10th by end of lap 1, while Max Agg was stuck back in 18th. Min moved steadily through the field, but there were two of the "slower" cars that broke away from the pack. So Min Agg was into 3rd by end of lap 5, but took several laps to chase these two down. Finally caught them and got through them on Lap 18. Max Agg took until about Lap 10 to break into top 5, and was 4th at end of Lap 13. And that's where Max finished (did not have enough time to chase down the two "rabbits")

(Opponents at 1, 2)
As mentioned above, on street circuits there's a noticeable difference in how easily faster cars can get through the field when opponents have these low aggression values. It took Min Agg two laps this time to get into top 10. And again Max struggled, in 19th at end of Lap 1 and 16th at end of Lap 2. This time Min Agg was only up to 6th by end of Lap 5. And again we had the same thing with a few cars breaking away from the pack while these cars struggled.

Min Agg took almost the whole race to chase down the leaders, but did ultimately end up getting it done in the last few laps, moving from 4th up to 1st in the last 5 laps.

Meanwhile, Max Agg could only manage to get up to 11th by around lap 7, and didn't make the pass up into 10th until the last half of the race. And that's where Max finished. 1st vs. 10th.


(B) DETROIT (The Detroit '99 track worked on by Pavel and I believe Bob?)
(Opponents at 998, 999)
Pretty steady rise for Min Agg this time. Start was slow - only up to 16th by end of first lap this time, and 15th by end of lap two. But rose steadily to 2nd by the mid-point of the race, and took the lead shortly after.

Meanwhile Max Agg kept pace for a while, was right with Min Agg for the first few laps, but eventually got held up a bit. At lap 5, Min Agg was in 8th, Max Agg in 12th. At lap 7, Min Agg was up to 5th, while Max Agg was at 7th (but a ways back). At halfway point, Max had only managed to get into 6th. Did ultimately finish 2nd, but took almost the full 40% distance to get there.


(Opponents at 1, 2)
Again, we see significant step up in the difficulty of the fast cars getting through the pack. At end of lap 10, Min Agg was only up to 6th, while Max Agg had just managed to make it into the Top 10.

At halfway point, Min Agg was in 4th, and Max in 8th. This ended up being the only race in my simulations that Min Agg did not win. Min finished 2nd, and Max finished 4th.



OTHER SIMS
I did do some other sims, but I thought these examples would give the best insight. In general, we ALWAYS saw Min Agg beat Max Agg. 20 out of 20 finished sims where they both finished. Although Max Agg was occasionally competitive. Max Agg had significant trouble on Road/Street courses when opponents were set at (1, 2) aggression, while Min Agg did not have as much trouble in those cases. And Max Agg ended up a lap down or more on about half of the ovals.


LIMITATIONS/FUTURE QUESTIONS
With these sims, I was just recording ranks at different lap intervals, but it could be the case that tracking actual gaps in seconds would be valuable to get more insight in the gaps between the two.

I also didn't look at any differences in pit behavior/pace over the course of stints - but wouldn't put it past ICR2 for the more aggressive car to wear through tires faster.


RECOMMENDATIONS
When creating your Drivers2 files - I would recommend having all drivers sub-500. We want them to be competent enough to get by each other, and especially on ovals we want to avoid clumping. Beyond that, it depends on your preferences. I would think of Aggression instead as "Racecraft". If the driver is better at passing/defending, move the values down - if not, move them up.

Hopefully helpful to those creating their own driver ratings

[samsepi0l]

So aggression is inversely proportional to the value. Thank you for your thoroughness for this test.

I found that making the DLAT gap between PASS1 and RACE LPs wider (also PASS2 and RACE) makes a HUGE difference in how easy the AI cars pass eachother and when they will attempt a pass. I found this out very successfully when Checkpoint10 and I were working on LPs for Detroit-91. My AI was really crappy at passing and lapping until I made the gaps just a little wider and then they were able to pass everywhere... This is something that I will be doing with all of my LPs going forward because it makes such an awesome difference.

When I do a carset- I just make the min and max values for aggression the same for all drivers. I think I use 300 and 800. I can't remember. This way there is a randomness to how well they will be able to move forward.

I am starting to wonder if Papy originally did not really optimize some of these things. It's almost like certain aspects of the game were rushed- and it leaves a lot of room for us to improve if we take the time to understand and optimize things.

[Pavel 69]

It's really interesting find, thank you! I always thought that higher aggression value make AI better in traffic But reverse is true! Now I'm curious if Nascar sims has same behavior?

[JoyfulPanda]

Yeah I was imagining there's some kind of interaction with the LP files - I'm hoping to learn more about track editing this year because it would be cool to understand really precisely the impact of some of the LP values with the driver ratings. On some circuits I've definitely noticed major struggles in overtaking, maybe this insight about DLAT could fix some of these otherwise great circuits...

I'll try to dig into it a bit over the next week

[checkpoint10]

Great to see this research. I was thinking if we wanted to do experiments in "controlled conditions" maybe we can use that "Ford Test Track" from the archives and make an LP that cruises along the center of the track at a constant (slower) speed, then make a couple of passing LPs to the left and right of it. I think with that, you could measure exactly how much slower from top speed an AI would go, depending on the values in drivers2.txt. And then maybe it would be easier to measure what happens when a faster car comes up on a slower car.

[samsepi0l]

Getting off topic from original post (but very interested in LP files) I love the idea of more LP research. Perhaps we move this portion of the topic to this thread? https://www.icr2.net/forum/showthread.php?tid=1420