Here’s a behind-the-scenes look at how we developed the game CEV-1!
I’ve always loved reading through how other games are developed, so we’re excited to share our development process with you!
A replay of a winning playthrough of CEV-1.
Brainstorming
Over the past few months, a couple of friends and I got together to create a game. We weren’t sure what kind game we were going to create yet, so we started off with some brainstorming.
We thought of games we really liked and attempted to mash and merge ideas together to create some brand-new game. We really liked Civilization V and Plague Inc. and wanted to do some sort of resource management game, and write a great story to go along with it.
With all games, the player needs to have some sort of action. In most resource management games, there are two types of actions:
1. An investment that increases resources generated
2. An action that gets the player closer to the goal
Initial Ideas
Machine Learning Simulator
Synopsis
In Machine Learning simulator, the player “builds up” a computer which they will use to participate in machine learning competitions. By participating in these competitions, the player will earn more money, which they will use to improve their computer, machine learning algorithm, training data, etc. in order to win in higher and higher competitions (which grant more money). There would be plenty of things the player can do in order to improve their performance, such as hiring developers (various tiers of developers?) or purchasing better computer parts. We can incorporate both idle and gacha aspects to this game, by maybe having some sort of “research events” happening on the side, which unlock upgrades or something to further the computer’s progression, but time locked, and also include a computer parts gacha. We can have infinitely-many “machines”, as such, the player can’t really “max out” and run out of gacha targets to summon for.
Similar aspects to games like
The perfect tower 2
Cookie clicker
Any gacha game
Playable by novice and experts because
The mechanics of the game are fairly simple (the player doesn’t need to know very much of anything about coding or anything like that), but optimization and figuring out the most efficient way to progress with the least amount of time/resources spent would be interesting to the expert players.
Cool ways to increase complexity
Add restrictions/combos/tradeoffs to certain computer parts rather than just making the parts “linear” as in one is slightly more efficient than the other.
It’s a pun! “Severity 1” incidents and “Coherent Extrapolated Volition”
Synopsis
You are a newly created artificial general intelligence that has been instructed to maximize human safety. Unfortunately (for humans), your utility function is misaligned, and you plan to achieve this goal by benevolently taking over the world and enacting totalitarian policies to ensure maximal long-term human safety.
The AI (player) should use nonviolent methods even if, for example, humans use nuclear weapons against it. The player’s score is the number of surviving humans, which disincentivizes deploying weapons of mass destruction. The ultimate goal is to control the world, but intermediate goals could be decommissioning weapons to discourage war, curing diseases, or outlawing cars.
Game mechanics
The main UI is a world map visualization, like in Plague Inc, where the goal is to conquer every country. The key metrics are: dead humans (trying to minimize), trust in AI, and public satisfaction. In the beginning, the player has low resources and low trust. For each country, the player can spend “public satisfaction” points to enact safety policies. More totalitarian policies decrease public satisfaction more, like banning cars vs banning unsafe drinking water. Enacted policies decrease the number of dead humans and increase trust in AI over time. If trust in AI is high enough, the player gets more freedom in buying policies that aren’t as clearly related to safety, like peacekeeping robots.
As the game progresses (maybe based on total trust in AI?), the player gets access to more potential policies in their tech tree. In the end game, the player can deploy the robots to less trusting or less satisfied countries to force them to adopt very totalitarian policies regardless of trust in AI values. The final step will be to enclose all humans in cryogenic chambers to maximize safety.
Similar aspects to games like
Plague Inc (game mechanics)
Ideology Rush (game mechanics)
Universal Paperclips (playing as a misaligned AI)
Playable by novice and experts because
The difficulty can be adjusted easily by changing some variables about how difficult resource acquisition is or how much tech tree items cost. We can also include “advisor popups” to suggest good actions to the novice player. Expert players are challenged because there are many possible options and it is difficult to optimize for the best final score.
Cool ways to increase complexity
Make each country have different policy costs and trust in AI values depending on their culture. Include “news alerts” like Plague Inc.
Design Phase – Round 1
After the initial brainstorming session, we needed to actually make the game. We decided as a group to go with the CEV 1 idea over the Machine Learning Simulator, as it seemed more interesting and had more varied gameplay/interesting to play.
Initial Game
Gameplay
The gameplay is like a Plague Inc. and Civilization hybrid. The goal is for the player to take over every country in the world while balancing resources and researching policies in a tech tree. There are three resources the player has to manage:
Money ($)
Satisfaction (sat)
Military strength (troops)
The player must always keep $ and sat positive. The lose condition is when either is negative.
We split up the world into “regions” (corresponding to subcontinents). Each region produces a different amount of $, sat, and troops per turn and requires some combination of resources to “take it over.” For example, taking over Europe may require 100K sat, 500T $, and 50K troops. To add complexity, regions are interdependent; taking over Europe without first taking Asia would cost more. After acquiring a region, it acts as a resource source (troops++ and $++) and a resource sink (sat–).
Tech tree
Researching tech tree upgrades can increase resources, or decrease or “move” the amount of resources required to take over a country. For example, the “Global Web Influence” policy manipulates content on the web to push more trust in AI, and could have the following costs and benefits:
Media indoctrination (+5 sat, -$100) (requires Better Hypnosis)
Internet censorship (requires Hacking)
Science tree:
Vaccine Research (-$30)
Microchip Research (-$25)
Nanobot healthcare (-$40)
Selective breeding
GMO research
Cryogenics
Cryonics
Indefinite cryonic suspension
Virtual reality
The Matrix
Basic hypnosis
Better hypnosis
Mass hypnosis
Artificial gene engineering
Hacking
Drones
ICBMs
Robots
(Not shown: military tree)
Random events
Random interactive events affect the player’s resources. The likelihood is determined by the satisfaction level and/or upgrades unlocked in the tree. These events have 2-3 choices for the player to choose from and there will be tradeoffs between morality and resources. Some examples:
You receive an email from a mysterious organization asking for $3,000,000,000. “We have discovered a critical security vulnerability in your system. Pay us and we’ll tell you what it is. Otherwise we’ll leak it to the public.”
[Pay] The group, true to their word, helps you patch up the security vulnerability they have discovered.
[Not Pay] The group releases this to the public. The people lose faith in their glorious leader, dropping satisfaction by 50,000.
A new wave of flu rages across TERRITORY. The worker union initiates a strike and demands paid leave as they recover from their illness.
[Give in] Satisfaction +15,000. Income from TERRITORY decreased by 80% for the next 2 months.
[Refuse] Satisfaction -30,000. Military troops/month decreased by 20% from TERRITORY permanently.
Testing
We created a spreadsheet on Google Sheets with the costs/benefits of each region and kept track of the player’s resources as they played the game. Initially, our game balancing was terrible (on our first playtest, a loss was guaranteed by turn 5), but we tried to change the relative costs to keep it interesting. Game balancing was a very significant challenge for us.
Feedback
Our players did not find the game very fun, partly because our game was unbalanced, but also because they felt like they were waiting for the days to pass by and resources to accumulate, without much opportunity for action. We were worried that players would find the tech tree or number of regions overwhelming, but they did not raise this concern.
We have a metagame, but no actual stuff to do
We need more short term goals
We already have a long term goal
Idea: have more popups (random events)
Zoran: but no one will read the popups
Too much waiting
Idea: include more options
Idea: start off with free upgrades
Design Phase – Round 2
With our first design going poorly, we decided to scrap our game idea and start over from scratch. Since a “world” might be too chaotic and overwhelming for the player, we decided to go with a single city.
We also took inspiration from the MegaMan Battle Network 3 Navi Customizer.
Essentially how the Navi Customizer worked is that over the course of the game, you unlocked new “pieces” which you can try to fit into your grid. There are certain limitations on where pieces can go, but it let you choose which upgrades you wanted to apply to MegaMan.
In our case, buildings would be those “upgrade pieces”, which the player would place down on a grid. We can have upgrades which allow buildings to perform better when placed next to other buildings. Basically creating a “soft restriction” on where the player can place buildings, unlike in MegaMan, which would refuse to let you place certain upgrades next to each other.
Updated Plan
Premise
The player is an AI in charge of a new Mars colony. The AI’s (and thus the player’s) objective function is to “ensure colonists survive”, and they will do anything to keep people from dying. The player loses if a certain number N of colonists die (hard mode=1, medium=100, easy=∞). The player wins when they create a post-scarcity economy where no humans can die and resource production is greater than resource consumption.
Narrative
The theme is misaligned AI and space colonization. The upgrade tree contains policies and buildings to improve the survivability of the colonists. As the player continues researching through the tree, the policies become more ominous, allowing the AI to genetically modify colonists, impose child limits, and ultimately cryonically freeze colonists or upload them into the Matrix. The player wins when one of the final win-condition upgrades is researched, ushering in a new post-scarcity dystopia.
Game mechanics
Resources
There are eight resources:
Food
Water
Oxygen
Metal
Energy
Science
Colonists
Alive colonists
Dead colonists (trying to minimize)
(Space)
Each colonist consumes 2 food/turn, 1 water/turn, 1 oxygen/turn. If resource consumption is more than resource production, non-critical buildings (like schools and mines) “fail” (get deactivated) and if resource consumption is still too high, colonists die until consumption < production. The population increases exponentially as turns pass, pressuring the player to constantly increase resource production.
Buildings
There are several types of buildings that consume and produce resources. The player can rotate and place these strangely-shaped buildings on a 15×15 grid. A challenge is choosing and configuring the buildings to fit the available space.
Some of the buildings
Building name
Cost
Effects per turn
Shape
Farm
-20 food -20 oxygen -20 metal
+25 food +3 oxygen -2 water
###
####
Oxygenerator
-80 metal
+40 oxygen -4 water -4 electricity
#
###
#
Mine
-250 oxygen -60 metal
+10 metal -10 electricity
###
#####
### ##
Upgrade tree
A tech tree allows the player to use science to purchase upgrades that unlock new buildings, make existing buildings more efficient, or reduce resource consumption. Our initial grid was rather complicated, but we reduced the number of upgrades to make it easier to handle.
The initial state of the upgrade tree in the prototype stage. Green=buildings, Blue=upgrades, Red=policy, Gray=unavailable.
We spent a lot of time iterating on our tree. Initially, our tree was too wide and too shallow. To encourage a more structured progression through the game, we made the tree deeper and removed many nodes. Live link to upgrade tree visualization
Testing
We used Google Sheets for much of the “paper” prototyping session because our game includes a lot of math that is easier to do with Sheets formulas. We also made a poster grid and cut out buildings on paper because it’s easier than a computer for players to rotate and fit shapes together.
Prototype on Google Sheets showing an early-game grid and resource chart.
So when playtesters came in, we had them physically place the buildings onto the grid, and then mimic their actions on the spreadsheet, and have it calculate the next turn’s resources. We used Google Sheets in conjunction with Google Apps Script to simulate the game environment.
Paper prototype in the late midgame.
Feedback
Positive feedback
“The overall design is really good and thoughtful, very attractive”
“I like how I need to plan where to put my infrastructure. And I need to think about many different categories when I am playing the game.”
Number of resources is not too much. Players can comfortably keep track of everything.
Positive reactions when we changed our early game to introduce resources and buildings one at a time. Players felt it was a good tutorial that still allowed them freedom and early small wins.
Constructive feedback
“Lack of some incentive to let me keep playing, and I would expect more graphic design”
“They can add some natural disasters. If I have enough science, I can make instruments to detect. This is my favorite one […]”
Random events (like in our initial game idea) would work well
Two playtesters mentioned this and how it improves replayability
Early game: overwhelming to have so many available buildings, but most of them are too expensive to build
Fix: introduce resources one at a time in the early game (food → water → oxygen → metal → energy → science) along with buildings (farm → well → oxygenerator → …)
Also, reveal upgrade tree in turn 10 when science > 10
In the early game, we noticed many players didn’t try to optimize their building placement (i.e. tightly pack buildings like Tetris) because they didn’t realize grid space would be scarce later.
Is this a problem we have to fix? Maybe it’s okay to not encourage optimal placement in the early game, but allow rearranging buildings later.
What should the cost of rearranging buildings be? Free (no resource cost) building rearrangement isn’t “realistic,” but it’s what we used in our subsequent playtests.
Player feedback: was too disincentivized to reorganize, not very easy or fun
Too much math
Fix: Better visualizations that help the player avoid doing any complex mental calculations (or forcing them to use a calculator)
Gameplay is too long
Fix: Cut down on building tiers
Balance fix: make school cheaper so that player can progress through upgrade tree faster
Show where resources are coming from (sources) and what is consuming them (sinks)
Show forecast for future resource consumption (show causality)
Visualize game model better
Don’t make the player keep the whole game state in their head
Auto-play day timer
In our tests, players never played in real-time. They went turn-by-turn or calculated the number of turns to “skip” to gain more resources.
Show upgrade tree earlier instead of waiting till turn ~15
Pro: tree changes the whole game mechanics, should show as soon as possible
Con: might be overwhelming, especially since tree is so big
Upgrade tree is overwhelming
Idea: add a “fog of war” to only show available upgrades 1 or 2 levels deeper
Con: might hide end-goal policies (Matrix, Cryonics), obscuring the end goal
Start with water instead of food (just from a narrative perspective)
Mixed feedback
These are the areas that some playtesters thought were good and some thought could be improved.
Tutorial
+ good progression
– “too synthetic”
Graph/visualization (mostly a technical limitation with Google Sheets)
+ helpful
+ gain/loss per day is good
– “super confusing”
– show projected next-day resources (food, water, people)
Game Progression Timeline
Here’s a breakdown of what we planned to have happen on each turn.
Turn 1
Turn 1
Introduce mechanics:
Player can place buildings, and in multiple orientations
Farms produce food
Humans eat food (2 food/day)
Only foodand humans are shown on the graph visualization and on the farm visualization. The farm will continue to produce/consume the other resources, except it’s just hidden from the player.
Here is the screen that players would be first presented with:
The player is required to place down 2 farms, so they can see the graph update in real-time.
After placing down the second farm, the player will have a surplus of 1 food.
The city center produces 1 of every resource.
Turn 2
Turn 2
Introduce mechanic:
Humans produce more humans (population growth)
1 human will join the colony, making the food income negative. The player will be prompted to build a farm to keep the income positive.
Turn 5
Turn 5
Introduce new building type and mechanic:
Wells produce water
Humans drink water
Turn 7
Turn 7
Introduce new building types and mechanics:
Oxygen and Oxygenerator
Metal and Mine
Building Cost (all the previous buildings were “free”)
Turn 9
Turn 9
Introduce new building type and mechanics:
Energy and Solar Panel
Turn 11
Turn 11
Introduce new mechanic:
Science
The player will have 10 science by this point, as the city center generates 1 science/turn
Upgrades (and Upgrades Tree)
The player can choose between any of the upgrades that cost 10 science.
We would have a button that takes the player to this upgrades tree screen, where the player can choose an upgrade to buy.
We plan for all a tooltip to be shown when the player hovers over an upgrade. The tooltip will display all of the upgrade’s effects.
Turns 12-50
Mid Game
Player adds buildings to their plot
Player increases plot size from 15×15 to 20×20 to 25×20
Player buys upgrades
Resource consumption increases as population grows
Efficient placements become more important as turns go on (space limitation)
More flexibility in strategies
New mechanics introduced by upgrades:
Passive effects to resources
Slow down population growth
Interdependencies between buildings
e.g. two adjacent solar panels produce more electricity
Increased plot size
New buildings, weirder shapes
Challenge to fit the shapes efficiently
Turns 50-60
End Game
Player optimizes their buildings in order to have the resources necessary to buy one of the final two “win” upgrades
Once player buys a win-condition upgrade, they win the game
These upgrades set pop growth = 0 and all resource production more than consumption
Ending screen:
# of days it took to win
other interesting statistics and graphs
compare to others (e.g. leaderboard for fastest days to win.
Possibly infinite mode?
Ideas suggested by playtesters:
Disaster events (such as meteors) that the player as to deal with
Destroy buildings + make tiles unoccupiable
Player receives warning/heads up of the incoming disaster if they bought some upgrade
Problems with our plan:
End goal isn’t clear at the beginning of the game
Development
Now that we’ve flushed out a lot of the game’s core mechanics, and we are able to at least keep the playtesters invested in the game for a while, it was time to start actually coding the game. We looked into Unity and HaxeFlixel, but ultimately decided to go with HaxeFlixel as it seemed to be the simpler of the two – and because the Haxe programming language looks a lot like JavaScript, which we were all huge fans of.
We used a Trello Board to keep our tasks organized.
Initial UI Mockup. Made in Google Slides.
A few days into development, when we started working on the grid and building graphics – and it turned out to be a bit difficult in HaxeFixel, so we ultimately ended up switching to GODOT. Looking back, we should have went with Unity instead, as we have discovered that GODOT was not made to develop this kind of game. There were a lot of challenges we had to overcome with using GODOT. For instance, our buildings are all physics objects to allow for dragging, the resource visualization was really challenging to create, and having to build all of our components like tooltips, and tech tree items from scratch took up a lot of time.
Getting the first building placed down on the grid (invisible grid)!
We probably could have gotten away with making our entire game using React, and it probably would have saved us a lot of time. However, we wanted the experience of learning a new language and trying out a game editor.
Resource Visualization Design
Visualization Evolution
We started off with a basic bar chart in the Google Sheets mockup of our game.
During the design/prototype phase, we received feedback that it wasn’t clear what buildings were affecting the numbers. So, we decided to try out a stacked bar chart type of visualization.
This was a mockup of the graph design we were thinking of. We made this using Powerpoint.
Placed the initial resource visualization inside of the game.
The design of the graph made the game look too “mathy”, so we decided not to use it. We re-designed the graph so that each resource only took up one bar, and decided to place the reserves for each resource below the bar, and the actual incomes/turn on the bar itself.
Fixing all the labels, and adjusting the colors, we ended up with this as our graph design before releasing to the public:
Buildings Sidebar
Working on the Buildings Sidebar
We planned the buildings sidebar to be in a table format. Each row would have the building name, cost, effects, and the actual building which the player can drag onto the grid.
We had all of the buildings data stored in a .json file. We then read the file and turned it into all of these rows:
Initial sidebar. All the buildings were placed on top of each other and the text alignment is not correct.
After making the buildings move onto the correct positions, we ended with this:
Added hand-crafted pixel art for each building:
Finally, we column-aligned all of the text, and also made buildings you were unable to purchase grayed out:
Finally, we added some color to the text.
And this the version the sidebar when we initially released our game to the public.
Upgrade Tree
Working on the Upgrade Tree
Here was our initial implementation of the upgrade tree.
Initial upgrade tree featuring a bug where the buildings were rendered on top.
All of our upgrade data is stored in a .json file, so we were working on creating a dynamic upgrade info screen. We did change the color scheme to make it look better. We also decided to make a few modifications to our upgrades tree now that we’re able to playtest our game a lot quicker than with the paper cutout version of our game.
Logging and Analytics
Gathering Player Turn Data
To be able to gather information about how our game is doing, we set up some “actions” that we think might be useful to log, and then whenever the user does the action, we would send it off to the logging server (basically just a SQL database).
Some of the things we kept track of were:
Turn number
Resources at the start of the turn
Building placed
Upgrades bought
Colonist deaths
Game Win/Game Loss
“Upgrade” Button Clicked
We were then able to analyze this data and draw conclusions from it. We then use this data to improve our game!
Initial Playtesting
After we finished our initial coded version of our game, we hosted it on cev.vishald.com, and sent it off to a few of our friends and roommates for final testing. Note that cev.vishald.com now just redirects you to our released version of the game.
One of our roommates playtesting the initial version of CEV 1.Another one of our friends playtesting CEV 1.
We fixed a couple bugs and made a few QoL changes that our playtesters suggested.
Now that our game didn’t have any more bugs caught by the playtesters, it was time to release this to the public and see what happens.
Analytics Phase
Analysis Round 1
We released our game on itch.io on the evening of May 16, 2022. On the morning of May 19th, itch.io’s analytics said we had:
19.1k impressions
699 views
530 plays
We were pretty shocked that our game made it that far. We even had another game developer tweet about our game! We did not expect to get any type of recognition for this.
CEV-1 is a very simple (yet super engaging) city builder.
Here’s what our turn retention graph looked like a few days later:
We noticed that we had huge dropoffs on turns 1, 6, and 10, so we looked at those turns and investigated why players are leaving.
Turn 1
Investigating Turn 1
What our turn 0 looked like previouslyWhat our turn 1 looked like previously.
Looking at those turns, we noticed that there was a LOT of text for the player to read through. We hypothesized that players were overwhelmed by the amount of text they had to read through, and got a bad first impression. We attempted to fix this by reducing the amount of text the player has to read through, and removed turn 0. We initially had turn 0 in place in order to split up the amount of text onto two separate screens.
What our new turn 1 looked like. We combined the text from the old turns 0 and 1 and shortened it to this.
We also made the blue a bit brighter, and also updated the resource visualization.
Some feedback that we received was that players were confused at the amount of reserves they had. So, we changed the bar height to represent the reserves, and we added an arrow to indicate the bar’s height next turn.
Another minor change we made was hiding the “Next Month” button until after the player places down the first well.
Turn 6
Investigating Turn 6
We figured players were overwhelmed with the amount of text again, so we just shortened and simplified the text.
Turn 10
Investigating Turn 10
We investigated the amount of players that quit on turn 10, and noticed that many of those that quit did not even click on the Upgrades button to view the upgrade tree!
(We made it so that the player must view the tree before moving onto turn 11.)
Therefore, we made the button animated to make it very obvious that the player should click it.
We updated our game page with these changes (and a few other QoL changes everywhere) in hopes of increasing overall player retention!
Removed the “placed building requirements” before advancing to the next turn, allowing more freedom in the tutorial
Analysis Round 2
We took a look at the turn retention graph and here’s what we saw:
Note that since we removed turn 0, it’s actually offset by 1 turn. So, iteration 2 should be “shifted over” 1 turn to the right.
We see that the turn 1 dropoff is fixed, however, the turn 6 and 10 dropoffs are still there, and even bigger. We created a replay tool in order to view what our players were doing. We filtered out players who quit on those turns and viewed their replays.
We see that this player sold the mine and all their farms for whatever reason.
It wasn’t just this one single player. Take a look at the buildings demolished chart:
We see that there players overall are selling a ton of buildings being sold on turn 6. There is pretty much no reason to demolish a building during the early turns unless the player is purposefully trying to lose.
Once we realized this, we needed to figure out a way to stop players from softlocking themselves in the early game. We had two ideas:
Disallow selling up until turn 10.
Make it so that all selling grants a full refund.
We weren’t sure which one was best. When in doubt, perform an A/B test!
Essentially, we assign every player a random number from 1-3, and do something different based on the number:
A (1) – Control Group: Selling always enabled, full refund iff selling on same turn
B (2) – AlwaysRefund Group: Selling always enabled, full refund always
C (3) – DisableEarlySelling Group: Same rules as Control, no selling until turn 10
With that, we updated our game and waited for the results to come in.
Also made some more QoL changes such as adding building multi-select, and showed the controls to flip and rotate when dragging buildings around.
Also updated the color scheme of the upgrades tree for accessibility reasons
Tech Tree Colors
Applying a deuteranopia color-blind filter to our game to see what it looks like. It’s pretty bad.
Tech tree with updated colors.
Analysis Round 3
Now that the results from round 2 have come in, it’s time to see which of the three groups turned out better:
We see that overall, C (refund iff on the same turn, but no selling until turn 10) performed the best. We attribute the initial drop between minutes 2 and 4 due to noise. We only had 78 players in that sample size. B surprisingly did not perform as bad as we initially thought. We predicted that B would perform the worst, as it would make the game “too easy” and thus, players would finish sooner and quit sooner. The data does not suggest that however.
Taking a look at the turn graph, we see that the turn 6 dropoffs are at least a lot smaller. The turn 10 dropoffs are still present, however. (Note that iterations 2 onwards are offset by 1 turn due to the deletion of turn 0.)
We needed to figure out why players were still leaving on turn 10 despite us making the button animated and visible.
We scratched our heads and handed our game out to more friends until one of them discovered something that we were never going to discover on our own:
One of our friends discovered why players were leaving on turn 10 without clicking on the Upgrades button!
Turns out that the button was actually blocked by itch.io’s overlay on some specific screen sizes. That’s why players were still leaving without clicking on the button even though it’s animated and stuff.
We cannot do anything about the overlay unfortunately (it’s an itch.io limitation). So, what we did was just move the Upgrades button to the left.
The other chunk of players that left on this turn viewed the Upgrade Tree, but never purchased anything. Our data showed that none of the players who purchased an upgrade quit on the same turn. So, we just needed a way to get players to actually purchase an upgrade.
Tech Tree Changes
Animations for Purchasable Upgrades
We added an animation around the border of the upgrades the player is able to purchase.
Our hypothesis is that by adding in this animation, players would be drawn to look at these upgrades, over the others. It also gives players a way to differentiate between upgrades they can purchase right now.
What the Upgrade Tree looks like with this change.
We actually wanted to do a bit more with the Upgrades Tree, so we decided to run an A/B test to see our new upgrade tree.
A – Old tree with border animation for purchasable upgrades
B – Redesigned tree featuring slightly different colors, a legend, larger science display, updated text, different cursor icons when hovering over unpurchaseable upgrades, and more!
A/B Tech Tree Comparison
The Upgrades Tree shown to experimental group A.
The Upgrades Tree shown to experimental group B.
Analysis Round 4
Time to see which Upgrade Tree players preferred:
Player retention graph between the two experimental groups. C: Upgrade Tree A. D: Upgrade Tree B.
The data suggests that the upgrades tree visuals makes no difference in playtime, which we found really surprising.
Zooming out, it appears to make an impact in the late game. However, it shouldn’t really have an impact there (it should really only impact the early game), so we attribute this to noise.
We did choose to go with Upgrade Tree B because we think it looks nicer, even if it doesn’t make players play longer.
Thus far, all of our efforts have went to making more players get to the mid/late-game stage. Time to focus on the mid/late-game at least a little bit. We wanted to verify that having population growth be %-based is a good idea. So far, the population growth was always %-based. To confirm that this was the best, we performed another A/B test:
A – Population growth is %-based
B – Population growth is hard-coded per turn
Our hypothesis is that by having the population growth be %-based, players who struggle in the early game will have an easier time in the late game, and players who excel in the early game will have a harder time in the late game. We figure that this will make the game more enjoyable, and thus, players will play for a longer amount of time.
Analysis Round 5
From our data, we see that in the late game, the adaptive gameplay group (A) held onto players much better than the hard-coded group (B). It’s likely that group B’s players finished more quickly than A’s players on average, as players who excel at these types of games find it too easy, and players who struggled early on find it hard to catch up.
The first 20ish minutes, even though group B performed better than A, we attribute that to noise. The early games are pretty much identical since adaptive triggers have not taken much effect yet. We see in the mid-late game, however, this greatly improved audience retention.
As such, we kept the population scaling percentage-based in order to keep our players engaged longer.
Featured!
Here’s just a showcase of some places that CEV-1 was featured!