It has been said that an army marches on its stomach, and Upkeep is the cost of keeping those stomachs full.
As you train more troops, the amount of unprotected Food subtracted for Upkeep will also increase. You can check your Upkeep cost estimate by tapping the Income board outside the wall in your city.
Please note that Wounded, Marching, or any troops outside your Keep will not be included in these calculations.
You will notice that Upkeep is subtracted periodically, such as when you first open the game or when you send your troops out on a march. Doing these particular actions will never remove more Upkeep than is intended overall, it will just change the intervals at which you notice it is subtracted. At the end of the day, the only thing that can change the total amount of Food consumed by Upkeep is losing troops or training more troops.
Upkeep will also only pull from loose Food above the protected level of your Storehouse. If the amount of loose Food in your inventory (not included Food in crates) is below the capacity protected by your Storehouse level, then no Upkeep will be subtracted until you have Food above that number.