i think that software optimization should happen on multiple levels.
Mostly i'd focus on three:
1. algorithmic optimization, so that programs scale well with amount of data to be processed,
2. bottleneck optimization, so that slowest (or for example, most memory consuming) parts of the program are given extra care,
3. low-level optimization, so that we utilize modern hardware to the fullest,
Problem is usually, a lack of resources such as time, programmers, money.
see also, if You wish or need: Software Costs & Optimization, Optimization Costs, Assembler Optimization, Cache.