Is it worth upgrading my context limit from default since 6.1 sol got 50% cheaper in cache price?

What are the drawbacks of doing it? Sometimes I wonder if its actually cheaper to have higher limit since useful info will be all cached instead of it looking it up for the first time again?