One thing that you should never be afraid of when gaming on the cheap is embracing the classics; unless you're one of those guys who can only play games with the newest graphics or something. Classic games are available on a variety of systems and platforms often for extremely cheap prices.
If you're a PS3 or Xbox 360 owner then the Playstation Network and Xbox Live offer a variety of older games for cheap prices. Sony has slowly added a variety of Playstation 1 and 2 games to PSN and almost all of them are $10 or less. Titles like Final Fantasy VII, Grand Theft Auto: Vice City and Okami should excite a lot of cheap gamers. Microsoft, likewise, has old Xbox games like Ninja Gaiden and Halo: Combat Evolved up for download.
PC gamers have the motherload though thanks to resources like Steam and GOG that I mentioned earlier. You can get a variety of classic games for cheap prices on services like them.
Don't dismiss buying an old console either. Getting an old PS2, Gamecube or Xbox should not only be easy but it should be cheap too. And you can usually find lots full of games for ridiculous prices on ebay or craigslist. This definitely is an option that should be considered for those who might be especially tight on money. Gamestop is also reportedly going to be clearing out their PS2 section soon, probably because of the PS4, so we could see some incredible deals. Something you might want to keep in mind.
I know that this option might not be for everyone, but it's probably the cheapest I can offer you. I doubt anyone has played all the classic games they want to so this should be a good option for anyone on a tight budget.