If you look closely in the tread grooves, you'll notice "wear bars" cast into the tire's radius at regular intervals. When the tread wears down to these bars, its definately time to replace. Street tires have to have a minimum tread depth to be legal for the street. The "wear bars" indicate you are at that limit.
If you have no sidewall cracking and enough tread depth, they are fine for normal street use. And, I use them until the tread is thin. 4 years is not very old in my opinion. Unless they have been baking exposed in the Arizona sun the entire duration.
The rubber compound does get incrementally harder as it ages. So, if you are routinely scraping the foot pegs in turns, a newer, stickier compound is better suited to your driving style.
It took me almost 15 years to wear out a front Continental tire on one of the 550s. It even had sidewall age cracks for the last 5 years. I wouldn't take it on a long trip. But, it was fine for commuting and I kept speeds down below 80 MPH.
The safety angle is often just fear mongering. Or, someone willing to spend your money for you, like tire manufacturers, tire dealers, the government, friends and family, etc. There is nothing 100% safe, not even brand new tires. If you think it is worth it to add 1 or 2 percent to your safety margin, then by all means, go ahead and spend the extra 200-300 dollars a year and brand spankin new tires. But, if they're not crackin and you're not racin', there is more benefit in putting that money in your retirement fund.
But, that's just my opinion.
