I always thought it was the rose bowl, but my co-worker is telling me that it is the sugar bowl

Orange bowl.

the rose bowl has the most history, but the sugar bowl usually has better teams...

Rose or Sugar Bowl.

You always have better games and teams (usually) in the Sugar Bowl

Rose, Sugar, Fiesta, Orange are all considered equal behind the NC game at this point in time.
That is why before last year the NC game rotated between the above 4 and why now, its site still rotates between those 4 stadiums.
Basically would depend on Conference loyalties as to which you thought was greater.
The Rose Bowl is the one called the Granddaddy of them all. And I believe it is the longest running.

EDIT- as you can see from other answers the conference loyalties. Sugar has always had SEC tie-ins, while Rose has had Big10vsPac10. These still are there but can and usually are pre-empted to get the #1vs #2 match-up and other "superior" matchups, It all depends on the order the bowls get to bid and who they think will draw better

I don't think that college bowl games is prejudice. I think that the Super Bowl can be prejudice somewhat.

Since "prestigious" is very subjective, it's all in the eye of the beholder. Most college football fans though would agree that the fanciest of the bowl games is historically the Rose Bowl. Established in 1902, they have the Parade of Roses... floats... debutants... it's pretty much a culmination of all things associated with the "classic" decades of college athletics (50s-60s). About as close to a high school homecoming weekend, but in a different context. Sugar Bowl doesn't really have that same feeling of a dignitary's ball. It's played in a dome stadium these days, and although established in 1935 in an outdoor college field, the Rose Bowl was the leader of the pack.

None of them...They have all lost a bit of their luster since the BCS came along. The Sugar usually has a good SEC team in it, so that one should be the best game.
Most prestige? The Grandaddy of 'em all...the Rose...only now the Big 10 and Pac-10 champs aren't a lock to go like the used to be - could be a #2 team from each conference, or different conferences completely.
The bowl system is a mess now.

Veggie...

do you mean prestigious? what do they teach in law school these days?

Your co-worker must be an SEC homer.

Nicknamed "The Granddaddy of Them All", the Rose Bowl is the oldest and, over the course of its history, the most prestigious bowl game. In the current BCS alignment, the Rose Bowl will host the Big Ten and Pacific 10 conference champions unless they are involved in the national championship game.

The Sugar Bowl is the second-oldest bowl game in the country, behind the Rose Bowl. Under the current BCS alignment the Sugar Bowl will host the Southeastern Conference (SEC) champion unless they are involved in the national championship game.

Probably the Rose Bowl.

gw