There is no doubt that video games and television have an impact on cognition. Whether this impact is positive or negative depends on the content and the individual. Strategic video games have been found to improve cognitive skills in older adults (so has searching the internet); video games have been found to improve mental rotation skills, visual and spatial memory, and multitasking skills. Playing the game Dance Revolution was found in one study to affect emotional arousal, and through that, creativity. More negatively, violent video games also can affect emotional arousal and attention.
But amount is also important, particularly for television, which is a far more passive activity. Active mental stimulation supports cognition, especially in older adults, and too many hours spent watching TV means less time available to engage in activities which truly stimulate your mind. You don't have to do crosswords -- even talking to other people is a far more stimulating activity than watching television.