Suspicious toys

So Jack is 4 weeks old and apparently we're supposed to start playing with toys with him. Just high contrasting colored toys. Seems a little early to me since he doesn't seem interested in anything but looking at our faces (besides breastfeeding, pooping, and spitting up). But we attempted it with his cute colorful friend "Elmer". An elephant character from an Italian/English series of kids books. (compliments of Ferrario and Swagner).

To me, he seemed at first oblivious and then suspicious. So not sure that's the desired effect...