Despite a mercurial performance from Toni Kroos and an early headed goal from Shkodran Mustafi, they were troubled at the back at times.
Jerome Boateng cleared off the line superbly while keeper Manuel Neuer made several good saves before a late scare.
Yevhen Konoplyanka threatened for Ukraine but they tired, with Bastian Schweinsteiger adding a late second.
The Manchester United midfielder, who has not played since 20 March because of a knee injury, came off the bench in stoppage time and two minutes later scored a sweet half-volley from a swift counter attack.
After Poland beat Northern Ireland earlier on Sunday, Germany now top Group C. On Thursday they face the Poles, who beat them in qualifying for Euro 2016.
