I couldn't find any instructions on how to do this and found it by trial and error, so I thought I'd share.
In the app, I only saw one icon/list of "All Sermons" for that page. I saw, however, that others had multiple playlists on that page. In order to do this, you must create a page on your site with all the playlists that you want listed (by sermon series and/or by speaker). Then, in your app settings for the sermon page, select that page from the list to display all the playlists.
If you don't want this page of playlists to appear on your site, in the sidekick option panel, you can drag that page/menu item to the "Additional Pages". This way it can still be linked to, but won't show up on your site menu.
The only thing I haven't figured out yet is how to change the text for each of these playlists in the app.
Maria is correct in the method to show different playlists on the App.
The tutorial covers the linking of the app to the page of those sermons: https://support.sharefaith.com/support/solutions/articles/1000142731