I connected the building with a footpath to the driveway. I guess this helps even more.
https://www.openstreetmap.org/changeset/177114259
I think this is a general problem of/with routing engines, with connecting the poi's (even manual start/end positions) to the correct nearest path. With a building we can help, by connecting it, but if you take the 'garden' between the house and the highway, it will probably always take the highway.