This will only work for files that you UPLOAD to Google Drive.
Use "External Media without Import" plugin to import images to WordPress.
Step 1: Go to Google Drive and right click the file that you want to share & and hit share or select the file and click the share or link icon on top.
Step 2: On the window that comes up, click "Get shareable link".
Step 3: Make sure the dropdown is set to one of the "Anyone with the link..." or "Anyone on the internet..." options, then click "Copy link".
Step 4: Now just paste that link into the text box above and create your direct link. Enjoy!